Fine-Tuning DeepSeek R1 (Reasoning Model)

在本教程中，我们将在 Hugging Face 的 Medical Chain-of-Thought 数据集上微调 DeepSeek-R1-Distill-Llama-8B 模型。该蒸馏版 DeepSeek-R1 是在由 DeepSeek-R1 生成的数据上对 Llama 3.1 8B 模型进行微调而得，展现出与原始模型相近的推理能力。

如果您刚接触大语言模型与微调，强烈建议先学习用 Python 认识大语言模型（LLM）课程。您也可以查看我们的指南：在 RTX 4090 上微调 DeepSeek R1-0528。

图片来源：作者

DeepSeek R1 简介

中国 AI 公司 DeepSeek AI 已开源其第一代推理模型 DeepSeek-R1 和 DeepSeek-R1-Zero，在数学、编程与逻辑等推理任务上的表现可与 OpenAI 的 o1 相媲美。您可以阅读我们的DeepSeek R1 全面指南以了解更多。

DeepSeek-R1-Zero

DeepSeek-R1-Zero 是首个在初始阶段完全采用大规模强化学习（RL）而非监督式微调（SFT）训练出的开源模型。此方法使模型能够自主探索链式思维（CoT）推理、解决复杂问题并迭代优化输出。但其也面临诸如推理步骤重复、可读性较差、语言混杂等挑战，影响清晰度与可用性。

DeepSeek-R1

DeepSeek-R1 在强化学习前引入冷启动数据，以弥补 DeepSeek-R1-Zero 的局限，为推理与非推理任务打下坚实基础。

这种多阶段训练使模型在数学、代码与推理基准上达到与 OpenAI-o1 相当的最先进水平，同时提升输出的可读性与连贯性。

DeepSeek 蒸馏

除了需要大量算力与内存的大语言模型外，DeepSeek 还推出了蒸馏模型。这些更小、更高效的模型同样展现出卓越的推理表现。

参数规模从 1.5B 到 70B 不等，这些模型依然保持强劲的推理能力，其中 DeepSeek-R1-Distill-Qwen-32B 在多个基准上超越 OpenAI-o1-mini。

小模型能继承大模型的推理模式，彰显蒸馏过程的有效性。

来源：deepseek-ai/DeepSeek-R1

阅读 这篇DeepSeek-R1：特性、与 o1 的对比、蒸馏模型等博文，了解其关键特性、研发流程、蒸馏模型、获取方式、定价，以及与 OpenAI o1 的比较。

微调 DeepSeek R1：分步指南

要微调 DeepSeek R1 模型，您可按以下步骤进行：

1. 环境准备

本项目将使用 Kaggle 作为云端 IDE，因为它提供免费的 GPU，且往往比 Google Colab 上的更强。首先，新建一个 Kaggle 笔记本，并将您的 Hugging Face 令牌与 Weights & Biases 令牌添加为机密。

您可以在 Kaggle 笔记本界面的 Add-ons 选项卡中选择 Secrets 来添加机密信息。

设置好机密后，安装 unsloth Python 包。Unsloth 是一个开源框架，旨在让大语言模型（LLM）的微调速度提升 2 倍并更省内存。

阅读我们的 Unsloth 指南：优化并加速 LLM 微调，了解其关键功能、常用方法及如何优化您的微调流程。

%%capture
!pip install unsloth
!pip install --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git技术支持

使用我们从 Kaggle Secrets 安全提取的 Hugging Face API 登录 Hugging Face CLI。

from huggingface_hub import login
from kaggle_secrets import UserSecretsClient
user_secrets = UserSecretsClient()

hf_token = user_secrets.get_secret("HUGGINGFACE_TOKEN")
login(hf_token)技术支持

使用您的 API key 登录 Weights & Biases（wandb），并创建一个新项目以跟踪实验与微调进度。

import wandb

wb_token = user_secrets.get_secret("wandb")

wandb.login(key=wb_token)
run = wandb.init(
    project='Fine-tune-DeepSeek-R1-Distill-Llama-8B on Medical COT Dataset', 
    job_type="training", 
    anonymous="allow"
)技术支持

2. 加载模型与分词器

在本项目中，我们将加载 Unsloth 版本的DeepSeek-R1-Distill-Llama-8B。此外，我们将以 4-bit 量化方式加载模型，以优化内存占用与性能。

from unsloth import FastLanguageModel

max_seq_length = 2048 
dtype = None 
load_in_4bit = True


model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/DeepSeek-R1-Distill-Llama-8B",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
    token = hf_token, 
)技术支持

3. 微调前的模型推理

为了为模型创建提示模版，我们将定义系统提示，并为问题与响应生成设置占位符。该提示将引导模型进行逐步思考，给出逻辑且准确的回答。

prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context. 
Write a response that appropriately completes the request. 
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.

### Instruction:
You are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. 
Please answer the following medical question. 

### Question:
{}

### Response:
<think>{}"""技术支持

在此示例中，我们将把医学问题传入 prompt_style，转换为 token，然后将这些 token 传给模型生成响应。

question = "A 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?"


FastLanguageModel.for_inference(model) 
inputs = tokenizer([prompt_style.format(question, "")], return_tensors="pt").to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("### Response:")[1])技术支持

即便未进行微调，我们的模型也能成功生成链式思维，在给出最终答案前先进行推理。推理过程被包裹在 <think></think> 标签内。

那么为何仍需微调？尽管推理详尽，但过程冗长、不够精炼。此外，最终答案以项目符号形式呈现，这与我们欲微调的数据集的结构与风格不一致。

<think>
Okay, so I have this medical question to answer. Let me try to break it down. The patient is a 61-year-old woman with a history of involuntary urine loss during activities like coughing or sneezing, but she doesn't leak at night. She's had a gynecological exam and a Q-tip test. I need to figure out what cystometry would show regarding her residual volume and detrusor contractions.

First, I should recall what I know about urinary incontinence. Involuntary urine loss during activities like coughing or sneezing makes me think of stress urinary incontinence. Stress incontinence typically happens when the urethral sphincter isn't strong enough to resist increased abdominal pressure from activities like coughing, laughing, or sneezing. This usually affects women, especially after childbirth when the pelvic muscles and ligaments are weakened.

The Q-tip test is a common diagnostic tool for stress urinary incontinence. The test involves inserting a Q-tip catheter, which is a small balloon catheter, into the urethra. The catheter is connected to a pressure gauge. The patient is asked to cough, and the pressure reading is taken. If the pressure is above normal (like above 100 mmHg), it suggests that the urethral sphincter isn't closing properly, which is a sign of stress incontinence.

So, based on the history and the Q-tip test, the diagnosis is likely stress urinary incontinence. Now, moving on to what cystometry would show. Cystometry, also known as a filling cystometry, is a diagnostic procedure where a catheter is inserted into the bladder, and the bladder is filled with a liquid to measure how much it can hold (residual volume) and how it responds to being filled (like during a cough or sneeze). This helps in assessing the capacity and compliance of the bladder.

In a patient with stress incontinence, the bladder's capacity might be normal, but the sphincter's function is impaired. So, during the cystometry, the residual volume might be within normal limits because the bladder isn't overfilled. However, when the patient is asked to cough or perform a Valsalva maneuver, the detrusor muscle (the smooth muscle layer of the bladder) might not contract effectively, leading to an increase in intra-abdominal pressure, which might cause leakage.

Wait, but detrusor contractions are usually associated with voiding. In stress incontinence, the issue isn't with the detrusor contractions but with the sphincter's inability to prevent leakage. So, during cystometry, the detrusor contractions would be normal because they are part of the normal voiding process. However, the problem is that the sphincter doesn't close properly, leading to leakage.

So, putting it all together, the residual volume might be normal, but the detrusor contractions would be normal as well. The key finding would be the impaired sphincter function leading to incontinence, which is typically demonstrated during the Q-tip test and clinical history. Therefore, the cystometry would likely show normal residual volume and normal detrusor contractions, but the underlying issue is the sphincter's inability to prevent leakage.
</think>

Based on the provided information, the cystometry findings in this 61-year-old woman with stress urinary incontinence would likely demonstrate the following:

1. **Residual Volume**: The residual volume would be within normal limits. This is because the bladder's capacity is typically normal in cases of stress incontinence, where the primary issue lies with the sphincter function rather than the bladder's capacity.

2. **Detrusor Contractions**: The detrusor contractions would also be normal. These contractions are part of the normal voiding process and are not impaired in stress urinary incontinence. The issue is not with the detrusor muscle but with the sphincter's inability to prevent leakage.

In summary, the key findings of the cystometry would be normal residual volume and normal detrusor contractions, highlighting the sphincteric defect as the underlying cause of the incontinence.<｜end▁of▁sentence｜>技术支持

4. 加载与处理数据集

我们将对提示模版做小幅调整，为复杂链式思维列新增第三个占位符，以便处理数据集。

train_prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context. 
Write a response that appropriately completes the request. 
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.

### Instruction:
You are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. 
Please answer the following medical question. 

### Question:
{}

### Response:
<think>
{}
</think>
{}"""技术支持

编写 Python 函数，在数据集中创建一个包含训练提示模版的 "text" 列。用问题、链式思维文本与答案填充相应占位符。

EOS_TOKEN = tokenizer.eos_token  # Must add EOS_TOKEN


def formatting_prompts_func(examples):
    inputs = examples["Question"]
    cots = examples["Complex_CoT"]
    outputs = examples["Response"]
    texts = []
    for input, cot, output in zip(inputs, cots, outputs):
        text = train_prompt_style.format(input, cot, output) + EOS_TOKEN
        texts.append(text)
    return {
        "text": texts,
    }技术支持

我们将从 Hugging Face Hub 加载 FreedomIntelligence/medical-o1-reasoning-SFT 数据集的前 500 条样本。之后，我们将使用 formatting_prompts_func 函数映射 text 列。

from datasets import load_dataset
dataset = load_dataset("FreedomIntelligence/medical-o1-reasoning-SFT","en", split = "train[0:500]",trust_remote_code=True)
dataset = dataset.map(formatting_prompts_func, batched = True,)
dataset["text"][0]技术支持

如您所见，text 列包含系统提示、指令、链式思维与答案。

"Below is an instruction that describes a task, paired with an input that provides further context. \nWrite a response that appropriately completes the request. \nBefore answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.\n\n### Instruction:\nYou are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. \nPlease answer the following medical question. \n\n### Question:\nA 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?\n\n### Response:\n<think>\nOkay, let's think about this step by step. There's a 61-year-old woman here who's been dealing with involuntary urine leakages whenever she's doing something that ups her abdominal pressure like coughing or sneezing. This sounds a lot like stress urinary incontinence to me. Now, it's interesting that she doesn't have any issues at night; she isn't experiencing leakage while sleeping. This likely means her bladder's ability to hold urine is fine when she isn't under physical stress. Hmm, that's a clue that we're dealing with something related to pressure rather than a bladder muscle problem. \n\nThe fact that she underwent a Q-tip test is intriguing too. This test is usually done to assess urethral mobility. In stress incontinence, a Q-tip might move significantly, showing urethral hypermobility. This kind of movement often means there's a weakness in the support structures that should help keep the urethra closed during increases in abdominal pressure. So, that's aligning well with stress incontinence.\n\nNow, let's think about what would happen during cystometry. Since stress incontinence isn't usually about sudden bladder contractions, I wouldn't expect to see involuntary detrusor contractions during this test. Her bladder isn't spasming or anything; it's more about the support structure failing under stress. Plus, she likely empties her bladder completely because stress incontinence doesn't typically involve incomplete emptying. So, her residual volume should be pretty normal. \n\nAll in all, it seems like if they do a cystometry on her, it will likely show a normal residual volume and no involuntary contractions. Yup, I think that makes sense given her symptoms and the typical presentations of stress urinary incontinence.\n</think>\nCystometry in this case of stress urinary incontinence would most likely reveal a normal post-void residual volume, as stress incontinence typically does not involve issues with bladder emptying. Additionally, since stress urinary incontinence is primarily related to physical exertion and not an overactive bladder, you would not expect to see any involuntary detrusor contractions during the test.<｜end▁of▁sentence｜>"技术支持

5. 配置模型

借助目标模块，我们将为模型添加低秩适配器以完成设置。

model = FastLanguageModel.get_peft_model(
    model,
    r=16,  
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
    lora_alpha=16,
    lora_dropout=0,  
    bias="none",  
    use_gradient_checkpointing="unsloth",  # True or "unsloth" for very long context
    random_state=3407,
    use_rslora=False,  
    loftq_config=None,
)技术支持

接下来，我们将设置训练参数与训练器，提供模型、分词器、数据集及其他关键训练参数，以优化微调流程。

from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        # Use num_train_epochs = 1, warmup_ratio for full training runs!
        warmup_steps=5,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=10,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
    ),
)技术支持

6. 训练模型

运行以下命令开始训练。

trainer_stats = trainer.train()技术支持

训练过程耗时 44 分钟完成。训练损失逐步下降，这是模型性能改善的良好迹象。

您可以登录 Weights & Biases 网站并查看项目，在仪表板上查看完整的模型评估报告。

如果运行上述代码时遇到问题，请参考 微调 DeepSeek R1（推理模型）的 Kaggle 笔记本。

7. 微调后的模型推理

为便于对比，我们将向微调后的模型提出与之前相同的问题，观察变化。

question = "A 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?"


FastLanguageModel.for_inference(model)  # Unsloth has 2x faster inference!
inputs = tokenizer([prompt_style.format(question, "")], return_tensors="pt").to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("### Response:")[1])
技术支持

效果更好也更准确。链式思维更直接，答案简洁明了且为单段表述。微调取得成功。

<think>
Okay, so let's think about this. We have a 61-year-old woman who's been dealing with involuntary urine loss during things like coughing or sneezing, but she's not leaking at night. That suggests she might have some kind of problem with her pelvic floor muscles or maybe her bladder.

Now, she's got a gynecological exam and a Q-tip test. Let's break that down. The Q-tip test is usually used to check for urethral obstruction. If it's positive, that means there's something blocking the urethra, like a urethral stricture or something else.

Given that she's had a positive Q-tip test, it's likely there's a urethral obstruction. That would mean her urethra is narrow, maybe due to a stricture or some kind of narrowing. So, her bladder can't empty properly during activities like coughing because the urethral obstruction is making it hard.

Now, let's think about what happens when her bladder can't empty. If there's a urethral obstruction, the bladder is forced to hold more urine, increasing the residual volume. That's because her bladder doesn't empty completely. So, her residual volume is probably increased.

Also, if her bladder can't empty properly, she might have increased detrusor contractions. These contractions are usually stronger to push the urine out. So, we expect her detrusor contractions to be increased.

Putting it all together, if she has a urethral obstruction and a positive Q-tip test, we'd expect her cystometry results to show increased residual volume and increased detrusor contractions. That makes sense because of the obstruction and how her bladder is trying to compensate by contracting more.
</think>
Based on the findings of the gynecological exam and the positive Q-tip test, it is most likely that the cystometry would reveal increased residual volume and increased detrusor contractions. The positive Q-tip test indicates urethral obstruction, which would force the bladder to retain more urine, thereby increasing the residual volume. Additionally, the obstruction can lead to increased detrusor contractions as the bladder tries to compensate by contracting more to expel the urine.<｜end▁of▁sentence｜>
技术支持

8. 本地保存模型

现在，我们将本地保存适配器、完整模型与分词器，以便在其他项目中复用。

new_model_local = "DeepSeek-R1-Medical-COT"
model.save_pretrained(new_model_local) 
tokenizer.save_pretrained(new_model_local)

model.save_pretrained_merged(new_model_local, tokenizer, save_method = "merged_16bit",)技术支持

9. 推送到 Hugging Face Hub

我们还将把适配器、分词器与模型推送到 Hugging Face Hub，便于社区将其集成到各自系统中。

new_model_online = "kingabzpro/DeepSeek-R1-Medical-COT"
model.push_to_hub(new_model_online)
tokenizer.push_to_hub(new_model_online)

model.push_to_hub_merged(new_model_online, tokenizer, save_method = "merged_16bit")技术支持

来源：kingabzpro/DeepSeek-R1-Medical-COT · Hugging Face

您学习旅程的下一步是将模型服务化并部署到云端。您可以参考如何使用 BentoML 部署 LLM指南，内含逐步流程，帮助您用 BentoML 与 vLLM 等工具高效且具成本效益地部署大语言模型。

或者，如果您更愿意本地使用模型，可以将其转换为 GGUF 格式并在本机运行。相关内容请参阅微调 Llama 3.2 并在本地使用指南，获取本地使用的详细说明。

结语

AI 领域正快速演进。开源社区正在崛起，挑战过去三年主导 AI 版图的专有模型。

开源大语言模型（LLM）正变得更强、更快、更高效，使在更低算力与内存资源上进行微调变得前所未有的容易。

在本教程中，我们探索了 DeepSeek R1 推理模型，并学习了如何将其蒸馏版用于医学问答任务的微调。经过微调的推理模型不仅提升性能，还能应用于医学、应急与医疗服务等关键领域。

为应对 DeepSeek R1 的发布，OpenAI 推出了两款强大工具：更先进的推理模型 o3，以及由全新计算机使用代理（CUA）模型驱动、可自主浏览网站并执行任务的 OpenAI Operator AI 代理。

• OpenAI 的 O3：特性、与 O1 的对比、发布日期等
• OpenAI 的 Operator：示例、用例、竞品与更多