在 RTX 4090 上微调 DeepSeek-R1-0528

DeepSeek-R1-0528 是 DeepSeek R1 模型的最新迭代，具备更强的推理与推断能力。其表现可与 o3 和 Gemini 2.5 Pro 等领先模型比肩，是目前全球最好的开源推理模型。

本教程将对 DeepSeek-R1-0528-Qwen3-8B 模型在医学推理多项选择题 (MCQ) 数据集上进行微调。微调将引入一种新的推理风格，并以 A、B、C、D 选项的形式生成答案。

最棒的是？这一切都可以在消费级 GPU（如 RTX 4090）上完成，方便您在本地顺利微调模型。您还可以查看我们关于 DeepSeek-R1 的完整指南，以及我们的微调指南以了解更多信息。

图片由作者提供

1. 环境设置

我们将创建一台搭载 RTX 4090 GPU、使用 PyTorch 2.4.0 镜像的 RunPod 实例。

来源： My Pods

进入 RunPod 控制台的“My Pods”区域，编辑该 pod 的配置：

将存储提升至 100 GB，以容纳更大的数据集和模型。
添加 Hugging Face Token 作为环境变量。该令牌将在与 Hugging Face API 交互时用于身份验证。

来源： My Pods

当 pod 运行后，安装所需的 Python 包。

注意，Transformers 库的最新版本存在一些已知问题，因此我们将安装一个较旧且稳定的版本。

%%capture
%pip install -U transformers==4.52.1
%pip install -U datasets 
%pip install -U accelerate 
%pip install -U peft 
%pip install -U trl 
%pip install -U bitsandbytes技术支持

如需加载模型或向 Hugging Face Hub 推送更新，请使用您的 API key 登录 Hugging Face CLI

from huggingface_hub import login
import os

hf_token = os.environ.get("HF_TOKEN")
login(hf_token)技术支持

2. 加载模型与分词器

尽管我们有 24GB 的显存，足以加载完整模型，但我们将使用 4 比特量化，以预留更多内存用于微调并确保稳定性。这样既能优化显存使用，又能保持性能。

我们将从 Hugging Face Hub 下载并加载 DeepSeek-R1-0528-Qwen3-8B 的模型与分词器。您可以通过我们的课程进一步了解如何使用 Hugging Face 及 Hub 指南。

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=False,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load tokenizer & model

model_dir = "deepseek-ai/DeepSeek-R1-0528-Qwen3-8B"

tokenizer = AutoTokenizer.from_pretrained(model_dir, use_fast=True)

model = AutoModelForCausalLM.from_pretrained(
    model_dir,
    quantization_config=bnb_config,   
    device_map="auto",  
    torch_dtype=torch.bfloat16,
    trust_remote_code=True             
)

model.config.use_cache = False
model.config.pretraining_tp = 1技术支持

为验证加载模型后的显存占用，运行以下命令：

!nvidia-smi技术支持

显存占用约为 8.3GB，约有三分之二的内存可用于微调。

3. 加载与处理数据集

此步骤中，我们将通过创建包含问题与回答占位符的模板来准备训练提示。同时定义一个 Python 函数，按提示样式格式化数据集，并创建新的“text”列。

train_prompt_style = """
Please answer with one of the options in the bracket. Write reasoning in between <analysis></analysis>. Write the answer in between <answer></answer>.
### Question:
{}

### Response:
{}"""


EOS_TOKEN = tokenizer.eos_token  # Must add EOS_TOKEN

def formatting_prompts_func(examples):
    inputs = examples["input"]
    outputs = examples["output"]
    texts = []
    for question, response in zip(inputs, outputs):
        # Remove the "Q:" prefix from the question
        question = question.replace("Q:", "")
        
        # Append the EOS token to the response if it's not already there
        if not response.endswith(tokenizer.eos_token):
            response += tokenizer.eos_token
            
        text = train_prompt_style.format(question, response)
        texts.append(text)
    return {"text": texts}技术支持

接着，我们将加载 mamachang/medical-reasoning，应用 formatting_prompts_func 函数，并展示第 11 个样本的“text”列。

from datasets import load_dataset

dataset = load_dataset(
    "mamachang/medical-reasoning",
    split="train",
    trust_remote_code=True,
)
dataset = dataset.map(
    formatting_prompts_func,
    batched=True,
)
print(dataset["text"][10])技术支持

text 列包含系统提示、问题、推理与答案，格式如下：

Please answer with one of the options in the bracket. Write reasoning in between <analysis></analysis>. Write the answer in between <answer></answer>.
### Question:
A research group wants to assess the relationship between childhood diet and cardiovascular disease in adulthood. A prospective cohort study of 500 children between 10 to 15 years of age is conducted in which the participants' diets are recorded for 1 year and then the patients are assessed 20 years later for the presence of cardiovascular disease. A statistically significant association is found between childhood consumption of vegetables and decreased risk of hyperlipidemia and exercise tolerance. When these findings are submitted to a scientific journal, a peer reviewer comments that the researchers did not discuss the study's validity. Which of the following additional analyses would most likely address the concerns about this study's design?? 
{'A': 'Blinding', 'B': 'Crossover', 'C': 'Matching', 'D': 'Stratification', 'E': 'Randomization'},

### Response:
<analysis>

This is a question about assessing the validity of a prospective cohort study. The study found an association between childhood diet and cardiovascular disease in adulthood. The peer reviewer is concerned that the researchers did not discuss the validity of the study design. 

To address concerns about validity in a prospective cohort study, we need to consider potential confounding factors that could influence the results. The additional analysis suggested should help control for confounding.技术支持

4. 微调前的模型推理

在微调前，我们将测试基座模型的表现，以建立对比基线。这包括创建推理提示、从数据集中选择一个示例问题，并使用基座模型生成回答。

我们将创建一个用于推理的提示模板，其中包含问题的占位符。

inference_prompt_style = """
Please answer with one of the options in the bracket. Write reasoning in between <analysis></analysis>. Write the answer in between <answer></answer>.

### Question:
{}

### Response:
<analysis>
"""技术支持

随后，我们将选择第 11 个样本中的问题，进行格式化并转为 tokens 后提供给模型。

question = dataset[10]['input']
question = question.replace("Q:", "")

inputs = tokenizer(
    [inference_prompt_style.format(question) + tokenizer.eos_token],
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    eos_token_id=tokenizer.eos_token_id,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(response[0].split("### Response:")[1])技术支持

模型的回答不尽如人意。分析部分过长，耗尽了 token 限制，导致未能生成答案部分。

5. 模型设置

在此步骤中，我们将为微调配置模型与分词器。

1. SFTTrainer 并不直接接收分词器。因此我们将使用 transformers 库中的 DataCollatorForLanguageModeling 类将分词器转换为数据整理器（data collator）。

from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False
)技术支持

我们将使用 LoRA （低秩自适应）对模型进行微调。LoRA 仅修改模型参数的一个小子集，相较于全量微调更节省内存且速度更快。

from peft import LoraConfig, get_peft_model

# LoRA config
peft_config = LoraConfig(
    lora_alpha=16,                           # Scaling factor for LoRA
    lora_dropout=0.05,                       # Add slight dropout for regularization
    r=64,                                    # Rank of the LoRA update matrices
    bias="none",                             # No bias reparameterization
    task_type="CAUSAL_LM",                   # Task type: Causal Language Modeling
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],  # Target modules for LoRA
)

model = get_peft_model(model, peft_config)技术支持

TrainingArguments 用于配置微调过程，而 SFTTrainer 通过将模型、数据集、数据整理器、训练参数与 LoRA 配置整合到单一流程中，从而简化微调。

from trl import SFTTrainer
from transformers import TrainingArguments


# Training Arguments
training_arguments = TrainingArguments(
    output_dir="DeepSeek-R1-0528-Qwen3-8B-Medical-Reasoning",
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=2,
    optim="paged_adamw_32bit",
    num_train_epochs=1,
    logging_steps=0.2,
    warmup_steps=10,
    logging_strategy="steps",
    learning_rate=2e-4,
    fp16=False,
    bf16=False,
    group_by_length=True,
    report_to="none"
)

# Initialize the Trainer
trainer = SFTTrainer(
    model=model,
    args=training_arguments,
    train_dataset=dataset,
    peft_config=peft_config,
    data_collator=data_collator,
)技术支持

6. 模型训练

在开始训练前，务必清理缓存并移除无用的内存与显存占用，以避免出现 OOM（内存溢出）错误。

清理内存后，即可使用 SFTTrainer 开始训练。

import gc, torch
gc.collect()
torch.cuda.empty_cache()
model.config.use_cache = False
trainer.train()技术支持

训练过程中，您可以在 RunPod 仪表板查看 GPU 状态。应能看到 GPU 利用率接近 100%，这表明模型正在使用 GPU（而非 CPU）进行训练。

随着训练推进，您应能观察到训练损失逐步下降。这表明模型正在学习，微调取得进展。

7. 微调后的模型推理

微调完成后，我们将使用数据集中同样的第 11 个样本问题进行测试。这样可以与基座模型进行对比，评估微调带来的改进。

question = dataset[10]['input']
question = question.replace("Q:", "")

inputs = tokenizer(
    [inference_prompt_style.format(question,) + tokenizer.eos_token],
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    eos_token_id=tokenizer.eos_token_id,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(response[0].split("### Response:")[1])技术支持

微调后模型的回答有明显改进：

分析部分更简洁，且正确使用了 <analysis> 标签，而非 <think>。
答案部分使用 <answer> 标签进行正确格式化。
但答案本身不正确，模型选择了 E: Randomization，与数据集中的正确答案不一致。

<analysis>
This is a question about evaluating the validity of a prospective cohort study design. The study looked at childhood diet and cardiovascular disease in adulthood. The peer reviewer raised concerns about the study's validity, likely because it was an observational study without randomization or control group. 

The question asks which additional analysis would most likely address the concerns about the study's design. The choices include blinding, crossover, matching, stratification, and randomization. 

Randomization is the key to reducing bias in observational studies. By randomly assigning participants to different groups, you can control for confounding factors. The other choices do not directly address the lack of randomization in the original study design.
</analysis>
<answer>
E: Randomization
</answer>技术支持

为了进一步评估微调模型，我们将其在数据集中的另一条样本上进行测试并观察结果。

question = dataset[100]['input']
question = question.replace("Q:", "")

inputs = tokenizer(
    [inference_prompt_style.format(question) + tokenizer.eos_token],
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    eos_token_id=tokenizer.eos_token_id,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(response[0].split("### Response:")[1])技术支持

模型在该样本上的表现非常出色：

分析部分简洁、准确，且与数据集风格一致。
答案部分正确且格式规范。

<analysis>

This is a clinical vignette describing a 55-year-old man with burning and shooting pain in his feet and lower legs that worsens at night. He has a history of type 2 diabetes mellitus and hypertension. The description of the pain being burning and shooting in a symmetric distribution in the lower extremities, worsening at night, along with his history of diabetes, is most consistent with distal symmetric sensorimotor polyneuropathy. Autonomic neuropathy would not cause sensory symptoms. Isolated cranial nerve or peripheral nerve neuropathy would not explain the symmetric distribution. Radiculopathy would not explain the sensory symptoms.
</analysis>
<answer>
D: Distal symmetric sensorimotor polyneuropathy
</answer>技术支持

8. 保存模型

下一步是保存微调后的模型并推送到 Hugging Face Hub。这样其他人即可访问并用于推理或进一步微调。

new_model_name = "DeepSeek-R1-0528-Qwen3-8B-Medical-Reasoning"
trainer.model.push_to_hub(new_model_name)
trainer.processing_class.push_to_hub(new_model_name)技术支持

成功推送后，模型将在 Hugging Face Hub 上可用： kingabzpro/DeepSeek-R1-0528-Qwen3-8B-Medical-Reasoning。

9. 从 Hugging Face 加载微调后的模型

将微调后的模型与适配器保存到 Hugging Face Hub 后，我们将回载模型进行测试，以确认其表现符合预期。步骤包括清理 GPU 内存、加载基座模型与 LoRA 适配器，并在示例提示上运行推理。

1. 清理 GPU 内存，避免潜在的 OOM 问题，确保在干净环境中加载模型。

del model
del trainer
torch.cuda.empty_cache()技术支持

2. 从 Hugging Face Hub 加载基座模型与微调后的 LoRA 适配器。基座模型将以 4 比特量化方式加载以优化内存占用，并与 LoRA 适配器合并。

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
import torch

# Base model
base_model_id = "deepseek-ai/DeepSeek-R1-0528-Qwen3-8B"

# Your fine-tuned LoRA adapter repository
lora_adapter_id = "kingabzpro/DeepSeek-R1-0528-Qwen3-8B-Medical-Reasoning"

# Load the model in 4-bit
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=False,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    quantization_config=bnb_config,
    trust_remote_code=True,
)

# Attach the LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    lora_adapter_id,
    device_map="auto",
    trust_remote_code=True,
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)技术支持

3. 现在我们将提供一个示例提示对微调模型进行测试。提示将被分词、送入模型，并对输出进行解码。

# Inference example
prompt = """
Please answer with one of the options in the bracket. Write reasoning in between <analysis></analysis>. Write the answer in between <answer></answer>.

### Question:
A research group wants to assess the relationship between childhood diet and cardiovascular disease in adulthood.
A prospective cohort study of 500 children between 10 to 15 years of age is conducted in which the participants' diets are recorded for 1 year and then the patients are assessed 20 years later for the presence of cardiovascular disease.
A statistically significant association is found between childhood consumption of vegetables and decreased risk of hyperlipidemia and exercise tolerance.
When these findings are submitted to a scientific journal, a peer reviewer comments that the researchers did not discuss the study's validity.
Which of the following additional analyses would most likely address the concerns about this study's design? 
{'A': 'Blinding', 'B': 'Crossover', 'C': 'Matching', 'D': 'Stratification', 'E': 'Randomization'},
### Response:
<analysis>

"""

inputs = tokenizer(
    [prompt + tokenizer.eos_token],
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    eos_token_id=tokenizer.eos_token_id,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(response[0].split("### Response:")[1])技术支持

微调后的模型会生成与数据集风格相近的回答，确认模型与适配器已正确加载。

<analysis>
This is a question about evaluating the validity of a prospective cohort study design. The study looked at childhood diet and cardiovascular disease in adulthood. The peer reviewer was concerned about the study's validity. 

To address concerns about validity in a prospective cohort study, we need to consider potential confounding factors. The choices given are different statistical methods that can help control for confounding. 

Blinding and crossover designs are not applicable to a prospective cohort study. Matching and stratification can help control for confounding by balancing the distribution of confounders between groups. Randomization is the best way to minimize confounding by randomly assigning participants to different exposure groups.
</analysis>
<answer>
E: Randomization
</answer>技术支持

如果您在运行上述代码时遇到任何问题，请参阅 fine-tuning-Deepseek-new-R1.ipynb 笔记本。该笔记本已多次测试，可直接在 RunPod 或类似环境中开箱即用。

结语

DeepSeek-R1-0528 是当前最优秀的开源推理模型之一。其强大的复杂推理能力与开源属性，使其成为开发者与研究人员的理想之选。虽然经微调的模型已能产出令人印象深刻的结果，但仍有多种方式可进一步提升微调效果：

加载完整模型：相较于量化版本，使用完整模型有望提升准确性。
增加训练轮数：将训练轮数从 1 提升至至少 3，可带来更好的收敛与表现。
优化提示样式：通过尝试与优化提示风格，帮助模型更好地理解并响应特定任务。
优化超参数：调整学习率、批量大小与梯度累积步数等训练超参数，能显著改善结果。
扩大数据集规模：在更大的数据集（如超过 1 万条样本）上训练，可提升模型泛化与推理能力。

此外，本指南也可用于将 DeepSeek R1 模型微调到任意推理数据集上，包括使用 OpenAI 的 o3 等其他模型合成的数据集。您还可以在我们的 Fine-Tuning With Llama 3 课程中亲手实践这里展示的一些技术。如果您对使用医学数据集微调其他模型感兴趣，请查看我们的教程：在脑部 MRI 数据集上微调 MedGemma。

作者

Abid Ali Awan

LinkedIn Twitter

作为一名持证的数据科学家，我热衷于利用前沿技术打造创新的机器学习应用。凭借在语音识别、数据分析与报告、MLOps、对话式人工智能以及自然语言处理方面的扎实背景，我不断打磨构建智能系统的能力，力求带来切实影响。除技术专长外，我也擅长沟通，能够将复杂概念提炼为清晰、简明的表述。因此，我成为数据科学领域备受关注的博主，与不断壮大的数据专业人士社区分享洞见与实践经验。目前，我专注于内容创作与编辑，借助大语言模型打造有力且吸引人的内容，帮助企业与个人更好地发挥数据价值。

主题

人工智能大语言模型

主题

技术