微调 Llama 4：含演示项目的指南

使用医学推理数据集微调 Llama 4

本节中，我们将学习在微调全新的 Llama 4 模型时如何应对常见挑战，例如显存不足问题以及 Transformers 库中的缺陷。我们还将介绍如何顺利加载、微调并保存 LoRA 适配器。按照这些步骤，您可以专注于微调过程，而不必为技术障碍分心。

1. 环境准备

我们将安装微调大型语言模型所需的 Python 包。

注意：

• Transformers 库的最新版本存在嵌入不匹配的缺陷，已在GitHub 仓库中报告。为避免该问题，我们将安装 4.51.0 版本。
• Hugging Face 服务器提供较 Git-LFS 更快的 xet 集成。此集成可将下载速度提升至三倍。

%%capture
!pip install transformers==4.51.0
%pip install -U datasets 
%pip install -U accelerate 
%pip install -U peft 
%pip install -U trl 
%pip install -U bitsandbytes
%pip install huggingface_hub[hf_xet]技术支持

从环境变量中加载 API 密钥用于登录 Hugging Face。登录后，您可以访问受限模型，并保存微调后的模型和分词器。

from huggingface_hub import login
import os

hf_token = os.environ.get("HF_TOKEN")
login(hf_token)技术支持

2. 加载模型与分词器

以 4 位量化的方式加载Llama-4-Scout-17B-16E-Instruct 模型以提高内存使用效率。请确保将 device_map 设为 auto 以使用全部三块 H200 GPU。

注意：请确保您已获得该模型的访问权限，因为此模型为受限访问，需要前往meta-llama/Llama-4-Scout-17B-16E-Instruct 网址填写表单。

import os
import torch
from transformers import AutoTokenizer, Llama4ForConditionalGeneration, BitsAndBytesConfig


model_id = "meta-llama/Llama-4-Scout-17B-16E-Instruct"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=False,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)


model = Llama4ForConditionalGeneration.from_pretrained(
    model_id,
    device_map="auto", 
    torch_dtype=torch.bfloat16,
    quantization_config=bnb_config,
    trust_remote_code=True,
)

model.config.use_cache = False
model.config.pretraining_tp = 1技术支持

我们还将使用相同的模型 ID 加载分词器。

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)技术支持

运行下方命令，可检查我们用于设置训练器和微调模型的剩余显存。

!nvidia-smi技术支持

3. 加载与处理数据

我们将为模型创建一种提示词样式，其中包含系统提示，并为问题、思维链和回答预留占位符。该提示将帮助模型逐步思考，给出清晰准确的答案。

train_prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context. 
Write a response that appropriately completes the request. 
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.

### Instruction:
You are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. 
Please answer the following medical question. 

### Question:
{}

### Response:
<think>
{}
</think>
{}"""技术支持

接下来，我们将创建 Python 函数，使用训练提示样式和数据集中的列来生成 “text” 列。

EOS_TOKEN = tokenizer.eos_token  # Must add EOS_TOKEN

def formatting_prompts_func(examples):
    inputs = examples["Question"]
    complex_cots = examples["Complex_CoT"]
    outputs = examples["Response"]
    texts = []
    for question, cot, response in zip(inputs, complex_cots, outputs):
        # Append the EOS token to the response if it's not already there
        if not response.endswith(tokenizer.eos_token):
            response += tokenizer.eos_token
        text = train_prompt_style.format(question, cot, response)
        texts.append(text)
    return {"text": texts}技术支持

我们将从 Hugging Face Hub 加载FreedomIntelligence/medical-o1-reasoning-SFT 数据集的前 500 条样本，然后应用 formatting_prompts_func 函数以创建 “text” 列。

from datasets import load_dataset


dataset = load_dataset("FreedomIntelligence/medical-o1-reasoning-SFT","en", split = "train[0:500]",trust_remote_code=True)
dataset = dataset.map(formatting_prompts_func, batched = True,)
dataset["text"][0]技术支持

“text” 列包含系统提示、说明、问题、思维链和回答。

新的 STF 训练器不接受分词器，因此我们会将分词器转换为数据整理器，并向训练器提供数据整理器而非分词器。

from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=False
)技术支持

4. 微调前的模型推理

我们将创建一个测试提示样式，包含训练提示样式中的所有内容，但不包含思维链与回答。

prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context. 
Write a response that appropriately completes the request. 
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.

### Instruction:
You are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. 
Please answer the following medical question. 

### Question:
{}

### Response:
<think>{}"""技术支持

我们将取数据集中的第一个问题，按测试提示样式转换为提示词，然后让模型生成回答。

question = dataset[0]['Question']
inputs = tokenizer(
    [prompt_style.format(question, "") + tokenizer.eos_token],
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    eos_token_id=tokenizer.eos_token_id,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(response[0].split("### Response:")[1])技术支持

模型的思维链很长，而其给出的答案较为简短，与数据集差异明显。

<think>assistant

To approach this question, let's break down the key elements provided and analyze them step by step:

1. **Symptoms**: The patient experiences involuntary urine loss during activities like coughing or sneezing but has no leakage at night. This pattern of urinary incontinence is suggestive of stress urinary incontinence (SUI), which is characterized by the involuntary leakage of urine on effort or exertion, or on sneezing or coughing.

2. **Diagnostic Tests Mentioned**:
   - **Gynecological Exam**: This is likely performed to assess the pelvic anatomy, including the position and support of the urethra and bladder neck, and to check for any pelvic organ prolapse.
   - **Q-tip Test**: This test is used to assess urethral mobility. A Q-tip (cotton swab) is inserted into the urethra, and its angle of movement is measured. Increased mobility (an angle change of more than 30 degrees) is often associated with stress urinary incontinence.

3. **Cystometry (Cystometrogram)**: This test measures the pressure within the bladder during filling and helps assess bladder function, including the residual volume and detrusor muscle contractions. The detrusor muscle is the smooth muscle in the wall of the bladder that contracts to allow urine to be expelled.

Given that the patient likely has stress urinary incontinence (SUI) based on her symptoms:
- **Residual Volume**: In patients with SUI, the bladder usually functions normally, and thus, the residual volume (the amount of urine left in the bladder after urination) is typically not significantly affected. Therefore, one would expect the residual volume to be normal or near-normal.
  
- **Detrusor Contractions**: In SUI, the problem primarily lies with the urethral sphincter mechanism and support rather than with the detrusor muscle itself. Hence, detrusor contractions are usually normal. The patient does not have symptoms suggestive of an overactive bladder (like urgency, urge incontinence, or nocturia), which would be more indicative of detrusor overactivity.

Based on this analysis, cystometry in this patient would most likely reveal:
- A **normal residual volume**, as her symptoms do not suggest a problem with bladder emptying.
- **Normal detrusor contractions**, as her condition (stress urinary incontinence) primarily involves issues with urethral support and continence mechanisms rather than detrusor function.

Therefore, cystometry would likely show that she has a normal residual volume and normal detrusor contractions. 

</think>

The final answer is: $\boxed{Normal residual volume and normal detrusor contractions}$技术支持

5. 实现 LoRA

我们现在将实现 LoRA（低秩适配）以进行参数高效微调，并将其应用于模型。LoRA 的思路是冻结模型的大部分参数，仅训练一小部分新增参数来实现微调。

这种方法更省内存、更快且更具性价比，同时仍能保持与全量微调相当的高准确率。

from peft import LoraConfig, get_peft_model

# LoRA config
peft_config = LoraConfig(
    lora_alpha=16,                           # Scaling factor for LoRA
    lora_dropout=0.05,                       # Add slight dropout for regularization
    r=64,                                    # Rank of the LoRA update matrices
    bias="none",                             # No bias reparameterization
    task_type="CAUSAL_LM",                   # Task type: Causal Language Modeling
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],  # Target modules for LoRA
)

model = get_peft_model(model, peft_config)技术支持

我们现在将配置并初始化 SFTTrainer（监督微调训练器），为其提供数据集、模型、数据整理器、训练参数和 LoRA 配置。SFTTrainer 将这些组件集成到一个简化流程中，使基于 LoRA 微调 Llama 4 等大型语言模型更为便捷。

from trl import SFTTrainer
from transformers import TrainingArguments


# Training Arguments
training_arguments = TrainingArguments(
    output_dir="output",
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=2,
    optim="paged_adamw_32bit",
    num_train_epochs=1,
    logging_steps=0.2,
    warmup_steps=10,
    logging_strategy="steps",
    learning_rate=2e-4,
    fp16=False,
    bf16=False,
    group_by_length=True,
    report_to="none"
)

# Initialize the Trainer
trainer = SFTTrainer(
    model=model,
    args=training_arguments,
    train_dataset=dataset,
    peft_config=peft_config,
    data_collator=data_collator,
)技术支持

6. 模型训练

运行以下命令开始训练：

trainer.train()技术支持

如果切换到 Pod 控制台，您会看到训练器正在使用三块 GPU 进行训练。

我们仅用 7 分钟就完成了模型微调，从头到尾运行整个项目总计耗时 30 分钟。

7. 微调后的模型推理

我们将使用与之前相同的样本测试微调后的模型，以比较其微调前后的表现，从而评估模型在推理与生成详细回答方面的改进程度。

question = dataset[0]['Question']
inputs = tokenizer(
    [prompt_style.format(question, "") + tokenizer.eos_token],
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    eos_token_id=tokenizer.eos_token_id,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(response[0].split("### Response:")[1])技术支持

模型的推理过程准确，回答详尽且合乎逻辑：

<think>
Alright, let's think about what's going on with this 61-year-old woman. She's experiencing involuntary urine loss when she coughs or sneezes, which is classic for stress urinary incontinence. But there's no leakage at night, which is interesting. That detail might help us rule out other types of incontinence, like overactive bladder.

Now, let's think about the tests she's had. The gynecological exam and the Q-tip test are usually done to check for any anatomical issues that could be causing her symptoms. If the Q-tip test is positive, it means there's some mobility in the urethra, which can be a sign of stress incontinence.

Okay, so we've got stress urinary incontinence in mind. What does cystometry tell us? It's a test that looks at how well the bladder functions by measuring pressure and volume during filling and voiding. 

For someone with stress incontinence, the cystometry would probably show that her bladder is doing its job properly. The residual volume, which is the amount of urine left in the bladder after she pees, should be normal. This means her bladder is emptying well enough.

Also, with stress incontinence, you wouldn't expect to see abnormal detrusor contractions. These are like muscle spasms in the bladder that can cause urgency and urge incontinence. But in stress incontinence, the problem is more about the urethral support and pressure control during activities, not the bladder muscle itself.

So, putting it all together, the cystometry should show a normal residual volume and no abnormal detrusor contractions. This fits perfectly with what we know about stress urinary incontinence. Yep, that's what I'd expect to see in this case.
</think>
In this scenario, the cystometry would most likely reveal a normal residual volume and no abnormal detrusor contractions. This is because stress urinary incontinence, as indicated by the patient's symptoms and the positive Q-tip test, is primarily related to issues with urethral support and pressure control during activities, not with the bladder muscle's function or emptying ability.技术支持

我们再用数据集中的另一条样本测试模型，以进一步评估其表现。

question = dataset[10]['Question']
inputs = tokenizer(
    [prompt_style.format(question, "") + tokenizer.eos_token],
    return_tensors="pt"
).to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    eos_token_id=tokenizer.eos_token_id,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(response[0].split("### Response:")[1])技术支持

回答合乎逻辑、细节充分，体现了模型有效运用医学推理的能力：

<think>
Alright, let's break this down. So, we have a 42-year-old guy who's just recovered from pneumonia, and his T3 levels are low. Now, his TSH is slightly elevated at 4.7 µU/mL, which isn't too high but is definitely not normal. T4 is at 6 µg/dL, which is kind of in the middle range, and T3 is 68 ng/dL, which is low. Hmm, these numbers make me think of a condition called sick euthyroid syndrome. It's like when the body gets sick, the thyroid function tests can get all wonky, even if the thyroid itself is working fine.

In this syndrome, T3 usually drops first, and T4 might stay normal or even rise a bit. This makes sense here because his T3 is low, but T4 is still around 6 µg/dL. Now, the TSH is slightly elevated, which is a bit puzzling, but in sick euthyroid syndrome, TSH can be normal or even slightly elevated. So, this fits.

But wait, what about the other hormones? In sick euthyroid syndrome, we often see an increase in reverse T3 (rT3). This is because the body might convert T4 to rT3 instead of T3 when it's stressed. So, if I were to guess, I'd say the reverse T3 level is probably elevated in this case.

This all makes sense because the clinical picture and lab results are lining up nicely with sick euthyroid syndrome. So, if I had to pick an additional hormone that's likely elevated, it would be the reverse T3. Yeah, that seems to fit the scenario perfectly.
</think>
In this scenario, considering the clinical context of a 42-year-old man recovering from pneumonia with decreased T3 levels and slightly elevated TSH, along with normal T4 levels, the likely condition is sick euthyroid syndrome. In this condition, the body often converts T4 to reverse T3 (rT3) instead of T3 during stress or illness. Therefore, the additional hormone level that is likely to be elevated in this patient is reverse T3 (rT3).技术支持

8. 保存模型

我们现在将把微调后的模型（LoRA 适配器）和分词器推送到 Hugging Face Hub。该过程会自动为您创建仓库并上传所有必要的模型文件，使模型可以公开访问以便进一步使用或分享。

model.push_to_hub("Llama-4-Scout-17B-16E-Instruct-Medical-ChatBot")
tokenizer.push_to_hub("Llama-4-Scout-17B-16E-Instruct-Medical-ChatBot")技术支持

完成后，您可以通过以下链接访问模型仓库：kingabzpro/Llama-4-Scout-17B-16E-Instruct-Medical-ChatBot

这里是配套笔记本，包含所有必要的代码、输出和详细说明，帮助您微调自己的 Llama 4 模型。

结语

编写本教程既具挑战也颇有收获。在使用 Llama 4 的过程中，我遇到了一些问题，凸显了使用该模型的复杂性。依我之见，Llama 4 目前尚未为广泛使用做好充分优化，尤其是对依赖消费级 GPU 的个人用户而言。以下是我面临的一些主要挑战：

1. 显存不足问题：Llama 4 需要大量显存，使其难以在标准 GPU 上运行或微调。
2. Transformers 库中的缺陷：与 Llama 4 搭配使用时存在多处兼容性与缺陷问题，需要排查和变通解决。
3. 量化挑战：Hugging Face 上可用的量化模型并不兼容，且需要巨大的显存。
4. 环境配置复杂：要支持新的机制，配置环境耗时且需要对 Pytorch 和 Hugging Face 生态有深入理解。

尽管如此，只要逐步按照本指南操作，您应能较为顺利地在任意新数据集上微调 Llama 4。

要进一步了解 Llama 4，请查看以下资源：

• Llama 4 概览
• 搭配 vLLM 的 Llama 4