课程
使用 Llama 3 进行微调
2 小时
4.1K
Phi-4-reasoning 是一款拥有 140 亿参数的开放权重模型，专为复杂推理任务设计。通过对精选数据集的精细微调，它可与体量更大的模型媲美。该模型能够生成详尽的推理链，并在推理阶段充分利用算力，从而实现卓越表现。
其增强版本 Phi-4-reasoning-plus 结合了强化学习，并使用了 1.5 倍的 Token 以获得更高的准确率。尽管规模更小，这两款模型都优于更大的同类模型（如 DeepSeek-R1-Distill-70B），甚至在基准测试中超越了拥有 6710 亿参数的 DeepSeek-R1。
在本教程中，我们将使用 Phi-4-reasoning-plus 模型，并在 Financial Q&A 推理数据集上对其进行微调。内容涵盖设置 Runpod 环境、加载模型、分词器与数据集、为训练准备数据、配置训练、运行评估，以及保存微调后的模型适配器。
您还可以通过以下教程了解 Phi 4 并构建多模态语言导师：Phi-4-Multimodal：含演示项目的指南。
图片来源：作者
1. 环境搭建
本教程将使用 RunPod 作为计算环境。请按照以下步骤进行设置：
1. 进入 RunPod 控制台，点击“Deploy On-Demand”，创建一个新 pod。
2. 选择 A100 GPU，足以加载并微调该模型。
3. 自定义部署：
- 将存储空间增加到 150GB。
- 将 Hugging Face Token 添加为环境变量。该令牌用于加载 gated 模型并将模型推送到 Hugging Face Hub。
4. 部署完成后，点击 Connect 按钮以启动 JupyterLab 实例。
5. 在 JupyterLab 中，新建一个笔记本并安装所需的 Python 包。
%%capture
%pip install -U transformers
%pip install -U datasets
%pip install -U accelerate
%pip install -U peft
%pip install -U trl
%pip install -U bitsandbytes
%pip install huggingface_hub[hf_xet]6. 接下来，使用之前保存的令牌登录 Hugging Face CLI。
from huggingface_hub import login
import os
hf_token = os.environ.get("HF_TOKEN")
login(hf_token)2. 加载模型与分词器
本指南将从 Hugging Face 加载完整的 microsoft/Phi-4-reasoning-plus 模型及其分词器。
这里不使用 4 位量化，因为硬件可以轻松承载全精度权重。配备 80GB 显存的 A100 空间充裕，完整的 Phi-4-Reasoning 模型仅占用约 28GB，仍有 50GB 以上可用于激活值、优化器状态等训练开销。
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load tokenizer & model
model_dir = "microsoft/Phi-4-reasoning-plus"
tokenizer = AutoTokenizer.from_pretrained(model_dir, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
model_dir,
device_map="auto",
torch_dtype=torch.bfloat16,
trust_remote_code=True
)
model.config.use_cache = False
model.config.pretraining_tp = 1
模型加载后，您可以运行以下命令查看其占用的 GPU 显存：
!nvidia-smi完整模型大约使用 28GB 显存。A100 GPU 具有 80GB 显存，因此仍有约 55GB 可用于微调。
3. 加载与处理数据集
在加载数据集之前，我们先定义一种训练提示模板。该提示包括：
- 系统指令： 为模型提供应答规范。
- 开放式金融问题：用户提出的问题，需要进行推理。
- 助手带思考过程的回答： 先给出逐步推理链，再给出最终答案。
这种提示风格旨在引导模型进行审慎思考，并在最终回答的同时给出推理过程。
train_prompt_style="""
<|im_start|>system<|im_sep|>
Below is an instruction that describes a task, paired with an input that provides further context.
Write a response that appropriately completes the request.
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.
<|im_end|>
<|im_start|>user<|im_sep|>
{}<|im_end|>
<|im_start|>assistant<|im_sep|>
<think>
{}
</think>
{}
"""我们将创建一个 Python 函数，把数据集中的列（问题、推理与回答）应用到该模板上，生成名为“text”的新列，包含所有提示组件。
EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN
def formatting_prompts_func(examples):
inputs = examples["Open-ended Verifiable Question"]
complex_cots = examples["Complex_CoT"]
outputs = examples["Response"]
texts = []
for input, cot, output in zip(inputs,complex_cots,outputs):
text = train_prompt_style.format(input,cot,output) + EOS_TOKEN
texts.append(text)
return {
"text": texts,
}现在我们将下载数据集（TheFinAI/Fino1_Reasoning_Path_FinQA），加载 1000 条样本，应用格式化函数，并创建新的 “text” 列。
from datasets import load_dataset
dataset = load_dataset(
"TheFinAI/Fino1_Reasoning_Path_FinQA", split="train[0:1000]", trust_remote_code=True
)
dataset = dataset.map(
formatting_prompts_func,
batched=True,
)
dataset["text"][20]数据集的 "text" 列包含系统指令、用户问题、逐步推理链以及最终答案。这种结构化格式有助于模型理解任务并生成高质量回答。
新的 SFT 训练器函数 不直接接受分词器，因此需要使用 Transformers 中的 DataCollatorForLanguageModeling 将分词器转换为数据整理器（collator）。
from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(
tokenizer=tokenizer,
mlm=False
)4. 微调前的模型推理
在微调之前，我们先对数据集中的一个问题进行推理，以便与微调后的表现进行对比。推理用的提示模板与训练模板相似，但不包含推理过程和答案。
inference_prompt_style = """
<|im_start|>system<|im_sep|>
Below is an instruction that describes a task, paired with an input that provides further context.
Write a response that appropriately completes the request.
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.
<|im_end|>
<|im_start|>user<|im_sep|>
{}<|im_end|>
<|im_start|>assistant<|im_sep|>
<think>
{}
"""我们将选择数据集中的第 21 个问题，进行格式化并分词以用于推理：
question = dataset[20]['Open-ended Verifiable Question']
inputs = tokenizer(
[inference_prompt_style.format(question, "") + tokenizer.eos_token],
return_tensors="pt"
).to("cuda")
outputs = model.generate(
input_ids=inputs.input_ids,
attention_mask=inputs.attention_mask,
max_new_tokens=1200,
eos_token_id=tokenizer.eos_token_id,
use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("<|im_start|>assistant<|im_sep|>")[1])模型会生成较长的推理过程，但未能给出明确答案。将 max_new_tokens 增加到 2000 也许能得到答案，但这种方式低效且不稳定：推理零散，结论也不准确。
5. 模型设置
本节我们将实现LoRA（低秩适配），这是一种参数高效的微调技术，用于将大型语言模型（LLM）适配到新任务。LoRA 通过冻结大部分参数，并以低秩矩阵形式引入少量可训练参数，以低秩分解的方式加到模型中，使模型在无需修改或存储完整权重的情况下适应新任务。
借助 LoRA，我们能在保持与全量微调相当准确率的同时，大幅降低显存占用与训练时间。
from peft import LoraConfig, get_peft_model
# LoRA config
peft_config = LoraConfig(
lora_alpha=16, # Scaling factor for LoRA
lora_dropout=0.05, # Add slight dropout for regularization
r=64, # Rank of the LoRA update matrices
bias="none", # No bias reparameterization
task_type="CAUSAL_LM", # Task type: Causal Language Modeling
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
], # Target modules for LoRA
)
model = get_peft_model(model, peft_config)接下来设置 SFTTrainer。它将数据集、模型、数据整理器、训练参数以及 LoRA 配置整合为一体，简化微调流程，使之高效、易用且高度可定制。
from trl import SFTTrainer
from transformers import TrainingArguments
# Training Arguments
training_arguments = TrainingArguments(
output_dir="output",
per_device_train_batch_size=1,
per_device_eval_batch_size=1,
gradient_accumulation_steps=2,
optim="paged_adamw_32bit",
num_train_epochs=1,
logging_steps=0.2,
warmup_steps=10,
logging_strategy="steps",
learning_rate=2e-4,
fp16=False,
bf16=False,
group_by_length=True,
report_to="none"
)
# Initialize the Trainer
trainer = SFTTrainer(
model=model,
args=training_arguments,
train_dataset=dataset,
peft_config=peft_config,
data_collator=data_collator,
)6. 模型训练
开始训练前，我们将释放 RAM 与 GPU 显存，以避免 OOM（内存不足）问题，确保训练顺利进行。
import gc, torch
gc.collect()
torch.cuda.empty_cache()
model.config.use_cache = False
trainer.train()训练期间，内存使用率约 54%，GPU 利用率可达 99%，资源使用高效。
可以看到训练损失在逐步下降，表明模型正在有效学习并得到微调。
7. 微调后的模型推理
现在我们用与数据集对照的方式测试微调后的模型。对上一节的同一问题生成回答：
question = dataset[20]['Open-ended Verifiable Question']
inputs = tokenizer(
[inference_prompt_style.format(question, "") + tokenizer.eos_token],
return_tensors="pt"
).to("cuda")
outputs = model.generate(
input_ids=inputs.input_ids,
attention_mask=inputs.attention_mask,
max_new_tokens=1200,
eos_token_id=tokenizer.eos_token_id,
use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("<|im_start|>assistant<|im_sep|>")[1])推理部分详尽且准确，答案与数据集高度一致。
<think><|im_end|>First, I need to figure out what the employee contributions were for the years 2002 and 2003. Okay, looking at the numbers, I see that the company contributed $979,000 in 2002 and $825,000 in 2003. So, those are my two key figures.
Next, I'll calculate the change in the contribution amount. To find this, I subtract the 2003 contribution from the 2002 contribution. Let me do that: $979,000 minus $825,000 equals $154,000. So, there's a decrease of $154,000 from 2002 to 2003.
Now, I need to understand what this decrease means in terms of percentage. Percentage change is usually calculated by taking the change, dividing it by the original number, and then multiplying by 100. So, I'll take that $154,000 change and divide it by the 2002 amount, $979,000.
Let me do that math: $154,000 divided by $979,000... Hmm, this gives me a value of approximately 0.1573.
Finally, I multiply that by 100 to convert it to a percentage. That gives me about 15.73%.
Oh, wait, let me make sure everything makes sense. I calculated the change in contributions correctly by subtracting the 2003 amount from the 2002 amount. Then I used the original amount from 2002 as the base for the percentage calculation.
Yes, that all seems right. So, the employee contribution decreased by approximately 15.73% from 2002 to 2003.</think>The percentage decrease in the employee contribution from 2002 to 2003 was approximately 15.73%.<|im_end|>我们再测试一个示例：
question = dataset[200]['Open-ended Verifiable Question']
inputs = tokenizer(
[inference_prompt_style.format(question, "") + tokenizer.eos_token],
return_tensors="pt"
).to("cuda")
outputs = model.generate(
input_ids=inputs.input_ids,
attention_mask=inputs.attention_mask,
max_new_tokens=1200,
eos_token_id=tokenizer.eos_token_id,
use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("<|im_start|>assistant<|im_sep|>")[1])结果令人印象深刻：模型能够以数据集风格生成准确的推理与答案：
<think><|im_end|>Okay, let's figure out how much of the long-term debt should be considered current liabilities. So, the total long-term debt at the end of 2011 was $1859 million. Now, I remember that there's some information about the timing of cash payments related to this debt. In particular, there's a chunk of $29 million that is due in 2012. Hmm, that's the part that's likely to become a current liability.
Let’s check the table again to make sure. It says total long-term debt is $1859 million, and the breakdown of estimated cash payments includes $29 million for 2012. Yep, that seems to align. So, this $29 million should be classified as a current liability because it's due within the next year.
But wait, let me double-check. The table is pretty straightforward: long-term debt is $1859 million and the breakdown clearly shows $29 million due in 2012. So, that $29 million is the part that should be on the current liabilities section of the balance sheet.
Alright, so I'm confident that the portion of the long-term debt that's included in current liabilities as of December 31, 2011, is $29 million. That's it!</think>The portion of the long-term debt included in the current liabilities on the balance sheet as of December 31, 2011, is $29 million. This $29 million represents the amount of debt due within the next year, making it a current liability.<|im_end|>8. 保存模型
为便于后续使用并供他人下载，我们将保存模型适配器与分词器，并推送到 Hugging Face Hub：
new_model_name = "Phi-4-Reasoning-Plus-FinQA-COT"
model.push_to_hub(new_model_name)
tokenizer.push_to_hub(new_model_name)该模型现已公开可用：kingabzpro/Phi-4-Reasoning-Plus-FinQA-COT。
来源：kingabzpro/Phi-4-Reasoning-Plus-FinQA-COT · Hugging Face
如果您在运行微调脚本时遇到问题，请参考配套笔记本：fine-tuning-phi-4-reasoning.ipynb。
结语
Phi-4 Reasoning 模型为开源社区带来了重要进展。与其依赖 6850 亿参数的超大模型，Phi-4 Reasoning 以更小、更高效的架构取得了更好的结果，展示了专注高质量数据集与强化学习也能让小模型实现更优性能。
微调 Phi-4 Reasoning 的一大优势在于成本低、效率高。得益于较小的模型规模与优化架构，在 2 万条样本或以上的数据上微调，成本不到 10 美元且仅需数小时。
像 Phi-4 Reasoning 这样的中小型模型，不仅降低微调成本，也能节省推理时的云端基础设施开支，是在不引入超大模型负担的前提下追求高性能项目的理想选择。
更多信息请参阅我们关于不同大语言模型微调的指南：