Kursus
Fine-Tuning dengan Llama 3
2 jam
4.1K
Dalam tutorial ini, kita akan melakukan fine-tuning model DeepSeek-R1-Distill-Llama-8B pada Medical Chain-of-Thought Dataset dari Hugging Face. Model DeepSeek-R1 hasil distilasi ini dibuat dengan melakukan fine-tuning pada Llama 3.1 8B menggunakan data yang dihasilkan dengan DeepSeek-R1. Model ini menampilkan kemampuan penalaran yang serupa dengan model aslinya.
Jika Anda baru dalam LLM dan fine-tuning, saya sangat merekomendasikan Anda mengikuti Introduction to LLMs in Python. Anda juga dapat melihat panduan kami tentang Fine-Tuning DeepSeek R1-0528 pada RTX 4090.
Gambar oleh Penulis
Mengenal DeepSeek R1
Perusahaan AI asal Tiongkok, DeepSeek AI, telah membuka sumber model penalaran generasi pertamanya, DeepSeek-R1 dan DeepSeek-R1-Zero, yang menyaingi OpenAI o1 dalam kinerja pada tugas penalaran seperti matematika, pengodean, dan logika. Anda dapat membaca panduan lengkap DeepSeek R1 untuk mempelajari lebih lanjut.
DeepSeek-R1-Zero
DeepSeek-R1-Zero adalah model open-source pertama yang dilatih semata-mata dengan pembelajaran penguatan (RL) skala besar alih-alih supervised fine-tuning (SFT) sebagai langkah awal. Pendekatan ini memungkinkan model untuk secara mandiri mengeksplorasi penalaran chain-of-thought (CoT), menyelesaikan masalah kompleks, dan menyempurnakan keluarannya secara iteratif. Namun, pendekatan ini memiliki tantangan seperti langkah penalaran yang berulang, keterbacaan yang buruk, dan percampuran bahasa yang dapat memengaruhi kejelasan dan kegunaannya.
DeepSeek-R1
DeepSeek-R1 diperkenalkan untuk mengatasi keterbatasan DeepSeek-R1-Zero dengan memasukkan data cold-start sebelum pembelajaran penguatan, sehingga memberikan fondasi yang kuat untuk tugas penalaran dan non-penalaran.
Pelatihan multi-tahap ini memungkinkan model mencapai kinerja mutakhir, sebanding dengan OpenAI-o1, di berbagai tolok ukur matematika, kode, dan penalaran, sekaligus meningkatkan keterbacaan dan koherensi keluarannya.
Distilasi DeepSeek
Selain model bahasa besar yang memerlukan daya komputasi dan memori ekstensif untuk beroperasi, DeepSeek juga memperkenalkan model hasil distilasi. Model yang lebih kecil dan efisien ini telah menunjukkan bahwa mereka tetap dapat mencapai kinerja penalaran yang luar biasa.
Dengan rentang parameter dari 1,5B hingga 70B, model-model ini mempertahankan kemampuan penalaran yang kuat, dengan DeepSeek-R1-Distill-Qwen-32B mengungguli OpenAI-o1-mini di berbagai tolok ukur.
Model yang lebih kecil mewarisi pola penalaran model yang lebih besar, menunjukkan efektivitas proses distilasi.
Sumber: deepseek-ai/DeepSeek-R1
Baca DeepSeek-R1: Fitur, Perbandingan dengan o1, Model Distilasi & Lainnya untuk mempelajari fitur utama, proses pengembangan, model distilasi, akses, harga, dan perbandingannya dengan OpenAI o1.
Fine-Tuning DeepSeek R1: Panduan Langkah demi Langkah
Untuk melakukan fine-tuning model DeepSeek R1, Anda dapat mengikuti langkah-langkah di bawah ini:
1. Penyiapan
Untuk proyek ini, kami menggunakan Kaggle sebagai Cloud IDE karena menyediakan akses gratis ke GPU, yang sering kali lebih bertenaga daripada yang tersedia di Google Colab. Untuk memulai, buat notebook Kaggle baru dan tambahkan token Hugging Face serta token Weights & Biases Anda sebagai secrets.
Anda dapat menambahkan secrets dengan menavigasi ke tab Add-ons di antarmuka notebook Kaggle dan memilih opsi Secrets.
Setelah menyiapkan secrets, instal paket Python unsloth. Unsloth adalah kerangka kerja open-source yang dirancang untuk membuat fine-tuning model bahasa besar (LLM) 2X lebih cepat dan lebih hemat memori.
Baca Panduan Unsloth: Optimalkan dan Percepat Fine-Tuning LLM untuk mempelajari fitur utama Unsloth, berbagai fungsinya, dan cara mengoptimalkan alur kerja fine-tuning Anda.
%%capture
!pip install unsloth
!pip install --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.gitMasuk ke Hugging Face CLI menggunakan API Hugging Face yang telah kami ambil secara aman dari Kaggle Secrets.
from huggingface_hub import login
from kaggle_secrets import UserSecretsClient
user_secrets = UserSecretsClient()
hf_token = user_secrets.get_secret("HUGGINGFACE_TOKEN")
login(hf_token)Masuk ke Weights & Biases (wandb) menggunakan kunci API Anda dan buat proyek baru untuk melacak eksperimen serta progres fine-tuning.
import wandb
wb_token = user_secrets.get_secret("wandb")
wandb.login(key=wb_token)
run = wandb.init(
project='Fine-tune-DeepSeek-R1-Distill-Llama-8B on Medical COT Dataset',
job_type="training",
anonymous="allow"
)2. Memuat model dan tokenizer
Untuk proyek ini, kami memuat versi Unsloth dari DeepSeek-R1-Distill-Llama-8B. Selain itu, kita akan memuat model dalam kuantisasi 4-bit untuk mengoptimalkan penggunaan memori dan kinerja.
from unsloth import FastLanguageModel
max_seq_length = 2048
dtype = None
load_in_4bit = True
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "unsloth/DeepSeek-R1-Distill-Llama-8B",
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
token = hf_token,
)3. Inferensi model sebelum fine-tuning
Untuk membuat gaya prompt bagi model, kita akan mendefinisikan system prompt dan menyertakan placeholder untuk pertanyaan serta pembuatan respons. Prompt akan memandu model untuk berpikir selangkah demi selangkah dan memberikan jawaban yang logis serta akurat.
prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context.
Write a response that appropriately completes the request.
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.
### Instruction:
You are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning.
Please answer the following medical question.
### Question:
{}
### Response:
<think>{}"""Dalam contoh ini, kita akan memberikan sebuah pertanyaan medis ke prompt_style, mengonversinya menjadi token, lalu meneruskan token ke model untuk menghasilkan respons.
question = "A 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?"
FastLanguageModel.for_inference(model)
inputs = tokenizer([prompt_style.format(question, "")], return_tensors="pt").to("cuda")
outputs = model.generate(
input_ids=inputs.input_ids,
attention_mask=inputs.attention_mask,
max_new_tokens=1200,
use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("### Response:")[1])Bahkan tanpa fine-tuning, model kita berhasil menghasilkan chain of thought dan memberikan penalaran sebelum menyampaikan jawaban akhir. Proses penalaran dibungkus dalam tag <think></think>.
Lalu, mengapa kita masih perlu fine-tuning? Proses penalarannya, meski terperinci, terlalu bertele-tele dan tidak ringkas. Selain itu, jawaban akhir disajikan dalam format poin-poin, yang menyimpang dari struktur dan gaya dataset yang ingin kita gunakan untuk fine-tuning.
<think>
Okay, so I have this medical question to answer. Let me try to break it down. The patient is a 61-year-old woman with a history of involuntary urine loss during activities like coughing or sneezing, but she doesn't leak at night. She's had a gynecological exam and a Q-tip test. I need to figure out what cystometry would show regarding her residual volume and detrusor contractions.
First, I should recall what I know about urinary incontinence. Involuntary urine loss during activities like coughing or sneezing makes me think of stress urinary incontinence. Stress incontinence typically happens when the urethral sphincter isn't strong enough to resist increased abdominal pressure from activities like coughing, laughing, or sneezing. This usually affects women, especially after childbirth when the pelvic muscles and ligaments are weakened.
The Q-tip test is a common diagnostic tool for stress urinary incontinence. The test involves inserting a Q-tip catheter, which is a small balloon catheter, into the urethra. The catheter is connected to a pressure gauge. The patient is asked to cough, and the pressure reading is taken. If the pressure is above normal (like above 100 mmHg), it suggests that the urethral sphincter isn't closing properly, which is a sign of stress incontinence.
So, based on the history and the Q-tip test, the diagnosis is likely stress urinary incontinence. Now, moving on to what cystometry would show. Cystometry, also known as a filling cystometry, is a diagnostic procedure where a catheter is inserted into the bladder, and the bladder is filled with a liquid to measure how much it can hold (residual volume) and how it responds to being filled (like during a cough or sneeze). This helps in assessing the capacity and compliance of the bladder.
In a patient with stress incontinence, the bladder's capacity might be normal, but the sphincter's function is impaired. So, during the cystometry, the residual volume might be within normal limits because the bladder isn't overfilled. However, when the patient is asked to cough or perform a Valsalva maneuver, the detrusor muscle (the smooth muscle layer of the bladder) might not contract effectively, leading to an increase in intra-abdominal pressure, which might cause leakage.
Wait, but detrusor contractions are usually associated with voiding. In stress incontinence, the issue isn't with the detrusor contractions but with the sphincter's inability to prevent leakage. So, during cystometry, the detrusor contractions would be normal because they are part of the normal voiding process. However, the problem is that the sphincter doesn't close properly, leading to leakage.
So, putting it all together, the residual volume might be normal, but the detrusor contractions would be normal as well. The key finding would be the impaired sphincter function leading to incontinence, which is typically demonstrated during the Q-tip test and clinical history. Therefore, the cystometry would likely show normal residual volume and normal detrusor contractions, but the underlying issue is the sphincter's inability to prevent leakage.
</think>
Based on the provided information, the cystometry findings in this 61-year-old woman with stress urinary incontinence would likely demonstrate the following:
1. **Residual Volume**: The residual volume would be within normal limits. This is because the bladder's capacity is typically normal in cases of stress incontinence, where the primary issue lies with the sphincter function rather than the bladder's capacity.
2. **Detrusor Contractions**: The detrusor contractions would also be normal. These contractions are part of the normal voiding process and are not impaired in stress urinary incontinence. The issue is not with the detrusor muscle but with the sphincter's inability to prevent leakage.
In summary, the key findings of the cystometry would be normal residual volume and normal detrusor contractions, highlighting the sphincteric defect as the underlying cause of the incontinence.<｜end▁of▁sentence｜>4. Memuat dan memroses dataset
Kita akan sedikit mengubah gaya prompt untuk pemrosesan dataset dengan menambahkan placeholder ketiga untuk kolom chain of thought yang kompleks.
train_prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context.
Write a response that appropriately completes the request.
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.
### Instruction:
You are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning.
Please answer the following medical question.
### Question:
{}
### Response:
<think>
{}
</think>
{}"""Tulis fungsi Python yang akan membuat kolom "text" pada dataset, yang terdiri dari gaya train prompt. Isi placeholder dengan pertanyaan, rantai pemikiran, dan jawaban.
EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN
def formatting_prompts_func(examples):
inputs = examples["Question"]
cots = examples["Complex_CoT"]
outputs = examples["Response"]
texts = []
for input, cot, output in zip(inputs, cots, outputs):
text = train_prompt_style.format(input, cot, output) + EOS_TOKEN
texts.append(text)
return {
"text": texts,
}Kita akan memuat 500 sampel pertama dari dataset FreedomIntelligence/medical-o1-reasoning-SFT, yang tersedia di Hugging Face hub. Setelah itu, kita akan memetakan kolom text menggunakan fungsi formatting_prompts_func.
from datasets import load_dataset
dataset = load_dataset("FreedomIntelligence/medical-o1-reasoning-SFT","en", split = "train[0:500]",trust_remote_code=True)
dataset = dataset.map(formatting_prompts_func, batched = True,)
dataset["text"][0]Seperti yang terlihat, kolom teks memiliki system prompt, instruksi, chain of thought, dan jawabannya.
"Below is an instruction that describes a task, paired with an input that provides further context. \nWrite a response that appropriately completes the request. \nBefore answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.\n\n### Instruction:\nYou are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. \nPlease answer the following medical question. \n\n### Question:\nA 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?\n\n### Response:\n<think>\nOkay, let's think about this step by step. There's a 61-year-old woman here who's been dealing with involuntary urine leakages whenever she's doing something that ups her abdominal pressure like coughing or sneezing. This sounds a lot like stress urinary incontinence to me. Now, it's interesting that she doesn't have any issues at night; she isn't experiencing leakage while sleeping. This likely means her bladder's ability to hold urine is fine when she isn't under physical stress. Hmm, that's a clue that we're dealing with something related to pressure rather than a bladder muscle problem. \n\nThe fact that she underwent a Q-tip test is intriguing too. This test is usually done to assess urethral mobility. In stress incontinence, a Q-tip might move significantly, showing urethral hypermobility. This kind of movement often means there's a weakness in the support structures that should help keep the urethra closed during increases in abdominal pressure. So, that's aligning well with stress incontinence.\n\nNow, let's think about what would happen during cystometry. Since stress incontinence isn't usually about sudden bladder contractions, I wouldn't expect to see involuntary detrusor contractions during this test. Her bladder isn't spasming or anything; it's more about the support structure failing under stress. Plus, she likely empties her bladder completely because stress incontinence doesn't typically involve incomplete emptying. So, her residual volume should be pretty normal. \n\nAll in all, it seems like if they do a cystometry on her, it will likely show a normal residual volume and no involuntary contractions. Yup, I think that makes sense given her symptoms and the typical presentations of stress urinary incontinence.\n</think>\nCystometry in this case of stress urinary incontinence would most likely reveal a normal post-void residual volume, as stress incontinence typically does not involve issues with bladder emptying. Additionally, since stress urinary incontinence is primarily related to physical exertion and not an overactive bladder, you would not expect to see any involuntary detrusor contractions during the test.<｜end▁of▁sentence｜>"5. Menyiapkan model
Dengan menggunakan target modules, kita akan menyiapkan model dengan menambahkan low-rank adopter ke model.
model = FastLanguageModel.get_peft_model(
model,
r=16,
target_modules=[
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
],
lora_alpha=16,
lora_dropout=0,
bias="none",
use_gradient_checkpointing="unsloth", # True or "unsloth" for very long context
random_state=3407,
use_rslora=False,
loftq_config=None,
)Selanjutnya, kita akan menyiapkan argumen pelatihan dan trainer dengan menyediakan model, tokenizer, dataset, dan parameter pelatihan penting lainnya yang akan mengoptimalkan proses fine-tuning kita.
from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=dataset,
dataset_text_field="text",
max_seq_length=max_seq_length,
dataset_num_proc=2,
args=TrainingArguments(
per_device_train_batch_size=2,
gradient_accumulation_steps=4,
# Use num_train_epochs = 1, warmup_ratio for full training runs!
warmup_steps=5,
max_steps=60,
learning_rate=2e-4,
fp16=not is_bfloat16_supported(),
bf16=is_bfloat16_supported(),
logging_steps=10,
optim="adamw_8bit",
weight_decay=0.01,
lr_scheduler_type="linear",
seed=3407,
output_dir="outputs",
),
)6. Pelatihan model
Jalankan perintah berikut untuk memulai pelatihan.
trainer_stats = trainer.train()Proses pelatihan memakan waktu 44 menit untuk selesai. Training loss berangsur berkurang, yang merupakan pertanda baik peningkatan kinerja model.
Anda dapat melihat laporan evaluasi model lengkap di dasbor Weights & Biases dengan masuk ke situs web dan melihat proyeknya.
Jika Anda mengalami masalah saat menjalankan kode di atas, silakan merujuk ke notebook Kaggle Fine-tuning DeepSeek R1 (Reasoning Model).
7. Inferensi model setelah fine-tuning
Untuk membandingkan hasilnya, kita akan menanyakan pertanyaan yang sama seperti sebelumnya kepada model yang telah di-fine-tune untuk melihat apa yang berubah.
question = "A 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?"
FastLanguageModel.for_inference(model) # Unsloth has 2x faster inference!
inputs = tokenizer([prompt_style.format(question, "")], return_tensors="pt").to("cuda")
outputs = model.generate(
input_ids=inputs.input_ids,
attention_mask=inputs.attention_mask,
max_new_tokens=1200,
use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("### Response:")[1])
Ini jauh lebih baik dan lebih akurat. Rantai pemikirannya langsung, dan jawabannya lugas dalam satu paragraf. Fine-tuning berhasil.
<think>
Okay, so let's think about this. We have a 61-year-old woman who's been dealing with involuntary urine loss during things like coughing or sneezing, but she's not leaking at night. That suggests she might have some kind of problem with her pelvic floor muscles or maybe her bladder.
Now, she's got a gynecological exam and a Q-tip test. Let's break that down. The Q-tip test is usually used to check for urethral obstruction. If it's positive, that means there's something blocking the urethra, like a urethral stricture or something else.
Given that she's had a positive Q-tip test, it's likely there's a urethral obstruction. That would mean her urethra is narrow, maybe due to a stricture or some kind of narrowing. So, her bladder can't empty properly during activities like coughing because the urethral obstruction is making it hard.
Now, let's think about what happens when her bladder can't empty. If there's a urethral obstruction, the bladder is forced to hold more urine, increasing the residual volume. That's because her bladder doesn't empty completely. So, her residual volume is probably increased.
Also, if her bladder can't empty properly, she might have increased detrusor contractions. These contractions are usually stronger to push the urine out. So, we expect her detrusor contractions to be increased.
Putting it all together, if she has a urethral obstruction and a positive Q-tip test, we'd expect her cystometry results to show increased residual volume and increased detrusor contractions. That makes sense because of the obstruction and how her bladder is trying to compensate by contracting more.
</think>
Based on the findings of the gynecological exam and the positive Q-tip test, it is most likely that the cystometry would reveal increased residual volume and increased detrusor contractions. The positive Q-tip test indicates urethral obstruction, which would force the bladder to retain more urine, thereby increasing the residual volume. Additionally, the obstruction can lead to increased detrusor contractions as the bladder tries to compensate by contracting more to expel the urine.<｜end▁of▁sentence｜>
8. Menyimpan model secara lokal
Sekarang, mari simpan adopter, model lengkap, dan tokenizer secara lokal agar kita dapat menggunakannya dalam proyek lain.
new_model_local = "DeepSeek-R1-Medical-COT"
model.save_pretrained(new_model_local)
tokenizer.save_pretrained(new_model_local)
model.save_pretrained_merged(new_model_local, tokenizer, save_method = "merged_16bit",)
9. Mendorong model ke Hugging Face Hub
Kita juga akan mendorong adopter, tokenizer, dan model ke Hugging Face Hub agar komunitas AI dapat memanfaatkan model ini dengan mengintegrasikannya ke dalam sistem mereka.
new_model_online = "kingabzpro/DeepSeek-R1-Medical-COT"
model.push_to_hub(new_model_online)
tokenizer.push_to_hub(new_model_online)
model.push_to_hub_merged(new_model_online, tokenizer, save_method = "merged_16bit")
Sumber: kingabzpro/DeepSeek-R1-Medical-COT · Hugging Face
Langkah berikutnya dalam perjalanan belajar Anda adalah menyajikan dan menerapkan model Anda ke cloud. Anda dapat mengikuti panduan Cara Menerapkan LLM dengan BentoML, yang menyediakan proses langkah demi langkah untuk menerapkan model bahasa besar secara efisien dan hemat biaya menggunakan BentoML dan alat seperti vLLM.
Sebagai alternatif, jika Anda lebih suka menggunakan model secara lokal, Anda dapat mengonversinya ke format GGUF dan menjalankannya di mesin Anda. Untuk itu, lihat panduan Fine-tuning Llama 3.2 dan Menggunakannya Secara Lokal, yang menyediakan instruksi terperinci untuk penggunaan lokal.
Kesimpulan
Perkembangan di bidang AI berlangsung sangat cepat. Komunitas open-source kini mengambil alih, menantang dominasi model proprietary yang telah menguasai lanskap AI selama tiga tahun terakhir.
Model bahasa besar (LLM) open-source menjadi semakin baik, lebih cepat, dan lebih efisien, sehingga memudahkan fine-tuning pada sumber daya komputasi dan memori yang lebih rendah.
Dalam tutorial ini, kita mengeksplorasi model penalaran DeepSeek R1 dan mempelajari cara melakukan fine-tuning versi distilasinya untuk tugas tanya jawab medis. Model penalaran yang telah di-fine-tune tidak hanya meningkatkan performa, tetapi juga memungkinkan penerapannya di bidang-bidang kritis seperti kedokteran, layanan darurat, dan kesehatan.
Untuk menandingi peluncuran DeepSeek R1, OpenAI memperkenalkan dua alat yang kuat: o3 dari OpenAI, model penalaran yang lebih canggih, dan Operator AI agent dari OpenAI, didukung oleh model Computer-Using Agent (CUA) baru, yang dapat menavigasi situs web dan melakukan tugas secara otonom.
