Fine-tunen van DeepSeek R1 (Reasoning Model)

In deze tutorial fine-tunen we het DeepSeek-R1-Distill-Llama-8B-model op de Medical Chain-of-Thought-dataset van Hugging Face. Dit gedistilleerde DeepSeek-R1-model is gemaakt door Llama 3.1 8B te fine-tunen op data die met DeepSeek-R1 is gegenereerd. Het toont vergelijkbare redeneercapaciteiten als het oorspronkelijke model.

Ben je nieuw met LLM’s en fine-tunen, dan raad ik je zeker de Introductie tot LLM’s in Python-cursus aan. Bekijk ook onze gids over Fine-tunen van DeepSeek R1-0528 op een RTX 4090.

Afbeelding door de auteur

Introductie van DeepSeek R1

Het Chinese AI-bedrijf DeepSeek AI heeft zijn eerstegeneratie-redeneermodellen, DeepSeek-R1 en DeepSeek-R1-Zero, open-source gemaakt. Ze evenaren OpenAI’s o1 op redeneertaken zoals wiskunde, coderen en logica. Lees onze volledige gids over DeepSeek R1 voor meer info.

DeepSeek-R1-Zero

DeepSeek-R1-Zero is het eerste open-source model dat uitsluitend is getraind met grootschalige reinforcement learning (RL) in plaats van supervised fine-tuning (SFT) als eerste stap. Deze aanpak stelt het model in staat om zelfstandig chain-of-thought-redeneringen (CoT) te verkennen, complexe problemen op te lossen en iteratief zijn outputs te verfijnen. Er zijn echter uitdagingen, zoals repetitieve redeneerstappen, matige leesbaarheid en taalmenging, die de helderheid en bruikbaarheid kunnen beïnvloeden.

DeepSeek-R1

DeepSeek-R1 werd geïntroduceerd om de beperkingen van DeepSeek-R1-Zero te overwinnen door cold-startdata toe te voegen vóór reinforcement learning. Zo krijgt het model een sterke basis voor zowel redeneer- als niet-redeneertaken. 

Deze meerfasige training zorgt voor state-of-the-art prestaties, vergelijkbaar met OpenAI-o1, op benchmarks voor wiskunde, code en redeneren, terwijl de leesbaarheid en samenhang van de outputs verbeteren.

DeepSeek-distillatie

Naast grote taalmodellen die veel rekenkracht en geheugen vereisen, heeft DeepSeek ook gedistilleerde modellen geïntroduceerd. Deze kleinere, efficiëntere modellen laten zien dat ze nog steeds opvallende redeneerprestaties kunnen leveren. 

Met 1,5B tot 70B parameters behouden deze modellen sterke redeneercapaciteiten, waarbij DeepSeek-R1-Distill-Qwen-32B OpenAI-o1-mini overtreft op meerdere benchmarks. 

Kleinere modellen nemen de redeneerpatronen van grotere modellen over, wat de effectiviteit van het distillatieproces aantoont.

Bron: deepseek-ai/DeepSeek-R1

Lees de blog DeepSeek-R1: Features, o1-vergelijking, gedistilleerde modellen & meer om te leren over de belangrijkste features, het ontwikkelproces, gedistilleerde modellen, toegang, prijzen en de vergelijking met OpenAI o1.

DeepSeek R1 fine-tunen: stapsgewijze gids

Volg de onderstaande stappen om het DeepSeek R1-model te fine-tunen: 

1. Installatie en setup

Voor dit project gebruiken we Kaggle als onze Cloud IDE, omdat het gratis toegang tot GPU’s biedt die vaak krachtiger zijn dan die van Google Colab. Start een nieuw Kaggle-notebook en voeg je Hugging Face-token en Weights & Biases-token toe als secrets.

Je kunt secrets toevoegen via het tabblad Add-ons in de Kaggle-notebookinterface en vervolgens de optie Secrets selecteren. 

Installeer daarna het Python-pakket unsloth. Unsloth is een open-source framework dat het fine-tunen van grote taalmodellen (LLM’s) 2x sneller en geheugen-efficiënter maakt.

Lees onze Unsloth-gids: LLM-fine-tuning optimaliseren en versnellen om te leren over de belangrijkste features van Unsloth, de functies en hoe je je fine-tuningworkflow optimaliseert. 

%%capture
!pip install unsloth
!pip install --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git

Log in bij de Hugging Face CLI met de Hugging Face API die we veilig uit Kaggle Secrets hebben opgehaald. 

from huggingface_hub import login
from kaggle_secrets import UserSecretsClient
user_secrets = UserSecretsClient()

hf_token = user_secrets.get_secret("HUGGINGFACE_TOKEN")
login(hf_token)

Log in bij Weights & Biases (wandb) met je API-sleutel en maak een nieuw project aan om de experimenten en de voortgang van het fine-tunen bij te houden.

import wandb

wb_token = user_secrets.get_secret("wandb")

wandb.login(key=wb_token)
run = wandb.init(
    project='Fine-tune-DeepSeek-R1-Distill-Llama-8B on Medical COT Dataset', 
    job_type="training", 
    anonymous="allow"
)

2. Het model en de tokenizer laden

Voor dit project laden we de Unsloth-versie van DeepSeek-R1-Distill-Llama-8B.  Daarnaast laden we het model met 4-bit kwantisatie om het geheugengebruik en de performance te optimaliseren.

from unsloth import FastLanguageModel

max_seq_length = 2048 
dtype = None 
load_in_4bit = True


model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/DeepSeek-R1-Distill-Llama-8B",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
    token = hf_token, 
)

3. Modelinference vóór fine-tunen

Om een promptstijl voor het model te maken, definiëren we een systeemprompt en voegen we placeholders toe voor de vraag en de reactie. De prompt stuurt het model aan om stap voor stap te redeneren en een logische, nauwkeurige reactie te geven.

prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context. 
Write a response that appropriately completes the request. 
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.

### Instruction:
You are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. 
Please answer the following medical question. 

### Question:
{}

### Response:
<think>{}"""

In dit voorbeeld geven we een medische vraag mee aan prompt_style, zetten die om in tokens en voeren de tokens door het model om een reactie te genereren. 

question = "A 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?"


FastLanguageModel.for_inference(model) 
inputs = tokenizer([prompt_style.format(question, "")], return_tensors="pt").to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("### Response:")[1])

Zelfs zonder fine-tuning genereerde ons model succesvol een chain of thought en gaf het redenering voordat het het eindantwoord presenteerde. Het redeneerproces staat tussen de <think></think>-tags.

Waarom is fine-tunen dan nog nodig? Het redeneerproces was weliswaar gedetailleerd, maar ook omslachtig en niet bondig. Bovendien werd het eindantwoord als opsomming gepresenteerd, wat afwijkt van de structuur en stijl van de dataset waarop we willen fine-tunen. 

<think>
Okay, so I have this medical question to answer. Let me try to break it down. The patient is a 61-year-old woman with a history of involuntary urine loss during activities like coughing or sneezing, but she doesn't leak at night. She's had a gynecological exam and a Q-tip test. I need to figure out what cystometry would show regarding her residual volume and detrusor contractions.

First, I should recall what I know about urinary incontinence. Involuntary urine loss during activities like coughing or sneezing makes me think of stress urinary incontinence. Stress incontinence typically happens when the urethral sphincter isn't strong enough to resist increased abdominal pressure from activities like coughing, laughing, or sneezing. This usually affects women, especially after childbirth when the pelvic muscles and ligaments are weakened.

The Q-tip test is a common diagnostic tool for stress urinary incontinence. The test involves inserting a Q-tip catheter, which is a small balloon catheter, into the urethra. The catheter is connected to a pressure gauge. The patient is asked to cough, and the pressure reading is taken. If the pressure is above normal (like above 100 mmHg), it suggests that the urethral sphincter isn't closing properly, which is a sign of stress incontinence.

So, based on the history and the Q-tip test, the diagnosis is likely stress urinary incontinence. Now, moving on to what cystometry would show. Cystometry, also known as a filling cystometry, is a diagnostic procedure where a catheter is inserted into the bladder, and the bladder is filled with a liquid to measure how much it can hold (residual volume) and how it responds to being filled (like during a cough or sneeze). This helps in assessing the capacity and compliance of the bladder.

In a patient with stress incontinence, the bladder's capacity might be normal, but the sphincter's function is impaired. So, during the cystometry, the residual volume might be within normal limits because the bladder isn't overfilled. However, when the patient is asked to cough or perform a Valsalva maneuver, the detrusor muscle (the smooth muscle layer of the bladder) might not contract effectively, leading to an increase in intra-abdominal pressure, which might cause leakage.

Wait, but detrusor contractions are usually associated with voiding. In stress incontinence, the issue isn't with the detrusor contractions but with the sphincter's inability to prevent leakage. So, during cystometry, the detrusor contractions would be normal because they are part of the normal voiding process. However, the problem is that the sphincter doesn't close properly, leading to leakage.

So, putting it all together, the residual volume might be normal, but the detrusor contractions would be normal as well. The key finding would be the impaired sphincter function leading to incontinence, which is typically demonstrated during the Q-tip test and clinical history. Therefore, the cystometry would likely show normal residual volume and normal detrusor contractions, but the underlying issue is the sphincter's inability to prevent leakage.
</think>

Based on the provided information, the cystometry findings in this 61-year-old woman with stress urinary incontinence would likely demonstrate the following:

1. **Residual Volume**: The residual volume would be within normal limits. This is because the bladder's capacity is typically normal in cases of stress incontinence, where the primary issue lies with the sphincter function rather than the bladder's capacity.

2. **Detrusor Contractions**: The detrusor contractions would also be normal. These contractions are part of the normal voiding process and are not impaired in stress urinary incontinence. The issue is not with the detrusor muscle but with the sphincter's inability to prevent leakage.

In summary, the key findings of the cystometry would be normal residual volume and normal detrusor contractions, highlighting the sphincteric defect as the underlying cause of the incontinence.<｜end▁of▁sentence｜>

4. De dataset laden en verwerken

We passen de promptstijl licht aan voor de datasetverwerking door een derde placeholder toe te voegen voor de kolom met de complexe chain of thought. 

train_prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context. 
Write a response that appropriately completes the request. 
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.

### Instruction:
You are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. 
Please answer the following medical question. 

### Question:
{}

### Response:
<think>
{}
</think>
{}"""

Schrijf de Python-functie die een kolom "text" in de dataset maakt, bestaande uit de train-promptstijl. Vul de placeholders met vragen, chains of thought en antwoorden. 

EOS_TOKEN = tokenizer.eos_token  # Must add EOS_TOKEN


def formatting_prompts_func(examples):
    inputs = examples["Question"]
    cots = examples["Complex_CoT"]
    outputs = examples["Response"]
    texts = []
    for input, cot, output in zip(inputs, cots, outputs):
        text = train_prompt_style.format(input, cot, output) + EOS_TOKEN
        texts.append(text)
    return {
        "text": texts,
    }

We laden de eerste 500 samples uit de FreedomIntelligence/medical-o1-reasoning-SFT-dataset, beschikbaar op de Hugging Face Hub. Daarna mappen we de kolom text met de functie formatting_prompts_func.

from datasets import load_dataset
dataset = load_dataset("FreedomIntelligence/medical-o1-reasoning-SFT","en", split = "train[0:500]",trust_remote_code=True)
dataset = dataset.map(formatting_prompts_func, batched = True,)
dataset["text"][0]

Zoals je ziet bevat de tekstkolom een systeemprompt, instructies, chain of thought en het antwoord. 

"Below is an instruction that describes a task, paired with an input that provides further context. \nWrite a response that appropriately completes the request. \nBefore answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.\n\n### Instruction:\nYou are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. \nPlease answer the following medical question. \n\n### Question:\nA 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?\n\n### Response:\n<think>\nOkay, let's think about this step by step. There's a 61-year-old woman here who's been dealing with involuntary urine leakages whenever she's doing something that ups her abdominal pressure like coughing or sneezing. This sounds a lot like stress urinary incontinence to me. Now, it's interesting that she doesn't have any issues at night; she isn't experiencing leakage while sleeping. This likely means her bladder's ability to hold urine is fine when she isn't under physical stress. Hmm, that's a clue that we're dealing with something related to pressure rather than a bladder muscle problem. \n\nThe fact that she underwent a Q-tip test is intriguing too. This test is usually done to assess urethral mobility. In stress incontinence, a Q-tip might move significantly, showing urethral hypermobility. This kind of movement often means there's a weakness in the support structures that should help keep the urethra closed during increases in abdominal pressure. So, that's aligning well with stress incontinence.\n\nNow, let's think about what would happen during cystometry. Since stress incontinence isn't usually about sudden bladder contractions, I wouldn't expect to see involuntary detrusor contractions during this test. Her bladder isn't spasming or anything; it's more about the support structure failing under stress. Plus, she likely empties her bladder completely because stress incontinence doesn't typically involve incomplete emptying. So, her residual volume should be pretty normal. \n\nAll in all, it seems like if they do a cystometry on her, it will likely show a normal residual volume and no involuntary contractions. Yup, I think that makes sense given her symptoms and the typical presentations of stress urinary incontinence.\n</think>\nCystometry in this case of stress urinary incontinence would most likely reveal a normal post-void residual volume, as stress incontinence typically does not involve issues with bladder emptying. Additionally, since stress urinary incontinence is primarily related to physical exertion and not an overactive bladder, you would not expect to see any involuntary detrusor contractions during the test.<｜end▁of▁sentence｜>"

5. Het model configureren

Met behulp van de targetmodules richten we het model in door een low-rank adapter aan het model toe te voegen. 

model = FastLanguageModel.get_peft_model(
    model,
    r=16,  
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
    lora_alpha=16,
    lora_dropout=0,  
    bias="none",  
    use_gradient_checkpointing="unsloth",  # True or "unsloth" for very long context
    random_state=3407,
    use_rslora=False,  
    loftq_config=None,
)

Vervolgens stellen we de trainingsargumenten en de trainer in door het model, de tokenizer, de dataset en andere belangrijke trainingsparameters op te geven die ons fine-tuningproces optimaliseren.

from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        # Use num_train_epochs = 1, warmup_ratio for full training runs!
        warmup_steps=5,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=10,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
    ),
)

6. Modeltraining

Voer het volgende commando uit om de training te starten.  

trainer_stats = trainer.train()

Het trainingsproces duurde 44 minuten. De training loss daalde geleidelijk, wat een goed teken is voor betere modelprestaties.

Je kunt het volledige model-evaluatierapport bekijken op het Weights & Biases-dashboard door in te loggen op de website en het project te openen. 

Krijg je problemen met het draaien van bovenstaande code, raadpleeg dan het Fine-tunen van DeepSeek R1 (Reasoning Model)-Kaggle-notebook. 

7. Modelinference na fine-tunen

Om de resultaten te vergelijken, stellen we het fijngestemde model dezelfde vraag als eerder om te zien wat er is veranderd.

question = "A 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?"


FastLanguageModel.for_inference(model)  # Unsloth has 2x faster inference!
inputs = tokenizer([prompt_style.format(question, "")], return_tensors="pt").to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("### Response:")[1])

Dit is veel beter en nauwkeuriger. De chain of thought was to-the-point en het antwoord was helder en in één alinea. Het fine-tunen was succesvol.

<think>
Okay, so let's think about this. We have a 61-year-old woman who's been dealing with involuntary urine loss during things like coughing or sneezing, but she's not leaking at night. That suggests she might have some kind of problem with her pelvic floor muscles or maybe her bladder.

Now, she's got a gynecological exam and a Q-tip test. Let's break that down. The Q-tip test is usually used to check for urethral obstruction. If it's positive, that means there's something blocking the urethra, like a urethral stricture or something else.

Given that she's had a positive Q-tip test, it's likely there's a urethral obstruction. That would mean her urethra is narrow, maybe due to a stricture or some kind of narrowing. So, her bladder can't empty properly during activities like coughing because the urethral obstruction is making it hard.

Now, let's think about what happens when her bladder can't empty. If there's a urethral obstruction, the bladder is forced to hold more urine, increasing the residual volume. That's because her bladder doesn't empty completely. So, her residual volume is probably increased.

Also, if her bladder can't empty properly, she might have increased detrusor contractions. These contractions are usually stronger to push the urine out. So, we expect her detrusor contractions to be increased.

Putting it all together, if she has a urethral obstruction and a positive Q-tip test, we'd expect her cystometry results to show increased residual volume and increased detrusor contractions. That makes sense because of the obstruction and how her bladder is trying to compensate by contracting more.
</think>
Based on the findings of the gynecological exam and the positive Q-tip test, it is most likely that the cystometry would reveal increased residual volume and increased detrusor contractions. The positive Q-tip test indicates urethral obstruction, which would force the bladder to retain more urine, thereby increasing the residual volume. Additionally, the obstruction can lead to increased detrusor contractions as the bladder tries to compensate by contracting more to expel the urine.<｜end▁of▁sentence｜>

8. Het model lokaal opslaan

Laten we nu de adapter, het volledige model en de tokenizer lokaal opslaan zodat we ze in andere projecten kunnen gebruiken.

new_model_local = "DeepSeek-R1-Medical-COT"
model.save_pretrained(new_model_local) 
tokenizer.save_pretrained(new_model_local)

model.save_pretrained_merged(new_model_local, tokenizer, save_method = "merged_16bit",)

9. Het model pushen naar Hugging Face Hub

We pushen ook de adapter, tokenizer en het model naar Hugging Face Hub, zodat de AI-community dit model kan benutten en integreren in hun systemen.

new_model_online = "kingabzpro/DeepSeek-R1-Medical-COT"
model.push_to_hub(new_model_online)
tokenizer.push_to_hub(new_model_online)

model.push_to_hub_merged(new_model_online, tokenizer, save_method = "merged_16bit")

Bron: kingabzpro/DeepSeek-R1-Medical-COT · Hugging Face

De volgende stap in je leertraject is je model als service aanbieden en in de cloud deployen. Volg de gids How to Deploy LLMs with BentoML, die een stapsgewijs proces biedt om grote taalmodellen efficiënt en kosteneffectief te deployen met BentoML en tools als vLLM.

Wil je het model liever lokaal gebruiken, converteer het dan naar GGUF-formaat en draai het op je eigen machine. Bekijk hiervoor de gids Fine-tuning Llama 3.2 en lokaal gebruiken met gedetailleerde instructies voor lokaal gebruik.

Conclusie

Er verandert razendsnel veel in AI. De open-sourcecommunity neemt het nu over en daagt de dominantie uit van propriëtaire modellen die de afgelopen drie jaar het AI-landschap bepaalden. 

Open-source large language models (LLM’s) worden beter, sneller en efficiënter, waardoor fine-tunen met minder rekenkracht en geheugen eenvoudiger dan ooit is.

In deze tutorial hebben we het DeepSeek R1-redeneermodel verkend en geleerd hoe je de gedistilleerde versie fine-tunet voor medische Q&A-taken. Een fijngestemd redeneermodel verbetert niet alleen de prestaties, maar maakt ook toepassingen mogelijk in kritieke domeinen zoals geneeskunde, hulpdiensten en zorg.

Als reactie op de lancering van DeepSeek R1 introduceerde OpenAI twee krachtige tools: OpenAI’s o3, een geavanceerder redeneemodel, en OpenAI’s Operator AI-agent, aangedreven door het nieuwe Computer-Using Agent (CUA)-model, dat zelfstandig websites kan navigeren en taken kan uitvoeren. 

• OpenAI’s O3: Features, O1-vergelijking, releasedatum & meer
• OpenAI’s Operator: voorbeelden, use-cases, concurrentie & meer