Fine-tuning di DeepSeek R1 (modello di ragionamento)

In questo tutorial eseguiremo il fine-tuning del modello DeepSeek-R1-Distill-Llama-8B sul Medical Chain-of-Thought Dataset da Hugging Face. Questo modello DeepSeek-R1 distillato è stato creato effettuando il fine-tuning del modello Llama 3.1 8B sui dati generati con DeepSeek-R1. Mostra capacità di ragionamento simili al modello originale.

Se sei alle prime armi con gli LLM e il fine-tuning, ti consiglio vivamente di seguire il corso Introduzione agli LLM in Python. Puoi anche consultare la nostra guida su Fine-tuning di DeepSeek R1-0528 su una RTX 4090.

Immagine dell’autore

Presentazione di DeepSeek R1

La società cinese DeepSeek AI ha reso open source i suoi modelli di ragionamento di prima generazione, DeepSeek-R1 e DeepSeek-R1-Zero, che rivaleggiano con o1 di OpenAI nelle prestazioni su compiti di ragionamento come matematica, coding e logica. Puoi leggere la nostra guida completa a DeepSeek R1 per saperne di più.

DeepSeek-R1-Zero

DeepSeek-R1-Zero è il primo modello open source addestrato esclusivamente con reinforcement learning (RL) su larga scala invece che con supervised fine-tuning (SFT) come passo iniziale. Questo approccio consente al modello di esplorare in autonomia il ragionamento a chain-of-thought (CoT), risolvere problemi complessi e perfezionare iterativamente le risposte. Tuttavia, presenta sfide come passaggi di ragionamento ripetitivi, scarsa leggibilità e mescolanza di lingue che possono compromettere chiarezza e usabilità.

DeepSeek-R1

DeepSeek-R1 è stato introdotto per superare i limiti di DeepSeek-R1-Zero incorporando dati di cold-start prima del reinforcement learning, fornendo una base solida per compiti di ragionamento e non. 

Questo addestramento in più fasi consente al modello di raggiungere prestazioni all’avanguardia, paragonabili a OpenAI-o1, su benchmark di matematica, codice e ragionamento, migliorando al contempo la leggibilità e la coerenza dell’output.

Distillazione DeepSeek

Oltre ai large language model che richiedono molta potenza di calcolo e memoria, DeepSeek ha introdotto anche modelli distillati. Questi modelli più piccoli ed efficienti hanno dimostrato di poter comunque ottenere notevoli prestazioni nel ragionamento. 

Con parametri da 1,5B a 70B, questi modelli mantengono forti capacità di ragionamento, con DeepSeek-R1-Distill-Qwen-32B che supera OpenAI-o1-mini su più benchmark. 

I modelli più piccoli ereditano i pattern di ragionamento dei modelli più grandi, a testimonianza dell’efficacia del processo di distillazione.

Fonte: deepseek-ai/DeepSeek-R1

Leggi il post DeepSeek-R1: caratteristiche, confronto con o1, modelli distillati e altro per scoprire funzionalità chiave, processo di sviluppo, modelli distillati, accesso, prezzi e confronto con OpenAI o1.

Fine-tuning di DeepSeek R1: guida passo passo

Per eseguire il fine-tuning del modello DeepSeek R1, puoi seguire i passaggi qui sotto: 

1. Configurazione

Per questo progetto usiamo Kaggle come Cloud IDE perché offre accesso gratuito alle GPU, spesso più potenti di quelle disponibili su Google Colab. Per iniziare, avvia un nuovo notebook Kaggle e aggiungi il tuo token di Hugging Face e il token di Weights & Biases come secrets.

Puoi aggiungere i secrets andando alla scheda Add-ons nell’interfaccia del notebook Kaggle e selezionando l’opzione Secrets. 

Dopo aver impostato i secrets, installa il pacchetto Python unsloth. Unsloth è un framework open source progettato per rendere il fine-tuning dei large language model (LLM) 2 volte più veloce e più efficiente in memoria.

Leggi la nostra Guida a Unsloth: ottimizza e accelera il fine-tuning degli LLM per scoprire le funzionalità chiave di Unsloth, le varie funzioni e come ottimizzare il tuo workflow di fine-tuning. 

%%capture
!pip install unsloth
!pip install --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git

Accedi alla CLI di Hugging Face usando l’API di Hugging Face che abbiamo estratto in modo sicuro da Kaggle Secrets. 

from huggingface_hub import login
from kaggle_secrets import UserSecretsClient
user_secrets = UserSecretsClient()

hf_token = user_secrets.get_secret("HUGGINGFACE_TOKEN")
login(hf_token)

Accedi a Weights & Biases (wandb) usando la tua API key e crea un nuovo progetto per tracciare esperimenti e avanzamento del fine-tuning.

import wandb

wb_token = user_secrets.get_secret("wandb")

wandb.login(key=wb_token)
run = wandb.init(
    project='Fine-tune-DeepSeek-R1-Distill-Llama-8B on Medical COT Dataset', 
    job_type="training", 
    anonymous="allow"
)

2. Caricamento del modello e del tokenizer

Per questo progetto carichiamo la versione Unsloth di DeepSeek-R1-Distill-Llama-8B.  Inoltre, caricheremo il modello con quantizzazione a 4 bit per ottimizzare l’uso della memoria e le prestazioni.

from unsloth import FastLanguageModel

max_seq_length = 2048 
dtype = None 
load_in_4bit = True


model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "unsloth/DeepSeek-R1-Distill-Llama-8B",
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
    token = hf_token, 
)

3. Inferenza del modello prima del fine-tuning

Per creare uno stile di prompt per il modello, definiremo un system prompt e includeremo segnaposto per la domanda e la generazione della risposta. Il prompt guiderà il modello a pensare passo dopo passo e a fornire una risposta logica e accurata.

prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context. 
Write a response that appropriately completes the request. 
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.

### Instruction:
You are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. 
Please answer the following medical question. 

### Question:
{}

### Response:
<think>{}"""

In questo esempio forniremo una domanda medica a prompt_style, la convertiremo in token e poi passeremo i token al modello per la generazione della risposta. 

question = "A 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?"


FastLanguageModel.for_inference(model) 
inputs = tokenizer([prompt_style.format(question, "")], return_tensors="pt").to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("### Response:")[1])

Anche senza fine-tuning, il nostro modello ha generato con successo una chain of thought e ha fornito il ragionamento prima della risposta finale. Il processo di ragionamento è racchiuso tra i tag <think></think>.

Perché allora serve comunque il fine-tuning? Il ragionamento, sebbene dettagliato, era prolisso e poco conciso. Inoltre, la risposta finale era presentata in formato elenco puntato, discostandosi dalla struttura e dallo stile del dataset su cui vogliamo eseguire il fine-tuning. 

<think>
Okay, so I have this medical question to answer. Let me try to break it down. The patient is a 61-year-old woman with a history of involuntary urine loss during activities like coughing or sneezing, but she doesn't leak at night. She's had a gynecological exam and a Q-tip test. I need to figure out what cystometry would show regarding her residual volume and detrusor contractions.

First, I should recall what I know about urinary incontinence. Involuntary urine loss during activities like coughing or sneezing makes me think of stress urinary incontinence. Stress incontinence typically happens when the urethral sphincter isn't strong enough to resist increased abdominal pressure from activities like coughing, laughing, or sneezing. This usually affects women, especially after childbirth when the pelvic muscles and ligaments are weakened.

The Q-tip test is a common diagnostic tool for stress urinary incontinence. The test involves inserting a Q-tip catheter, which is a small balloon catheter, into the urethra. The catheter is connected to a pressure gauge. The patient is asked to cough, and the pressure reading is taken. If the pressure is above normal (like above 100 mmHg), it suggests that the urethral sphincter isn't closing properly, which is a sign of stress incontinence.

So, based on the history and the Q-tip test, the diagnosis is likely stress urinary incontinence. Now, moving on to what cystometry would show. Cystometry, also known as a filling cystometry, is a diagnostic procedure where a catheter is inserted into the bladder, and the bladder is filled with a liquid to measure how much it can hold (residual volume) and how it responds to being filled (like during a cough or sneeze). This helps in assessing the capacity and compliance of the bladder.

In a patient with stress incontinence, the bladder's capacity might be normal, but the sphincter's function is impaired. So, during the cystometry, the residual volume might be within normal limits because the bladder isn't overfilled. However, when the patient is asked to cough or perform a Valsalva maneuver, the detrusor muscle (the smooth muscle layer of the bladder) might not contract effectively, leading to an increase in intra-abdominal pressure, which might cause leakage.

Wait, but detrusor contractions are usually associated with voiding. In stress incontinence, the issue isn't with the detrusor contractions but with the sphincter's inability to prevent leakage. So, during cystometry, the detrusor contractions would be normal because they are part of the normal voiding process. However, the problem is that the sphincter doesn't close properly, leading to leakage.

So, putting it all together, the residual volume might be normal, but the detrusor contractions would be normal as well. The key finding would be the impaired sphincter function leading to incontinence, which is typically demonstrated during the Q-tip test and clinical history. Therefore, the cystometry would likely show normal residual volume and normal detrusor contractions, but the underlying issue is the sphincter's inability to prevent leakage.
</think>

Based on the provided information, the cystometry findings in this 61-year-old woman with stress urinary incontinence would likely demonstrate the following:

1. **Residual Volume**: The residual volume would be within normal limits. This is because the bladder's capacity is typically normal in cases of stress incontinence, where the primary issue lies with the sphincter function rather than the bladder's capacity.

2. **Detrusor Contractions**: The detrusor contractions would also be normal. These contractions are part of the normal voiding process and are not impaired in stress urinary incontinence. The issue is not with the detrusor muscle but with the sphincter's inability to prevent leakage.

In summary, the key findings of the cystometry would be normal residual volume and normal detrusor contractions, highlighting the sphincteric defect as the underlying cause of the incontinence.<｜end▁of▁sentence｜>

4. Caricamento e processazione del dataset

Modificheremo leggermente lo stile del prompt per processare il dataset aggiungendo un terzo segnaposto per la colonna con la complessa chain of thought. 

train_prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context. 
Write a response that appropriately completes the request. 
Before answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.

### Instruction:
You are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. 
Please answer the following medical question. 

### Question:
{}

### Response:
<think>
{}
</think>
{}"""

Scrivi la funzione Python che creerà una colonna "text" nel dataset, composta dallo stile di prompt per il training. Compila i segnaposto con domande, chain of thought e risposte. 

EOS_TOKEN = tokenizer.eos_token  # Must add EOS_TOKEN


def formatting_prompts_func(examples):
    inputs = examples["Question"]
    cots = examples["Complex_CoT"]
    outputs = examples["Response"]
    texts = []
    for input, cot, output in zip(inputs, cots, outputs):
        text = train_prompt_style.format(input, cot, output) + EOS_TOKEN
        texts.append(text)
    return {
        "text": texts,
    }

Caricheremo i primi 500 campioni dal dataset FreedomIntelligence/medical-o1-reasoning-SFT, disponibile su Hugging Face Hub. Dopodiché mapperemo la colonna text usando la funzione formatting_prompts_func.

from datasets import load_dataset
dataset = load_dataset("FreedomIntelligence/medical-o1-reasoning-SFT","en", split = "train[0:500]",trust_remote_code=True)
dataset = dataset.map(formatting_prompts_func, batched = True,)
dataset["text"][0]

Come possiamo vedere, la colonna text contiene un system prompt, istruzioni, chain of thought e la risposta. 

"Below is an instruction that describes a task, paired with an input that provides further context. \nWrite a response that appropriately completes the request. \nBefore answering, think carefully about the question and create a step-by-step chain of thoughts to ensure a logical and accurate response.\n\n### Instruction:\nYou are a medical expert with advanced knowledge in clinical reasoning, diagnostics, and treatment planning. \nPlease answer the following medical question. \n\n### Question:\nA 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?\n\n### Response:\n<think>\nOkay, let's think about this step by step. There's a 61-year-old woman here who's been dealing with involuntary urine leakages whenever she's doing something that ups her abdominal pressure like coughing or sneezing. This sounds a lot like stress urinary incontinence to me. Now, it's interesting that she doesn't have any issues at night; she isn't experiencing leakage while sleeping. This likely means her bladder's ability to hold urine is fine when she isn't under physical stress. Hmm, that's a clue that we're dealing with something related to pressure rather than a bladder muscle problem. \n\nThe fact that she underwent a Q-tip test is intriguing too. This test is usually done to assess urethral mobility. In stress incontinence, a Q-tip might move significantly, showing urethral hypermobility. This kind of movement often means there's a weakness in the support structures that should help keep the urethra closed during increases in abdominal pressure. So, that's aligning well with stress incontinence.\n\nNow, let's think about what would happen during cystometry. Since stress incontinence isn't usually about sudden bladder contractions, I wouldn't expect to see involuntary detrusor contractions during this test. Her bladder isn't spasming or anything; it's more about the support structure failing under stress. Plus, she likely empties her bladder completely because stress incontinence doesn't typically involve incomplete emptying. So, her residual volume should be pretty normal. \n\nAll in all, it seems like if they do a cystometry on her, it will likely show a normal residual volume and no involuntary contractions. Yup, I think that makes sense given her symptoms and the typical presentations of stress urinary incontinence.\n</think>\nCystometry in this case of stress urinary incontinence would most likely reveal a normal post-void residual volume, as stress incontinence typically does not involve issues with bladder emptying. Additionally, since stress urinary incontinence is primarily related to physical exertion and not an overactive bladder, you would not expect to see any involuntary detrusor contractions during the test.<｜end▁of▁sentence｜>"

5. Impostazione del modello

Usando i moduli target, imposteremo il modello aggiungendo il low-rank adapter al modello. 

model = FastLanguageModel.get_peft_model(
    model,
    r=16,  
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
    ],
    lora_alpha=16,
    lora_dropout=0,  
    bias="none",  
    use_gradient_checkpointing="unsloth",  # True or "unsloth" for very long context
    random_state=3407,
    use_rslora=False,  
    loftq_config=None,
)

Successivamente imposteremo gli argomenti di training e il trainer fornendo modello, tokenizer, dataset e altri parametri di training importanti che ottimizzeranno il nostro processo di fine-tuning.

from trl import SFTTrainer
from transformers import TrainingArguments
from unsloth import is_bfloat16_supported

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        # Use num_train_epochs = 1, warmup_ratio for full training runs!
        warmup_steps=5,
        max_steps=60,
        learning_rate=2e-4,
        fp16=not is_bfloat16_supported(),
        bf16=is_bfloat16_supported(),
        logging_steps=10,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
    ),
)

6. Training del modello

Esegui il seguente comando per avviare il training.  

trainer_stats = trainer.train()

Il processo di training ha impiegato 44 minuti per terminare. La training loss si è ridotta gradualmente, segnale positivo di un miglioramento delle prestazioni del modello.

Puoi visualizzare il report completo di valutazione del modello sulla dashboard di Weights & Biases accedendo al sito e aprendo il progetto.

Se riscontri problemi durante l’esecuzione del codice sopra, fai riferimento al notebook Kaggle Fine-tuning di DeepSeek R1 (modello di ragionamento). 

7. Inferenza del modello dopo il fine-tuning

Per confrontare i risultati, porremo al modello fine-tunato la stessa domanda di prima per vedere cosa è cambiato.

question = "A 61-year-old woman with a long history of involuntary urine loss during activities like coughing or sneezing but no leakage at night undergoes a gynecological exam and Q-tip test. Based on these findings, what would cystometry most likely reveal about her residual volume and detrusor contractions?"


FastLanguageModel.for_inference(model)  # Unsloth has 2x faster inference!
inputs = tokenizer([prompt_style.format(question, "")], return_tensors="pt").to("cuda")

outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1200,
    use_cache=True,
)
response = tokenizer.batch_decode(outputs)
print(response[0].split("### Response:")[1])

Molto meglio e più accurato. La chain of thought è stata diretta e la risposta chiara e in un unico paragrafo. Il fine-tuning è riuscito.

<think>
Okay, so let's think about this. We have a 61-year-old woman who's been dealing with involuntary urine loss during things like coughing or sneezing, but she's not leaking at night. That suggests she might have some kind of problem with her pelvic floor muscles or maybe her bladder.

Now, she's got a gynecological exam and a Q-tip test. Let's break that down. The Q-tip test is usually used to check for urethral obstruction. If it's positive, that means there's something blocking the urethra, like a urethral stricture or something else.

Given that she's had a positive Q-tip test, it's likely there's a urethral obstruction. That would mean her urethra is narrow, maybe due to a stricture or some kind of narrowing. So, her bladder can't empty properly during activities like coughing because the urethral obstruction is making it hard.

Now, let's think about what happens when her bladder can't empty. If there's a urethral obstruction, the bladder is forced to hold more urine, increasing the residual volume. That's because her bladder doesn't empty completely. So, her residual volume is probably increased.

Also, if her bladder can't empty properly, she might have increased detrusor contractions. These contractions are usually stronger to push the urine out. So, we expect her detrusor contractions to be increased.

Putting it all together, if she has a urethral obstruction and a positive Q-tip test, we'd expect her cystometry results to show increased residual volume and increased detrusor contractions. That makes sense because of the obstruction and how her bladder is trying to compensate by contracting more.
</think>
Based on the findings of the gynecological exam and the positive Q-tip test, it is most likely that the cystometry would reveal increased residual volume and increased detrusor contractions. The positive Q-tip test indicates urethral obstruction, which would force the bladder to retain more urine, thereby increasing the residual volume. Additionally, the obstruction can lead to increased detrusor contractions as the bladder tries to compensate by contracting more to expel the urine.<｜end▁of▁sentence｜>

8. Salvataggio del modello in locale

Ora salviamo localmente l’adapter, il modello completo e il tokenizer in modo da poterli usare in altri progetti.

new_model_local = "DeepSeek-R1-Medical-COT"
model.save_pretrained(new_model_local) 
tokenizer.save_pretrained(new_model_local)

model.save_pretrained_merged(new_model_local, tokenizer, save_method = "merged_16bit",)

9. Pubblicazione del modello su Hugging Face Hub

Pubblicheremo anche l’adapter, il tokenizer e il modello su Hugging Face Hub così che la community AI possa integrare questo modello nei propri sistemi.

new_model_online = "kingabzpro/DeepSeek-R1-Medical-COT"
model.push_to_hub(new_model_online)
tokenizer.push_to_hub(new_model_online)

model.push_to_hub_merged(new_model_online, tokenizer, save_method = "merged_16bit")

Fonte: kingabzpro/DeepSeek-R1-Medical-COT · Hugging Face

Il prossimo passo del tuo percorso di apprendimento è servire e distribuire il modello nel cloud. Puoi seguire la guida Come distribuire gli LLM con BentoML, che fornisce un processo passo passo per distribuire grandi modelli linguistici in modo efficiente ed economico usando BentoML e strumenti come vLLM.

In alternativa, se preferisci usare il modello in locale, puoi convertirlo in formato GGUF ed eseguirlo sulla tua macchina. Per questo, consulta la guida Fine-tuning di Llama 3.2 e utilizzo in locale, che fornisce istruzioni dettagliate per l’uso locale.

Conclusione

Le cose stanno cambiando rapidamente nel campo dell’IA. La community open source sta prendendo il sopravvento, mettendo in discussione il predominio dei modelli proprietari che hanno dominato il panorama dell’IA negli ultimi tre anni. 

I large language model (LLM) open source stanno diventando migliori, più veloci ed efficienti, rendendo più facile che mai eseguire il fine-tuning con risorse di calcolo e memoria ridotte.

In questo tutorial abbiamo esplorato il modello di ragionamento DeepSeek R1 e imparato come eseguire il fine-tuning della sua versione distillata per compiti di Q&A medico. Un modello di ragionamento fine-tunato non solo migliora le prestazioni, ma abilita anche l’applicazione in ambiti critici come medicina, servizi di emergenza e sanità.

Per controbilanciare il lancio di DeepSeek R1, OpenAI ha introdotto due potenti strumenti: o3 di OpenAI, un modello di ragionamento più avanzato, e l’agente Operator di OpenAI, alimentato dal nuovo modello Computer-Using Agent (CUA), in grado di navigare autonomamente sui siti web ed eseguire attività. 

• OpenAI O3: caratteristiche, confronto con O1, data di rilascio e altro
• Operator di OpenAI: esempi, casi d’uso, concorrenza e altro