RankGPT come agente di riordinamento per RAG (Tutorial)

Capire la Retrieval Augmented Generation (RAG)

La retrieval augmented generation (RAG) è un metodo che combina LLM e sistemi di information retrieval. Significa che quando si chiede a un LLM di generare testo, può attingere informazioni rilevanti da fonti esterne, rendendo le sue risposte più accurate e informate.

RAG è composto da due componenti principali — il retriever e il generator — e da un componente opzionale, il reranker:

1. Retriever — Il compito del retriever è trovare documenti o segmenti di testo rilevanti da un ampio insieme di documenti in base alla query dell’utente. Usa algoritmi come BM25 per classificare i documenti per rilevanza.
2. Reranker (opzionale) — Il reranker prende il set iniziale di documenti recuperati e li riordina per assicurare che i più rilevanti siano in cima. Aiuta a filtrare le informazioni meno utili e a concentrarsi su ciò che conta.
3. Generator — Il generator è l’LLM che usa i documenti recuperati per generare l’output finale. L’accesso a dati esterni pertinenti può produrre risposte più accurate.

Ruolo e vantaggi di RankGPT in RAG

RankGPT utilizza gli LLM per valutare la rilevanza dei documenti o dei segmenti di testo recuperati, assicurando che i più importanti siano in cima. Con RankGPT, il generator nella pipeline RAG riceve input di qualità superiore, ottenendo risposte più accurate.

Rilevanza e prestazioni migliorate

RankGPT va oltre il semplice matching di keyword comprendendo il significato profondo e il contesto di query e documenti. Questo gli consente di fornire informazioni più accurate agli LLM, identificando i contenuti più rilevanti in base al loro significato reale, non solo alle parole chiave.

Quando si usa GPT-4 con zero-shot instructional permutation generation, RankGPT supera sistemi supervisionati leader su vari benchmark come TREC, BEIR e Mr.TyDi.

Distillazione efficiente ed economica

RankGPT usa la distillazione per permutazioni per trasferire le capacità di ranking di modelli grandi come GPT-4 in modelli più piccoli e specializzati.

Questi modelli più piccoli mantengono alte prestazioni pur essendo molto più efficienti. Per esempio, un modello distillato da 440M ha superato un modello supervisionato da 3B nel benchmark BEIR, riducendo significativamente i costi computazionali e ottenendo risultati migliori.

Gestione di informazioni nuove e sconosciute

RankGPT include il test set NovelEval per garantire robustezza e affrontare i rischi di contaminazione dei dati. Questo set valuta la capacità del modello di classificare i passaggi in base a informazioni recenti e sconosciute.

GPT-4 ha raggiunto prestazioni allo stato dell’arte in questo test, dimostrando di saper gestire in modo efficace query nuove e mai viste.

Prestazioni di RankGPT nei benchmark

RankGPT (gpt-4) supera tutti gli altri modelli su TREC e BEIR, con un punteggio medio nDCG@10 di 53,68 come mostrato nella tabella seguente. Ha ottenuto i risultati più alti nei dataset BEIR, battendo modelli supervisionati di fascia alta come monoT5 (3B) e Cohere Rerank-v2. Anche con gpt-3.5-turbo, RankGPT ottiene punteggi competitivi, a riprova della sua efficacia come reranker.

Fonte: Weiwei Sun et al., 2023

RankGPT (gpt-4) si comporta molto bene anche sui dataset Mr.TyDi, con un punteggio medio nDCG@10 di 62,93, superando sia BM25 sia mmarcoCE. Supera costantemente BM25 e batte anche mmarcoCE in molte lingue, in particolare in indonesiano e swahili.

Nel complesso, RankGPT ha ottenuto i punteggi più alti in molte lingue, come bengalese, indonesiano e giapponese, con solo pochi casi in cui è rimasto leggermente dietro a mmarcoCE.

Fonte: Weiwei Sun et al., 2023

Infine, RankGPT è stato testato sul dataset NovelEval, che misura quanto bene un modello sa classificare i passaggi in base a informazioni recenti e poco familiari. RankGPT (gpt-4) ha ottenuto il punteggio più alto in tutte le metriche di valutazione (nDCG@1, nDCG@5 e nDCG@10), in particolare con un nDCG@10 di 90,45. Ha superato altri modelli forti come monoT5 (3B) e monoBERT (340M), evidenziando le sue ottime prestazioni come reranker.

Fonte: Weiwei Sun et al., 2023

In tutti i risultati dei benchmark, RankGPT (gpt-4) supera costantemente gli altri metodi, supervisionati e non, dimostrando la sua superiore capacità di riordinamento.

Implementare RankGPT nelle pipeline RAG

Ecco come possiamo integrare RankGPT in una pipeline RAG.

Passo 1: Clonare il repository RankGPT

Per prima cosa, devi clonare il repository RankGPT. Esegui il seguente comando nel terminale:

git clone https://github.com/sunnweiwei/RankGPT

Passo 2: Configurare l’ambiente

Vai alla directory RankGPT e installa i pacchetti richiesti. Potresti voler creare un ambiente virtuale e installare i pacchetti usando il file requirements.txt fornito:

pip install -r requirements.txt

Passo 3: Implementazione di RankGPT

Qui utilizziamo l’esempio semplificato di query e documenti recuperati fornito dal repository originale di RankGPT.

item = {
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'}
    ]
}

Puoi usare la pipeline di permutazione fornita per riordinare facilmente i documenti recuperati con RankGPT.

from rank_gpt import permutation_pipeline
new_item = permutation_pipeline(
    item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo',
    api_key='Your OPENAI Key!'
)
print(new_item)

Questo produrrà il seguente nuovo ordine dei documenti:

{
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}
    ]
}

Generazione di permutazioni istruzionali passo dopo passo

Per un’implementazione più dettagliata della pipeline di permutazione, puoi interagire direttamente con RankGPT per creare ed elaborare istruzioni di permutazione come segue:

from rank_gpt import (
    create_permutation_instruction,
    run_llm,
    receive_permutation
)
# Create permutation generation instruction
messages = create_permutation_instruction(
    item=item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo'
)

[{'role': 'system',
  'content': 'You are RankGPT, an intelligent assistant that can rank passages based on their relevancy to the query.'},
 {'role': 'user',
  'content': 'I will provide you with 3 passages, each indicated by number identifier []. \\nRank the passages based on their relevance to query: How much impact do masks have on preventing the spread of the COVID-19?.'},
 {'role': 'assistant', 'content': 'Okay, please provide the passages.'},
 {'role': 'user',
  'content': '[1] Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
 {'role': 'assistant', 'content': 'Received passage [1].'},
 {'role': 'user',
  'content': '[2] Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
 {'role': 'assistant', 'content': 'Received passage [2].'},
 {'role': 'user',
  'content': '[3] Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
 {'role': 'assistant', 'content': 'Received passage [3].'},
 {'role': 'user',
  'content': 'Search Query: How much impact do masks have on preventing the spread of the COVID-19?. \\nRank the 3 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [] > [], e.g., [1] > [2]. Only response the ranking results, do not say any word or explain.'}]

# Get ChatGPT predicted permutation
permutation = run_llm(
    messages,
    api_key='Your OPENAI Key!',
    model_name='gpt-3.5-turbo'
)

'[1] > [3] > [2]'

# Use permutation to re-rank the passage
item = receive_permutation(
    item,
    permutation,
    rank_start=0,
    rank_end=3
)

{'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
 'hits': [{'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
  {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
  {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}]}

Strategia della finestra scorrevole (SWA) per RankGPT

Se devi classificare più documenti di quanti il modello possa gestire in una volta, usa una strategia a finestra scorrevole. Ecco come applicarla per riordinare i documenti:

from rank_gpt import sliding_windows
api_key = "Your OPENAI Key"
new_item = sliding_windows(
    item,
    rank_start=0,
    rank_end=3,
    window_size=2,
    step=1,
    model_name='gpt-3.5-turbo',
    api_key=api_key
)
print(new_item)

In questo esempio, la finestra scorrevole ha dimensione 2 e passo 1, cioè elabora due documenti alla volta spostandosi avanti di un documento per la passata di ranking successiva.

Conclusione

Usando gli LLM per valutare meglio la rilevanza delle informazioni, RankGPT migliora l’accuratezza dell’ordinamento e del riordinamento dei contenuti.

Questo affronta problemi comuni come mantenere il focus dei contenuti, migliorare l’efficienza e ridurre la probabilità di generare informazioni fuorvianti.

Nel complesso, RankGPT contribuisce a costruire applicazioni RAG più affidabili e accurate.