RankGPT comme agent de re-classement pour RAG (Tutoriel)

Comprendre la retrieval augmented generation (RAG)

La retrieval augmented generation (RAG) combine des LLM avec des systèmes de recherche d'information. Concrètement, lorsqu'on demande à un LLM de générer du texte, il peut intégrer des informations pertinentes issues de sources externes, ce qui rend ses réponses plus précises et mieux étayées.

RAG se compose de deux éléments principaux — le retriever et le generator — et d'un composant optionnel, le reranker :

1. Retriever — Sa mission est de trouver, à partir de la requête de l'utilisateur, les documents ou segments de texte pertinents dans un large ensemble. Il utilise des algorithmes comme BM25 pour classer les documents selon leur pertinence.
2. Reranker (optionnel) — Il reprend l'ensemble initial de documents récupérés et les réordonne pour placer les plus pertinents en tête. Cela permet d'écarter les informations moins utiles et de se concentrer sur l'essentiel.
3. Generator — Le générateur est le LLM qui s'appuie sur les documents récupérés pour produire la sortie finale. L'accès à des données externes pertinentes améliore la précision des réponses.

Rôle et avantages de RankGPT dans RAG

RankGPT utilise des LLM pour évaluer la pertinence des documents ou segments récupérés et garantir que les plus importants apparaissent en premier. Avec RankGPT, le générateur d'une pipeline RAG reçoit des entrées de meilleure qualité, ce qui se traduit par des réponses plus exactes.

Pertinence et performances accrues

RankGPT va au-delà de la simple correspondance de mots-clés en comprenant le sens profond et le contexte des requêtes et des documents. Il peut ainsi fournir aux LLM des informations plus justes, en identifiant le contenu le plus pertinent selon sa signification réelle, et non seulement selon des mots-clés.

Avec GPT-4 et la génération d'instructions par permutations en zero-shot, RankGPT surpasse des systèmes supervisés de premier plan sur divers benchmarks comme TREC, BEIR et Mr.TyDi.

Distillation efficace et économique

RankGPT utilise la distillation par permutations pour transférer les capacités de classement de grands modèles comme GPT-4 vers des modèles plus petits et spécialisés.

Ces modèles plus compacts conservent des performances élevées tout en étant bien plus efficients. Par exemple, un modèle distillé de 440 M a surpassé un modèle supervisé de 3 B sur le benchmark BEIR, réduisant considérablement les coûts de calcul tout en obtenant de meilleurs résultats.

Gestion d'informations nouvelles et inconnues

RankGPT inclut le jeu de test NovelEval pour garantir la robustesse et traiter les risques de contamination des données. Ce jeu évalue la capacité du modèle à classer des passages fondés sur des informations récentes et inédites.

GPT-4 y atteint des performances de pointe, démontrant sa capacité à traiter efficacement des requêtes nouvelles et jamais vues.

Performances de RankGPT sur les benchmarks

RankGPT (gpt-4) surpasse tous les autres modèles sur TREC et BEIR, avec un score moyen nDCG@10 de 53,68 comme indiqué dans le tableau ci-dessous. Il obtient les meilleurs résultats sur les jeux de données BEIR, devant des modèles supervisés puissants comme monoT5 (3B) et Cohere Rerank-v2. Même avec gpt-3.5-turbo, RankGPT affiche des scores compétitifs, confirmant son efficacité élevée en tant que reranker.

Source : Weiwei Sun et al., 2023

RankGPT (gpt-4) affiche également d'excellents résultats sur les jeux de données Mr.TyDi, avec un score moyen nDCG@10 de 62,93, devant BM25 et mmarcoCE. Il dépasse systématiquement BM25 et surpasse même mmarcoCE dans de nombreuses langues, notamment en indonésien et en swahili.

Globalement, RankGPT obtient les meilleurs scores dans de nombreuses langues, comme le bengali, l'indonésien et le japonais, avec seulement quelques cas où il est légèrement derrière mmarcoCE.

Source : Weiwei Sun et al., 2023

Enfin, RankGPT a été évalué sur le jeu de données NovelEval, qui mesure la capacité d'un modèle à classer des passages fondés sur des informations récentes et peu familières. RankGPT (gpt-4) obtient les meilleurs scores sur toutes les métriques (nDCG@1, nDCG@5 et nDCG@10), en particulier un nDCG@10 de 90,45. Il surpasse d'autres modèles robustes comme monoT5 (3B) et monoBERT (340M), ce qui souligne sa performance en tant que reranker.

Source : Weiwei Sun et al., 2023

Sur l'ensemble des benchmarks, RankGPT (gpt-4) surpasse systématiquement les autres approches, supervisées comme non supervisées, démontrant sa supériorité en re-classement.

Implémenter RankGPT dans des pipelines RAG

Voici comment intégrer RankGPT dans une pipeline RAG.

Étape 1 : cloner le dépôt RankGPT

Commencez par cloner le dépôt RankGPT. Exécutez la commande suivante dans votre terminal :

git clone https://github.com/sunnweiwei/RankGPT

Étape 2 : configurer votre environnement

Accédez au répertoire RankGPT et installez les dépendances requises. Vous pouvez créer un environnement virtuel et installer les paquets via requirements.txt :

pip install -r requirements.txt

Étape 3 : implémenter RankGPT

Ici, nous utilisons l'exemple simple de requête et de documents récupérés fourni par le dépôt RankGPT d'origine.

item = {
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'}
    ]
}

Vous pouvez utiliser le pipeline de permutations fourni pour reclasser facilement les documents récupérés avec RankGPT.

from rank_gpt import permutation_pipeline
new_item = permutation_pipeline(
    item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo',
    api_key='Your OPENAI Key!'
)
print(new_item)

Vous obtiendrez ainsi le nouvel ordre de documents suivant :

{
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}
    ]
}

Génération d'instructions par permutations, étape par étape

Pour une implémentation plus pas-à-pas du pipeline de permutations, vous pouvez interagir directement avec RankGPT pour créer et traiter des instructions de permutation comme suit :

from rank_gpt import (
    create_permutation_instruction,
    run_llm,
    receive_permutation
)
# Create permutation generation instruction
messages = create_permutation_instruction(
    item=item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo'
)

[{'role': 'system',
  'content': 'You are RankGPT, an intelligent assistant that can rank passages based on their relevancy to the query.'},
 {'role': 'user',
  'content': 'I will provide you with 3 passages, each indicated by number identifier []. \\nRank the passages based on their relevance to query: How much impact do masks have on preventing the spread of the COVID-19?.'},
 {'role': 'assistant', 'content': 'Okay, please provide the passages.'},
 {'role': 'user',
  'content': '[1] Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
 {'role': 'assistant', 'content': 'Received passage [1].'},
 {'role': 'user',
  'content': '[2] Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
 {'role': 'assistant', 'content': 'Received passage [2].'},
 {'role': 'user',
  'content': '[3] Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
 {'role': 'assistant', 'content': 'Received passage [3].'},
 {'role': 'user',
  'content': 'Search Query: How much impact do masks have on preventing the spread of the COVID-19?. \\nRank the 3 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [] > [], e.g., [1] > [2]. Only response the ranking results, do not say any word or explain.'}]

# Get ChatGPT predicted permutation
permutation = run_llm(
    messages,
    api_key='Your OPENAI Key!',
    model_name='gpt-3.5-turbo'
)

'[1] > [3] > [2]'

# Use permutation to re-rank the passage
item = receive_permutation(
    item,
    permutation,
    rank_start=0,
    rank_end=3
)

{'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
 'hits': [{'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
  {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
  {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}]}

Stratégie de fenêtre glissante (SWA) pour RankGPT

Si vous devez classer plus de documents que le modèle ne peut en traiter en une seule fois, utilisez une stratégie de fenêtre glissante. Voici comment l'appliquer pour reclasser des documents :

from rank_gpt import sliding_windows
api_key = "Your OPENAI Key"
new_item = sliding_windows(
    item,
    rank_start=0,
    rank_end=3,
    window_size=2,
    step=1,
    model_name='gpt-3.5-turbo',
    api_key=api_key
)
print(new_item)

Dans cet exemple, la fenêtre glissante a une taille de 2 et un pas de 1 : deux documents sont traités à la fois, puis on décale d'un document pour le passage de classement suivant.

Conclusion

En mobilisant des LLM pour mieux évaluer la pertinence de l'information, RankGPT améliore la précision du tri et du re-classement des contenus.

Cela permet de résoudre des problèmes courants : rester centré sur le sujet, gagner en efficacité et réduire le risque de produire des informations trompeuses.

Au final, RankGPT contribue à concevoir des applications RAG plus fiables et plus précises.