RankGPT als herordeneringsagent voor RAG (Tutorial)

Retrieval Augmented Generation (RAG) begrijpen

Retrieval augmented generation (RAG) is een methode die LLM’s combineert met informatieretrievalsystemen. Dit betekent dat wanneer een LLM wordt gevraagd om tekst te genereren, het relevante informatie uit externe bronnen kan ophalen, waardoor de antwoorden nauwkeuriger en beter onderbouwd zijn.

RAG bestaat uit twee hoofdcomponenten—de retriever en de generator—en een optionele component, de reranker:

1. Retriever—De taak van de retriever is om relevante documenten of tekstsegmenten te vinden uit een grote set documenten op basis van de vraag van de gebruiker. Hij gebruikt algoritmen zoals BM25 om documenten op relevantie te rangschikken.
2. Reranker (optioneel) — De reranker neemt de initiële set opgehaalde documenten en ordent ze opnieuw zodat de meest relevante bovenaan staan. Dit helpt minder nuttige informatie te filteren en te focussen op wat belangrijk is.
3. Generator — De generator is de LLM die de opgehaalde documenten gebruikt om de uiteindelijke output te genereren. Toegang tot relevante externe data kan zorgen voor nauwkeurigere antwoorden.

De rol en voordelen van RankGPT in RAG

RankGPT gebruikt LLM’s om de relevantie van opgehaalde documenten of tekstsegmenten te beoordelen, zodat de belangrijkste bovenaan staan. Met RankGPT krijgt de generator in de RAG-pijplijn input van hogere kwaliteit, wat resulteert in accuratere antwoorden.

Betere relevantie en prestaties

RankGPT gaat verder dan eenvoudige zoekwoordmatching door de diepere betekenis en context van vragen en documenten te begrijpen. Zo kan het accuratere informatie aan LLM’s bieden en de meest relevante content identificeren op basis van de werkelijke betekenis, niet alleen op basis van trefwoorden.

Bij gebruik van GPT-4 met zero-shot instructional permutation generation presteert RankGPT beter dan toonaangevende gesuperviseerde systemen op diverse benchmarks zoals TREC, BEIR en Mr.TyDi.

Efficiënte en kosteneffectieve distillatie

RankGPT gebruikt permutatie-distillatie om de rankingvaardigheden van grote modellen zoals GPT-4 over te dragen naar kleinere, gespecialiseerde modellen.

Deze kleinere modellen behouden hoge prestaties terwijl ze veel efficiënter zijn. Zo presteerde een gedistilleerd 440M-model beter dan een 3B gesuperviseerd model op de BEIR-benchmark en verlaagde het de rekenkosten aanzienlijk met betere resultaten.

Omgaan met nieuwe en onbekende informatie

RankGPT bevat de NovelEval-testset om robuustheid te waarborgen en zorgen over datacontaminatie aan te pakken. Deze set evalueert het vermogen van het model om passages te rangschikken op basis van recente en onbekende informatie.

GPT-4 behaalde state-of-the-art prestaties op deze test, wat aantoont dat het effectief kan omgaan met nieuwe en ongeziene vragen.

RankGPT-benchmarkprestaties

RankGPT (gpt-4) presteert beter dan alle andere modellen op TREC en BEIR, met een gemiddelde nDCG@10-score van 53,68 zoals in de onderstaande tabel. Het behaalde de hoogste resultaten in de BEIR-datasets en versloeg sterke gesuperviseerde modellen zoals monoT5 (3B) en Cohere Rerank-v2. Zelfs met gpt-3.5-turbo scoort RankGPT competitief, wat bewijst dat het een zeer effectieve reranker is.

Bron: Weiwei Sun et al., 2023

RankGPT (gpt-4) presteert ook sterk op de Mr.TyDi-datasets, met een gemiddelde nDCG@10-score van 62,93, beter dan zowel BM25 als mmarcoCE. Het presteert consistent beter dan BM25 en overtreft zelfs mmarcoCE in veel talen, vooral in het Indonesisch en Swahili.

Over het geheel genomen scoorde RankGPT het hoogst in veel talen, zoals Bengaals, Indonesisch en Japans, met slechts enkele gevallen waarin het net achter mmarcoCE bleef.

Bron: Weiwei Sun et al., 2023

Tot slot is RankGPT getest op de NovelEval-dataset, die meet hoe goed een model passages kan rangschikken op basis van recente en onbekende informatie. RankGPT (gpt-4) scoorde het hoogst op alle evaluatiemetrics (nDCG@1, nDCG@5 en nDCG@10), met name met de nDCG@10-score van 90,45. Het presteerde beter dan andere sterke modellen zoals monoT5 (3B) en monoBERT (340M), wat de sterke prestaties als reranker benadrukt.

Bron: Weiwei Sun et al., 2023

In alle benchmarkresultaten presteert RankGPT (gpt-4) consistent beter dan andere methoden, zowel gesuperviseerd als ongesuperviseerd, wat de superieure herordeningcapaciteiten aantoont.

RankGPT implementeren in RAG-pijplijnen

Zo kun je RankGPT integreren in een RAG-pijplijn.

Stap 1: Kloon de RankGPT-repository

Eerst moet je de RankGPT-repository klonen. Voer de volgende opdracht uit in je terminal:

git clone https://github.com/sunnweiwei/RankGPT

Stap 2: Stel je omgeving in

Navigeer naar de RankGPT-map en installeer de vereiste pakketten. Je kunt een virtuele omgeving aanmaken en pakketten installeren met de meegeleverde requirements.txt:

pip install -r requirements.txt

Stap 3: RankGPT-implementatie

Hier gebruiken we de eenvoudige voorbeeldquery en opgehaalde documenten die door de originele RankGPT-repository zijn geleverd.

item = {
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'}
    ]
}

Je kunt de meegeleverde permutatiepijplijn gebruiken om de opgehaalde documenten eenvoudig te herordenen met RankGPT.

from rank_gpt import permutation_pipeline
new_item = permutation_pipeline(
    item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo',
    api_key='Your OPENAI Key!'
)
print(new_item)

Dit resulteert in de volgende nieuwe volgorde van documenten:

{
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}
    ]
}

Stapsgewijze instructional permutation generation

Voor een meer stapsgewijze implementatie van de permutatiepijplijn kun je direct met RankGPT interageren om permutatie-instructies te maken en te verwerken zoals volgt:

from rank_gpt import (
    create_permutation_instruction,
    run_llm,
    receive_permutation
)
# Create permutation generation instruction
messages = create_permutation_instruction(
    item=item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo'
)

[{'role': 'system',
  'content': 'You are RankGPT, an intelligent assistant that can rank passages based on their relevancy to the query.'},
 {'role': 'user',
  'content': 'I will provide you with 3 passages, each indicated by number identifier []. \\nRank the passages based on their relevance to query: How much impact do masks have on preventing the spread of the COVID-19?.'},
 {'role': 'assistant', 'content': 'Okay, please provide the passages.'},
 {'role': 'user',
  'content': '[1] Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
 {'role': 'assistant', 'content': 'Received passage [1].'},
 {'role': 'user',
  'content': '[2] Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
 {'role': 'assistant', 'content': 'Received passage [2].'},
 {'role': 'user',
  'content': '[3] Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
 {'role': 'assistant', 'content': 'Received passage [3].'},
 {'role': 'user',
  'content': 'Search Query: How much impact do masks have on preventing the spread of the COVID-19?. \\nRank the 3 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [] > [], e.g., [1] > [2]. Only response the ranking results, do not say any word or explain.'}]

# Get ChatGPT predicted permutation
permutation = run_llm(
    messages,
    api_key='Your OPENAI Key!',
    model_name='gpt-3.5-turbo'
)

'[1] > [3] > [2]'

# Use permutation to re-rank the passage
item = receive_permutation(
    item,
    permutation,
    rank_start=0,
    rank_end=3
)

{'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
 'hits': [{'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
  {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
  {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}]}

Sliding Window Strategy (SWA) voor RankGPT

Als je meer documenten moet rangschikken dan het model in één keer aankan, gebruik dan een sliding window-strategie. Zo pas je een sliding window toe om documenten te herordenen:

from rank_gpt import sliding_windows
api_key = "Your OPENAI Key"
new_item = sliding_windows(
    item,
    rank_start=0,
    rank_end=3,
    window_size=2,
    step=1,
    model_name='gpt-3.5-turbo',
    api_key=api_key
)
print(new_item)

In dit voorbeeld heeft het sliding window een grootte van 2 en een stapgrootte van 1, wat betekent dat het twee documenten tegelijk verwerkt en voor de volgende rangschikking één document opschuift.

Conclusie

Door LLM’s te gebruiken om de relevantie van informatie beter te beoordelen, verbetert RankGPT de nauwkeurigheid van het sorteren en herordenen van content.

Dit pakt veelvoorkomende problemen aan, zoals zorgen dat content to the point is, de efficiëntie verhogen en de kans op misleidende output verkleinen.

Alles bij elkaar draagt RankGPT bij aan het bouwen van betrouwbaardere en nauwkeurigere RAG-applicaties.