RankGPT als Re-Ranking-Agent für RAG (Tutorial)

RAG (Retrieval Augmented Generation) verstehen

Retrieval Augmented Generation (RAG) kombiniert LLMs mit Informationsabrufsystemen. Das bedeutet: Wenn ein LLM Text generiert, kann es relevante Informationen aus externen Quellen einbeziehen und dadurch treffendere, fundiertere Antworten liefern.

RAG besteht aus zwei Hauptkomponenten — dem Retriever und dem Generator — sowie einer optionalen Komponente, dem Reranker:

1. Retriever — Der Retriever sucht basierend auf der Nutzeranfrage relevante Dokumente oder Textsegmente aus einer großen Sammlung heraus. Er verwendet Algorithmen wie BM25, um die Dokumente nach Relevanz zu ordnen.
2. Reranker (optional) — Der Reranker nimmt die initial gefundenen Dokumente und sortiert sie neu, sodass die relevantesten ganz oben stehen. So filtert er weniger hilfreiche Informationen heraus und fokussiert auf das Wesentliche.
3. Generator — Der Generator ist das LLM, das die abgerufenen Dokumente nutzt, um die finale Antwort zu erzeugen. Der Zugriff auf relevante externe Daten führt zu präziseren Ergebnissen.

Rolle und Vorteile von RankGPT in RAG

RankGPT nutzt LLMs, um die Relevanz der gefundenen Dokumente oder Textsegmente zu bewerten und die wichtigsten nach oben zu priorisieren. So erhält der Generator in der RAG-Pipeline hochwertigere Eingaben, was zu genaueren Antworten führt.

Höhere Relevanz und bessere Performance

RankGPT geht über simples Keyword-Matching hinaus, indem es Bedeutung und Kontext von Anfragen und Dokumenten versteht. Dadurch stellt es LLMs präzisere Informationen bereit und identifiziert Inhalte anhand ihrer tatsächlichen Aussage — nicht nur anhand von Schlagwörtern.

Beim Einsatz von GPT-4 mit Zero-Shot Instructional Permutation Generation übertrifft RankGPT führende überwachte Systeme auf verschiedenen Benchmarks wie TREC, BEIR und Mr.TyDi.

Effiziente und kostengünstige Distillation

RankGPT nutzt Permutations-Distillation, um die Ranking-Fähigkeiten großer Modelle wie GPT-4 in kleinere, spezialisierte Modelle zu übertragen.

Diese kleineren Modelle halten eine hohe Performance bei deutlich höherer Effizienz. So übertraf ein destilliertes 440M-Modell ein überwachtes 3B-Modell im BEIR-Benchmark und senkte die Rechenkosten deutlich — bei gleichzeitig besseren Ergebnissen.

Umgang mit neuen und unbekannten Informationen

RankGPT umfasst den NovelEval-Testdatensatz, um Robustheit sicherzustellen und Datenkontamination zu adressieren. Dieser Datensatz prüft, wie gut das Modell Passagen zu aktuellen und unbekannten Informationen ranken kann.

GPT-4 erreichte in diesem Test State-of-the-Art-Ergebnisse und zeigt damit, dass es neue und ungesehene Anfragen effektiv verarbeiten kann.

RankGPT-Benchmark-Ergebnisse

RankGPT (gpt-4) übertrifft in TREC und BEIR alle anderen Modelle mit einem durchschnittlichen nDCG@10 von 53,68, wie in der Tabelle unten zu sehen ist. Es erzielte die höchsten Werte in den BEIR-Datensätzen und schlug starke überwach­te Modelle wie monoT5 (3B) und Cohere Rerank-v2. Selbst mit gpt-3.5-turbo liefert RankGPT konkurrenzfähige Ergebnisse — ein klarer Beleg für seine Wirksamkeit als Reranker.

Quelle: Weiwei Sun et al., 2023

Auch auf den Mr.TyDi-Datensätzen liegt RankGPT (gpt-4) vorn, mit einem durchschnittlichen nDCG@10 von 62,93, und schlägt sowohl BM25 als auch mmarcoCE. Es übertrifft BM25 durchweg und ist in vielen Sprachen sogar besser als mmarcoCE — insbesondere auf Indonesisch und Swahili.

Insgesamt erzielte RankGPT in vielen Sprachen — etwa Bengalisch, Indonesisch und Japanisch — die höchsten Werte, mit nur wenigen Fällen, in denen es knapp hinter mmarcoCE lag.

Quelle: Weiwei Sun et al., 2023

Schließlich wurde RankGPT auf dem NovelEval-Datensatz getestet, der misst, wie gut ein Modell Passagen zu aktuellen und ungewohnten Informationen ranken kann. RankGPT (gpt-4) erzielte bei allen Metriken (nDCG@1, nDCG@5 und nDCG@10) die höchsten Werte — besonders mit einem nDCG@10 von 90,45. Es übertraf starke Modelle wie monoT5 (3B) und monoBERT (340M) und unterstreicht damit seine Stärke als Reranker.

Quelle: Weiwei Sun et al., 2023

Über alle Benchmarks hinweg übertrifft RankGPT (gpt-4) konstant andere Methoden — ob überwacht oder unüberwacht — und zeigt seine überlegene Fähigkeit im Re-Ranking.

RankGPT in RAG-Pipelines implementieren

So integrierst du RankGPT in eine RAG-Pipeline.

Schritt 1: RankGPT-Repository klonen

Zuerst musst du das RankGPT-Repository klonen. Führe dazu folgenden Befehl im Terminal aus:

git clone https://github.com/sunnweiwei/RankGPT

Schritt 2: Umgebung einrichten

Wechsle in das Verzeichnis RankGPT und installiere die benötigten Pakete. Du kannst dafür eine virtuelle Umgebung anlegen und die Pakete aus der requirements.txt installieren:

pip install -r requirements.txt

Schritt 3: RankGPT implementieren

Hier nutzen wir die einfache Beispielanfrage und die dazugehörigen abgerufenen Dokumente aus dem ursprünglichen RankGPT-Repository.

item = {
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'}
    ]
}

Mit der bereitgestellten Permutations-Pipeline kannst du die abgerufenen Dokumente bequem mit RankGPT neu ranken.

from rank_gpt import permutation_pipeline
new_item = permutation_pipeline(
    item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo',
    api_key='Your OPENAI Key!'
)
print(new_item)

Das führt zur folgenden neuen Reihenfolge der Dokumente:

{
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}
    ]
}

Schritt-für-Schritt: Instructional Permutation Generation

Für eine kleinteiligere Implementierung der Permutations-Pipeline kannst du RankGPT direkt ansteuern und wie folgt Permutationsanweisungen erstellen und verarbeiten:

from rank_gpt import (
    create_permutation_instruction,
    run_llm,
    receive_permutation
)
# Create permutation generation instruction
messages = create_permutation_instruction(
    item=item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo'
)

[{'role': 'system',
  'content': 'You are RankGPT, an intelligent assistant that can rank passages based on their relevancy to the query.'},
 {'role': 'user',
  'content': 'I will provide you with 3 passages, each indicated by number identifier []. \\nRank the passages based on their relevance to query: How much impact do masks have on preventing the spread of the COVID-19?.'},
 {'role': 'assistant', 'content': 'Okay, please provide the passages.'},
 {'role': 'user',
  'content': '[1] Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
 {'role': 'assistant', 'content': 'Received passage [1].'},
 {'role': 'user',
  'content': '[2] Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
 {'role': 'assistant', 'content': 'Received passage [2].'},
 {'role': 'user',
  'content': '[3] Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
 {'role': 'assistant', 'content': 'Received passage [3].'},
 {'role': 'user',
  'content': 'Search Query: How much impact do masks have on preventing the spread of the COVID-19?. \\nRank the 3 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [] > [], e.g., [1] > [2]. Only response the ranking results, do not say any word or explain.'}]

# Get ChatGPT predicted permutation
permutation = run_llm(
    messages,
    api_key='Your OPENAI Key!',
    model_name='gpt-3.5-turbo'
)

'[1] > [3] > [2]'

# Use permutation to re-rank the passage
item = receive_permutation(
    item,
    permutation,
    rank_start=0,
    rank_end=3
)

{'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
 'hits': [{'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
  {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
  {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}]}

Sliding-Window-Strategie (SWA) für RankGPT

Wenn du mehr Dokumente ranken musst, als das Modell auf einmal verarbeiten kann, nutze eine Sliding-Window-Strategie. So setzt du sie zum Neu-Ranking von Dokumenten ein:

from rank_gpt import sliding_windows
api_key = "Your OPENAI Key"
new_item = sliding_windows(
    item,
    rank_start=0,
    rank_end=3,
    window_size=2,
    step=1,
    model_name='gpt-3.5-turbo',
    api_key=api_key
)
print(new_item)

In diesem Beispiel hat das Sliding Window die Größe 2 und einen Schritt von 1. Das bedeutet, es verarbeitet jeweils zwei Dokumente und rückt für den nächsten Ranking-Durchlauf um ein Dokument weiter.

Fazit

Indem LLMs die Relevanz von Informationen besser einschätzen, verbessert RankGPT die Genauigkeit beim Sortieren und Neu-Ranken von Inhalten.

Das adressiert typische Probleme wie inhaltliche Zielgenauigkeit, höhere Effizienz und eine geringere Wahrscheinlichkeit irreführender Ausgaben.

In Summe trägt RankGPT dazu bei, verlässlichere und präzisere RAG-Anwendungen zu entwickeln.