将 RankGPT 用作 RAG 的重排序代理（教程）

理解检索增强生成（RAG）

检索增强生成（RAG）是一种将 LLM 与信息检索系统结合的方法。也就是说，当 LLM 被要求生成文本时，它可以从外部来源调取相关信息，使其回答更准确、更有依据。

RAG 由两个主要组件——检索器与生成器——以及一个可选组件重排序器组成：

1. 检索器——检索器的工作是根据用户的查询，从大量文档中找到相关的文档或文本片段。它使用诸如 BM25 等算法按相关性对文档进行排序。
2. 重排序器（可选）——重排序器对初始检索到的文档重新排序，确保最相关的位于顶部。这有助于过滤较无用的信息，聚焦重点。
3. 生成器——生成器是利用检索到的文档来生成最终输出的 LLM。访问到相关的外部数据可以带来更准确的回答。

RankGPT 在 RAG 中的角色与优势

RankGPT 使用 LLM 来评估检索到的文档或文本片段的相关性，确保最重要的内容排在最前。借助 RankGPT，RAG 流水线中的生成器能获得更高质量的输入，从而产生更准确的回答。

更高的相关性与性能

RankGPT 超越了简单的关键词匹配，能够理解查询与文档更深层的含义和语境。这使其能够为 LLM 提供更准确的信息，基于实际含义而非仅仅关键词来识别最相关的内容。

当使用GPT-4并采用零样本的指令排列生成时，RankGPT 在多项基准测试（如 TREC、BEIR 和 Mr.TyDi）上优于领先的有监督系统。

高效且具成本效益的蒸馏

RankGPT 使用排列蒸馏，将 GPT-4 等大型模型的排序能力迁移到更小、更专门化的模型中。

这些小模型在保持高性能的同时更加高效。例如，一个蒸馏后的 4.40 亿参数模型在 BEIR 基准上超越了 30 亿参数的有监督模型，显著降低计算成本的同时实现了更佳结果。

处理新颖与未知信息

RankGPT 包含 NovelEval 测试集，以确保稳健性并应对数据污染问题。该测试集评估模型基于最新与未知信息对段落进行排序的能力。

GPT-4 在该测试中达到了最先进水平，展示了其有效处理新颖且未见过查询的能力。

RankGPT 基准测试表现

RankGPT（gpt-4）在 TREC 与 BEIR 上均优于其他模型，平均 nDCG@10 得分为 53.68，如下表所示。它在 BEIR 数据集上取得最高成绩，超越了 monoT5（3B）与 Cohere Rerank-v2 等强有监督模型。即便使用 gpt-3.5-turbo，RankGPT 也有竞争力，证明其作为重排序器的高效性。

来源：Weiwei Sun 等, 2023

RankGPT（gpt-4）在 Mr.TyDi 数据集上同样表现强劲，平均 nDCG@10 得分为 62.93，领先于 BM25 与 mmarcoCE。它稳定地优于 BM25，并在多种语言中超越 mmarcoCE，尤其是在印尼语与斯瓦希里语上。

总体而言，RankGPT 在孟加拉语、印尼语、日语等多种语言中取得最高分，仅在少数情况下略低于 mmarcoCE。

来源：Weiwei Sun 等, 2023

最后，RankGPT 在 NovelEval 数据集上的测试结果显示其能够基于最新且不熟悉的信息对段落进行出色排序。RankGPT（gpt-4）在所有评估指标（nDCG@1、nDCG@5 和 nDCG@10）上均取得最高分，尤其是 nDCG@10 达到 90.45。它优于 monoT5（3B）与 monoBERT（340M）等强模型，凸显其作为重排序器的强劲表现。

来源：Weiwei Sun 等, 2023

综合各项基准结果，RankGPT（gpt-4）不论在有监督还是无监督方法中都持续领先，展现出其在重排序任务上的卓越能力。

在 RAG 流水线中实现 RankGPT

以下是我们如何将 RankGPT 集成到 RAG 流水线中的方法。

步骤 1：克隆 RankGPT 仓库

首先，您需要克隆 RankGPT 仓库。在终端中运行以下命令：

git clone https://github.com/sunnweiwei/RankGPT技术支持

步骤 2：设置您的环境

进入RankGPT 目录并安装所需依赖。您可以创建一个虚拟环境，并使用提供的requirements.txt 安装包：

pip install -r requirements.txt技术支持

步骤 3：实现 RankGPT

这里我们使用原始 RankGPT 仓库提供的简化示例查询与检索到的文档。

item = {
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'}
    ]
}技术支持

您可以使用提供的排列流水线，轻松用 RankGPT 对检索到的文档进行重排序。

from rank_gpt import permutation_pipeline
new_item = permutation_pipeline(
    item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo',
    api_key='Your OPENAI Key!'
)
print(new_item)技术支持

这将产生如下新的文档顺序：

{
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}
    ]
}技术支持

分步的指令式排列生成

若需更细化的排列流水线实现，您可以直接与 RankGPT 交互，按如下方式创建并处理排列指令：

from rank_gpt import (
    create_permutation_instruction,
    run_llm,
    receive_permutation
)
# Create permutation generation instruction
messages = create_permutation_instruction(
    item=item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo'
)技术支持

[{'role': 'system',
  'content': 'You are RankGPT, an intelligent assistant that can rank passages based on their relevancy to the query.'},
 {'role': 'user',
  'content': 'I will provide you with 3 passages, each indicated by number identifier []. \\nRank the passages based on their relevance to query: How much impact do masks have on preventing the spread of the COVID-19?.'},
 {'role': 'assistant', 'content': 'Okay, please provide the passages.'},
 {'role': 'user',
  'content': '[1] Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
 {'role': 'assistant', 'content': 'Received passage [1].'},
 {'role': 'user',
  'content': '[2] Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
 {'role': 'assistant', 'content': 'Received passage [2].'},
 {'role': 'user',
  'content': '[3] Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
 {'role': 'assistant', 'content': 'Received passage [3].'},
 {'role': 'user',
  'content': 'Search Query: How much impact do masks have on preventing the spread of the COVID-19?. \\nRank the 3 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [] > [], e.g., [1] > [2]. Only response the ranking results, do not say any word or explain.'}]技术支持

# Get ChatGPT predicted permutation
permutation = run_llm(
    messages,
    api_key='Your OPENAI Key!',
    model_name='gpt-3.5-turbo'
)技术支持

'[1] > [3] > [2]'技术支持

# Use permutation to re-rank the passage
item = receive_permutation(
    item,
    permutation,
    rank_start=0,
    rank_end=3
)技术支持

{'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
 'hits': [{'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
  {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
  {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}]}技术支持

RankGPT 的滑动窗口策略（SWA）

如果您需要排序的文档数量超过模型一次能处理的上限，可使用滑动窗口策略。以下展示如何应用滑动窗口策略对文档进行重排序：

from rank_gpt import sliding_windows
api_key = "Your OPENAI Key"
new_item = sliding_windows(
    item,
    rank_start=0,
    rank_end=3,
    window_size=2,
    step=1,
    model_name='gpt-3.5-turbo',
    api_key=api_key
)
print(new_item)技术支持

在该示例中，滑动窗口大小为2，步长为1，意味着每次处理两个文档，并在下一次排序时向前滑动一个文档。

结论

通过利用 LLM 更好地评估信息的相关性，RankGPT 提升了内容排序与重排序的准确性。

这有助于解决常见问题，如确保内容切题、提升效率并降低产生误导性信息的可能性。

总体而言，RankGPT 有助于构建更可靠、更准确的 RAG 应用。