RankGPT sebagai Agen Re-Ranking untuk RAG (Tutorial)

Memahami Retrieval Augmented Generation (RAG)

Retrieval augmented generation (RAG) adalah metode yang menggabungkan LLM dengan sistem temu kembali informasi. Artinya, ketika sebuah LLM diminta menghasilkan teks, ia dapat menarik informasi relevan dari sumber eksternal, sehingga responsnya lebih akurat dan berpengetahuan.

RAG terdiri dari dua komponen utama—retriever dan generator—serta satu komponen opsional, yaitu reranker:

1. Retriever—Tugas retriever adalah menemukan dokumen atau segmen teks yang relevan dari sekumpulan besar dokumen berdasarkan kueri pengguna. Ia menggunakan algoritma seperti BM25 untuk memberi peringkat dokumen menurut relevansinya.
2. Reranker (opsional) — Reranker mengambil kumpulan awal dokumen yang ditemukan dan menyusunnya kembali agar yang paling relevan berada di urutan teratas. Ini membantu menyaring informasi yang kurang berguna dan fokus pada hal yang penting.
3. Generator — Generator adalah LLM yang menggunakan dokumen yang ditemukan untuk menghasilkan output akhir. Akses ke data eksternal yang relevan dapat menghasilkan respons yang lebih akurat.

Peran dan Manfaat RankGPT dalam RAG

RankGPT menggunakan LLM untuk mengevaluasi relevansi dokumen atau segmen teks yang ditemukan, memastikan yang terpenting berada di posisi teratas. Dengan RankGPT, generator dalam pipeline RAG menerima masukan berkualitas lebih baik, sehingga menghasilkan respons yang lebih akurat.

Peningkatan relevansi dan kinerja

RankGPT melampaui pencocokan kata kunci sederhana dengan memahami makna dan konteks yang lebih dalam dari kueri dan dokumen. Hal ini memungkinkannya menyediakan informasi yang lebih akurat untuk LLM, mengidentifikasi konten paling relevan berdasarkan makna sebenarnya, bukan sekadar kata kunci.

Saat menggunakan GPT-4 dengan zero-shot instructional permutation generation, RankGPT mengungguli sistem terawasi terkemuka pada berbagai tolok ukur seperti TREC, BEIR, dan Mr.TyDi.

Distilasi yang efisien dan hemat biaya

RankGPT menggunakan distilasi berbasis permutasi untuk mentransfer kemampuan pemeringkatan dari model besar seperti GPT-4 ke model yang lebih kecil dan terspesialisasi.

Model yang lebih kecil ini mempertahankan kinerja tinggi sekaligus jauh lebih efisien. Misalnya, model hasil distilasi berukuran 440M melampaui model terawasi 3B pada tolok ukur BEIR, secara signifikan mengurangi biaya komputasi sekaligus mencapai hasil yang lebih baik.

Menangani informasi baru dan tidak dikenal

RankGPT menyertakan himpunan uji NovelEval untuk memastikan ketangguhan dan mengatasi kekhawatiran kontaminasi data. Himpunan ini mengevaluasi kemampuan model dalam memberi peringkat bagian teks berdasarkan informasi terbaru dan tidak dikenal.

GPT-4 meraih kinerja mutakhir pada uji ini, menunjukkan kemampuannya menangani kueri baru dan yang belum pernah dilihat secara efektif.

Kinerja Benchmark RankGPT

RankGPT (gpt-4) mengungguli semua model lain pada TREC dan BEIR, dengan skor nDCG@10 rata-rata 53,68 seperti ditunjukkan pada tabel di bawah. Model ini meraih hasil tertinggi di dataset BEIR, melampaui model terawasi kuat seperti monoT5 (3B) dan Cohere Rerank-v2. Bahkan dengan gpt-3.5-turbo, RankGPT tetap meraih skor yang kompetitif, membuktikan bahwa ini adalah reranker yang sangat efektif.

Sumber: Weiwei Sun dkk., 2023

RankGPT (gpt-4) juga berkinerja kuat pada dataset Mr.TyDi, memimpin dengan skor nDCG@10 rata-rata 62,93, melampaui BM25 dan mmarcoCE. Metode ini secara konsisten mengungguli BM25 dan bahkan melampaui mmarcoCE di banyak bahasa, terutama dalam bahasa Indonesia dan Swahili.

Secara keseluruhan, RankGPT meraih skor tertinggi di banyak bahasa, seperti Bengali, Indonesia, dan Jepang, dengan hanya beberapa kasus tertinggal tipis dari mmarcoCE.

Sumber: Weiwei Sun dkk., 2023

Terakhir, RankGPT diuji pada dataset NovelEval, yang mengukur seberapa baik model dapat memberi peringkat bagian teks berdasarkan informasi terbaru dan tidak familier. RankGPT (gpt-4) meraih skor tertinggi di semua metrik evaluasi (nDCG@1, nDCG@5, dan nDCG@10), khususnya dengan skor nDCG@10 sebesar 90,45. Model ini melampaui model kuat lainnya seperti monoT5 (3B) dan monoBERT (340M), yang menegaskan performanya sebagai reranker.

Sumber: Weiwei Sun dkk., 2023

Di seluruh hasil benchmark, RankGPT (gpt-4) secara konsisten mengungguli metode lain, baik terawasi maupun tanpa pengawasan, menunjukkan kemampuan unggulnya dalam re-ranking.

Mengimplementasikan RankGPT dalam Pipeline RAG

Berikut cara mengintegrasikan RankGPT ke dalam pipeline RAG.

Langkah 1: Clone repositori RankGPT

Pertama, Anda perlu meng-clone repositori RankGPT. Jalankan perintah berikut di terminal Anda:

git clone https://github.com/sunnweiwei/RankGPT

Langkah 2: Siapkan lingkungan Anda

Buka direktori RankGPT dan instal paket yang diperlukan. Anda dapat membuat virtual environment dan memasang paket menggunakan requirements.txt yang disediakan:

pip install -r requirements.txt

Langkah 3: Implementasi RankGPT

Di sini, kita menggunakan contoh kueri sederhana dan dokumen hasil temu kembali yang disediakan oleh repositori RankGPT asli.

item = {
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'}
    ]
}

Anda dapat menggunakan pipeline permutasi yang disediakan untuk dengan mudah mengurutkan ulang dokumen yang ditemukan dengan RankGPT.

from rank_gpt import permutation_pipeline
new_item = permutation_pipeline(
    item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo',
    api_key='Your OPENAI Key!'
)
print(new_item)

Ini akan menghasilkan urutan dokumen baru sebagai berikut:

{
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}
    ]
}

Pembuatan Permutasi Instruksional Langkah demi Langkah

Untuk implementasi pipeline permutasi yang lebih bertahap, Anda dapat berinteraksi langsung dengan RankGPT untuk membuat dan memproses instruksi permutasi sebagai berikut:

from rank_gpt import (
    create_permutation_instruction,
    run_llm,
    receive_permutation
)
# Create permutation generation instruction
messages = create_permutation_instruction(
    item=item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo'
)

[{'role': 'system',
  'content': 'You are RankGPT, an intelligent assistant that can rank passages based on their relevancy to the query.'},
 {'role': 'user',
  'content': 'I will provide you with 3 passages, each indicated by number identifier []. \\nRank the passages based on their relevance to query: How much impact do masks have on preventing the spread of the COVID-19?.'},
 {'role': 'assistant', 'content': 'Okay, please provide the passages.'},
 {'role': 'user',
  'content': '[1] Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
 {'role': 'assistant', 'content': 'Received passage [1].'},
 {'role': 'user',
  'content': '[2] Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
 {'role': 'assistant', 'content': 'Received passage [2].'},
 {'role': 'user',
  'content': '[3] Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
 {'role': 'assistant', 'content': 'Received passage [3].'},
 {'role': 'user',
  'content': 'Search Query: How much impact do masks have on preventing the spread of the COVID-19?. \\nRank the 3 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [] > [], e.g., [1] > [2]. Only response the ranking results, do not say any word or explain.'}]

# Get ChatGPT predicted permutation
permutation = run_llm(
    messages,
    api_key='Your OPENAI Key!',
    model_name='gpt-3.5-turbo'
)

'[1] > [3] > [2]'

# Use permutation to re-rank the passage
item = receive_permutation(
    item,
    permutation,
    rank_start=0,
    rank_end=3
)

{'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
 'hits': [{'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
  {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
  {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}]}

Strategi Sliding Window (SWA) untuk RankGPT

Jika Anda perlu memberi peringkat lebih banyak dokumen daripada yang dapat ditangani model sekaligus, gunakan strategi sliding window. Berikut cara menerapkan strategi sliding window untuk mengurutkan ulang dokumen:

from rank_gpt import sliding_windows
api_key = "Your OPENAI Key"
new_item = sliding_windows(
    item,
    rank_start=0,
    rank_end=3,
    window_size=2,
    step=1,
    model_name='gpt-3.5-turbo',
    api_key=api_key
)
print(new_item)

Dalam contoh ini, sliding window berukuran 2 dengan langkah 1, artinya memproses dua dokumen sekaligus, lalu maju satu dokumen untuk proses pemeringkatan berikutnya.

Kesimpulan

Dengan menggunakan LLM untuk menilai relevansi informasi secara lebih baik, RankGPT meningkatkan akurasi pengurutan dan re-ranking konten.

Ini mengatasi masalah umum seperti memastikan konten tetap tepat sasaran, meningkatkan efisiensi, dan mengurangi kemungkinan menghasilkan informasi yang menyesatkan.

Secara keseluruhan, RankGPT berkontribusi pada pembangunan aplikasi RAG yang lebih andal dan akurat.