RAG için Yeniden Sıralama Aracısı Olarak RankGPT (Eğitim)

Retrieval Augmented Generation (RAG) Nedir?

Retrieval augmented generation (RAG), LLM’leri bilgi erişim sistemleriyle birleştiren bir yöntemdir. Bu sayede bir LLM’den metin üretmesi istendiğinde, yanıtlarını daha doğru ve bilgi dolu kılmak için harici kaynaklardan ilgili bilgileri çekebilir.

RAG iki ana bileşenden—getirici (retriever) ve üretici (generator)—ve isteğe bağlı bir bileşenden, yeniden sıralayıcıdan (reranker) oluşur:

1. Retriever—Retriever’ın görevi, kullanıcının sorgusuna göre büyük bir belge kümesinden ilgili belgeleri veya metin parçalarını bulmaktır. Belgeleri ilgililiklerine göre sıralamak için BM25 gibi algoritmalar kullanır.
2. Reranker (isteğe bağlı) — Reranker, ilk getirilen belge kümesini yeniden sıralar ve en ilgili olanların üstte yer almasını sağlar. Bu, daha az faydalı bilgileri ayıklamaya ve önemli olana odaklanmaya yardımcı olur.
3. Generator — Generator, getirilen belgeleri kullanarak nihai çıktıyı üreten LLM’dir. İlgili harici verilere erişim, daha doğru yanıtlar üretebilir.

RAG’de RankGPT’nin Rolü ve Faydaları

RankGPT, getirilen belgelerin veya metin parçalarının ilgililiğini değerlendirmek için LLM’leri kullanır ve en önemlilerin üstte yer almasını sağlar. RankGPT ile RAG hattındaki generator daha yüksek kaliteli girdiler alır ve bu da daha isabetli yanıtlar üretir.

Daha iyi ilgililik ve performans

RankGPT, sorguların ve belgelerin daha derin anlamını ve bağlamını kavrayarak basit anahtar kelime eşleştirmenin ötesine geçer. Böylece sadece anahtar kelimelere değil, gerçek anlama dayalı olarak en ilgili içeriği belirleyip LLM’lere daha doğru bilgi sağlar.

GPT-4 sıfır örnekli yönerge permütasyon üretimiyle kullanıldığında, RankGPT TREC, BEIR ve Mr.TyDi gibi çeşitli kıyaslamalarda önde gelen denetimli sistemleri geride bırakır.

Verimli ve uygun maliyetli damıtma

RankGPT, GPT-4 gibi büyük modellerin sıralama yeteneklerini daha küçük, uzmanlaşmış modellere aktarmak için permütasyon damıtımını kullanır.

Bu daha küçük modeller, çok daha verimli olurken yüksek performansı korur. Örneğin, damıtılmış 440M’lik bir model, BEIR kıyaslamasında 3B denetimli bir modeli geride bırakarak hesaplama maliyetlerini önemli ölçüde azalttı ve daha iyi sonuçlar elde etti.

Yeni ve bilinmeyen bilgileri ele alma

RankGPT, sağlamlığı sağlamak ve veri bulaşması endişelerini gidermek için NovelEval test setini içerir. Bu set, modelin yakın tarihli ve bilinmeyen bilgilere dayalı pasajları sıralama becerisini değerlendirir.

GPT-4 bu testte son teknoloji performansına ulaştı ve yeni ve görülmemiş sorguları etkili şekilde ele alabildiğini gösterdi.

RankGPT Kıyaslama Performansı

RankGPT (gpt-4), aşağıdaki tabloda gösterildiği gibi ortalama nDCG@10 skoru 53,68 ile TREC ve BEIR genelinde diğer tüm modelleri geride bırakıyor. BEIR veri kümelerinde en yüksek sonuçları alarak monoT5 (3B) ve Cohere Rerank-v2 gibi güçlü denetimli modelleri geçti. gpt-3.5-turbo ile bile RankGPT rekabetçi skorlar elde ederek son derece etkili bir yeniden sıralayıcı olduğunu kanıtlıyor.

Kaynak: Weiwei Sun ve diğ., 2023

RankGPT (gpt-4) ayrıca Mr.TyDi veri kümelerinde de güçlü bir performans sergileyerek 62,93 ortalama nDCG@10 skoru ile BM25 ve mmarcoCE’yi geride bırakıyor. Birçok dilde tutarlı şekilde BM25’i aşıyor ve özellikle Endonezce ve Svahili’de mmarcoCE’yi de geçiyor.

Genel olarak RankGPT, Bengalce, Endonezce ve Japonca gibi birçok dilde en yüksek puanları aldı; yalnızca birkaç durumda mmarcoCE’nin biraz gerisinde kaldı.

Kaynak: Weiwei Sun ve diğ., 2023

Son olarak, RankGPT yakın tarihli ve yabancı bilgilere dayalı olarak bir modelin pasajları ne kadar iyi sıralayabildiğini ölçen NovelEval veri setinde test edildi. RankGPT (gpt-4), özellikle 90,45’lik nDCG@10 skoru ile tüm değerlendirme metriklerinde (nDCG@1, nDCG@5 ve nDCG@10) en yüksek puanları aldı. monoT5 (3B) ve monoBERT (340M) gibi güçlü modelleri geride bırakarak yeniden sıralayıcı olarak güçlü performansını ortaya koydu.

Kaynak: Weiwei Sun ve diğ., 2023

Tüm kıyaslama sonuçlarında RankGPT (gpt-4), denetimli ya da denetimsiz fark etmeksizin diğer yöntemleri istikrarlı biçimde geride bırakarak yeniden sıralamada üstün yeteneğini göstermektedir.

RAG Hatlarında RankGPT’yi Uygulama

RankGPT’yi bir RAG hattına şu şekilde entegre edebiliriz.

Adım 1: RankGPT deposunu klonlayın

Öncelikle RankGPT deposunu klonlamanız gerekir. Terminalinizde aşağıdaki komutu çalıştırın:

git clone https://github.com/sunnweiwei/RankGPT

Adım 2: Ortamınızı hazırlayın

RankGPT dizinine gidin ve gerekli paketleri yükleyin. Sanal bir ortam oluşturup, sağlanan requirements.txt dosyasını kullanarak paketleri kurmak isteyebilirsiniz:

pip install -r requirements.txt

Adım 3: RankGPT uygulaması

Burada, orijinal RankGPT deposu tarafından sağlanan basit örnek sorguyu ve getirilen belgeleri kullanıyoruz.

item = {
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'}
    ]
}

RankGPT ile getirilen belgeleri kolayca yeniden sıralamak için sağlanan permütasyon hattını kullanabilirsiniz.

from rank_gpt import permutation_pipeline
new_item = permutation_pipeline(
    item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo',
    api_key='Your OPENAI Key!'
)
print(new_item)

Bu, belgelerin aşağıdaki yeni sırasıyla sonuçlanacaktır:

{
    'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
    'hits': [
        {'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
        {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
        {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}
    ]
}

Adım Adım Yönergesel Permütasyon Üretimi

Permütasyon hattının daha adım adım bir uygulaması için, aşağıdaki gibi doğrudan RankGPT ile etkileşime geçerek permütasyon yönergeleri oluşturup işleyebilirsiniz:

from rank_gpt import (
    create_permutation_instruction,
    run_llm,
    receive_permutation
)
# Create permutation generation instruction
messages = create_permutation_instruction(
    item=item,
    rank_start=0,
    rank_end=3,
    model_name='gpt-3.5-turbo'
)

[{'role': 'system',
  'content': 'You are RankGPT, an intelligent assistant that can rank passages based on their relevancy to the query.'},
 {'role': 'user',
  'content': 'I will provide you with 3 passages, each indicated by number identifier []. \\nRank the passages based on their relevance to query: How much impact do masks have on preventing the spread of the COVID-19?.'},
 {'role': 'assistant', 'content': 'Okay, please provide the passages.'},
 {'role': 'user',
  'content': '[1] Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
 {'role': 'assistant', 'content': 'Received passage [1].'},
 {'role': 'user',
  'content': '[2] Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'},
 {'role': 'assistant', 'content': 'Received passage [2].'},
 {'role': 'user',
  'content': '[3] Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
 {'role': 'assistant', 'content': 'Received passage [3].'},
 {'role': 'user',
  'content': 'Search Query: How much impact do masks have on preventing the spread of the COVID-19?. \\nRank the 3 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [] > [], e.g., [1] > [2]. Only response the ranking results, do not say any word or explain.'}]

# Get ChatGPT predicted permutation
permutation = run_llm(
    messages,
    api_key='Your OPENAI Key!',
    model_name='gpt-3.5-turbo'
)

'[1] > [3] > [2]'

# Use permutation to re-rank the passage
item = receive_permutation(
    item,
    permutation,
    rank_start=0,
    rank_end=3
)

{'query': 'How much impact do masks have on preventing the spread of the COVID-19?',
 'hits': [{'content': 'Title: Universal Masking is Urgent in the COVID-19 Pandemic: SEIR and Agent Based Models, Empirical Validation, Policy Recommendations Content: We present two models for the COVID-19 pandemic predicting the impact of universal face mask wearing upon the spread of the SARS-CoV-2 virus--one employing a stochastic dynamic network based compartmental SEIR (susceptible-exposed-infectious-recovered) approach, and the other employing individual ABM (agent-based modelling) Monte Carlo simulation--indicating (1) significant impact under (near) universal masking when at least 80% of a population is wearing masks, versus minimal impact when only 50% or less of the population is wearing masks, and (2) significant impact when universal masking is adopted early, by Day 50 of a regional outbreak, versus minimal impact when universal masking is adopted late. These effects hold even at the lower filtering rates of homemade masks. To validate these theoretical models, we compare their predictions against a new empirical data set we have collected'},
  {'content': 'Title: To mask or not to mask: Modeling the potential for face mask use by the general public to curtail the COVID-19 pandemic Content: Face mask use by the general public for limiting the spread of the COVID-19 pandemic is controversial, though increasingly recommended, and the potential of this intervention is not well understood. We develop a compartmental model for assessing the community-wide impact of mask use by the general, asymptomatic public, a portion of which may be asymptomatically infectious. Model simulations, using data relevant to COVID-19 dynamics in the US states of New York and Washington, suggest that broad adoption of even relatively ineffective face masks may meaningfully reduce community transmission of COVID-19 and decrease peak hospitalizations and deaths. Moreover, mask use decreases the effective transmission rate in nearly linear proportion to the product of mask effectiveness (as a fraction of potentially infectious contacts blocked) and coverage rate (as'},
  {'content': 'Title: Masking the general population might attenuate COVID-19 outbreaks Content: The effect of masking the general population on a COVID-19 epidemic is estimated by computer simulation using two separate state-of-the-art web-based softwares, one of them calibrated for the SARS-CoV-2 virus. The questions addressed are these: 1. Can mask use by the general population limit the spread of SARS-CoV-2 in a country? 2. What types of masks exist, and how elaborate must a mask be to be effective against COVID-19? 3. Does the mask have to be applied early in an epidemic? 4. A brief general discussion of masks and some possible future research questions regarding masks and SARS-CoV-2. Results are as follows: (1) The results indicate that any type of mask, even simple home-made ones, may be effective. Masks use seems to have an effect in lowering new patients even the protective effect of each mask (here dubbed"one-mask protection") is'}]}

RankGPT için Kayan Pencere Stratejisi (SWA)

Modelin aynı anda işleyebileceğinden daha fazla belgeyi sıralamanız gerekiyorsa, kayan pencere stratejisini kullanın. Belgeleri yeniden sıralamak için kayan pencere stratejisini şu şekilde uygulayabilirsiniz:

from rank_gpt import sliding_windows
api_key = "Your OPENAI Key"
new_item = sliding_windows(
    item,
    rank_start=0,
    rank_end=3,
    window_size=2,
    step=1,
    model_name='gpt-3.5-turbo',
    api_key=api_key
)
print(new_item)

Bu örnekte kayan pencerenin boyutu 2, adım boyutu ise 1’dir; yani her seferinde iki belge işler ve bir sonraki sıralama geçişi için bir belge ileri kayar.

Sonuç

RankGPT, bilgilerin ilgililiğini daha iyi değerlendirmek için LLM’leri kullanarak içeriklerin sıralama ve yeniden sıralama doğruluğunu artırır.

Bu yaklaşım, içeriğin isabetliliğini sağlama, verimliliği artırma ve yanıltıcı bilgi üretme olasılığını azaltma gibi yaygın sorunları ele alır.

Genel olarak RankGPT, daha güvenilir ve doğru RAG uygulamaları geliştirilmesine katkı sağlar.