Improving Single Token Compression for Retrieval-Augmented Generation

What I did :

1. Improved answer accuracy by increasing Exact Match from 0.51 to 0.53 through the generation of up to 20 synthetic queries per question.
2. Enhanced retrieval robustness by using ensemble reranking across multi-query results to consistently identify the most relevant documents.
3. Reduced inference latency from approximately 43.2 seconds to 15.6 seconds by aggregating multi-document evidence before single-token compression.

1. Generate Distractor Choices

To save time and storage when setting up the corpus, we generate distractor context for each query. The prompt to generate the context is as shown:

prompt = (
    f"Generate {k} completely unrelated, misleading long contexts for the following question: \"{original_query}\". "
    "Each snippet should be a plain-text, Wikipedia-like excerpt about a random, generic topic (e.g., history, science, art) that is entirely irrelevant to the question. "
    "Do not mention any keywords related to the original query (or similar terms). "
    "Each snippet should be at least 4 SENTENCES LONG and must not reference the question or its subject in any way. "
    "Do not include numbering, bullet points, extra characters, headers, or any extra labels. "
    "Do not include numbered lists or any type of ordered lists. "
)

We use the "meta-llama/Llama-2-7b-chat-hf" model provided in huggingface. To create the distractor choices, run this:

cd xRAG/src/eval
python run_eval_multi_query.py \
    --data triviaqa_topk \
    --model_name_or_path Hannibal046/xrag-7b \
    --use_rag \
    --create_distractors \
    --save_dir ../../data/eval/triviaqa/retrieval/colbertv2

You can optionally use the --max_test_samples argument for debugging.

2. Generate Synthetic Queries and Ensemble Score

After generating your distractor choices, you can now generate k number of synthetic queries. Each synthetic query can now choose their own top 1 document. Afterwards, an ensemble scoring is done to select the overall top 1 document.

To do this run:

cd xRAG/src/eval
python run_eval_multi_query.py \
    --data triviaqa \
    --model_name_or_path Hannibal046/xrag-7b \
    --use_rag \
    --k_samples 5 \
    --retriever_name_or_path Salesforce/SFR-Embedding-Mistral \
    --ensemble_rerank \
    --save_dir ../../data/eval/triviaqa/retrieval/colbertv2

3. Evaluate Approach

cd xRAG/src/eval
python run_eval_multi_query.py \
    --data triviaqa \
    --model_name_or_path Hannibal046/xrag-7b \
    --use_rag \
    --k_samples 5 \
    --retriever_name_or_path Salesforce/SFR-Embedding-Mistral \
    --save_dir ../../data/eval/triviaqa/retrieval/colbertv2

If you defined --max_test_samples from step 1, please define it here as well.