Reciprocal Rank Fusion

Click to play video

Score normalization can be messy. Min-max normalization doesn't work well with major outliers, or if the data you're combining has different kinds of distributions.

What if we could skip scores entirely? That's what Reciprocal Rank Fusion (RRF) does!

def rrf_score(rank, k=60):
    return 1 / (k + rank)

It uses rankings instead of scores. A ranking is just a position in a list, so instead of:

Document A = 15.2
Document B = 8.7
Document C = 6.3

We use numbered ranks:

Document A = 1st
Document B = 2nd
Document C = 3rd

Example Result Set

Say we run a query through keyword and semantic search, and get these score-based results:

BM25:
- Brother Bear (15.2)
- Jungle Book (6.3)
- Paddington (8.7)
Semantic:
- Paddington (0.8)
- Brother Bear (0.7)
- We Bare Bears (0.6)

We can use the rrf_score function to convert these into rrf rankings:

BM25:
1. Brother Bear – 1 / (60 + 1) = 0.0164
2. Jungle Book – 1 / (60 + 2) = 0.0161
3. Paddington – 1 / (60 + 3) = 0.0159
Semantic:
1. Paddington – 1 / (60 + 1) = 0.0164
2. Brother Bear – 1 / (60 + 2) = 0.0161
3. We Bare Bears – 1 / (60 + 3) = 0.0159

To create a hybrid ranking, we can just sum the RRF scores for each document across both result sets:

Brother Bear: 0.0164 + 0.0161 = 0.0325
Paddington: 0.0164 + 0.0159 = 0.0323
Jungle Book: 0.0161
We Bare Bears: 0.0159

What's the K Parameter?

The k parameter (a constant) controls how much more weight we give to higher-ranked results vs. lower-ranked ones.

Lower k values like 20: Gives more weight to top-ranked results, creating a steep drop-off in scores.
Higher k values like 100: Creates a more gradual decline, giving lower-ranked results more influence.

A good "default" value for k is around 60; this tends to work well across many datasets and queries.

Assignment

Build hybrid search using Reciprocal Rank Fusion.

Add a new rrf-search command to your hybrid search CLI script.
- It should accept a required positional query parameter.
- It should accept an optional -k parameter (default to 60).
- It should accept an optional --limit parameter (default to 5).
Implement the missing rrf_search method in your HybridSearch class. It should:
- Call the _bm25_search method to get BM25 results. Again, get 500 times the actual limit.
- Call the search method of ChunkedSemanticSearch to get semantic chunk results for the same query. Again, get 500 times the actual limit.
- Combine the results from both searches using Reciprocal Rank Fusion as follows:
  - Create a dictionary mapping document IDs to the documents themselves and their BM25 and semantic ranks (not scores).
  - For each document, calculate the RRF score using the rrf_score function and add that score to each document as well. If a document shows up in both searches, sum its RRF scores.
  - Return the results sorted by the RRF score in descending order.
Hook up your rrf-search command to your HybridSearch class's rrf_search method. It should call the method and return the results, truncated to the specified limit, in this format:

1. Paddington
   RRF Score: 0.033
   BM25 Rank: 1, Semantic Rank: 1
   Deep in the rainforests of Peru, a young bear lives peacefully with his Aunt Lucy and Uncle Pastuzo,...

2. The Indian in the Cupboard
   RRF Score: 0.031
   BM25 Rank: 2, Semantic Rank: 8
   On his ninth birthday, Omri receives an old cupboard from his brother Gillon (Vincent Kartheiser) an...

Run and submit the CLI tests.