Is Contextual Retrieval worth it? Here is what each technique added.

01What Contextual Retrieval is

To search long documents, you cut them into small pieces called chunks. The problem is that a chunk loses its context. A chunk that says “the fee was 8% of total development cost” no longer says which project or which document it came from.

Contextual Retrieval, published by Anthropic, fixes this before anything is searched:

1. An AI model reads the whole document and one chunk.
2. It writes a short sentence saying where the chunk comes from, such as “This is from the final cost report for the Oak Street project.”
3. That sentence is added to the chunk, and the search index is built from the result.

Anthropic reported 35–67% fewer retrieval failures on its own data. We wanted to see how it does on documents that look like a real company archive, and how it compares with the other tools a team would reach for.

02The dataset, and why it is hard

287 synthetic housing-finance documents: mortgages, liens, payoff letters, grant agreements, budgets and policies. Because we generated them from a known set of facts, we know every right answer to every question. The archive was built to be hard in the same ways real ones are:

The 77% matters most here: it is exactly the problem Contextual Retrieval is meant to solve, so the test gives it every chance. We asked 81 questions of two kinds:

• Find one thing (60 questions): “Pull up the payoff letter for this client.” One or two right documents each.
• Find everything (21 questions): “Show everything about this project” or “How many loans did we issue in 2021?” Often six or more right documents, up to 69.

03Keyword search

Keyword search finds chunks that contain the words in the question. We used Meilisearch, a popular open-source search engine, exactly as it comes out of the box. Its defaults are built for short queries typed into a search box, and two of them hurt on full questions:

• It doesn’t weigh rare words. It ranks chunks by how many question words they contain, so a client name counts no more than “what” or “the”.
• It drops words when matches run short. If too few chunks contain every word, it removes words from the end of the question. In “What developer fee was charged on [project]?”, the project name goes first.

15.9% recall@20. Its “frequency matching” setting, which drops common words first, reached 35.4%. This is a mismatch between search-box defaults and full questions, not a verdict on keyword search.

04BM25 search

BM25 is also keyword search, with a different way of ranking. It gives more weight to rare words, so a client name or a loan number counts far more than “what” or “the”.

84.3% recall@20, on the same documents and the same chunks. Only the ranking changed, and it was the biggest single improvement in the whole study.

05Similarity search

Similarity search finds chunks with a similar meaning, even when they use different words. It handles paraphrasing well, and is weaker on exact names and numbers. We used bge-m3 embeddings in Qdrant.

83.1% recall@20, close to BM25 but strong on different questions.

06BM25 + similarity search

A hybrid runs both searches and merges the two result lists, so each covers the other’s blind spot: BM25 catches exact names, similarity catches paraphrase.

87.7% recall@20, the best of the four. This is our starting point for everything that follows.

A good search finds single documents easily. Every method struggles when a question needs many documents at once. The next three techniques are attempts to fix that, and each is added on its own to the same starting point.

07Reranker

After the search returns its top 30 results, a second AI model reads the question and each result together and re-orders them, keeping the best 20. It can lift a good document from 25th place to 5th. It cannot rescue one the search never returned. We used Voyage rerank-2.5-lite, at about $0.0001 per search.

87.7% → 92.1% recall@20. That is +4.4 points.

08Filtering

Filtering puts context back as structured tags instead of text. Most archives hold a limited set of document types with predictable fields: a mortgage has a borrower, a lender, an amount and a date.

• When a document is added, an AI model reads it once and tags it with facts such as client, project, year and document type.
• When a question comes in, one short AI call turns it into filters. “Every loan issued in 2021” becomes type = loan and year = 2021, and the search only looks at documents that match.
• If a filter is wrong, it loosens step by step, and an ordinary search fills any empty places, so a mistake costs ranking, not the answer.

87.7% → 94.4% recall@20. That is +6.7 points, the largest gain of any single technique.

09Contextual chunking

This is Contextual Retrieval itself: every chunk gets its AI-written sentence of context before indexing. For this archive that took 5,171 AI calls, with 59% of the prompt text served from cache.

87.7% → 88.1% recall@20. That is +0.4 points.

One likely reason the gain is smaller than Anthropic’s: our documents already carry strong clues such as names, loan numbers and addresses, so a sentence of context has less to add. It should do better on documents where a chunk means nothing on its own, like manuals, contracts with numbered clauses, and code.

10What if we use all of them together?

Contextual chunking, a reranker and filters, all on top of BM25 + similarity search:

All together reached 94.6%, the best result. Each technique fixes a different gap: filters narrow the search to the right documents, the reranker puts the best of them first, and contextual chunking gives each chunk back the context it lost when the document was split. On an archive whose documents already name their clients and loans, filters and the reranker carried most of the gain.

• “Find everything” questions are where the gains are: from 59.8% to 91.1% with everything on. Filtering did most of that work.
• “Find one thing” questions were already above 97%. Filters added a few small mistakes there, from 97.5% to 95.8% with everything on.

11What each one costs

The two make different trade-offs. Contextual chunking does all its work once, when the index is built, and costs nothing per search. Filtering is lighter to build but adds one short AI call to every question. The reranker needs no work at indexing and costs about $0.0001 per search.

12What we would recommend

1. Check how your keyword search ranks. If it is a search-box engine at default settings, switch to BM25 ranking.
2. Combine BM25 with similarity search, and add a reranker. That reached 92.1% here with no AI work at indexing.
3. Add filtering if people ask “everything about…” or “how many…” questions. It is the technique that moved them most.
4. Test Contextual Retrieval on your own documents, especially if your chunks mean little without their source. Measure recall before and after.

13Limitations and method

• The documents are synthetic. They were generated from the same facts the filters look for, which favours filtering. Real archives contain documents that fit no template, and fields that are missing or wrong.
• 81 questions is a small set. Differences under about 1.5 points are noise.
• Tuned filters are left out of the comparison. A tuned version reached 96.5% recall@20 with BM25 + similarity, but its fixes were found by reading failures on these same questions, so the number is optimistic.
• Only search was scored, not the quality of the final answers.
• One of each. One similarity model, one AI model for context sentences and filters, Meilisearch at default and frequency settings only, and a simple BM25 with no stemming.

14All 88 setups

Want a different combination? Filter by search method, filtering, chunking or reranker. Sorted best first; the outlined row is the starting point.

Synthetic documents, questions and answer key. Search: Meilisearch 1.54, BM25, and Qdrant with bge-m3 for similarity. AI model for context sentences and filters: Qwen3.8-27B. Reranker: Voyage rerank-2.5-lite.