Kapa.ai's RAG Pruner: Drop 68% of Context, Keep 96% Recall

Kapa.ai builds AI assistants that answer complex questions over large knowledge bases — think API docs, forums, and support threads. Their RAG pipeline has a retriever and a generator, but they added a third step: a small, cheap LLM that reads the question and all retrieved chunks together, then discards the ones the answer won't need before the expensive model ever sees them. It drops about 68% of the context, keeps about 96% of recall, and cuts query cost by a third, net of its own cost.

Why prune at all?

Retrievers aim for maximum recall, so they return many chunks, most of which are noise. The generator pays for every chunk it reads — in Kapa's setup, retrieved chunks account for about two-thirds of query cost. Every chunk removed cuts cost by ~4%. In agentic workflows, context grows fast with tool call outputs, so tighter retrieval leaves room for everything else. The challenge is recall: drop a needed chunk and you trade cents for wrong answers.

Why reranker scores don't work

Reranker scores are ordinal, not calibrated — Cohere's docs say as much. A fixed cutoff like "keep everything above 0.7" fails because scores aren't comparable across queries. The deeper problem: relevance isn't a property of a single chunk. Pointwise cross-encoders score each query-chunk pair alone, missing that a chunk may be useless by itself but essential when combined with another. For example, one chunk defines a term, another explains its usage — each alone seems irrelevant, together they answer the question.

Anchor documents don't fix it

Kapa tried anchor documents (Sinhababu et al., 2025): plant synthetic chunks of known relevance into the ranking, then drop real chunks below a threshold. It fixes calibration but not the underlying scoring — the reranker still placed partially relevant chunks below irrelevant ones. To keep them, the anchor had to sit so low that hardly anything got pruned.

The solution: listwise LLM grading

Kapa ships a single listwise LLM call between reranker and generator. It gets the question and all chunks, then grades each on a five-level scale:

• 5 (ESSENTIAL): answer impossible without this chunk
• 4 (CONTRIBUTING): supplies something needed in combination with others
• 3 (SUPPORTING): on topic but likely unnecessary
• 2 (TANGENTIAL): same domain, no concrete contribution
• 1 (UNRELATED): no meaningful connection

Chunks at or above a threshold survive. Because each level is defined in words, a fixed cutoff works across queries. And because the model sees all chunks together, it can judge the set — partial and indirect relevance finally land somewhere.

Three knobs to tune

• Model: small, fast, cheap — paid for out of what it saves
• Threshold: the main dial between compression and recall
• keep-top-k: protect the strongest chunks from grading mistakes

Kapa also tested simpler designs (budget-select and direct "which chunks to keep?") but the five-level scale won on both compression and recall. The result: 68% context reduction, 96% recall, and a third off query costs.