RAG evaluation · Research tooling
RAGlign
Point it at your documents and it tells you which RAG configuration to use — and proves it.
RAG evaluation has a bias larger than the differences it is trying to measure. RAGlign measures that bias, removes it, and then uses the corrected metrics to rank 30 pipeline configurations against your own documents — offline, on CPU, with no vector database.
Problem
Building RAG means choosing a chunking strategy, a retriever, and whether to rerank. Those choices interact, the right answer depends on your documents, and most teams pick by intuition. Measuring instead does not help as much as it should, because the standard measurement is biased: ground truth is recorded as a chunk ID, but chunk 57 only exists inside one chunking strategy. Score a different strategy against it and you are measuring how similar its cut points are, not how well it retrieves.
Objectives
- Replace chunk-ID ground truth with something that belongs to the documents rather than to one chunking strategy.
- Quantify what merely authoring the ground truth is worth, and whether it changes the conclusion.
- Rank 30 configurations on a real corpus and explain why each one placed where it did.
- Name the stage responsible for every missed question, because the stages need opposite fixes.
- Check whether better retrieval actually produces better answers.
Implementation
- Ground truth is a character span in the original file. Each question carries a short quote copied from the corpus; the tool locates it and derives the offsets, and a quote that no longer appears verbatim is rejected rather than silently rotting.
- One invariant is asserted on every chunk of every strategy: the text sliced from the document at a chunk's offsets must equal the chunk itself. Offset drift does not crash anything — it produces plausible, wrong metrics. It caught a real bug where a CRLF checkout broke every multi-line quote.
- The grid builds five chunkers against three retrievers with reranking on or off, and runs every question through all thirty pipelines.
- Retrieval metrics are arithmetic over span overlap rather than LLM-judged, so they rerun bit-identically. Using an LLM to validate a methodology whose selling point is determinism would be circular.
- No vector database: roughly 2,000 chunks at 384 dimensions is a 3 MB NumPy matrix, and brute force is exact and sub-millisecond. Approximate search would be a confound in a study about retrieval correctness.
- A validation study scores every strategy twice against the same questions and the same retrievals — once by chunk ID, once by span alignment — then rotates which strategy authored the ground truth.
- A generation loop closes the gap to the user: one model answers from each configuration's retrieved context, and a different model judges, because judges show self-preference toward their own output.
Architecture
Ground truth
- documents
- ~40 questions
- verbatim quotes
- quote resolved to character span
- offset invariant asserted
Evaluation
- 5 chunkers
- 3 retrievers
- reranker on / off
- span overlap
- hit@k
- MRR
- nDCG
- bootstrap CIs
Analysis
- chunker destroyed / degraded
- retriever missed
- reranker demoted
- quality / latency / context weights
- templated explanation
- Streamlit dashboard
- CLI scripts
Technologies
- Python
- sentence-transformers
- BM25
- NumPy
- Streamlit
- pytest
- bootstrap CIs
Challenges & solutions
Challenge
Ground truth expressed as a chunk ID belongs to whichever strategy produced it, so comparing strategies against it measures agreement with its cut points rather than retrieval quality.
Solution
Store the span in the source document instead. Every strategy is then asked whether it retrieved the answering text, and none of them wrote the question.
Challenge
Any claim that the bias is real could be an artefact of which strategy happened to author the ground truth.
Solution
Rotate the author across strategies. A convenient effect would dissolve; this one held, and replicated on a corpus with no markup at all.
Challenge
The obvious objection is that this is just strict matching dressed up.
Solution
Sweep the overlap threshold. Relaxing the gold-chunk criterion from IoU ≥ 0.9 down to 0.3 decays the bias smoothly to zero — chunk-ID matching becomes unbiased only once it has been loosened into being span overlap, which is the argument rather than a counterexample.
Challenge
Measuring overlap as IoU would rank strategies by chunk size rather than by quality.
Solution
Measure coverage instead. Heading chunking covers 11 of 11 evidence spans perfectly while scoring worst on IoU purely because its chunks are larger; chunk bloat is reported separately as a context-budget cost.
Challenge
A score says a question failed but not which stage lost it, and the stages need opposite fixes — no embedding model can rescue evidence the chunker destroyed.
Solution
Ask whether any chunk in the index could have answered it, which is only expressible when ground truth is a position, and classify each failure as chunker destroyed, chunker degraded, retriever missed, reranker demoted, or ranked low.
Ground-truth bias
| Corpus | Questions | Chunk-ID ground truth | Span alignment |
|---|---|---|---|
| FastAPI docs | 41 | +0.569 … +0.675 | +0.016 … +0.049 |
| Public-domain prose | 43 | +0.589 … +0.744 | −0.116 … +0.101 |
Results
- Merely authoring the ground truth is worth roughly 0.6 hit@5 — several times the genuine quality spread between the strategies being compared — and it flips the recommended configuration in 3 of 4 rotations.
- The effect replicates on a corpus with no markup at all, so it is not a property of one document style.
- Per-question diagnosis separates failures that need a different chunker from ones that need a different retriever. On 300-character chunks, 8 of 11 failures were the chunker and none were the retriever — standard evaluation reports that as low hit@5 and sends you shopping for embedding models.
- Retrieval quality predicts answer quality at r ≈ +0.95, measured by having one model answer from each configuration's context and a different model judge it.
Lessons learned
- The measurement was the hard part, not the search. Most of the engineering went into making the numbers trustworthy rather than into retrieval itself.
- Check invariants instead of trusting them. The offset assertion caught a CRLF bug that would otherwise have produced plausible and entirely wrong results.
- Sample size humbles you. A finding that chunking mattered far more on structured documents held at 11–14 questions and evaporated at 41–43 — and when the sample was small, the effect that replicated and the one that vanished were reported with equal confidence.
- Positioning honestly beats claiming novelty. Span-level ground truth already exists in the literature; what was missing was a reusable tool rather than a new technique, and that is the smaller, accurate claim.