XGZXiangguo Zhang
Back to discipline

Deterministic RAG evaluation

RAG Quality Lab

A controlled regression test caught a knowledge-base update degrading the strongest pipeline; I then extended the same evidence lifecycle into tracked evaluation infrastructure for 11,309 enterprise documents.

PythonChromaBM25 + cross-encoder rerankSentence TransformersLangChainDeterministic manifests
Problem
RAG changes often look harmless while silently reducing answer quality. Teams need a repeatable gate that catches regressions and still works when the corpus becomes too large to inspect manually.
Audience
Applied-AI, retrieval, and evaluation teams that need reviewable evidence before a RAG change reaches users.
What I built
I built the A/B and regression harness, then added enterprise-corpus adapters, deterministic manifests, backend seams, a judge-free retrieval runner, verifiers, tests, and operating documentation.
Result
On the controlled 12-question set, a document-only update degraded four questions and every reported quality metric. The repository now carries the same versioned-data and verification lifecycle to 11,309 synthetic enterprise documents and 130 answerable questions.
Links
GitHub repositoryOpens external site

How it works

  1. 01
    Compare

    Run naive vector and hybrid-rerank pipelines on the same controlled questions.

  2. 02
    Regress

    Compare one pipeline across knowledge-base versions and bucket question-level changes.

  3. 03
    Scale

    Adapt 11,309 synthetic enterprise documents and 130 grounded questions.

  4. 04
    Manifest

    Bind datasets, adapters, and outputs to deterministic hashes.

  5. 05
    Gate

    Fail on data drift, contract breaks, or measured regression.

Try it

The follow-on engineering question: can that same evidence lifecycle remain auditable when the corpus grows beyond manual inspection?

Enterprise-scale extension

From the regression finding to a gated 11,309-document evaluation path

The public repository now contains the adapters, question set, manifest writer, runner seam, backend contract, and tests. The ~88 MB generated knowledge base is reproducible but intentionally excluded from Git.

  1. 01 · LockPin the EnterpriseRAG-Bench S1 source slice and license boundary.
  2. 02 · AdaptNormalize 5,189 Confluence and 6,120 Jira records into 11,309 documents.
  3. 03 · ManifestWrite deterministic counts, hashes, adapter identity, and generation parameters.
  4. 04 · MapBind 130 S1-answerable questions to the bounded evaluation scope.
  5. 05 · PreflightValidate the backend contract, corpus identity, and output destination before a run.
  6. 06 · RunInvoke the judge-free retrieval runner only when corpus, contract, and output gates pass.
Loading claim registry...
Documents in the bounded S1 scope11,309
S1-answerable questions130
Passing tests68
S1 corpus adapterS1-answerable question adapterDeterministic manifestBackend-aware verifierRetrieval contract layerRuff and test suite

How this was verified

  • The saved 2026-04 controlled run was deterministically re-parsed in 2026-07: Pipeline B moved from 0.988 to 0.867 faithfulness after the V1-to-V2 document update, with 4 degraded, 0 improved, and 8 stable questions.
  • The public repository at commit 88879a2 contains the enterprise adapters, manifest, backend seam, retrieval runner, tests, and bilingual operating documentation; its CI passed before merge.

What this does not prove

  • The reported quality and latency measurements belong to the controlled saved runs; they do not transfer to the 11,309-document enterprise corpus.
  • At enterprise scale, the current evidence establishes data integrity and runnable evaluation infrastructure, not a new answer-quality result.
  • The lab is a single-machine evaluation workbench, not a released package or a production serving system.