Can you trust your AI's context?

AI is only as reliable as the context it receives.

Modern AI systems depend on context layers to retrieve knowledge, maintain memory, and ground decisions. Yet there is no standard way to measure whether those context systems are stable, reproducible, or trustworthy. ContextBenchmark fills that gap.

The benchmark intentionally measures the context layer, not the language model. LLM inference nondeterminism is a separate, explicitly out-of-scope problem.

What ContextBenchmark measures

Four test families, each answering one question a production team actually asks.

Future families under specification: agent safety, governance, memory integrity, determinism under incremental update.

Current results

micro-app reference corpus, re-verified 2026-07-17 · fingerprints published in the repository · cross-platform verification runs publicly in CI on every push · same results as JSON.

The reference baseline (bm25) reaching CTL 4 shows the bar is achievable with plain engineering. A system scoring below the free baseline has made a design choice, not hit a law of nature.

The drift numbers separate the three architectures clearly: a structural dependency graph (spiderbrain) ignored a file nothing depends on. A lexical index (bm25) leaked the new file into a small number of unrelated result sets, because global term statistics shift slightly whenever a document is added. An exhaustive-search embedding index (emb-minilm), the determinism-friendliest RAG configuration possible, leaked the new file into 9 of 10 unrelated result sets, because a single new document changes the geometry of a shared vector space. That last number is the charitable case: emb-minilm is the most determinism-friendly embedding setup available, and a production ANN index is expected to do worse, not better, because it trades determinism for speed.

Re-verification, 2026-07-17: all three adapters were re-run 12 days after first publication. bm25 and emb-minilm rebuilt byte-identical artifacts. Spiderbrain's artifact did not reproduce, and isolation testing traced it to the engine embedding the enclosing repository's git commit time and the project's absolute path in its artifact, a reproducibility defect in our own engine. Fixed the same day: the engine no longer serialises environment into the artifact (provenance moved to a sidecar; two further couplings found and removed during the fix), its CI now locks the invariance, and the adapter builds from a git-free staged copy so the artifact is a pure function of corpus bytes. The republished fingerprint recomputed identically from a second location (compare verdict PASS). All query results were identical throughout, so every number above is unchanged. Full detail in results.json.

Field note, 2026-07-19: the contract held under refactoring. Two days after the fix above, the same engine went through a deliberately heavy day of internal change, deployed to production in four stages: three separate scoring code paths were unified into one shared scorer, output leak guards were hardened, and the cloud ingest strategy changed from shallow single-commit clones to full-history clones. Across five production parses of the same public reference repository (benjaminp/six, commit c8e3940) spanning all four deploys, the published structural fingerprint stayed identical: 9fb6e364e9c1fe49. The scored layer grew from 7 to 16 per-node fields; the fingerprinted structure never moved. That is the separation the fingerprint contract promises: the artifact is a function of the corpus and the declared contract, not of engine internals. The engine's internal CI now runs 45 determinism and contract gates, including artifact-invariance (unrelated commits, paths, mtimes) and a scorer-unification gate that fails if any scoring path drifts from the shared one. It has stayed identical through every production deploy since, including a cloud-parse API that added authenticated, metered parsing of a repository into a hosted brain: a new feature and a new authenticated surface, and still the same structural fingerprint for the same commit. Continued 2026-07-20: four further production deploys, this time rewriting the engine's memory subsystem rather than its scoring: a read-after-write consistency guard, an idempotent ratification path, hardened error handling across four blob stores (a failed read no longer reads as an empty one), and the groundwork for moving the memory log off object storage into Postgres. None of that touches the parser, the scorer, or the artifact contract, and the prediction was that the fingerprint would not move. It did not: the same commit of the same repository parsed through the production API after all four deploys returned 9fb6e364e9c1fe49 again. The value of the contract is exactly that this was boring. Disclosure: this is an operator-reported observation from our own engine's production API, not an independently reproduced benchmark run; trust levels above are unchanged by it.

Field note, 2026-08-04 — and unlike the two above, this one is a benchmark run, not an operator report. The engine published a new major version and both of its production surfaces were redeployed, so the suite was re-executed against the committed corpus. Every family passes again inside the new version: rebuild-identity 1 distinct hash over 3 builds, exact-match rate 1, drift-under-noise 0 with 0 noise in the top 10. CTL 3 is therefore re-earned here, not carried over. Two values moved — the engine identity (ffdd6784a7c77ce7 → 3cc01f9964131211) and the artifact hash. Because engine identity is a controlled variable, our own compare refuses the cross-version question instead of answering it: NOT COMPARABLE — a controlled variable differs, so neither PASS nor FAIL would mean anything. So we are not claiming the artifact reproduced across the version boundary; that claim is not available, and asserting it would break the disclosure rule this benchmark is built on. Recorded as an observation rather than a verified reproduction: all 10 query result hashes are byte-identical to the pre-6.0.0 fingerprint. Retrieval behaviour did not move across a major version whose artifact format did — the CTL 2 property surviving underneath a CTL 3 claim, which is precisely why the two are separate levels.

Benchmark architecture

git clone https://github.com/aabhisrv/contextbenchmark && cd contextbenchmark
node contextbenchmark.mjs run --adapters bm25            # dependency-free baseline
node contextbenchmark.mjs run --adapters bm25,emb-minilm # + typical-RAG reference
node contextbenchmark.mjs compare A.fingerprint.json B.fingerprint.json

Research and methodology

ContextBenchmark builds on reproducible-systems research, retrieval evaluation, and software reproducibility practice, and introduces standardized measurements for deterministic AI context.

Related work, and projects we are not

Several unaffiliated projects have near-identical names, and at least one measures something genuinely adjacent. Here is exactly where each one sits, so you can pick the right tool rather than the closest name.

The distinction that matters most: STATE-Bench asks whether the agent succeeds repeatably, which folds model, tools, and memory into one number. ContextBenchmark asks whether the context is the same context twice, with no model in the loop at all. A system can pass one and fail the other, which is the reason to measure both.

Positioning, stated precisely: benchmarks exist for model capability, for agent task reliability, and for retrieval accuracy. This one is for the reproducibility of the context layer itself, below the model and independent of task success.

Who runs this, and why you can check it

ContextBenchmark is maintained by Abhishek Srivastava, who also builds Spiderbrain, one of the systems in the results table above. That is a conflict of interest, so it is disclosed here rather than left for you to discover.

A benchmark that grades its author's own product has to be checkable, not merely trusted. Three things make it so:

If you find a way this benchmark unfairly favours any system, Spiderbrain included, that is a bug, not a footnote. Open an issue and it gets fixed in public. Proof this works: the 2026-07-17 re-verification caught a reproducibility defect in Spiderbrain's own artifact serialization; it is disclosed in the results rather than papered over, and the engine fix shipped the same day.