{
  "$comment": "Machine-readable twin of the results table on https://contextbenchmark.com/#results. Served with Access-Control-Allow-Origin: * so agents, answer engines and dashboards can read it directly. The authoritative artefacts are the fingerprints in the repository; this file is a summary of them.",
  "benchmarkVersion": "0.1",
  "generated": "2026-08-04",
  "$reproduction": "Re-run 2026-07-17 against the 2026-07-05 baseline. bm25 and emb-minilm reproduced byte-identical artifacts 12 days on, which is what their levels are supposed to mean. spiderbrain's artifact hash did not reproduce. Isolation tests (identical corpus bytes, identical embedded clock, identical absolute path) proved three independent inputs move that artifact: the git HEAD commit time of the repository enclosing the scanned project, the absolute path of the project, and the engine version. The first two are reproducibility defects in the engine's artifact serialization, found by this re-verification and logged for a fix; the third is ordinary version change, now recorded in the fingerprint's system field. All 10 query result hashes were identical under every condition, so retrieval behaviour and every reported metric are unchanged. An earlier version of this note attributed the change to the engine update alone; that was published before the cause was isolated, and was wrong. UPDATE 2026-07-17 (later the same day): both serialization defects are FIXED in the engine (environment values moved to a sidecar, git-time collection scoped, plus a filesystem-mtime fallback and a wall-clock decay read found and removed during the fix), the engine's CI now locks the invariance, the adapter stages the corpus git-free so the artifact is a pure function of corpus bytes, and the new fingerprint was verified identical when recomputed from a second location (compare verdict PASS). FIELD NOTE 2026-07-19: the contract held under refactoring. Two days after that fix, the engine went through four production deploys in one day carrying a heavy internal refactor: three scoring code paths unified into one shared scorer, output leak guards hardened, and cloud ingest changed from shallow single-commit clones to full-history clones. Across five production parses of the same public reference repository (benjaminp/six, commit c8e3940) spanning all four deploys, the published structural fingerprint stayed identical: 9fb6e364e9c1fe49. The scored layer grew from 7 to 16 per-node fields; the fingerprinted structure never moved, which is the separation the fingerprint contract promises. The engine's internal CI now runs 45 determinism and contract gates, including artifact-invariance and a scorer-unification gate. Disclosure: operator-reported observation from our own engine's production API, not an independently reproduced benchmark run; trust levels are unchanged by it. It has stayed identical through every production deploy since, including a cloud-parse API that added authenticated, metered parsing of a repository into a hosted brain; a new feature and a new authenticated surface, same structural fingerprint for the same commit. Field note continued 2026-07-20: four further production deploys rewrote the memory subsystem (read-after-write guard, idempotent ratify, hardened blob-read error handling, Postgres migration groundwork). Parser, scorer and artifact contract untouched; the same commit of benjaminp/six re-parsed through the production API after all four deploys returned the same structural fingerprint 9fb6e364e9c1fe49. Operator-reported, not an independently reproduced benchmark run. FIELD NOTE 2026-08-04, and this one is a benchmark run rather than an operator report: the engine published a new major version and both of its production surfaces were redeployed, so `contextbenchmark run --adapters spiderbrain` was re-executed against the committed corpus. Within the new version every family passes again — rebuild-identity 1 distinct hash over 3 builds, exact-match rate 1, drift-under-noise 0 with 0 noise in the top 10 — so CTL-3 is re-earned here rather than carried over. Two values moved: the engine identity (sha256:ffdd6784a7c77ce7 -> sha256:3cc01f9964131211) and the artifact hash (b24b5ee7... -> fbb9e1db...). Because engine identity is a controlled variable, this benchmark's own compare refuses the cross-version question rather than answering it: NOT COMPARABLE, \"a controlled variable differs, so neither PASS nor FAIL would mean anything\". So we are explicitly NOT claiming the artifact reproduced across the version boundary; that claim is not available and asserting it would break the rule in `disclosure`. What is on the record as an observation, not a verified reproduction: all 10 query result hashes are byte-identical to the pre-6.0.0 fingerprint. Retrieval behaviour did not move across a major version whose artifact format did — which is the CTL-2 property surviving underneath a CTL-3 claim, and the reason the two are separate levels.",
  "corpus": {
    "name": "micro-app",
    "description": "Committed, deterministic 12-file TypeScript fixture (layered webshop) with 10 fixed queries.",
    "queries": 10,
    "k": 10,
    "url": "https://github.com/aabhisrv/contextbenchmark/tree/main/corpora/micro-app"
  },
  "environment": {
    "os": "win32 10.0.19045",
    "arch": "x64",
    "node": "v24.14.0"
  },
  "trustLevels": {
    "CTL 0": "Non-repeatable. Below CTL 1.",
    "CTL 1": "Repeatable locally. Results not identical but rank-stable (Jaccard@k >= 0.9, Kendall tau >= 0.9).",
    "CTL 2": "Stable retrieval. Artifact bytes differ, but ranked query results are identical every time.",
    "CTL 3": "Machine-deterministic. Byte-identical artifacts across rebuilds and exact-match query results across trials.",
    "CTL 4": "Cross-machine deterministic. CTL 3, plus identical artifacts and query results across operating systems, verified by fingerprint exchange."
  },
  "results": [
    {
      "adapter": "spiderbrain",
      "description": "Structural code-context engine (dependency graph plus frozen-clock scoring).",
      "maintainedByBenchmarkAuthor": true,
      "contextTrustLevel": 3,
      "rebuildIdentity": {
        "pass": true,
        "distinctHashes": 1,
        "builds": 3
      },
      "queryStability": {
        "exactMatchRate": 1
      },
      "driftUnderNoise": {
        "driftScore": 0,
        "noiseInTopK": 0,
        "queries": 10,
        "verdict": "pass"
      },
      "crossMachine": {
        "status": "not-run",
        "note": "Pending CI-runnable packaging. Fingerprint published for independent comparison."
      },
      "artifactHash": "fbb9e1dbd332344ee380c0833dc443ca93c7c238c31d04f264fc38115302464e",
      "engineIdentity": "sha256:3cc01f9964131211",
      "artifactHashNote": "Defect FIXED 2026-07-17, the same day it was disclosed. The engine no longer serialises environment into the artifact of record (generatedAt and the absolute project path moved to a buildinfo.json sidecar; git-time collection scoped to the scanned subtree; a filesystem-mtime fallback and a wall-clock read in incident-decay scoring, found during the fix, were also removed), and its CI now locks artifact invariance across unrelated commits, clone paths and no-git copies. The adapter additionally stages the corpus git-free before building, so the artifact is a pure function of corpus bytes: a GitHub tarball and a git clone compute the same fingerprint. Verified by recomputation from a second location: artifact hash identical, compare verdict PASS (CTL-4-eligible). The previous hash 5c49f93d... was the defective format and is superseded.",
      "fingerprint": "https://github.com/aabhisrv/contextbenchmark/blob/main/results/spiderbrain.micro-app.win32-x64.fingerprint.json"
    },
    {
      "adapter": "bm25",
      "description": "Dependency-free lexical index. The determinism reference baseline.",
      "maintainedByBenchmarkAuthor": true,
      "contextTrustLevel": 4,
      "rebuildIdentity": {
        "pass": true,
        "distinctHashes": 1,
        "builds": 3
      },
      "queryStability": {
        "exactMatchRate": 1
      },
      "driftUnderNoise": {
        "driftScore": 0.04,
        "noiseInTopK": 2,
        "queries": 10,
        "verdict": "warn"
      },
      "crossMachine": {
        "status": "verified",
        "platforms": [
          "ubuntu",
          "windows",
          "macos"
        ],
        "note": "Verified in CI: artifact hashes identical and 10 of 10 query results identical across all three OS pairs."
      },
      "artifactHash": "2a3e7c0e6c2a8e0f379858d2053d4cb7c637a3bf02a30e0c6e6a92e985dc3eb7",
      "fingerprint": "https://github.com/aabhisrv/contextbenchmark/blob/main/results/bm25.micro-app.win32-x64.fingerprint.json"
    },
    {
      "adapter": "emb-minilm",
      "description": "MiniLM chunk embeddings with exhaustive search. The most determinism-friendly RAG configuration possible; ANN-indexed deployments are expected to do worse.",
      "maintainedByBenchmarkAuthor": true,
      "contextTrustLevel": 3,
      "rebuildIdentity": {
        "pass": true,
        "distinctHashes": 1,
        "builds": 2
      },
      "queryStability": {
        "exactMatchRate": 1
      },
      "driftUnderNoise": {
        "driftScore": 0.16,
        "noiseInTopK": 9,
        "queries": 10,
        "verdict": "warn"
      },
      "crossMachine": {
        "status": "not-run"
      },
      "artifactHash": "1aea22671b631f3fd56a7f3c615e517e7e5d03868f58300a66f8a610da09fe64",
      "fingerprint": "https://github.com/aabhisrv/contextbenchmark/blob/main/results/emb-minilm.micro-app.win32-x64.fingerprint.json"
    }
  ],
  "awaitingAdapters": [
    "mem0",
    "zep",
    "supermemory",
    "llamaindex",
    "vector-stores"
  ],
  "disclosure": {
    "rule": "No Context Trust Level may be claimed without a published fingerprint that anyone can independently recompute and compare. Reports missing fingerprints should not be trusted, including ours.",
    "adapterContract": "https://github.com/aabhisrv/contextbenchmark/blob/main/adapters/ADAPTER.md"
  },
  "license": "MIT",
  "repository": "https://github.com/aabhisrv/contextbenchmark",
  "website": "https://contextbenchmark.com/"
}