Add source version audit to track adapter fixture coverage - #18
Open
adewale wants to merge 4 commits into
Open
Conversation
`projection-table.json` mapped key-path -> classification globally, so
semantically different fields from different producers collided on one
entry. Fixtures now carry a declared source and every classification is a
statement about exactly one adapter's on-disk shape.
Two live collisions, both found by the red test before the migration:
- `version` was classified once, as "session header metadata (Pi)". Pi's
`version` is an on-disk schema revision (integer 3); Claude Code's is a
producer version (string 2.1.92). Claude Code's producer version was
being governed by Pi's entry.
- `message.content[].thinking` was classified `projected:messages.text
... via projectPiContentBlocks`. That pass runs for Pi alone, and
extractContent has no `thinking` case, so under Claude Code the block
body reaches neither messages.text nor FTS. The table asserted a
Claude Code reasoning field was searchable when it is raw-only.
`internal/adapters/fixtureaudit` is the new home for fixture attribution
and key-path walking. Attribution is declared, not inferred: Inventory
fails on an unattributed fixture, an unknown corpus directory, and a
declared fixture that has gone missing, so the corpus cannot grow or
shrink without the change being named. fidelity_test's duplicate dirname
prefix-matching now reads the same declaration.
Attributing fixtures also brought `opencode_realish.jsonl` and
`codex_modern_realish.jsonl` into coverage for the first time, which is 28
newly classified pairs — the entire OpenCode projection surface plus two
Codex modern fields.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FmC8EeZ3tkWBf8khvaZYZv
Where a source records its version, and what that version measures, was knowledge held nowhere. The audit needs both, and needs them for adapter number five as much as for today's four, so the declaration goes on the interface: an adapter that does not state where its version lives — or that it has none — cannot be registered in Builtins(). model.VersionSignal closes the set by construction. Its fields are unexported and its only constructors are NoVersion, ProducerVersion and SchemaVersion, so no signal can carry a kind outside the closed set and VersionKindNone cannot be paired with a value contradicting it. The zero value is the one invalid state Go leaves representable; its kind is deliberately not a member of VersionKinds(), so a forgotten implementation fails the exhaustive check over the registry rather than reading as "this source has no version". The kind distinction is the point. Pi's `version` is an on-disk schema revision and is declared VersionKindSchema; reporting it as a producer version would stratify coverage on the wrong axis and buy false confidence. A test asserts Pi's signal is specifically not VersionKindProducer, so a later change that unifies them fails loudly. Enforcement is checked both ways: testdata/compilefail holds an adapter implementing every method but this one, and a second test builds the same source with the method supplied, so the failure is attributable to the missing method rather than to a broken fixture. Signals read the record's raw line rather than parsed fields, because Pi's schema version lives on the session header that ParseSession strips. Numbers are decoded as json.Number so the revision stays "3". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FmC8EeZ3tkWBf8khvaZYZv
`TestKeyPathCoverage` answers "is every field we have seen classified?".
Nothing answered "have we seen enough?", and the measurement is
uncomfortable: the entire Claude Code projection surface is calibrated
against one observation of one release, 2.1.92, which sits below the whole
band a published external audit covers.
The audit records, per source: the declared version kind, every distinct
version observed, the band those versions span, file and record counts,
and the version band in which each key path was first and last seen. It is
generated, never hand-written, and committed in two renderings —
testdata/source-version-audit.json and docs/source-version-audit.md.
Three gates keep it honest, and each closes a different escape route:
- a corpus version outside the recorded band fails, naming the source,
the version and the band, so vendoring a newer corpus re-derives the
calibration instead of arriving as a diff of unexplained new fields;
- a stale artefact fails a regenerate-and-diff check, with a companion
test proving the check fires on a hand-widened band rather than only
on a missing file;
- a static guard in testquality fails if either artefact is deleted or
emptied, which is otherwise the one edit that turns a failing gate
into a passing one.
The privacy tests were written first. The audit is derived from files
holding prompts, source code, commands and private paths, and one
rendering lands in docs/, so a leak there would be worse than no audit.
They work in both directions: a blacklist derived from the fixtures
themselves rather than a hand-kept list, and a whitelist asserting every
string in the generated JSON is a source name, a version kind, an observed
version, or a key path. Fixture labels are deliberately absent — corpus
filenames carry session identifiers.
BuildAuditFrom takes the fixture order as a parameter so the claim that
the audit is a set rather than a sequence is tested under permutation
rather than assumed, and CompareVersions orders releases rather than
strings, without which 2.1.206 would read as in-band below 2.1.92.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FmC8EeZ3tkWBf8khvaZYZv
The per-adapter cases stated each kind and the field its version is read from, but nothing forced a new adapter to appear there. The closed-set check catches an out-of-set kind; this catches an adapter whose kind and version field are never stated where a reviewer looks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FmC8EeZ3tkWBf8khvaZYZv
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a source-version audit system that tracks how much of each adapter's release history the committed fixture corpus covers. This provides visibility into version stratification across producers (Claude Code, Codex, Pi) and surfaces when new fixtures add coverage for previously unseen versions.
Key Changes
Version signal infrastructure (
model.VersionSignal,adapters.VersionSignal): Every adapter now declares where its version lives and what it measures (producer,schema, ornone). This is a closed set enforced at compile time via a compile-fail test.Fixture audit package (
internal/adapters/fixtureaudit): New package that inventories committed fixtures, reads their records, and derives two generated artefacts:testdata/source-version-audit.json: Aggregate structure and counts (files, records, key paths) stratified by version band per sourcedocs/source-version-audit.md: Human-readable markdown rendering of the same dataProjection table refactoring: Updated
testdata/projection-table.jsonto key classifications by(source, path)pairs instead of path alone, since two producers can spell a field identically but mean different things (e.g., Pi'sversionis schema revision; Claude Code's is producer version).Regenerate-and-diff gates: Two test suites enforce that the audit is always generated, never hand-edited:
TestGeneratedVersionAuditIsUpToDate: Fails if either JSON or markdown rendering is staleTestHandEditingTheGeneratedAuditFails: Verifies the gate catches deliberate editsTestAuditLeaksNoFixtureContent: Ensures no transcript content leaks into the committed artefact (only version values are exempt)Generator command (
cmd/aha-gen-version-audit): Standalone tool to regenerate both renderings and write them to explicit outputs.Property-based invariant:
TestAuditIsInvariantToFixtureOrderuses rapid to verify the audit is a set, not a sequence, so filesystem walk order never affects the artefact.Compile-time exhaustiveness:
TestVersionSignalCasesCoverEveryBuiltinandTestAdapterWithoutVersionSignalDoesNotCompileensure every registered adapter declares its version signal and that a fifth adapter cannot be added without doing so.Implementation Details
Version comparison is semantic (component-wise, numerically where both components are numeric) rather than lexicographic, so
2.1.92correctly precedes2.1.206.The audit reports aggregate structure only: file counts, record counts, and key-path bands. No fixture content is included, preventing accidental leaks of prompts, code, or private paths into the committed artefact.
Fixture attribution is declared in
fixtureaudit.handFixtureSourcesandfixtureaudit.corpusSourcesmaps, not inferred from filenames. Inventory fails on any missing or undeclared fixture, so the corpus cannot grow or shrink without the change being named.The projection table loader now validates that every
(source, path)pair in fixtures has a classification entry, catching silent data-shape changes that would otherwise pass through unnoticed viaentries.raw_json.https://claude.ai/code/session_01FmC8EeZ3tkWBf8khvaZYZv