[ddickmann]

Summary

This PR records the owner's direction of 2026-11-08, charts the route as a wayfinder map (linked decision tickets), removes the query-time retrieval tooling, and reports the resolution feasibility spike, which the owner decided as go on 2026-11-11. After that decision it adds a scorer fitted on reviewer labels that links two real company registers, and a prototype that induces an ontology over a warehouse estate for a host's read-only screen: concepts and relations, entities, a second source linked to a master list under a certified line, hierarchies the data proves, and a class tree proposed for a person to decide. It also fixes two pre-existing CI test failures, a secret-scan gate whose verdict depended on the network, an erasure gap in latence purge, a silent empty run on an unfilled stack template, an over-broad manifest redaction and config keys that were ignored in silence, rewrites the front-door pages, and adds a clickable prototype of the expert review workbench as scratch code.

The branch history was rewritten once on 2026-11-10 to correct the author field of its commits; content is unchanged.

Decisions (eight ADRs; the Decision Log goes from 65 to 73 records, numbered 0001–0074)

• ADR-0067, the re-scope. Latence becomes a traceable semantic-resolution and ontology engine, standalone and as a sidecar to Metis. Warehouse metadata (Unity Catalog and metric views first) is a new input. The text path and Graph Completion stay.
• ADR-0068, retrieval goes. Query-time retrieval tooling, hyperedges and Context Enrichment are removed. Embedders stay as a resolution signal only.
• ADR-0069, the domain model. The semantic layer is a replay of an append-only ledger of Assertions and Decisions. Entities may be accepted by the engine inside a calibrated band; Concepts are proposals a human approves. Bindings are keyed on Unity Catalog ids.
• ADR-0070, the boundary with Metis. One Metis-specific interface, a reviewable proposal file; everything else reaches Metis through Unity Catalog.
• ADR-0071, the warehouse pipeline. Warehouse readers are new Source kinds in the one Pipeline and meet the text path in one resolution phase. Row values are matched inside the customer's workspace. Filters and masks are labelled, and masked data is excluded.
• ADR-0072, runtime on Databricks. The engine runs as jobs inside the workspace and keeps its state in Delta tables in Unity Catalog. The review surface is a Databricks App with a local mode. Explanations are built in by default; a hosted model is opt-in.
• ADR-0073, the schema level. Schema proposals come from voting signals, with declared warehouse constraints strongest and a model as one voice. Every proposed constraint is validated on the data before a person sees it. Exports use OWL 2 QL, SHACL, SKOS and PROV-O, with opaque identifiers under a per-customer prefix.
• ADR-0074, the host screen. A host shows the engine's export in a read-only screen of its own, off the answer path. A reviewer's decisions leave the screen as a file the engine reads on its next run.

The retrieval removal (ADR-0068 executed)

• Packages latence-retrieval, latence-sink-qdrant, latence-splade and latence-muvera leave the workspace (34 → 30).
• latence-core loses hyperedges, Context Enrichment, graph features, the BM25 / sparse / multi-vector / FDE export signals, redaction survival, the SPLADE index driver, and the protocols, descriptors, conformance cases, CLI group and setup-recipe parts that existed only for them.
• SCHEMA_VERSION is 22 and QUALITY_SCHEMA_VERSION is 21.
• Three retrieval stacks, benchmark/s5, benchmark/serving, the campaign's s5-retrieval stage and the retrieval docs are removed. About 78,000 lines go.
• A new chapter, docs/anatomy/04-knowledge-graph-export.md, documents the graph export's files, columns and redaction posture.
• Leftovers, code. The benchmark campaign's check for an earlier export is removed; the labelling tool's hyperedge judgment mode is removed; a help example and a stage diagram name things that exist.
• Leftovers, benchmark datasets (owner's decision). musique and hotpotqa only fed retrieval benchmarks and leave the campaign's dataset registry, together with the code only they reached. The MuSiQue licence row stays because MuSiQue data is still tracked; the HotpotQA row goes because none is.
• Leftovers, scripts that could no longer run. Fifteen scripts and four test files across benchmark/official/, benchmark/official-data/, benchmark/musique-subset/, benchmark/rfc/ and benchmark/enterprise-gold/ are deleted: they imported a removed package or read a hyperedge export nothing writes. Their recorded results, question sets and data stay as dated records, each folder with a note. A test fails if a script that could run imports a removed package; one report script under results/ is kept as a record and exempted by name.
• Leftovers, config keys. Nine Stage config keys the removal retired were accepted and ignored. A Pipeline that carries one is now refused with a ConfigError naming the key, the Stage and ADR-0068, before anything is written; latence stack check lists them. Unknown keys in general are still allowed: a Provider may read keys the core does not know.
• Leftovers, documentation. ROADMAP.md and anatomy chapters 2 and 3 are rewritten against the current tree, and every file:line reference in the two chapters was re-read (38 corrected). A test fails a listed page whose paragraph names a removed capability without saying it is gone.
• Doc statements checked against code. Six pages had statements nobody had checked; each is now verified or reworded. Removed for lack of any record: a GPU sizing figure, a "2.7× faster" served-OCR claim (now gone from seven places, with a test), a vendor throughput figure, and a literature figure.

Front-door pages (owner: the name stays Latence; lead with what exists, then the direction)

• README.md and docs/index.md now open with what the code does today, each claim checked against the tree, and end with a marked section on the decided direction, none of it described as built.
• CONTEXT.md gains a section "Decided, not yet in code" for the vocabulary ADR-0069 fixed (ledger, assertion, locator, decision, actor, concept, entity, binding, accept band). The review and reject bands are in no accepted ADR, and the glossary says so.
• A test requires the front door to state the direction as not built, and fails the day a record type of the re-scope appears in the code without the pages being updated.
• Still old: the opening framing of AGENTS.md, and two retrieval pages outside the site navigation.

Release gates

• RELEASE-GATES.md required SPLADE proven against a live vector store and a graph path proven to retrieve. Those two gates are removed, and a new Gate 6 holds the resolution bars: auto-accept precision of at least 0.95, recall of at least 0.60 at that precision on the synthetic estate, and a text path that beats the recorded baseline.
• RELEASE-GATES.md and ROADMAP.md are now under the doc-truth link check.

Fixes outside the re-scope

• Secret scan (6d2b52c). detect-secrets verified Stripe-shaped strings against the vendor's API and dropped a fixture the vendor reported invalid, so the gate's verdict depended on the network. The gate now scans with verification off, and the one fixture is audited into the baseline.
• Purge (392e51e). A purge left the document's file name, id and content hash in every Corpus Version manifest while the CLI said nothing is retained. A purge now drops every live listing of the document from every Version. The content-addressed id stays as a suppression tombstone; the help text and tutorials say so.
• Unfilled stack template. latence run on a stack whose placeholders were never filled exited 0 after creating a directory literally named {STORAGE_URI}. It is now refused with a ConfigError before anything is written. Known limits: a missing source folder still gives an empty run with exit 0, and a legitimate config string of the form {UPPER_SNAKE} would be refused.
• Manifest redaction. manifest.json redacted every key containing "token", which also hid the chunk budget. A closed list of five numeric token-count keys is now shown; every other key containing "token" stays redacted. The owner confirmed the list.
• Tuner. The endpoint entity extractor was mapped under a name no Provider has and was never swept; it is mapped by its registered name. The --sample help and the report labels now say what the figure is. Found and not fixed: the endpoint knob family sweeps a key none of its three Providers reads.
• Benchmark campaign. run --fresh now redoes a stage whose artifacts are intact. The budget prompt says what the cap does: it is an estimate turned into a call budget, and calls past it fall back to configured labels. Whether it should be a hard stop is left to the owner.
• Deploy comments. The compose file, the Helm values and the install notes no longer say a run writes a knowledge graph or that the CronJob is incremental. Wording only.
• CI (abe4a8b). Two legs of the test matrix stalled on 2026-11-10 on unrelated commits; the cause is not found. The test job gets timeout-minutes: 90 and pytest gets faulthandler_timeout = 900, which writes every thread's traceback to the log when one test runs longer than 15 minutes. A readiness test requires both.
• A timing test that failed once on macOS. test_chunking_pathological_input_stays_near_linear compared one wall-clock sample per size and read 3.3× against a limit of 3.0×. It now takes the fastest of seven interleaved runs per size. It still fails a walk that rescans from the start (3.7×). Test-only.

The map: .scratch/semantic-sidecar/

• Resolved: 01–07, 10, 11, 12, 14, 17, 18, 19, 20 (go, owner, 2026-11-11), 21, 22, 23, 25, and of the ontology route 26, 27, 28, 30, 31, 32.
• Open, waiting for the owner: 29 (the hand-checked list is a draft; no score counts until the owner has corrected and signed it), 33 (the Unity Catalog reader is built and proven against a stand-in; one run against a real catalog is outstanding), 34 (the gate "is the ontology at v1": seven of nine checks met, the two open ones are 29 and 33).
• Open, started 2026-11-11 on the owner's word: 35–39, the demonstration on a host's local backend. Their code is in the host's repository, not here.
• Open, older: 08 (needs the GPU machine), 09 (needs a Databricks sandbox), 13 (it holds the spike's findings and the owner's decision that the scorer is fitted per domain), 15, 16, 24 (one item left: the compose and Helm commands have never been executed).
• Done check 6, "no retrieval code remains", is met.

The resolution spike (ticket 20, scratch code, not shipped): report in .scratch/semantic-sidecar/spike/REPORT.md

• Decision. Go (owner, 2026-11-11). Budget for a million records: 4 hours on one 24 GB GPU.

• Method. Splink on DuckDB scores candidate pairs; Learn-then-Test certifies an accept and a reject line; the pairs left undecided are scored again by neural models and tested in a second stage; a constrained clusterer builds the entities. Reviewer labels are simulated from the truth.

• A million records on a 15 GB machine. Pairs and clusters are held as integer arrays; the matcher's prediction table stays in DuckDB. 989,658 records run end to end in 72 s at 5.4 GB peak memory on 4 CPUs. The method is unchanged: on the 12k estate, the 100k estate and the public 50k set the rewrite gives the same pairs, bands and clusterings as the code before it, and the 12k comparison is a test against a recorded run.

• Model arms, measured on 2026-11-11 on one 24 GB GPU (extractor, reranker, two decision models and an embedder, all zero-shot; revisions in the report).

  • Synthetic estate, 99k records: auto-accept precision 0.986 to 0.990 in every arm. The machine alone finds 75.7% of the true pairs with the matcher only and 89.4% with the smallest GLiDE decision model (0.4B) re-scoring the undecided pairs; the review band falls from 23,607 pairs to 10,390, and to 8,962 with the reranker as well. Zero-shot GLiNER2.5-Decide changes nothing.
  • Synthetic estate, 990k records: with 5,000 uniformly drawn labels no accept line is certified; with 2,500 more labels drawn from the undecided pairs, auto-accept precision is 0.9955 and the machine alone finds 71.0% of the true pairs at pairwise precision 0.997. The labels did that, not the model: a stage with labels and no model gives the same result in 83 s.
  • Public person dataset, 50k records: auto-accept precision 0.971 with the matcher alone; neither model moves a pair out of the 21.7% review band.
  • Text, 3,000 synthetic documents: auto-link precision 0.959 to 0.965 in every model arm; 55.9% of mentions auto-linked with the reranker, 66.6% with GLiDE 0.4B as well. The extractor finds 99.5% of the gold mentions at precision 0.845.

• A pass below the reject line (negative result). The best-scored rejected pairs are scored by the model as a stage of their own. It reaches 93 to 95% of the true matches lost below the line and accepts none of them at the bar, at 10⁵ and at 10⁶ records (68 minutes for the whole run, inside the budget).

• Two real company registers. 120,192 UK legal-entity records against the UK company register, 102,732 true links, the shared company number withheld; one estate has 302,924 records, one 1,222,924. The data is downloaded where the run happens and is never committed.

  • First result: the matcher alone certified nothing; with the model the machine found 36.5% of the true links and left 594,359 pairs in review.
  • Cause: the first stage learns without labels what a field is worth, and where thousands of companies share a registered office it learned that the address makes the match. 98,924 pairs between the registers carry the identical name, 98,866 of them are true links, and about 23,000 true links scored near zero.
  • Fix (run_a.py --learned): boosted trees fitted on the reviewer's labels over the evidence of every pair, calibrated across folds; training labels spread over patterns of evidence; the accept and the reject line each certified on fresh labels from their own side of even odds. Same 5,000 labels, no GPU.

  Estate	Machine alone before (best arm), pairwise P / R	Machine alone now	Left for review now	Time
  Real registers, 302,924 records	0.991 / 0.365	0.998 / 0.972	6 pairs	61 s
  Real registers, 1,222,924 records	nothing certified	0.997 / 0.971	none	158 s
  Synthetic 10⁵	0.991 / 0.900 (two models)	0.993 / 0.913	2,179 pairs	24 s
  Synthetic 10⁶	0.997 / 0.710 (56 min, GPU)	0.984 / 0.874	8,196 pairs	6 min
  Public person records	0.917 / 0.840	0.949 / 0.841 (one seed; 0.916 / 0.848 over three)	35,239 pairs	14 s

  • Limits. On the registers 96% of the true links carry the same name, so that estate is easy once the scorer knows what counts. The 10⁶ accept band is at 0.960, close to the bar. On the person records the clusters' pairwise precision is under the bar (0.916 over three seeds) although the accepted pairs are 98% right. The evidence is written by hand for organisations and persons. The neural models were not run on top of it.
  • What failed on the way, each kept as a test or a note in the code: a fixed grid of lines (287,000 pairs went to review), a floor that followed the scorer down (pairs right one time in three were accepted), an accept test that began on 100 labels (nothing certified at 10⁶), rank among a record's candidates (cost precision), trees fitted once without calibration.

• How it carries to new data.

  • Labels per estate (three seeds, five budgets): about 600 labels give the automatic merges on the registers (recall 0.972); on the noisier synthetic names recall rises from 0.815 at 600 to 0.913 at 2,500. A certified reject line needs about 300 labels on its own side. The number does not grow with the estate.
  • A scorer carried from one estate to another does not work: recall 0.26 and 0.52, against 0.87 and 0.97 with the same 1,000 labels spent on the target. The owner decided (2026-11-11) that the scorer is fitted per domain, as deployment work.
  • A small estate of another shape (211 dealers, 1,274 seeded accounts) certified nothing at first. Found and fixed, each with a test that failed first: the first stage stopped without postcodes; a source can be named as holding no duplicates (its own pairs are ruled out, no cluster holds two of its records, the scorer sees whether a record fits one master record clearly or two equally); linking to a master list compares no two records of one source; a candidate pass over a shared word in the same town (candidates held 69% of the true pairs before, 98% after).

• Labels from a reviewer (ticket 30). The matcher asks for labels in two rounds: first the pairs it learns from, then two test samples drawn from the fitted scorer's own scores and frozen. A reviewer's label wins over the seed's truth; a pair that was picked by its score can train and decide and is never a test label; with open labels and no truth nothing is linked. The answer file (resolution/3) says per sample how many labels came from a person.

• Acronyms (owner's priority, 2026-11-11). aliases.py gathers the evidence for a short form (the full name in the same thread, a definition in some document, an alias column, the letters alone) and a small ranker fitted on the calibration labels weighs it with popularity and context. On a synthetic set written for it, acronym mentions linked correctly go from 0.1% to 81.8% (97.4% with the full name in the thread, 42.3% with context only). On 685 real annual reports the definition miner finds 6,852 definitions; on hand-checked samples of 80 it was right in 92% before two fixes and 97% after. Linking those definitions to the right filer has not been measured.

• Research notes with sources and unverified points marked: real data sources and their terms (22), licences of English entity-linking benchmarks (23, one clean candidate, not run), methods for linking company registers (24).

• Defects the runs found, each fixed with a test that failed first: the embedder searched with the whole document and retrieved the other organisation a document names; the public person dataset was scored with the organisation question.

• Not done: no customer estate and no real text linked end to end; no entity-linking benchmark run; no Spark run; no larger GLiDE variant and no fine-tuned arm; no review-item explanations from the hosted model; the no-duplicates rule has not been measured on the registers or the synthetic estates.

• run_a.py --export-review writes the review band as items a reviewer can decide, with a suggestion per pair, and --export-graph writes the machine clusters as a graph. Both feed the workbench and cover the synthetic estate only.

The estate prototype (tickets 25 to 33, scratch code, not shipped): .scratch/semantic-sidecar/estate/

It reads a warehouse estate through one small reader and writes one graph-export/3 file with a JSON Schema, and the same ontology as one Turtle file (OWL 2 QL and SKOS). Measured on a local mirror of a synthetic vehicle and warranty estate (29 objects with the seeded one, 13 metric views). Nothing has run against a Databricks workspace: the Unity Catalog reader has met a stand-in server only.

• Concepts, relations, measures. Induced from objects, columns, comments and metric-view definitions, each with the signals it rests on and its support counted on the data, kept as an append-only ledger whose replay survives a rebuild. A relation a metric view or a host-verified join states is stated; a mined one is proposed. A mined relation that holds for fewer than 90% of its rows is dropped. 39 concepts, 87 relations, 38 measures.
• Entities. Values of value sets and rows of master data are nodes. A fact row is never a node: facts are counts on edges, each with the object and columns it was counted over. 2,514 nodes.
• A second source linked to the master list. estate seed writes service accounts of a partner portal into a schema of its own in a copy of the mirror: generated data, marked as such in the catalog comments, the export and the summary. The build hands dealers and accounts to the resolution engine and brings its answer into the export as same_as edges with the probability and what each rests on, review items, and a summary. 938 of 1,274 accounts are linked to a dealer under a certified line, 916 of them right (97.7%), which is 85% of the accounts that belong to a dealer; 304 pairs are left for a person. It cost 1,300 labels, taken from the seed's truth in place of a reviewer, and the export says so. Run once with 64 answers written as a reviewer would give them: 60 training and 2 test labels and 31 links were then a person's, and one machine link was taken away.
• Structure the data proves (the owner's objection: linking records is disambiguation, not an ontology). From counts alone: roll-ups between value sets with the values that break them, value sets that cut the rows the same way, part-of from keys, derived objects that describe an entity, and broader edges between values. 9 roll-up steps (market ← dealer group ← dealer; make ← model; make ← manufacturer; system ← component), 4 part-of and links, 301 broader edges, and one value set named as no level because it names single things. Names that may be the same are read against the hierarchy.
• A class tree, proposed. What kind of thing a concept is cannot be proved from counts. Two voices propose it and a person decides: attribute patterns from the catalog, and a language model that is shown names, types, comments and relations and never a row or a value. The top classes are six of our own (owner's decision: no outside vocabulary at first). The model's voice ran live once (ticket 31): one call to a small hosted model on the provider's SDK with a strict answer schema, 15,033 tokens in and 4,034 out, USD 0.0035 at list price (prices taken from third-party listings, the provider's page could not be opened). The answer is kept as a file; a rebuild with the same prompt makes no call and gives the same export; a call whose worst case is above the limit on the command is not made.
• Statements as codes (ticket 26). The export carries no sentence the engine wrote. A signal, a line of evidence and a reason are each a code with parameters (63 codes, listed in schemas/codes.json), and a host renders them in its own languages. Free text says who wrote it and in what language.
• Decisions on every kind of element (ticket 28). Every decidable element carries a digest of what it states. A decision is applied only while the element still states that; otherwise it is reported as stale. A rejected hierarchy step is removed before the steps are reduced, so a step it hid returns. In the Turtle file an approved statement is plain and a proposed one is a marked axiom. Nine decisions, one per kind, survive two rebuilds byte for byte.
• Six defects on the real mirror (ticket 27), each with a test that failed first: a wrong singular ("Investigation cas"); a reference to one column of a two-column key read as part-of; a roll-up proposed from a nearly unique title; broader edges drawn under a step with breaking values; two concepts with one label; a key column credited to the wrong one of two tables with the same name. Two of the causes were other than first assumed; the ticket says which.
• What people ask, as a signal (ticket 32). A log of recorded questions (usage/1: measures and dimensions by name, no value) adds "used in N answered questions" to elements the model already holds and lists what was asked for and is not in the model. It creates nothing and changes no status. On this estate the log is the host's 408 golden questions, which are synthetic and written from the same metric views, so the counts say what the host's tests cover.
• A reader for Unity Catalog (ticket 33). Every statement takes its quoting and its text casts from the reader, and a reader over the catalog and statement APIs (standard library only) gives the same model as the mirror against a stand-in server that pages, stalls, splits results, refuses a token and fails a statement. A column tagged sensitive appears in no statement. Lineage, row filters and column masks are not read.
• A score (ticket 29). estate score gives precision and recall per kind against a hand-checked list. The list in the tree is a draft written before the score was run; nobody has checked it, and the command marks its numbers as not measured.
• Personal data. A column marked personal is never selected. An object with such a column gives no rows unless the estate file names it as shown without those columns; an object about persons gives none in any case. The truth of the seed stays outside the catalog and the export. Tests cover each.
• The screen that shows this export is in the host's own repository (ADR-0074); nothing of the host is copied here.
• Not built: mentions from the estate's documents; classes from an outside vocabulary; SHACL and PROV-O exports; real packages in place of the prototype folder. HANDOFF.md in that folder is the owner's run as seven blocks to paste, and says what was verified where.

The workbench prototype (ticket 22, scratch code, not shipped): .scratch/semantic-sidecar/workbench/

• One React component, built to be mounted as a route in a Carbon host's own router, plus a standalone app around it. The host's breadcrumb, key handling, notices, controls, translation and data are typed props with defaults; styles read only --rw-* variables. INTEGRATION.md lists the edits a mount needs.
• Review: a queue grouped by cluster and ranked by records touched times uncertainty; the evidence for a pair; one-key decisions on an append-only log with actor and time; a read-only audit sample of auto-accepted pairs.
• Group decisions: merge calls a group's open pairs the same; a split, given as a letter per record, calls pairs inside a letter the same and pairs across letters different. A pair already decided is never replaced. Undo appends withdrawals.
• Suggestions: each review pair can carry a suggested verdict with the combination of field agreement it rests on and how already decided pairs with that combination turned out. One key takes it; every decision records what the suggestion said. On the 12k synthetic estate the suggestion matches the truth for 96.4% of review pairs out of sample, against 69.8% for "always the same"; real data will be harder. A suggestion decides nothing.
• Graph: the schema level as a layered map with counts, and a walkable neighbourhood view per node.
• The owner settled the interaction model on this prototype (ticket 22).
• Limits: synthetic data only. The schema shown is declared for the synthetic estate, not induced. The mount is tested against a stand-in router, chrome and button, not a real host.

CI test fixes (test-only): b2db2a5 and 3a78ee4 pin the core-count probes in five thread-cap tests.

Licence

• I can sign the Contributor Licence Agreement for this contribution. The owner is the contributor.
• AI assistance disclosed. Left for the owner to complete.

Definition of Done

• Baseline bar: not applicable to the decisions and the map.
• Gate bar for the removal: see Gates.
• Hardening bar: not applicable.

The test that was red

• The five thread-cap tests were reproduced red first, then shown passing.
• Documented lint commands must name paths that exist: red on benchmark/s5 and benchmark/serving.
• The secret-scan gate must pass --no-verify: red before the change.
• An open release gate must not name a capability the tree no longer has, and the gates must cover the new claims: both red on the old file.
• After a purge no file under the store names the purged document, and an abandoned transaction's staged manifest is scrubbed too: both red before the change.
• A Pipeline with an unfilled placeholder is refused before anything is written: 31 of 32 red before the change, plus one for the Airflow Runner.
• A Pipeline with a removed config key is refused before anything is written, and stack check lists every such key: red before the change.
• The manifest shows the chunk budget and hides an inlined token: two tests red before the change; two more guard the other direction.
• calibrate is not a command, the capability help example names real capabilities, the stage diagram names only stages that exist, the campaign stage needs no earlier export, and every dataset the campaign offers has a plan and a pipeline: each red before its change.
• Every Provider name in the tuner's knob mapping is one the workspace declares; the --sample help says what the flag does; --fresh redoes a done stage; the budget prompt says what the cap does: each red before its change.
• No script that could run imports a removed package; the served-OCR speed-up is stated nowhere without a record: both red before the change.
• ROADMAP.md and anatomy chapters 2 and 3 name no removed capability as current: red on each page before its rewrite. The front door states the direction as not built: 7 tests red on the old pages.
• The test job has a time limit and pytest a stuck-test watchdog: red before the change.
• The removal deletes tests only together with the behaviour they tested.
• Spike (scratch code): the embedder is asked with the mention, not the document; the pair scorers are asked about the kind of record at hand; the stages below the reject line; two registers with shared addresses are linked once the labels say what counts; a reject line is found among scores that are all close to zero; a few wrong pairs at the very top do not end the accept test; no pair is accepted that the scorer rates below even odds; an estate without a single postcode still runs; a source without duplicates is never merged with itself and ties are not guessed; linking to a master list compares no two records of one source; two spellings that share a common word and the town are candidates: each red before its change.
• Estate prototype (scratch code): an object with a personal column gives no rows unless the estate says so; an account of a dealer with a namesake can still be told apart; the induction defects the real mirror showed (a line-number column as a key, relations mined into aggregate tables, a transitive hierarchy, a composite key that was not minimal); a concept the model declines to place and the columns do place is marked as disputed; the six defects of ticket 27; the same model through a reader that speaks another SQL dialect; an approval that no longer fits what the element states is not applied: each red before its fix.
• Not red first: the roster entries that put ROADMAP.md, README.md, docs/index.md, CONTEXT.md, llms.txt and the agent briefing under existing checks (the pages already passed them); the workbench's tests, the acronym resolver's tests and most of the estate prototype's tests (seed, merge of the engine's answer, structure, classes, codes, usage, the score, the catalog reader against its stand-in, the label loop), which were written with the code; the property test that fitted scores are probabilities; the change to the chunking timing test, which failed once on one CI leg and passes locally before and after.

Gates

Run on the removal tree before its merge:

• ruff check packages benchmark/sota-campaign scripts: exit 0.
• scripts/typecheck.sh (strict mypy, 30 packages, src and tests): exit 0.
• PYTHONHASHSEED=0 uv run pytest packages: 4,332 passed, 17 skipped, exit 0.
• scripts/verify-local.sh, every lane after the secret scan: all passed.
• uv lock --check: exit 0.
• Same graph as before: the demo, the example pipelines and every kept stack give equal node and edge records before and after, apart from the schema version.

Run on the later changes:

• CI on ee3686c: the test matrix (three Python versions on Linux, one on macOS), lint, types, secret scan, stack validation, bake-offs, the Airflow lane and the build passed, in both runs. Only the MinIO lane is red (see below). Earlier heads that passed in full: d346eb4, 798a6f7, 63c41e3, 5e3ab6f, 5401ddb, 5e8a8f4, 34e5f8e.
• On c64b5c1 one of the two runs lost its Python 3.11 leg to the 90-minute limit; the other run passed on every leg, and the push changed only files under .scratch/. The cause of the stall is still not found (see CI above and the PR comment).
• CI on the current head, 40f96b3, has not reported yet. Its five commits change only files under .scratch/semantic-sidecar/.
• On the eleven commits of the ticket 24 fixes and the front-door rewrite, locally, before they were rebased onto 34e5f8e: ruff exit 0; strict mypy on 30 packages exit 0; PYTHONHASHSEED=0 uv run pytest packages: 4,503 passed, 17 skipped, exit 0; secret scan exit 0.
• scripts/verify-local.sh has not been re-run on the final tree. The Helm templates were not rendered and the compose and Helm commands were not executed (no helm or docker daemon on the machine).

Workbench, spike and estate prototype (scratch code, outside the repository's gates):

• Spike tests: 155 passed, including the equivalence test against the recorded run, the two clusterers on random inputs, the acronym resolver, the label-fitted scorer and the two-round label loop. The spike folder is not held to the repository's lint rules. The record linker's environment is pinned in spike/requirements-link.txt; a clean install of it gave the kept result byte for byte.
• Estate prototype: 114 tests passed on toy estates with planted structure and against the catalog stand-in; the export validates against its JSON Schema; the Turtle file is parsed back; the build on the mirror with the seed runs end to end, the resolution engine included, and two builds of the same inputs are byte-identical. The owner's run in HANDOFF.md was repeated from nothing and gave the same files.
• Workbench: strict TypeScript check; 40 unit and mount tests (Vitest); library, standalone and single-file builds.
• Workbench in a browser: 118 checks on the built pages at phone and desktop width, light and dark, with WCAG 2.2 AA rule scans with axe.

Contracts, packaging, licences

• Contract change: SCHEMA_VERSION 21 → 22 and QUALITY_SCHEMA_VERSION → 21. Nothing after the removal changes a contract's shape.
• A purge now rewrites the delta.entries of earlier Corpus Version manifests. The contract's shape is unchanged.
• Behaviour changes: executing a Pipeline that still carries a {UPPER_SNAKE} placeholder, or one of nine removed config keys, raises ConfigError where it used to run. latence-campaign run --fresh redoes a stage.
• Benchmark campaign: two datasets are no longer selectable. The corpus manifest keeps its publishable field, now False for every remaining lane.
• The rendered tuning report's table header changes from baseline r/s to baseline figure; the keys of the tuned YAML are unchanged.
• Heavy deps stay out of latence-core.
• New Provider: none.
• Licence table: the HotpotQA row is removed (no HotpotQA data is tracked); the MuSiQue row stays and says where the data is.
• The workbench and the estate prototype are not workspace packages and are not built or published by any workflow. Their dependencies are pinned in committed files; their licences have not been audited one by one.
• One model the spike measured, fastino/GLiDE-0.4B, is under a vendor community licence, not a permissive one. No model weights are in the repository; using it in a product needs the agreement the owner is arranging. The label-fitted scorer needs no model.
• Terms of the real register data: the legal-entity file is CC0 (read on its primary page). The company register's download page names no licence for the file itself; the site's content is under the Open Government Licence v3.0. Left for the owner to accept or to drop that arm. No register data is in the repository.

Security, secrets, PII

• No secrets and no PII. One synthetic fixture key is newly audited in .secrets.baseline.
• The secret-scan gate no longer sends key-shaped strings from the tree to vendor APIs.
• Erasure: see the purge fix above. Retained by design: the purged document's content-addressed id.
• Manifest redaction: five numeric token-count keys are shown; every other key containing "token" stays redacted, and a listed key with a non-numeric value is redacted.
• Redaction surface: the graph export's name posture is unchanged and stays pinned by test_kg_export_name_posture.py.
• The workbench's sample data is generated: invented organisation names from the spike's seeded generator.
• The spike's model revisions are listed in its report, not in a tracked JSON file: the secret scan reads revision hashes inside JSON strings as secrets, and the audited baseline was left alone.
• Real register records were read on the machine that ran the arms and never left it; only aggregate result files are tracked. No officer or owner data was read. Only synthetic and public data was sent to any model.
• The estate export holds synthetic and public data, no value of a personal column and no row of an object about persons.
• What a language model is shown about an estate is written to a file first (classes_prompt.json): names, types, comments, keys and relations, no row and no value. Sending it to a hosted model needs the estate file to allow it and a flag on the command. One such call was made, on the synthetic and public vehicle estate; the key came from the environment's secret store and is in no file.
• The catalog reader takes its host, warehouse and credentials from the environment only, and selects no column the estate file or a catalog tag marks as sensitive.

Docs and decisions

• Doc-truth gates green; ADR, package and stack counts corrected to the tree.
• ADRs 0067–0074 added, each with the rejected alternatives.
• README.md, docs/index.md, ROADMAP.md and the CONTEXT.md glossary describe the tree after the removal first and state the re-scope as decided direction, none of it built.
• AGENTS.md still opens with the earlier framing.

Not run locally

• Pipeline e2e on MinIO (s3://) is red on this PR and on main: the image pull for minio/minio:latest is refused before any test runs. The cause is in a PR comment; the fix is kept out of this PR.
• mkdocs build --strict stops on one warning: the Mermaid script URL check, which the sandbox cannot reach. The docs workflow's build job passes in CI.
• Compose and Helm commands.
• Anything against Databricks: no credentials are present where this work ran. The Unity Catalog reader was run against a stand-in server only; the commands for the run on a real catalog are in HANDOFF.md.
• The score of the class tree and the hierarchies against a list a person checked: the list is a draft.

[ddickmann]

CI (heavy lanes) → Pipeline e2e on MinIO (s3://) is red, but the cause is not this PR.

The job fails in its first step, before any test runs:

docker: Error response from daemon: pull access denied for minio/minio, repository does not exist or may require 'docker login'

main fails the same way. Every scheduled CI (heavy lanes) run on main since at least 2 Nov fails with this identical image-pull error. The latest is run 42362732404, on 7 Nov, at d96b30d. This PR changes only docs, ADRs and .scratch/ planning files, and it doesn't touch .github/workflows/ci-heavy.yml.

No fix exists on main yet. I'm keeping this PR to its scope. A follow-up would change the docker run … minio/minio:latest step in ci-heavy.yml (and minio/mc:latest) in one of two ways:

• pin images that are still published, at a fixed tag or digest, after checking the registry the project now publishes to;
• or switch the lane to a maintained S3-compatible emulator.

Either way the lane keeps exercising a real s3:// round-trip.

---

Generated by Claude Code

[ddickmann]

test · conformance · coverage (macos-latest · py3.12) failed on 9080e1c: 1 failed, 5573 passed.

FAILED packages/latence-extract-gliner2/tests/test_gliner2_fused_provider.py::test_intra_op_threads_config_override
  assert [3] == [4]

This PR didn't cause it: the test depends on how many cores the runner has. resolve_intra_op_threads (latence_core/providers/perf.py) clamps an explicit intra_op_threads to os.cpu_count(). The test asks for 4 threads and hard-codes [4]. GitHub's macos-latest runner has 3 cores, so the provider correctly returns 3.

The fix is in this PR as its own test-only commit, 06df36e, because it's the only thing standing between this PR and a green test job:

• The test now pins os.cpu_count() to 128, the same way test_perf_hardware_derived_defaults.py already does.
• The clamp itself is still covered by test_perf_seam.py::test_resolve_intra_op_threads_default_and_clamp.
• I reproduced the failure locally by forcing the core count to 3: it gave [3] == [4] before the fix and passes after.
• All 56 tests in the file pass, and ruff check and strict mypy on the package are clean.

Pipeline e2e on MinIO (s3://) is still red, for the separate reason in my earlier comment: Docker Hub no longer serves the MinIO image, and main fails the same way.

---

Generated by Claude Code

[ddickmann]

test · conformance · coverage (ubuntu-latest · py3.11) failed on 9080e1c: 4 failed, 5570 passed. Fixed in e9c9ba0. Like the macOS failure, these tests depend on the runner's core count, and this PR didn't cause them.

• test_perf_load_choreography::test_gliner_load_runs_the_whole_sequence_in_order
• test_perf_load_choreography::test_gliner2_caps_intra_op_threads_before_importing_the_backbone
• test_perf_torch_runtime::test_st_load_runs_the_whole_sequence_in_order
• test_gliner2_pii_thread_cap::test_redaction_gliner2_caps_intra_op_threads_at_load

Cause. Each of these tests pins the core count by overriding os.cpu_count only. But the default thread count is computed by effective_cpu_count(), which asks os.process_cpu_count (Python 3.13+) and os.sched_getaffinity (Linux) before it ever looks at cpu_count. On a Linux or 3.13+ host the override therefore does nothing. The default comes from the real 4-core runner, giving 4 threads instead of the 8 the tests expect.

Fix. The three pin helpers now override all three functions. That is the same approach test_perf_hardware_derived_defaults.py already takes. Only tests changed.

Checked locally on a 4-core Linux host with Python 3.13:

• the four tests failed exactly as in CI before the fix and pass after it;
• 333 thread and performance tests across latence-core, latence-extract-gliner2 and latence-pii-gliner2 pass;
• ruff check and strict mypy are clean.

main has the same failure: its last push (d96b30d) also has red test jobs. Pipeline e2e on MinIO (s3://) stays red for the separate reason given earlier: the MinIO image is no longer on Docker Hub.

---

Generated by Claude Code

[ddickmann]

test · conformance · coverage (ubuntu-latest · py3.11) failed on c64b5c1 in one of its two runs: it ran into the job's 90-minute limit. No test failed.

• What happened. The leg in run 43087039110 started at 19:26 UTC and was ended at 21:01 UTC, 95 minutes later. Its log cannot be downloaded (HTTP 404), so there is no traceback to read, including the one the 15-minute stuck-test watchdog would have written.
• Why this PR's commits are not the cause. The same commit's twin run (43087049716) passed the same leg in 49 minutes, and every other leg of both runs passed. The three commits in that push change only files under .scratch/semantic-sidecar/, which no test in packages/ reads.
• This is the stall seen on 2026-11-10. Two legs stalled then on unrelated commits; the time limit and the watchdog were added for it in abe4a8b. The limit did its job. The cause is still not known, and without a log this run adds nothing to it.
• Not re-run. The branch has moved on to ee3686c, where the same leg is running now; a second run on the old head would test a commit nobody merges.

What blocks a fix: a log from a stalled leg. If the leg stalls again and the log is again missing, the next step is to upload pytest's output as an artifact from an always() step, so that a job killed at the limit still leaves its last lines behind.

---

Generated by Claude Code

[ddickmann]

CI on 0d61ec7 (the map and ticket files only): "all green" fails because the macOS test leg (test · conformance · coverage (macos-latest · py3.12)) is cancelled after 15 minutes without a runner ever picking it up. That happened on the first attempt and again on a re-run of the failed jobs.

Every job that ran passed: lint and types, the secret scan, the three Linux test legs (py3.11, 3.12, 3.13) and the stack validation.

Not caused by this push. What blocks a green run is macOS runner availability for this repository (the job never starts), which needs a look at the account's Actions settings; no change to the workflow is proposed here, since dropping or skipping the leg would weaken the gate.

---

Generated by Claude Code