Claims, checks, evidence — the truth layer (plans/claims.md) - #9
Merged
Merged
Conversation
The truth layer's foundation (plans/claims.md step 1). Claim / Check / Evidence are jarvis types resolved from the :Schema meta-graph, so nothing here adds a Strut node type; writes keep going through NodeWriter/EdgeWriter. - fixture: re-dump jarvis-ontology.ts from a post-125 jarvis (153 schemas, 346 edge schemas): Check, Evidence, Claim keyed on `claim-id` in the Epistemic domain, and the ABOUT / TESTS / PRODUCED_BY / SUPERSEDES pairs. Also un-mangles HiveInitiative's node_key (a vein->strut rename casualty) - claim-schema-upgrade.ts: one-shot standalone mirror of jarvis 124. The ontology seed is add-only, so an already-seeded DB would keep Claim at `claim-claim_text-speaker_name` and reject every claim. Runs before the seed and ONLY with STRUT_GRAPH_SEED_ONTOLOGY: it deletes old-shape :Claim nodes, which on a jarvis-hosted graph could be real podcast claims - strut-schemas: StrutRun -EXECUTED-> StrutStepVersion (single-step runs) - claims.ts: ids (lowercase alphanumeric; deterministic Evidence.id over check|run|path), the check subject/result contract, claimStatus() — computed on read per (claim, subject), never stored — and ClaimsReader (claimsFor / checksFor / evidenceFor / statusFor; muted edges invisible) - status rule refinement: a stream's latest is its newest evidence plus everything it observed in the same source run, so a foreach's last passing iteration cannot paper over an earlier failing one - gate: WorkspaceStore.graph (set by Neo4jWorkspaceStore) -> strut.claims, null on a filesystem workspace. Keyed on the workspace, not StrutOptions.graph: the lab host passes no `graph` and still needs claims - schema-crud.test: its live cases invented `Evidence` and extended Claim with `verdict` as examples; jarvis ships both now, so they use FieldNote / review_note instead
…ves a record Only runs recorded after this can ever be verified (plans/claims.md §3): evidence is ABOUT the exact version observed, and a workflow version does not pin its steps, so without a record the executed step version is unrecoverable once the step is republished. - run.start gains `stepHashes`, `cassette`, `origin` (RunEvent + RunOptions). stepHashes is re-recorded on `run.resumed`: a resume loads whatever is active THEN - closure.ts: walkSteps / flowClosure (loop+foreach bodies, onError, nested subflows through the workspace, agentTools grants; a templated or missing child makes it unresolvable) and stepHashesFor — the active hash of every workspace step in reach, or EVERY active hash when unresolvable (a superset is still true; a subset would lose a version). validate.ts and the agent step now share its walker and glob. Step 4's check closure reuses it - WorkspaceStore.getActiveStepHashes() on both stores (+ conformance); wired into all four launch sites (HTTP detached, strut.run, the authoring capability, the chat tool) - run stores: `step:<type>` and `check:<id>` keys map to steps/<type>/runs/ and checks/<id>/runs/ — the prefix is parsed, never written to disk, and path-traversal segments are refused. No workflow listing can see them. MemoryRunStore no longer lists `step:clip/trim` runs under `step:clip` - run-step.ts: runStep behind the chat tool, meta/run-step and POST /steps/:type/run. Still executes in memory; persisted under `step:<type>` only when the step has an active claim or `keep: true`. Results gain `runId` and `kept`. An unreachable claims graph never fails the run - projector: projectRun (one run -> its StrutRun ref) with EXECUTED -> StrutStepVersion for a step key, read from the recorded hash — never "whatever is active". projectRuns is behaviourally unchanged - get_step.recentRuns; meta run reads of a `step:` key are scoped by the step's publisher
…roof
Claims and checks become writable (plans/claims.md §2). Nothing produces
evidence yet — that is step 4 — so every status reads `unknown`.
- graph/claims-writer.ts: the graph's invariants, with no opinion about who
writes. A claim always has >=1 check (retiring the last is refused). Edits
SUPERSEDE: a successor claim carries its ABOUT attachments and its checks
(TESTS edges — an instrument is never cloned) and starts unknown; a
successor check takes over TESTS and the old one's evidence stops
counting. Retire = a timestamp, detach = a muted ABOUT edge (restored by
re-attaching); a claim's last subject cannot be detached
- claims-authoring.ts: one policy layer behind both doors. Validates check
specs before anything is written; applies defaults at write time (run,
and `always` for code vs `on_change` for anything presumed paid — an
agent/llm step anywhere in the check closure, or an unresolvable one — and
for external checks). Fixed point 1: a scoped (meta) author touches only
claims/checks it stamped and subjects it published. Fixed point 2: an
`ai`-stamped check may not reach a grader (gaia/*, harvey/*, eval/*,
meta/*, + STRUT_VERIFY_DENY) by name, through a subflow child, or via an
agentTools grant; an unresolvable closure is refused
- door one: `claims` on create_step / edit_step / create_workflow /
edit_workflow and on meta/create-step / meta/edit-step /
meta/publish-workflow. Validated BEFORE the publish, so a broken contract
blocks it; additive and idempotent by exact text, never retires. Results
carry claims: { count, added, existing } and a warning on zero
- door two: add_claim, list_claims (with computed status), edit_claim,
retire_claim, attach_claim, detach_claim, add_check, edit_check,
retire_check — and nine meta/* twins over the scoped capability. The chat
surface is human-supervised, so it is not scoped (like edit_step), but it
stamps `ai`, which keeps the deny-list on its checks
- ids are now time-sortable (base36 ms + per-process sequence + random):
still lowercase alphanumeric and never derived from the text, but a
contract lists in the order it was written
- prompt: a short claims section (rules 1-4), only when the tools are
offered. On a filesystem workspace there is no tool, no arg, no section,
and a contract handed to the meta surface is an error, never dropped
A post-run consumer in the projector's mould (plans/claims.md §4): it reads a
finished run's event log, rebuilds every subject the run observed, and for
each active claim runs each check whose policy fires — one Evidence per
(check, run, path), or a planned slot for an external check.
- subjectsOfRun: a step at a path (one per foreach iteration), a nested
subflow as an execution of the CHILD workflow, the workflow itself, and
{ input, error } for a failure so "fails loudly" is checkable. Replayed
steps and agent tool calls yield nothing
- versions are never guessed: stepHashes / workflowHash from run.start, and
the runner now records the child a subflow step resolves ON its step.start
(workflow, version, hash) — it resolves at execution, not launch. No record
-> no evidence (skipped: unknown-version), never "the active version"
- the check contract (mapCheckResult): { supports, content, locator? } as the
output, under `object` / `json`, or as JSON on stdout; a bare exec maps from
its exit code (run with allowFailure: 0 supports, 126/127/killed cannot-run,
else refutes), by shape so a workflow ending in an exec reads the same. A
check that cannot run writes NOTHING — a broken check is never a pass
- policy: always / on_change (subject version, the check's own resolved
version, no evidence, or older than freshness_days) / sample (deterministic
per check|run|path) / manual (verify_run only)
- budget: presumed-paid checks skip at STRUT_VERIFY_BUDGET_USD (per run) or
STRUT_VERIFY_BUDGET_USD_PER_DAY (per subject, computed from the store). A
check that REPORTS cost is persisted under check:<id> with run.start.verify
and counted — which is how a presumed-free check that costs money is caught
- guards: check runs carry origin "verify" and are never verified; the grader
deny-list is re-applied to the check closure at run time; one verifier per
deployment single-flights passes on a run id; Evidence.id is deterministic
and written in create mode, so a second pass writes nothing and a re-verify
runs only checks with no evidence for that (run, path)
- planned slots for external checks: EVIDENCED_BY with no strength, at most
one open per (check, subject) — a newer run mutes and replaces the question
- add_evidence: asserted, check-less, about the version the run executed;
fills the open slot in place when there is one. meta/add-evidence records
`observed` ONLY for a DAG step of a workflow the meta surface did not
publish; an agent tool call (StepContext.agentTool) is always `asserted`
- run_when: publish checks fire on every publish path over { source | yaml };
their evidence is ABOUT the new version and sourced to that version node
- triggers: services.onRunEnd(runId, info) for every top-level run (so a
candidate launched by meta/run-workflow is verified too), and runStep once
a single-step run reaches the real store. Always detached
- evidence ordering: observed_at is whole seconds, so same-second verdicts
order by the source run's ms id, then the write stamp — a regression run
right after a pass must win
- check configs are validated against the step's schema at write time
- surface: strut.verifier, POST /workflows/:name/runs/:runId/verify,
verify_run / add_evidence, meta/verify-run / meta/add-evidence
The forcing function (plans/claims.md §5): the builder reads what its work is claimed to do, and what the evidence says, in the RESULT of the call it just made — not in an instruction it can rationalize past. - ledger.ts: buildLedger — per subject, each claim's computed status, assertedOnly, unverified, latest evidence, and every check's lastVerify (pending | ran | skipped: policy|budget|cannot-launch|unknown-version|denied | planned). A foreach's several outcomes for one check read as the most informative one; workflow and step claims are listed separately (no roll-up) - run_workflow / run_step results carry `claims` with every RUN check `pending` (the contract of what the launch can execute: the workflow, its nested children, every step type in reach); verify_run returns the settled ledger directly - ai/verify-waker.ts: the chat that launched a run is woken with its verdict. A [verify-notification] when the pass settles — or, when it settles within 5s of the run (free checks take well under a second), the ledger rides on the run's own [run-notification] and no second wake-up is spent. Exactly one of the two ever carries it; queue-and-drain and the autoTurns cap apply unchanged. Runs nobody watches just get their evidence written. The synchronous results still never wait, as specified - verifier.costOf(subject): what keeping a subject's claims true has cost, from the run store alone; shown as verifyCostUsd on get_step / list_claims - prompt rules 5-6: not done while a claim is unknown or refuted; finish the turn after launching and act on the notification; cannot-launch means fix the CHECK; a `planned` check is a question — answer it only with something observed with a tool, else relay it to the user and stop looping - the chat flyout renders [verify-notification] as a notice
- claims-routes.ts: GET /claims?kind=&name= returns a subject's contract —
claims with computed status, the newest evidence behind each verdict (and
the run it came from), their checks, and open slots as to-dos. POST / PATCH
/ DELETE for claims and checks, attach / detach, and POST
/claims/:id/evidence — which is also how an open slot is ANSWERED. The
actor is a person: not publisher-scoped, stamped `person` (so an `ai`
author can never edit it, and the producer's grader deny-list does not
apply); what they say about a run is `asserted`, `by: person`. Mutations sit
behind requireApiKey. On a filesystem workspace GET answers
{ enabled: false } and mutations 409
- the listing (shared with list_claims) gains `latest` and `slots`; the
reader reports the source run's store key; statusFor returns the evidence
it computed from
- web: ClaimsPanel — status badge, asserted-only / unverified flags, latest
evidence with a link that opens the run, the checks under each claim,
add / reword / retire, and a check editor for step checks (type, YAML
config, fires, policy) and external checks. Open slots render FIRST as
to-dos: the question, the run and its artifacts, a note, Supports /
Refutes. Every write re-reads, because an edit replaces the node it touched
- StepEditFlyout: a collapsible Claims section on the step TYPE (custom steps
only) that opens by itself on a refutation or a to-do. A workflow-level
ClaimsFlyout behind a topbar button whose dot says what needs attention
(refuted > waiting on someone > unverified). Hidden where there is no
claims layer
The layer was gated in eight places, each re-deriving `workspace.graph` for itself (createStrut ×2, authoring ×2, claims-routes, tools ×2, prompts). Now createStrut decides once — on where the workspace is graph-backed, unless `claims: false` or STRUT_CLAIMS=0 — builds ONE ClaimsAuthoring, and threads `claims: ClaimsAuthoring | null` to the authoring capability, the chat tools, the system prompt and the /claims routes. Nothing downstream reads `workspace.graph` any more, so the switch cannot be missed by a future consumer. Default on a graph workspace is unchanged (on). Graph test covers: option off, env off, option winning over env, and the chat surface offering nothing when handed no claims layer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the whole of
plans/claims.md— the truth layer: a Claim states how a step or workflow should behave, a Check is an instrument that tests it, Evidence is what one check observed on one run, and a claim's status (supported | refuted | stale | unknown) is computed from evidence on the active version, never asserted. (Branch name says "step-1"; the PR is all eight steps plus the switch.)Walkthrough, by commit
claims.ts, status rule, graph gate (2446990) — Claim / Check / Evidence are jarvis types resolved from the:Schemameta-graph (fixture re-dumped from post-125 jarvis); a one-shotclaim-schema-upgrademirrors jarvis 124 for already-seeded DBs (only withSTRUT_GRAPH_SEED_ONTOLOGY).claimStatus()computed on read per (claim, subject);ClaimsReader. Gate is the workspace (WorkspaceStore.graph), notStrutOptions.graph.f1d346d) —run.startgainsstepHashes/cassette/origin;closure.tswalks a flow's reach (loops, onError, subflows, agentTools).step:<type>/check:<id>run-store buckets.runSteppersists a single-step run when the step has claims orkeep: true. Only runs recorded from here on can ever be verified.420d438) —claims-writer.tsholds the graph invariants (≥1 check per claim, edits SUPERSEDE, retire = timestamp, detach = muted edge).claims-authoring.tsis the one policy layer behind both doors: theclaimsarg on the publish tools, andadd_claim/list_claims/edit_*/retire_*/attach_*/add_check+ ninemeta/*twins. Scoped authors touch only what they stamped;aichecks can't reach a grader.6a20648) — post-run consumer: rebuilds every subject a run observed and runs each check whose policy fires (always/on_change/sample/manual), one Evidence per (check, run, path), or a planned slot for an external check. Versions never guessed. Budgets (STRUT_VERIFY_BUDGET_USD[_PER_DAY]). Check runs carryorigin: verifyand are never themselves verified.add_evidencefor tool-backed observations.strut.verifier,POST …/runs/:id/verify,verify_run.[verify-notification](7182c5d) —run_workflow/run_stepresults carry the contract with every checkpending; the chat that launched a run is woken with the verdict (rides on the[run-notification]when the pass settles within 5s, else its own wake-up).verifier.costOf. Prompt rules 5–6.1e102f0) —claims-routes.ts(GET/POST/PATCH/DELETE /claims…, actor = person); webClaimsPanel, a collapsible section inStepEditFlyout, a workflow-levelClaimsFlyoutbehind a topbar button whose dot says what needs attention. Open slots render first, as to-dos.47eb690,f9ec09f).ee8d975) — the layer had been gated in eight places, each re-derivingworkspace.graph. NowcreateStrutdecides once and threadsclaims: ClaimsAuthoring | nullto every consumer. On a graph workspace it is on by default;STRUT_CLAIMS=0orcreateStrut({ claims: false })turns the whole thing off (no tools, noclaimsarg, no prompt section, no verify pass, panel hidden). Filesystem workspaces never have it.Not in scope
The Hive eval chain / LLM-judged graph nodes (jarvis side) — see
plans/claims.md.Testing
npm test— 813/813npm run test:graphagainst a throwaway Neo4j (STRUT_TEST_NEO4J_URI=bolt://localhost:7688) — claims-authoring, verify (incl. the switch), query, schema tests all green