[FIX] Rectify: Evidence-Bound Injected Instruction Policy#4378
Merged
Conversation
…le registry (#4351) Part A only. CODEX_INTAKE_DISCIPLINE_DIGEST was a free-prose constant whose rule 2 ("Never read a file end-to-end") contradicted upstream Codex base instructions and the co-injected OUTPUT_DISCIPLINE_DIGEST. Replace it with IntakeRuleDef/CODEX_INTAKE_RULES in a new _type_intake_policy.py module, each rule carrying a machine-checkable basis/evidence/evidence_anchor and a mandatory exception whenever it states an absolute. Declare the co-injected policy subject matrix (CODEX_CO_INJECTED_POLICIES) so OUTPUT_DISCIPLINE_DIGEST, the intake digest, and the recipe delivery calling contract can be checked for disjoint subjects. Reconcile ADR-0005's Forward Obligations, which described a "do not spawn sub-agents" guard the shipped rule never enforced. Add scripts/measure_codex_read_repetition.py so the report's repeat-read rate is a re-runnable command, keyed by the v1/v2 digest version already on the wire. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… relationship (#4351) Adds structural guards for CODEX_INTAKE_RULES (unique kebab-case ids, mandatory exceptions on absolute imperatives, evidence resolution against live capabilities/ ADRs/issues, and no rule naming a harness-denied path class), contracts on the co-injected policy subject matrix (disjoint subjects, scope-marker honesty, byte budgets), an updated anchor list on the v2 digest text, delivery-matrix coverage for the resume surface and generated agent TOMLs, and the ADR-0005 ratchet for the sub-agent rule the shipped digest actually enforces. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Exercises the bounded-read classifier, per-session repeat-read counting, v1/v2 cohort split, and malformed/unclassified record handling in scripts/measure_codex_read_repetition.py so it is covered code rather than an unwired artifact. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…guards (#4351) Three suite-wide completeness guards independently track every core/types/*.py module and needed the new _type_intake_policy stem added: the singleton-locality allowlist (its digest is deliberately derived once at import time, like _type_constants), the test-filter core cascade map and its exact-membership ratchet, and the resolve-review Architectural Constraint Catalog so reviewer suggestions that would reintroduce an unevidenced or unqualified intake rule are rejected rather than accepted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Trecek
commented
Jul 27, 2026
Trecek
left a comment
Trecek
left a comment
Collaborator
Author
There was a problem hiding this comment.
AutoSkillit PR Review — Verdict: changes_requested
…e repeat-read measurement script (#4351) Applies four accepted findings from the PR #4378 review: - The rollout directory glob and date-filter pattern assumed a three-level YYYY/MM/DD path. The real on-disk layout is two-level YYYY/MM, so the glob matched zero files and date filtering was a silent no-op against real logs. - The rg path-extraction regex truncated at the first pipe, corrupting the extracted path whenever the searched pattern used POSIX ERE alternation -- the syntax AGENTS.md documents as correct for this repo. - An unguarded read_text in a loop over many files could abort a whole batch run on one unreadable file. The swallowed JSONDecodeError counting is deliberately unchanged: that finding was classified DISCUSS, since changing it would flip the existing assertion in test_unparseable_records_are_skipped_not_fatal and the shared-versus-distinct counter question needs a human decision. Co-Authored-By: Claude <noreply@anthropic.com>
Trecek
commented
Jul 27, 2026
Trecek
left a comment
Trecek
left a comment
Collaborator
Author
There was a problem hiding this comment.
AutoSkillit PR Review (round 2) — Verdict: changes_requested
…/--until (#4351) Round-two review regression from 97d8e8b. Correcting the rollout path to the real two-level YYYY/MM layout left the extracted date at month precision, but --since and --until accept day precision. The plain string comparison then made "2026-07" sort before "2026-07-18", so a day-precision --since silently dropped the entire target month and --until symmetrically over-included it. The rollout filename already carries the full timestamp, so the date is now read from there. _date_within_bound compares against bounds of either precision and honours the inclusive-boundary contract documented on both arguments. test_find_rollouts_date_filter_actually_filters only exercised month-precision values, which is why it missed this; it now covers day-precision boundaries on both bounds including the exact boundary date. Co-Authored-By: Claude <noreply@anthropic.com>
Trecek
commented
Jul 27, 2026
Trecek
left a comment
Trecek
left a comment
Collaborator
Author
There was a problem hiding this comment.
AutoSkillit review (round 3) — Verdict: approved (posted as COMMENT, not APPROVE — GitHub disallows self-approval on this self-authored PR). Verified the round-2 day-precision date-filter fix (commit 7d59e27) is correct and complete: manually traced _date_within_bound against all boundary cases and bisected the rewritten test against pre-fix logic to confirm it genuinely pins the regression, corroborated by two independent audit subagents. Round-1 fixes (97d8e8b) remain intact; the swallowed-JSONDecodeError item remains the deliberate DISCUSS deferral, correctly left unaddressed. No critical/warning findings — only two info-level test-coverage suggestions (non-blocking). task test-check: 30702 passed, 0 failures (an initial run hit an unrelated /dev/shm xdist-contention flake, traced and confirmed not caused by this PR; clean re-run passed). pre-commit run --all-files: clean. This PR implements Part A of #4351 only; Part B is tracked separately as #4375.
# Conflicts: # tests/contracts/AGENTS.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements PART A ONLY. Part B was split into issue #4375, which must be sequenced after #4369.
CODEX_INTAKE_DISCIPLINE_DIGESTis a 790-byte, six-rule free-prose string atsrc/autoskillit/core/types/_type_constants.py:170-197, injected into every Codex session throughfive delivery surfaces and into no Claude session. Two of its six rules have a cited basis; four do
not. Rule 2 — "Never read a file end-to-end" — contradicts both OpenAI Codex's shipped
base_instructions(verified verbatim from a first-party rollout) and the siblingOUTPUT_DISCIPLINE_DIGESTconcatenated into the same injected string two bytes away.Rewording rule 2 is a one-line diff, and the report's own evidence says it will not hold: between
one in four and one in two closed issues here is later contradicted by a recurrence, and the
dominant cause is aiming at the wrong layer. The wrong layer here is prose. The right layer is
the absence of any structure that could represent, and therefore check, a relationship between
two instruction texts.
Part A replaces the free-prose constant with an evidence-bound rule registry — each rule
carries a machine-checkable
basis,evidence,evidence_anchor,subject,exceptionandpath_classes— and renders the wire text from it. Six guards then make the bug class fail loudly:basisresolves to a live capability value, an existing ADR, or a cited issue/doc containing the rule's own anchor2, R4's2) frozen into a contract by a substring testrecipe_read_guarddeniestomllibcodex.py:847-849, which inspects only the agent body and never the appended suffixOUTPUT_DISCIPLINE_DIGEST↔ intake-digest contradiction, with no reference to upstream neededBehavioural verification needs zero new plumbing: the digest header carries its own version at
the wire (
Context Intake Discipline v2:), andsessions.jsonlalready storescodex_log, adirect path to each Codex session's rollout. Bumping the version to 2 is the cohort key. Part A
ships a reproducible measurement script so re-measuring the 54.6% repeat-read rate is a command,
not archaeology.
Scope Note
This PR is Part A only of a two-part rectification. Part B — the remainder of the
rectification — was split out into issue #4375, which must be sequenced after #4369.
Closes #4351
Implementation Plan
Plan file:
/home/talon/projects/autoskillit-runs/fix-4351-intake-digest-20260726-144540-308354/.autoskillit/temp/rectify/rectify_codex_intake_discipline_digest_2026-07-26_154500_part_a.md🤖 Generated with Claude Code via AutoSkillit