Output Budget Protocol Audit Remediation by Trecek · Pull Request #4277 · TalonT-Org/AutoSkillit

Trecek · 2026-07-17T05:20:33Z

Summary

Apply this remediation as a delta on the existing implementation branch at 093179c8f, whose diff already contains the four-layer Output Budget Protocol. The remediation makes every model-visible reduction artifact-backed, makes projected_utf8_bytes describe the exact canonical handler response, enforces an explicit registered-schema/wire-schema/handler-type/post-conversion policy, ratchets both response exemptions to independent measured ceilings, and emits only bounded, path-free telemetry.

The implementation is split into four ordered phases, each independently gated by clean pre-commit and task test-all runs with evidence recorded in .autoskillit/evidence/output-budget-remediation/manifest.json:

Phase 1: Lossless response formatting, exact projected_utf8_bytes canonical finalization, four-representation conformance middleware, closed ResponseBackstopExemptionDef registry with independent measured ceilings, privacy-safe bounded telemetry
Phase 2: Close JSONL wc -l command guard bypass with literal operand validation, preserve both incident commands from issue Codex investigate deep mode can consume 60K+ parent tokens from two tool outputs #4272 byte-for-byte, exercise large JSONL fixture through the guard
Phase 3: Installed Codex TOML config parse gate with Taskfile target and workflow ordering assertions
Phase 4: Live generated-subagent delivery probe with spawn/wait/rollout linkage, deep-investigate E2E proving completed agent waves + inter-batch synthesis + D6 validators + Claude 200K provenance, fresh GraphQL closure postcondition for issues open_kitchen response exceeds Claude Code MCP output cap — step graph truncated from orchestrator context, causing out-of-order step execution (review_approach before rectify) #4253/Codex Sous-Chef Recipe Context Loss via Auto-Compaction #3938 and PR Server-authoritative payload delivery for open_kitchen output cap #4259

Implementation Plan

Plan file: .autoskillit/temp/make-plan/output_budget_protocol_audit_remediation_plan_2026-07-15_194745.md

🤖 Generated with Claude Code via AutoSkillit

Add generated Codex child delivery probe (T13) with spawn/wait/rollout linkage assertions, strengthen deep-investigate E2E (T14) with workflow event normalization proving completed agent waves, inter-batch synthesis, post-report D6 validators, and Claude 200K provenance. Register generated agent TOMLs in session config. Add Taskfile smoke target. Raise investigate spawn ceiling to 16 with wave completion ordering contracts. Index fresh GraphQL closure postcondition for issues #4253/#3938 and PR #4259. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Phase 4 gate: 30941 passed, 611 skipped, 55 xfailed at 3e10de464. Evidence manifest validated with --mode incremental. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Remediation recipe grew to 183,103 chars on develop. Bump load_recipe ceiling to 185,000 and open_kitchen to 186,000. Update measurement IDs, registry digest, ADR doc, Codex token limit fixture, and all pinned test values. Fix shifted line numbers in schema version allowlist and symbol count in subpackage structure test. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The maxima assertion in test_canonical_recipe_responses_fit_independent_registry_ceilings hardcoded byte counts that included the absolute path to builtin_scripts_dir(), which varies by checkout location (100 chars locally vs 41 in CI = 59-byte gap). Normalize the rendered payload by replacing the resolved scripts path with its template placeholder before measuring, making the pinned maxima environment-independent. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Trecek

AutoSkillit PR Review — Verdict: approved_with_comments

Trecek · 2026-07-17T16:43:18Z

+                max_utf8_bytes=exemption.max_utf8_bytes,
+            )
+            return result
+        return bounded_response_budget_failure(


[warning] defense: Exemption ceiling breach drops the entire response with no artifact persisted: when an exempt tool (open_kitchen/load_recipe) exceeds its measured ceiling, this branch returns bounded_response_budget_failure WITHOUT writing a spill artifact — the only enforcement path that is not artifact-backed, contradicting the PR's stated 'every model-visible reduction artifact-backed' goal. The reused cause code internal_invariant_failed also obscures the actual condition (exemption ceiling exceeded). For load_recipe, ceilings were measured against bundled recipes (measurement_id bundled-recipes-all-modes-2026-07-16); a larger user-local recipe reaches this path at runtime and loses the recipe delivery entirely, in tension with ADR-0004's re-delivery obligation. Consider persisting the artifact before failing closed and/or introducing a distinct cause code.

Valid observation — flagged for design decision, with the cause-code half applied in 0645ba6: exemption ceiling breaches now report a distinct exemption_ceiling_exceeded cause instead of reusing internal_invariant_failed. The artifact-persistence half is deliberately not applied: ADR-0005 "Accepted Gaps" item 3 documents fail-closed-without-echo as intended, and test_exemption_overage_fails_closed_and_does_not_spill (tests/server/test_response_backstop.py:302) explicitly encodes the no-spill behavior. Persisting a spill artifact on this path — and resolving the ADR-0004 re-delivery tension for user-local recipes — requires a human decision to amend the ADR.

Trecek

AutoSkillit review: warning-only findings detected. See inline comments — no blocking changes required.

logger.exception attaches exc_info that logger.error dropped, keeping the structured event name while restoring post-hoc debuggability of MCP tool crashes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

assert isinstance(...) disappears under PYTHONOPTIMIZE, and convert_result exists on the base Tool class, so an invariant violation would proceed silently rather than fail. An explicit TypeError keeps behavior identical in both configurations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

_ProjectionNonconvergentError now embeds measured/projected/max_bytes and attempted state count, and the catch site surfaces the detail via the module's structured event pattern instead of swallowing the message. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Reusing internal_invariant_failed obscured the actual condition when an exempt tool exceeds its measured ceiling. The fail-closed no-spill behavior itself is unchanged (deliberate per ADR-0005 accepted gap 3). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

PROBE_POLICY_IDENTITY previously hashed only OUTPUT_DISCIPLINE_BLOCK, so editing the runtime-injected OUTPUT_DISCIPLINE_DIGEST left stale cached probe results valid. A combined digest over both policy texts now feeds the cache key; OUTPUT_DISCIPLINE_BLOCK_SHA256 remains for SKILL.md byte-identity checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

OUTPUT_DISCIPLINE_COMBINED_SHA256 raises the split-module __all__ union from 108 to 109 symbols. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Trecek

AutoSkillit PR Review — Verdict: changes_requested

Trecek

AutoSkillit review: 3 critical and 10 warning findings detected. See inline comments for details.

Outside Diff Range

These findings target lines not in the diff and could not be posted as inline comments:

tests/hooks/test_codex_hooks_format_contract.py

L19 [info/tests]: test_output_budget_guard_is_wired_with_identical_matchers uses next() without a default over HOOK_REGISTRY; if the guard

When combined JSON was under inline_max_chars but individual condensed fields exceeded it, preview/truncation markers embedded literal 'None' for the artifact path. Now forces a re-spill with inline_max_chars=0 when any condensed field needs truncation but no artifact was created. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Previously only -q/--quiet had combined-flag detection (e.g. -qn). Now extracts all single-char short flags from names and checks bundled forms, fixing bypass where 'grep -rn' passed but 'grep -r -n' was denied. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Commit 4a8d65a updated exemption ceilings but missed two prose sentences in Ceiling and Backstop Reconciliation: 53,000 -> 54,500 tokens, 212,000 -> 218,000 implied bytes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

File-level exemption silently suppressed detection of any future personal-path leak. Now uses (path, lineno) granularity targeting only lines 17-18 (the INCIDENT_LOG_SEARCH fixture). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Guard has 5 silent exit paths that all produce empty stdout. Requiring non-empty output ensures classification actually ran rather than passing vacuously on a bail-out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ible path shape_json_response had zero direct unit tests; added passthrough and spill-with-metadata cases. _plain_spill_envelope's preview_limit==0 None-return (irreducible_shape failure) was untested; added a tiny-config test that forces convergence failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Instead of requiring non-empty stdout from BOUNDED commands (which correctly produce none), add a separate test proving the guard actually classifies a known-hazardous command (grep -r) as deny. This catches bail-out regressions without breaking BOUNDED command semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

_bounded_failure cascades to progressively smaller responses; max_bytes=10 was too small, yielding '{}' without 'success' key. Set max_bytes=50 which allows '{"success":false}' while still forcing _plain_spill_envelope to return None. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

historical_context entries have bound_to_commit=false and closure.historical_artifacts are marked non-reconstructible — these are informal, non-portable evidence that was intentionally never tracked. The validator now checks structural metadata (field presence, SHA format) without requiring the referenced files to exist on disk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

historical_context and closure.historical_artifacts now validate field presence and format only (not file existence or hash match), matching the manifest's own bound_to_commit=false / non-reconstructible semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Route RecursionError from deep-nesting projection to the irreducible_shape failure path so the already-persisted artifact pointer survives, degrade deep JSON strings to the non-recursive plain spill, and reject nonpositive response_max_bytes at config construction with a terminating preview loop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Trecek and others added 13 commits July 16, 2026 21:37

feat: enforce lossless output response budgets

014333e

feat: guard unbounded command output

e2523b7

feat: deliver output discipline policy

a13c6ff

feat: derive output ceilings and add capability coverage

42bd9a1

fix: harden output budget response contracts

e8a7cca

test: route output budget evidence changes

bc357c3

fix: close JSONL command guard bypasses

54cb743

test: require installed Codex config parsing

033fed6

test: record phase 3 gate evidence

1d630fa

test: record phase 4 gate evidence

54ae181

Phase 4 gate: 30941 passed, 611 skipped, 55 xfailed at 3e10de464. Evidence manifest validated with --mode incremental. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Trecek commented Jul 17, 2026

View reviewed changes

Trecek and others added 6 commits July 17, 2026 10:41

fix(review): restore traceback capture in track_response_size catch-all

f979783

logger.exception attaches exc_info that logger.error dropped, keeping the structured event name while restoring post-hoc debuggability of MCP tool crashes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(review): bump type-constants symbol count for combined policy digest

4b02c90

OUTPUT_DISCIPLINE_COMBINED_SHA256 raises the split-module __all__ union from 108 to 109 symbols. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Trecek mentioned this pull request Jul 17, 2026

Output Budget Protocol: ADR-0005 exemption-breach documentation gap + validate_recipe size pre-check #4278

Open

Trecek commented Jul 17, 2026

View reviewed changes

Trecek and others added 6 commits July 17, 2026 12:48

fix(review): correct stale token/byte numbers in ADR-0005

daf1bf5

Commit 4a8d65a updated exemption ceilings but missed two prose sentences in Ceiling and Backstop Reconciliation: 53,000 -> 54,500 tokens, 212,000 -> 218,000 implied bytes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(review): assert non-empty guard output in battery test

f47de57

Guard has 5 silent exit paths that all produce empty stdout. Requiring non-empty output ensures classification actually ran rather than passing vacuously on a bail-out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Trecek and others added 5 commits July 17, 2026 12:58

Trecek mentioned this pull request Jul 18, 2026

Bound Codex-session context intake: read-discipline instructions, fresh-context sub-agents, and per-repo output ceilings #4280

Closed

Trecek added this pull request to the merge queue Jul 18, 2026

Merged via the queue into develop with commit 9716858 Jul 18, 2026
3 checks passed

Trecek deleted the impl-rectify_output_budget_protocol_2026-07-15_135615-20260715-163305 branch July 18, 2026 01:10

Uh oh!

Conversation

Trecek commented Jul 17, 2026

Summary

Implementation Plan

Uh oh!

Trecek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Trecek Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

Trecek Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Trecek left a comment

Choose a reason for hiding this comment

Uh oh!

Trecek left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Trecek left a comment

Choose a reason for hiding this comment

Outside Diff Range

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant