Skip to content

Streamline encode-policy-v2 and strengthen review handoffs - #89

Merged
hua7450 merged 3 commits into
PolicyEngine:mainfrom
hua7450:enhance-encode-policy-v2
Sep 6, 2026
Merged

hua7450 merged 3 commits into
PolicyEngine:mainfrom
hua7450:enhance-encode-policy-v2

Conversation

@hua7450

@hua7450 hua7450 commented Sep 5, 2026 •

Copy link
Copy Markdown
Collaborator

The encoding workflow spread routine work across many successive agents, and the MO LIHEAP benchmark exposed command-stub resolution failures, expensive review handoffs and unsupported scenarios entering the critical-fix loop. This change consolidates encoding ownership and makes the review handoff distinguish confirmed defects from unresolved claims.

Closes #87.

Changes

  • Use a coordinator with persistent implementation and test-author roles; overlap independent preparation, reuse source evidence, bound acquisition and waits, and retain targeted validation and independent review after repairs.
  • Support local encoding and unpublished local-diff reviews without GitHub writes, preserving worktrees and recording phase timings.
  • Require model-development to load in each model worker's own runtime, with actual load evidence. Add lightweight SessionStart/SubagentStart reminders; reminders do not themselves load or certify skills.
  • Remove the same-named encode-policy-v2, review-program and fix-pr command stubs, and reject duplicate skill/command registration in the wrapper builder. The skill entrypoints retain their slash commands.
  • Dispatch two reviewers together by default, scope incremental checks to affected behavior, distinguish required evidence gaps from ordinary limitations, and require supported scenario premises before counting critical defects.
  • Assemble reports without rewriting reviewer files: allocate new IDs above reserved prior IDs, select unchanged prior findings directly, attach adjudication notes and cap the summary at 20 lines. Encode dispatches repairs from confirmed_critical_ids, excluding UNVERIFIED findings.
  • Add no-delegation worker profiles: review-worker for both reviewers and model-worker for the encode implementer and test-author (Skill access, no Agent/Task; model-worker also has no web tools). Brief both encode owners from one shared contract block, make the coordinator's bounded test invocation the run of record with at most one worker self-check, add encode progress logs and canonical timing keys, forbid post-adjudication source probing in review, and define the adjudication/consolidation timing overlap. Launchers gain the compatibility-stub fallback, the nested-dispatch name rule and a 540-second bounded test deadline under the tool maximum.
  • Harden the SessionStart/SubagentStart hook against malformed input and add a behavioral test; regenerate the catalog with a PyYAML interpreter.
  • Refresh workflow documentation and the generated catalog. Extend existing mechanical tests for local snapshots, packaging and the repair/report contracts; these tests do not establish policy accuracy.

Validation

  • Final local repository checks: python -m pytest --ignore=tests/test_skill_examples.py -q — 175 passed.
  • Claude wrapper build, changed-helper lint, skill metadata validation and git diff --check passed.
  • Fresh native Claude entrypoint smoke test: both actual skills and canonical paths loaded in 23.8 seconds, with no compatibility stub.
  • Replay of saved MO review artifacts: 0.05–0.08 seconds, 20-line summaries, original evidence retained; marking C3 UNVERIFIED removes it from automatic repair. This measures assembly mechanics, not review duration.
  • Focused fresh-context C3 assessment loaded review-program and model-development and withheld automatic repair because the applicant premise was unestablished. It also volunteered unsupported side remarks, which were rejected; adjudication scope was narrowed afterwards.

The final workflow still needs a full real-PR benchmark to establish end-to-end speed and review quality. This PR does not claim a new 10–15-minute runtime. Detailed observations and limitations are recorded in skills/review-program/references/benchmark.md. No MO model implementation, local source cache or user research notes are included.

Catalog export now reports 23 skills, 36 agents, 20 command files and 8 bundles; the separately maintained app-v2 website stats need the corresponding companion update.

hua7450 and others added 3 commits September 5, 2026 12:19
Two benchmark runs under this branch's workflow exposed one failure class:
delegated contexts doing model work outside the contract. Encode workers now
use a no-delegation `model-worker` profile mirroring `review-worker`; both
coordinators brief owners from one shared contract block and treat the
coordinator's bounded test invocation as the run of record (a worker may
self-check its owned files once). Launchers gain the compatibility-stub
fallback, the nested-dispatch `name` rule and Bash timeout guidance. Review
adjudication forbids post-report source probing and defines the
adjudication/consolidation timing overlap. Encode gains canonical progress
logs and timing keys. Benchmark notes record the MO v6 interruption and the
Kansas LIEAP end-to-end run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A6rihVz3MtZQKCzzkCib35
Regenerate the dashboard manifest with a PyYAML interpreter: the previous
regeneration fell back to the subset parser, turning analysis numbers into
strings and dropping the nested corroboration verdict; it also lacked the two
worker profiles. Set the bounded test deadline to 540 seconds so it fires
under the tool's 600-second maximum and the wrapper's exit status survives.
Harden the model-development hook against missing cwd or malformed input and
add a behavioral test. State the fallback when model-worker cannot be a
retained named worker, drop Edit from review-worker, and correct the agent
count in the guide.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A6rihVz3MtZQKCzzkCib35
@hua7450
hua7450 merged commit 80b3ece into PolicyEngine:main Sep 6, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enhance the encode-policy-v2 command

1 participant