From 1842cd983090077e82fa084f8b1dd5be23604690 Mon Sep 17 00:00:00 2001 From: Mother Seara Date: Wed, 5 Aug 2026 06:41:55 +0900 Subject: [PATCH 1/2] fix: close the verify-gate bypass, reach the last two gates from MCP, correct the docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An audit of every doc claim against the code turned up three things that mattered. 1. The verify gate could be bypassed by deleting the thing it keys on. `verify-gate` matched `- [x] … verify: \`CMD\``, so an item that was ticked *after* its verify clause was removed matched nothing and passed silently. `ralph`'s eligibility check only looked at unchecked items, and only once, before the loop. So the documented guarantee — "a checkbox survives only if its verify command actually passes when the harness runs it" — did not hold on that path. Fixed with `--require-verify`: a checked item with no verify clause is a failure and is reverted; `ralph` passes the flag and now refuses a TODO where any item, checked or not, lacks the clause. Default stays lenient so a mixed TODO can still be scanned standalone. 2. The two strongest gates had no MCP surface. `arc-prereg` (link a seal so `arc-close` injects the kill-condition verbatim) and `verify-gate` were shell-only, so an agent driving Yeoul through MCP — the setup the README prescribes — could run neither, while the bootstrap prompt told it the verify gate existed as a tool. Added both; 10 tools → 12, CI assertion updated. 3. Docs. `docs/METHODOLOGY.md` pointed at `bin/yeoul-graduate`, which does not exist (`bin/graduate` does). The verify-gate claims in README, README_KO, METHODOLOGY and the bootstrap prompt now state what is enforced and what is not: the harness owns the re-run and the presence of the clause, but a verify command that cannot fail is still the author's problem. Backend A's obligation to run the gate itself is now explicit rather than a parenthetical. Tests: 5 new assertions for the bypass and its controls. Verified they fail against the previous scripts (3 of 5; the other 2 are the compat and no-false-positive controls, which pass on both). Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 7 ++++--- CHANGELOG.md | 17 +++++++++++++++++ README.md | 4 +++- README_KO.md | 4 +++- bin/ralph | 14 +++++++++----- bin/verify-gate | 33 ++++++++++++++++++++++++++------- docs/BOOTSTRAP_PROMPT.md | 4 +++- docs/METHODOLOGY.md | 16 +++++++++++----- mcp/README.md | 8 ++++++-- mcp/yeoul_mcp/server.py | 18 ++++++++++++++++++ tests/test_gates.sh | 16 ++++++++++++++++ 11 files changed, 116 insertions(+), 25 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index d1fb93a..889f24e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -48,7 +48,8 @@ jobs: from yeoul_mcp import server tools = [n for n in dir(server) if n in ( 'yeoul_new','arc_open','arc_ticket','loop_guard_init','loop_guard_tick', - 'arc_close','build_handoff','ralph_gate_check','status','arc_list')] - assert len(tools) == 10, tools - print("yeoul-mcp OK — 10 tools") + 'arc_close','build_handoff','ralph_gate_check','arc_prereg','verify_gate', + 'status','arc_list')] + assert len(tools) == 12, tools + print("yeoul-mcp OK — 12 tools") PY diff --git a/CHANGELOG.md b/CHANGELOG.md index ef3ea2b..6936c24 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,6 +23,23 @@ Initial extraction of the Yeoul harness (de-personalized structure only). cross-linked row (verdict · stop · sealed claim · one-line conclusion · source) to `KNOWLEDGE_INDEX.md`. Hands-off, sourced only from the sealed _SUMMARY — it can't rot the way a hand-maintained wiki does. +### Audit fixes (2026-08-05 — found by auditing docs against code) +- **Verify-gate bypass closed.** The gate keyed on the `verify:` clause, which lives in a file the agent + edits: deleting the clause while ticking the box made the item invisible to the gate and it survived + unverified. `verify-gate --require-verify` now treats a checked item with no verify clause as a failure, + `ralph` passes the flag and refuses a TODO where *any* item (checked or not) lacks the clause. The docs + that claimed "a checkbox survives only if its verify command passes" were corrected to state what is + actually enforced — and what is not (a verify command that cannot fail is still on the author). +- **The two strongest gates reached MCP.** `arc_prereg` (link a seal so `arc_close` injects the + kill-condition verbatim) and `verify_gate` (re-run a round's verify commands) had no MCP tool, so an + agent driving Yeoul purely through MCP — the setup the README prescribes — could run neither. Added; + tool surface 10 → 12. +- **Docs**: `bin/yeoul-graduate` → `bin/graduate` (the referenced command did not exist); backend A's + obligation to run the verify gate itself made explicit in METHODOLOGY and the bootstrap prompt. +- **`index-append`**: numbered lists no longer index an empty conclusion, and a failed extraction warns + instead of passing silently (see below). +- **`mcp<2` pin**: an unbounded floor let CI install a breaking major (`mcp.server.fastmcp` moved). + ### Review fixes - **Portability**: replaced GNU-only `sed -i` with a portable temp-file edit (was silently failing the In-Progress→Closed status update on macOS/BSD sed, which the README targets). diff --git a/README.md b/README.md index efe77a1..ccccd48 100644 --- a/README.md +++ b/README.md @@ -65,7 +65,9 @@ New to the method? Read [`docs/METHODOLOGY.md`](docs/METHODOLOGY.md), then paste - **KILL-defense 5-check** — a "failed" verdict cannot be sealed until anchor-reproduction, ≥2 independent angles, implementation-defect ruled out, catalog cross-check, and verbatim kill-wording are all recorded. - **Verify-gated dev loop** — each automated development round advances one TODO item and may only check it - off after its own machine verification command exits 0. Items without a verify command are refused. + off after its own machine verification command exits 0, re-run by the harness rather than reported by the + agent. Items without a verify command are refused — including checked ones, so deleting the clause is not + a way out. (A verify command that cannot fail is still a hole: that part is on the author.) ## Layers diff --git a/README_KO.md b/README_KO.md index e153c41..c5c8473 100644 --- a/README_KO.md +++ b/README_KO.md @@ -63,7 +63,9 @@ tests/test_gates.sh # 모든 게이트의 단언 기반 스모크 테스트 - **KILL 방어 5항 체크** — "실패" 판정은 앵커 재현·독립각도 ≥2·구현결함 배제·도감 대조·kill 문언 일치가 전부 기록되기 전엔 봉인되지 않는다. - **검증 게이트 개발 루프** — 각 자동 개발 라운드는 TODO 한 항목을 진행하고, 그 항목의 기계 검증 명령이 exit 0일 - 때만 체크할 수 있다. 검증 명령 없는 항목은 거부된다. + 때만 체크할 수 있다. 판정은 에이전트 보고가 아니라 **하네스 재실행**이다. 검증 명령 없는 항목은 거부되며 + **체크된 항목도 마찬가지** — 절을 지우고 체크하는 우회가 막힌다. (실패할 수 없는 검증 명령은 여전히 구멍이고, + 그건 작성자 몫이다.) ## 계층 diff --git a/bin/ralph b/bin/ralph index 451d4b7..4140807 100755 --- a/bin/ralph +++ b/bin/ralph @@ -27,10 +27,13 @@ for a in "${@:2}"; do case "$a" in --agent-cmd=*) AGENT_CMD="${a#*=}" ;; esac; done -# Gate: any unchecked item without a `verify:` command → refuse the whole loop (form = eligibility). -BAD="$(grep '^- \[ \]' "$TODO" | grep -v 'verify:' || true)" +# Gate: any item without a `verify:` command → refuse the whole loop (form = eligibility). +# Checked items are included: the loop hands the file to an agent every round, and a checked item +# with no verify clause is exactly what the clause-deletion bypass produces. Requiring the clause on +# every item is what lets verify-gate run with --require-verify below (no exemption bookkeeping). +BAD="$(grep -E '^- \[( |x)\]' "$TODO" | grep -v 'verify:' || true)" if [ -n "$BAD" ]; then - echo "⛔ unchecked items without a 'verify:' command — not loop-eligible (manual/session work):" + echo "⛔ items without a 'verify:' command — not loop-eligible (manual/session work):" echo "$BAD" | sed 's/^/ /' exit 3 fi @@ -68,8 +71,9 @@ $(cat "$TODO") # ★ harness enforces the gate: independently re-run verify for whatever got checked off; revert failures. # The agent's claim that "verify passed" is not trusted — a checkbox survives only if the command - # actually exits 0 when the harness runs it. - "$SCRIPT_DIR/verify-gate" "$TODO" --revert || echo " ⚠ harness reverted a checked item (its verify did not pass on re-run)" + # actually exits 0 when the harness runs it. --require-verify additionally reverts a checked item + # whose verify clause went missing, so deleting the clause is not a way out either. + "$SCRIPT_DIR/verify-gate" "$TODO" --revert --require-verify || echo " ⚠ harness reverted a checked item (verify failed on re-run, or its verify clause was missing)" TOK=0 if command -v python3 >/dev/null 2>&1; then diff --git a/bin/verify-gate b/bin/verify-gate index 9a10f4a..fe8c776 100755 --- a/bin/verify-gate +++ b/bin/verify-gate @@ -1,21 +1,40 @@ #!/usr/bin/env bash set -uo pipefail -# verify-gate [--revert] +# verify-gate [--revert] [--require-verify] # Harness-enforced verify gate. For every checked item (`- [x]` … verify: `CMD`), the harness # RE-RUNS CMD itself (cwd = current dir) — it does not trust the agent's claim that verify passed. # With --revert, any item whose command exits != 0 is flipped back to `- [ ]`. -# This is what makes "resists self-deception" true rather than requested: a checked box only -# survives if its verify command actually passes when the harness runs it. # Exit 0 = every checked item's verify passed · non-zero = at least one failed. +# +# --require-verify: a checked item with NO `verify:` clause counts as a failure too. +# Why it exists: without it the gate keys on text the agent itself writes, so deleting the +# `verify:` clause while ticking the box made the item invisible to the gate and it survived +# unverified (observed, not hypothetical). `ralph` always passes this flag — its eligibility gate +# has already established that every item in a loop-driven TODO carries a verify command. +# Default stays off so a mixed TODO (loop items + manual items) can still be scanned standalone. -TODO="${1:-}"; REVERT=0 -for a in "${@:2}"; do case "$a" in --revert) REVERT=1 ;; esac; done -[ -n "$TODO" ] && [ -f "$TODO" ] || { echo "usage: verify-gate [--revert]" >&2; exit 2; } +TODO="${1:-}"; REVERT=0; REQUIRE=0 +for a in "${@:2}"; do case "$a" in + --revert) REVERT=1 ;; + --require-verify) REQUIRE=1 ;; +esac; done +[ -n "$TODO" ] && [ -f "$TODO" ] || { echo "usage: verify-gate [--revert] [--require-verify]" >&2; exit 2; } FAIL=0; TMP="$TODO.vg.$$" : > "$TMP" while IFS= read -r line || [ -n "$line" ]; do - if printf '%s' "$line" | grep -qE '^- \[x\].*verify:[[:space:]]*`[^`]+`'; then + if [ "$REQUIRE" -eq 1 ] && printf '%s' "$line" | grep -q '^- \[x\]' \ + && ! printf '%s' "$line" | grep -qE 'verify:[[:space:]]*`[^`]+`'; then + # checked but carries no verify command — the gate has nothing to re-run, so it cannot vouch + # for this box. Treat as failed rather than skipping it (skipping is how the clause-deletion + # bypass worked). + FAIL=1 + echo " ✗ checked item has no verify command: ${line:0:60}…" >&2 + if [ "$REVERT" -eq 1 ]; then + line="$(printf '%s' "$line" | sed 's/^- \[x\]/- [ ]/')" + echo " ↩ reverted to [ ] (nothing for the harness to verify)" >&2 + fi + elif printf '%s' "$line" | grep -qE '^- \[x\].*verify:[[:space:]]*`[^`]+`'; then # extract the FIRST verify block via python (sed's greedy `.*verify:` grabbed the LAST block, # letting an appended decoy `verify: `true`` override a real failing command). cmd="$(printf '%s' "$line" | python3 -c 'import sys,re; m=re.search(r"verify:\s*`([^`]+)`", sys.stdin.read()); sys.stdout.write(m.group(1) if m else "")')" diff --git a/docs/BOOTSTRAP_PROMPT.md b/docs/BOOTSTRAP_PROMPT.md index 147a912..7b3fd33 100644 --- a/docs/BOOTSTRAP_PROMPT.md +++ b/docs/BOOTSTRAP_PROMPT.md @@ -29,7 +29,9 @@ HARD RULES (do not violate — these are the point of the tool) - Never seal before pre-registering the kill-condition. Never edit a kill-condition after seeing results (amend, don't overwrite). Record negatives and retractions indelibly. - Never automate: pre-registration sealing, PASS/KILL judgment, graduation, external publishing. Those are mine. -- In the dev loop, only check off a TODO item after its verify command exits 0; never edit verify commands. +- In the dev loop, only check off a TODO item after its verify command exits 0; never edit or delete verify + commands. Run `verify_gate` (or `bin/verify-gate --revert --require-verify`) after every round — + in backend A that is your job, not the loop's. - Separate the tool from your judgment: only say "a gate refused X" when a script/tool actually returned an error you can quote; say "applying the discipline, I suspect X" for your own reasoning. Never borrow the tool's credibility for a judgment call. diff --git a/docs/METHODOLOGY.md b/docs/METHODOLOGY.md index 9798098..c36a1ec 100644 --- a/docs/METHODOLOGY.md +++ b/docs/METHODOLOGY.md @@ -87,13 +87,19 @@ Run the dev loop two ways (Ralph pattern): **Loop-forbidden (belongs to the human/session):** measurement runs, sealing, PASS/KILL judgment. The loop's safety comes from the machine-verification gate, not from the loop — it is only as good as your verify commands. -The gate is **harness-enforced, not requested**: after each round the harness runs `bin/verify-gate --revert`, -which independently re-runs the `verify:` command of every checked item and flips any that don't exit 0 back to -`[ ]`. `ralph` does this automatically; in backend A, the session must run it too. A checkbox survives only if -its verify command actually passes when the *harness* runs it — the agent's word is not trusted. +The gate is **harness-enforced, not requested**: after each round the harness runs +`bin/verify-gate --revert --require-verify`, which independently re-runs the `verify:` command of every +checked item and flips any that don't exit 0 back to `[ ]`. `ralph` does this automatically; **in backend A the +session must run it too** (MCP: the `verify_gate` tool) — otherwise nothing has been verified but the agent's word. + +Scope, honestly: the gate keys on the `verify:` clause, which lives in a file the agent can edit. Deleting the +clause while ticking the box used to make the item invisible to the gate, so it survived unverified. That is what +`--require-verify` closes — a checked item with no verify clause is now reverted too, and `ralph` refuses a TODO +where any item lacks the clause. What remains outside the machine's reach is the *content* of a verify command: +a command that does not actually test the item will still exit 0. Write commands that can fail. ## Exits -- **graduate** (success): `bin/yeoul-graduate` → moves the project out of the incubator, leaves a pointer. +- **graduate** (success): `bin/graduate` → moves the project out of the incubator, leaves a pointer. - **close-project** (retire/reject): `bin/close-project` → archives with a `_CLOSED.md`, freezes open arcs, optionally seals the closure. History is preserved — no silent deletion. diff --git a/mcp/README.md b/mcp/README.md index cd5447f..4a59ec1 100644 --- a/mcp/README.md +++ b/mcp/README.md @@ -23,8 +23,12 @@ Set `YEOUL_BIN` if the `bin/` scripts are not adjacent to the package. ## Tools -`yeoul_new`, `arc_open`, `arc_ticket`, `loop_guard_init`, `loop_guard_tick`, `arc_close`, `build_handoff`, -`ralph_gate_check`, `status`, `arc_list`. +`yeoul_new`, `arc_open`, `arc_ticket`, `loop_guard_init`, `loop_guard_tick`, `arc_close`, `arc_prereg`, +`build_handoff`, `ralph_gate_check`, `verify_gate`, `status`, `arc_list`. + +`arc_prereg` and `verify_gate` are the two enforcement halves that used to be reachable only from the shell: +linking a seal (so `arc_close` injects the kill-condition verbatim) and re-running a round's verify commands. +An agent driving Yeoul purely through MCP could not run either — so neither gate held on that path. Each returns `{exit_code, stdout, stderr}`. A non-zero `exit_code` on `arc_close` (4 = blanks, 5 = KILL-defense) or `ralph_gate_check` (3 = ungated item) is an enforced gate, not an error to route around. diff --git a/mcp/yeoul_mcp/server.py b/mcp/yeoul_mcp/server.py index ef2abb3..ca67098 100644 --- a/mcp/yeoul_mcp/server.py +++ b/mcp/yeoul_mcp/server.py @@ -118,6 +118,24 @@ def ralph_gate_check(name: str, workspace: str = ".") -> dict: return {"exit_code": 0, "stdout": "loop-eligible: all unchecked items have verify commands", "stderr": ""} +@mcp.tool() +def arc_prereg(arc_dir: str, claim_id: str, ledger: str = "", workspace: str = ".") -> dict: + """Link a sealed pre-registration to an arc so arc_close injects its kill-condition VERBATIM instead of + trusting an agent-typed field. Seal the claim first (mirror-stack). Without this, closes are UNSEALED.""" + args = [arc_dir, claim_id] + ([ledger] if ledger else []) + return _run("arc-prereg", *args, cwd=workspace) + + +@mcp.tool() +def verify_gate(todo_path: str, revert: bool = True, require_verify: bool = True, + workspace: str = ".") -> dict: + """Re-run the `verify:` command of every checked TODO item and revert the boxes that do not pass. This is + the harness half of the dev loop — backend A (in-session) MUST call it each round, or nothing has been + verified but the agent's word. require_verify also reverts a checked item whose verify clause is missing.""" + args = [todo_path] + (["--revert"] if revert else []) + (["--require-verify"] if require_verify else []) + return _run("verify-gate", *args, cwd=workspace) + + @mcp.tool() def status(workspace: str = ".", md: bool = False) -> dict: """One line per active project: name · latest arc verdict · dev TODO progress.""" diff --git a/tests/test_gates.sh b/tests/test_gates.sh index 6ca1a92..f11cc0f 100755 --- a/tests/test_gates.sh +++ b/tests/test_gates.sh @@ -83,6 +83,22 @@ printf -- '- [x] decoy. verify: `false` verify: `true`\n' > "$T3" "$BIN/verify-gate" "$T3" --revert >/dev/null 2>&1 grep -q '^- \[ \] decoy' "$T3" && echo " ✓ decoy double-verify reverted (first block wins)" || { echo " ✗ decoy passed (greedy bug)"; FAIL=1; } +# --- clause-deletion bypass: ticking a box after deleting `verify:` must not survive --- +# Regression: the gate keys on text the agent writes, so a checked item with the clause removed used +# to be invisible to it and passed silently. Observed, not hypothetical. +T4="$WS/t4.md" +printf -- '- [x] deleted its own verify clause\n' > "$T4" +"$BIN/verify-gate" "$T4" >/dev/null 2>&1; assert "clause-less checked item passes without --require-verify (compat)" 0 $? +"$BIN/verify-gate" "$T4" --revert --require-verify >/dev/null 2>&1; assert "clause-less checked item fails with --require-verify" 1 $? +grep -q '^- \[ \] deleted its own verify clause' "$T4" && echo " ✓ clause-less box reverted to [ ]" || { echo " ✗ clause-less box survived"; FAIL=1; } +# a genuine passing item must still survive under the same flag (no false positives) +printf -- '- [x] real. verify: `true`\n' > "$T4" +"$BIN/verify-gate" "$T4" --revert --require-verify >/dev/null 2>&1; assert "genuine checked item still passes under --require-verify" 0 $? +grep -q '^- \[x\] real' "$T4" && echo " ✓ genuine box left checked" || { echo " ✗ genuine box wrongly reverted"; FAIL=1; } +# ralph must refuse a TODO whose CHECKED item lacks a verify clause (was only checking unchecked ones) +printf -- '- [x] checked without verify\n- [ ] ok. verify: `true`\n' > "$WS/projects/p/dev/TODO.md" +"$BIN/ralph" p >/dev/null 2>&1; assert "ralph refuses a checked item without verify" 3 $? + # --- index-append: the conclusion must survive any list style, and a miss must be loud --- # Regression: the extractor used to assume a `- ` bullet, so a numbered list silently indexed # an empty conclusion while still logging success. Observed in the wild, not hypothetical. From fe3626de3ae77911d4757aa591684470c915e5ba Mon Sep 17 00:00:00 2001 From: Mother Seara Date: Wed, 5 Aug 2026 06:44:52 +0900 Subject: [PATCH 2/2] fix(ci): restore block-scalar indentation broken by the tool-list edit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The added continuation line was dedented out of the `run: |` block, which made the workflow file itself invalid — the run failed before any job started. Validated the YAML parses and the embedded python compiles before pushing this time. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 889f24e..9d12c67 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -49,7 +49,7 @@ jobs: tools = [n for n in dir(server) if n in ( 'yeoul_new','arc_open','arc_ticket','loop_guard_init','loop_guard_tick', 'arc_close','build_handoff','ralph_gate_check','arc_prereg','verify_gate', - 'status','arc_list')] + 'status','arc_list')] assert len(tools) == 12, tools print("yeoul-mcp OK — 12 tools") PY