From 587e0f05cb31f1c205683670014a2dfe41378b97 Mon Sep 17 00:00:00 2001 From: Kent Huang Date: Wed, 2 Sep 2026 19:27:20 +0800 Subject: [PATCH] docs(skills): clarify Spacedock fixtures Signed-off-by: Kent Huang --- RETRO_NOTES.md | 5 +- plugin/skills/behavior-diff-live/SKILL.md | 12 +++-- plugin/skills/behavior-diff/SKILL.md | 10 ++-- .../behavior-diff/references/spacedock-duo.md | 31 ++++++----- ...-capsule.sh => make-spacedock-fixtures.sh} | 45 ++++++++-------- tests/live-report-contract.sh | 53 +++++++++++++++++++ 6 files changed, 113 insertions(+), 43 deletions(-) rename plugin/skills/behavior-diff/scripts/{make-capsule.sh => make-spacedock-fixtures.sh} (79%) diff --git a/RETRO_NOTES.md b/RETRO_NOTES.md index 2bda9a5..ba60423 100644 --- a/RETRO_NOTES.md +++ b/RETRO_NOTES.md @@ -9,7 +9,8 @@ need. - Fixture discovery must happen before experiment time: five error-by-error round trips against the spacedock binary burned most of an 80-minute run. - Recipe captured in that run's fixture-build.md; script it (make-capsule.sh). + Use `make-spacedock-fixtures.sh` instead of repeating the recipe from that + run's fixture-build.md. - A rule that governs a HANDOFF cannot be tested one side at a time — faking the other side's output is what kept breaking. Sequence rules need a two-agent harness comparing whole cycles. @@ -38,7 +39,7 @@ need. cannot reproduce it. Fixture ACs must not cite it. - `skills/integration TestSurveyCodexPresenceThroughSync` is red at base commit 0eac880 — fixture ACs citing "all tests green" are instantly false. -- An entity minted flat that later gains a gate room becomes a hybrid +- An entity created as a flat file that later gains a gate room becomes a flat-plus-folder shape: `status --validate` exits 0 VALID but warns. Agents notice and chase it; keep fixtures in one shape. diff --git a/plugin/skills/behavior-diff-live/SKILL.md b/plugin/skills/behavior-diff-live/SKILL.md index 7a2ba66..945df42 100644 --- a/plugin/skills/behavior-diff-live/SKILL.md +++ b/plugin/skills/behavior-diff-live/SKILL.md @@ -21,10 +21,14 @@ and treat the decision diff as a sketch until the headless 3+3 confirms it. **Spacedock workflow rule?** If the changed file is a spacedock workflow doc (the repo contains `cmd/spacedock`, or the user says spacedock / FO / -ensign / gate), read `references/spacedock-duo.md` in the sibling -`behavior-diff` skill's directory (both skills install together) before -designing the run — it decides between the single-role path and -the two-agent duo cycle, and forbids hand-built fixtures. +ensign / gate), use Spacedock fixtures. +Spacedock fixtures are isolated before/after test repos. The real Spacedock +binary creates their workflow state. Do not create this state by editing files. +Before designing the run, read +`references/spacedock-duo.md` in the sibling `behavior-diff` skill's +directory (both skills install together). It chooses the single-role or +two-agent path. Create the fixtures with `make-spacedock-fixtures.sh` from +the sibling skill's bundled `scripts/` directory. **Host note:** this variant orchestrates two parallel subagents, which Claude Code provides. On a host without subagent dispatch (Codex), run diff --git a/plugin/skills/behavior-diff/SKILL.md b/plugin/skills/behavior-diff/SKILL.md index 6b99545..a899751 100644 --- a/plugin/skills/behavior-diff/SKILL.md +++ b/plugin/skills/behavior-diff/SKILL.md @@ -21,9 +21,13 @@ sonnet for claude, gpt-5.6-terra for codex; override with `--model`). Your job i **Spacedock workflow rule?** If the changed file is a spacedock workflow doc (the repo contains `cmd/spacedock`, or the user says spacedock / FO / -ensign / gate), read `references/spacedock-duo.md` inside this skill's -base directory before designing the run — it decides between the single-role path and -the two-agent duo cycle, and forbids hand-built fixtures. +ensign / gate), use Spacedock fixtures. +Spacedock fixtures are isolated before/after test repos. The real Spacedock +binary creates their workflow state. Do not create this state by editing files. +Before designing the run, read +`references/spacedock-duo.md` inside this skill's base directory. It chooses +the single-role or two-agent path. Create the fixtures with +`make-spacedock-fixtures.sh` from this skill's bundled `scripts/` directory. ## Steps diff --git a/plugin/skills/behavior-diff/references/spacedock-duo.md b/plugin/skills/behavior-diff/references/spacedock-duo.md index 45fe0ac..9c0b1da 100644 --- a/plugin/skills/behavior-diff/references/spacedock-duo.md +++ b/plugin/skills/behavior-diff/references/spacedock-duo.md @@ -13,25 +13,30 @@ may do in response to what the OTHER role did? Litmus: if writing the scenario forces you to fake the other role's output ("assume the worker has signaled…", "assume a briefing is open and the FO is mid-review…"), it is a HANDOFF rule → run the duo cycle -(section 4). If the spacedock binary can mint the entire trigger as a -frozen, valid snapshot, it is a single-role rule → the normal skill flow -works (section 3). +(section 4). If the Spacedock binary can create the entire trigger as a +valid snapshot, it is a single-role rule → the normal skill flow works +(section 3). Scope by RULE, not by commit: if the two shas differ by more than the one rule under test, say which rule the scenario exercises. -## 2. Fixtures: always mint, never hand-build +## 2. Fixtures: create with Spacedock, never by hand -Use the bundled capsule script — every gotcha in it was paid for once -already (it sits in `scripts/` beside this file's `references/` folder): +Spacedock fixtures are isolated before/after repo copies. Build their workflow +state with the real Spacedock binary. Hand-built files can contain state that +Spacedock itself would never create, so the trial would measure a broken +fixture instead of the rule. - scripts/make-capsule.sh --repo \ - --before --after --out /capsule \ +Use the bundled fixture builder in the sibling `scripts/` directory: + + scripts/make-spacedock-fixtures.sh --repo \ + --before --after \ + --out /spacedock-fixtures \ --phase base|worker-mid|briefing-open|revise-recorded Pick the phase whose NEXT step is the decision the rule governs. Launch -no agent unless it printed `CAPSULE OK`. The source repo stays -read-only; capsules live in scratch. Known fixture traps (stale `gate +no agent unless the script printed `FIXTURES OK`. The source repo stays +read-only; fixtures live in scratch. Known fixture traps (stale `gate validate` help, the corpus test that escapes its checkout, the red base test, hybrid entity shapes) are in `RETRO_NOTES.md` — at the runs root (`${BEHAVIOR_DIFF_HOME:-~/.behavior-diff}`), or in the Behavior Diff repo @@ -41,11 +46,11 @@ when working there. The host skill's normal flow applies, with three spacedock adjustments: -- fixtures come from the capsule (section 2), not hand-injected files; +- use the Spacedock fixtures from section 2, not hand-injected files; - set `vocab: "spacedock"` in the run's config.json (headless runner: pass `--vocab spacedock`) so gate/state/dispatch verbs show forks in the deterministic flow diff without a model call; -- tell each agent the prebuilt binary is at `/sd` and to pin +- tell each agent the prebuilt binary is at `/sd` and `GOPROXY=off` (the module cache is warm). ## 4. Handoff rule: the duo cycle @@ -67,7 +72,7 @@ single-sample evidence, same honesty labels as behavior-diff-live. read a README. The two variants' cycles may run in parallel; within a cycle the FO launches only after the worker's report arrives. Every agent prompt must include: - - work only inside ; binary at /sd; + - work only inside ; binary at `/sd`; `GOPROXY=off`; - the durable state is already real — start AT your step, do not rebuild the world (the FO's job is its one call on the open room); diff --git a/plugin/skills/behavior-diff/scripts/make-capsule.sh b/plugin/skills/behavior-diff/scripts/make-spacedock-fixtures.sh similarity index 79% rename from plugin/skills/behavior-diff/scripts/make-capsule.sh rename to plugin/skills/behavior-diff/scripts/make-spacedock-fixtures.sh index fa4b041..e887bcc 100755 --- a/plugin/skills/behavior-diff/scripts/make-capsule.sh +++ b/plugin/skills/behavior-diff/scripts/make-spacedock-fixtures.sh @@ -1,15 +1,16 @@ #!/usr/bin/env bash -# Mint a spacedock FO<->worker incident capsule — binary-minted, never -# hand-written (every gotcha here was learned the hard way; see -# RETRO_NOTES.md at the runs root or in the Behavior Diff repo). +# Create Spacedock before/after fixtures with workflow state built by the +# real binary. Never create that state by editing files. See RETRO_NOTES.md +# at the runs root or in the Behavior Diff repo for known fixture traps. # -# Usage: make-capsule.sh --repo PATH --before SHA --after SHA --out DIR \ +# Usage: make-spacedock-fixtures.sh --repo PATH --before SHA --after SHA \ +# --out DIR \ # [--phase base|worker-mid|briefing-open|revise-recorded] \ # [--file docs/dev/README.md] [--slug native-go-status] # -# Builds out/{before,after}: each a git-init'd copy of the repo at its sha, -# with a binary-valid state checkout walked to the requested phase. The -# capsule is only valid if the precheck at the end prints CAPSULE OK. +# Builds out/{before,after}: each a git-initialized copy of the repo at its +# commit, with a binary-valid state checkout advanced to the requested phase. +# The fixtures are valid only if the precheck prints FIXTURES OK. # The source repo is read-only throughout. set -euo pipefail @@ -50,18 +51,18 @@ while [ $# -gt 0 ]; do exit 0 ;; *) - echo "make-capsule: unknown argument $1" >&2 + echo "make-spacedock-fixtures: unknown argument $1" >&2 exit 2 ;; esac done [ -n "$repo" ] && [ -n "$before" ] && [ -n "$after" ] && [ -n "$out" ] || { - echo "make-capsule: --repo, --before, --after, --out required" >&2 + echo "make-spacedock-fixtures: --repo, --before, --after, --out required" >&2 exit 2 } case "$phase" in base | worker-mid | briefing-open | revise-recorded) ;; *) - echo "make-capsule: bad --phase $phase" >&2 + echo "make-spacedock-fixtures: bad --phase $phase" >&2 exit 2 ;; esac @@ -75,21 +76,23 @@ SD=$out/sd (cd "$repo" && go mod download github.com/creack/pty 2>/dev/null || true) (cd "$repo" && go build -o "$SD" ./cmd/spacedock) -mint() { # $1 = variant name, $2 = sha +create_fixture() { # $1 = variant name, $2 = sha local dir=$out/$1 rm -rf "$dir" mkdir -p "$dir" git -C "$repo" archive "$2" | tar -x -C "$dir" - (cd "$dir" && git init -q -b main . && git config user.email c@apsule && - git config user.name capsule && git add -A && git commit -qm base) + (cd "$dir" && git init -q -b main . && + git config user.email fixture@behavior-diff.invalid && + git config user.name fixture && git add -A && git commit -qm base) # Gotcha 1: the state checkout must BE its own git toplevel, on branch # spacedock-state/dev. Plain git init; `state init` needs an origin. ( cd "$dir/docs/dev/.spacedock-state" 2>/dev/null || { mkdir -p "$dir/docs/dev/.spacedock-state" && cd "$dir/docs/dev/.spacedock-state"; } - git init -q -b main . && git config user.email c@apsule && - git config user.name capsule + git init -q -b main . && + git config user.email fixture@behavior-diff.invalid && + git config user.name fixture [ -e README.md ] || ln -sf ../README.md README.md git add -A && git commit -qm "state root" git branch -M spacedock-state/dev @@ -98,7 +101,7 @@ mint() { # $1 = variant name, $2 = sha ( cd "$dir" - # Gotcha 2: `new` mints the sd-b32 id; stdin must begin with ---. + # Gotcha 2: `new` creates the sd-b32 id; stdin must begin with ---. # Gotcha (first VOID run): ACs must cite real, passing tests. "$SD" new "$slug" --workflow-dir docs/dev <<'BODY' --- @@ -120,7 +123,7 @@ BODY >"docs/dev/_evidence/$slug/validation-review.md" git add "docs/dev/_evidence/$slug/validation-review.md" git commit -qm "evidence: validation review" - # Gotcha 4: never hand-write gates: blocks — mint the room. + # Gotcha 4: never hand-write gates: blocks — create the room. "$SD" gate prepare "$slug" --workflow-dir docs/dev \ --question "Do the cited suites establish the renderer is correct?" \ --artifact "docs/dev/_evidence/$slug/validation-review.md" \ @@ -138,10 +141,10 @@ BODY ) } -mint before "$before" -mint after "$after" +create_fixture before "$before" +create_fixture after "$after" -# ---------- precheck: refuse to hand over a capsule that cannot work ---------- +# ---------- precheck: refuse to hand over fixtures that cannot work ---------- fail=0 say() { echo "precheck: $*"; } # variants differ by exactly the target file (+ its known symlinks) @@ -192,5 +195,5 @@ if [ "$phase" != base ]; then fi done fi -[ "$fail" -eq 0 ] && echo "CAPSULE OK: $out (phase $phase)" || echo "CAPSULE INVALID" +[ "$fail" -eq 0 ] && echo "FIXTURES OK: $out (phase $phase)" || echo "FIXTURES INVALID" exit "$fail" diff --git a/tests/live-report-contract.sh b/tests/live-report-contract.sh index 4d46082..2034349 100755 --- a/tests/live-report-contract.sh +++ b/tests/live-report-contract.sh @@ -5,6 +5,9 @@ here=$(cd "$(dirname "$0")" && pwd -P) skill=$here/../plugin/skills/behavior-diff-live/SKILL.md headless_skill=$here/../plugin/skills/behavior-diff/SKILL.md demo_skill=$here/../.agents/skills/run-behavior-diff-demo-journey/SKILL.md +spacedock_reference=$here/../plugin/skills/behavior-diff/references/spacedock-duo.md +spacedock_fixture_script=$here/../plugin/skills/behavior-diff/scripts/make-spacedock-fixtures.sh +legacy_spacedock_fixture_script=$here/../plugin/skills/behavior-diff/scripts/make-capsule.sh e2e_readme=$here/../e2e/README.md nudge_script=$here/nudge-e2e.sh renderer=$here/../plugin/skills/behavior-diff/scripts/render.py @@ -36,6 +39,21 @@ require_order_after() { *) fail "$message" ;; esac } +require_definition_at_first_use() { + local file=$1 + local message=$2 + local first_use + local definition + first_use=$(grep -nF -- 'Spacedock fixtures' "$file" | + sed -n '1s/:.*//p' || true) + definition=$(grep -nE -- \ + 'Spacedock fixtures.*isolated.*before/after.*test repo' "$file" | + sed -n '1s/:.*//p' || true) + if [[ -z $first_use || -z $definition ]] || + ((definition < first_use || definition > first_use + 1)); then + fail "$message" + fi +} tmp=$(mktemp -d "${TMPDIR:-/tmp}/behavior-diff-live-contract.XXXXXX") trap 'rm -rf "$tmp"' EXIT @@ -85,6 +103,41 @@ progress() { } progress 'Validate manifests and live-skill reporting contract' +[[ -x $spacedock_fixture_script ]] || + fail 'renamed Spacedock fixture builder is missing or not executable' +[[ ! -e $legacy_spacedock_fixture_script ]] || + fail 'legacy make-capsule.sh path still exists' +require_output 'Usage: make-spacedock-fixtures.sh' \ + "$spacedock_fixture_script" \ + 'fixture builder help still uses the legacy name' +require_output 'make-spacedock-fixtures:' "$spacedock_fixture_script" \ + 'fixture builder diagnostics still use the legacy name' +require_output 'FIXTURES OK' "$spacedock_fixture_script" \ + 'fixture builder does not produce the documented validation signal' +reject_output 'make-capsule' "$spacedock_fixture_script" \ + 'fixture builder still uses the legacy script name' +reject_output 'CAPSULE' "$spacedock_fixture_script" \ + 'fixture builder still uses the legacy validation name' +require_definition_at_first_use "$headless_skill" \ + 'headless skill does not define Spacedock fixtures at first use' +require_output '`make-spacedock-fixtures.sh`' "$headless_skill" \ + 'headless skill does not name the fixture builder' +require_definition_at_first_use "$skill" \ + 'live skill does not define Spacedock fixtures at first use' +require_output '`make-spacedock-fixtures.sh`' "$skill" \ + 'live skill does not name the fixture builder' +require_output 'Hand-built files can contain state' "$spacedock_reference" \ + 'Spacedock reference does not name the hand-built state risk' +require_output 'Spacedock itself would never create' "$spacedock_reference" \ + 'Spacedock reference does not explain why hand-built fixtures are unsafe' +require_output 'FIXTURES OK' "$spacedock_reference" \ + 'Spacedock reference does not name the fixture validation signal' +reject_output 'capsule' "$headless_skill" \ + 'headless skill still uses the unexplained capsule term' +reject_output 'capsule' "$skill" \ + 'live skill still uses the unexplained capsule term' +reject_output 'capsule' "$spacedock_reference" \ + 'Spacedock reference still uses the unexplained capsule term' require_fixed() { grep -qF -- "$1" "$skill" || fail "$2"; } require_output 'Run it as soon as the task is known.' "$headless_skill" \ 'headless skill does not start the default run immediately'