Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions RETRO_NOTES.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@ need.

- Fixture discovery must happen before experiment time: five error-by-error
round trips against the spacedock binary burned most of an 80-minute run.
Recipe captured in that run's fixture-build.md; script it (make-capsule.sh).
Use `make-spacedock-fixtures.sh` instead of repeating the recipe from that
run's fixture-build.md.
- A rule that governs a HANDOFF cannot be tested one side at a time —
faking the other side's output is what kept breaking. Sequence rules need
a two-agent harness comparing whole cycles.
Expand Down Expand Up @@ -38,7 +39,7 @@ need.
cannot reproduce it. Fixture ACs must not cite it.
- `skills/integration TestSurveyCodexPresenceThroughSync` is red at base
commit 0eac880 — fixture ACs citing "all tests green" are instantly false.
- An entity minted flat that later gains a gate room becomes a hybrid
- An entity created as a flat file that later gains a gate room becomes a
flat-plus-folder shape: `status --validate` exits 0 VALID but warns.
Agents notice and chase it; keep fixtures in one shape.

Expand Down
12 changes: 8 additions & 4 deletions plugin/skills/behavior-diff-live/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,14 @@ and treat the decision diff as a sketch until the headless 3+3 confirms it.

**Spacedock workflow rule?** If the changed file is a spacedock workflow
doc (the repo contains `cmd/spacedock`, or the user says spacedock / FO /
ensign / gate), read `references/spacedock-duo.md` in the sibling
`behavior-diff` skill's directory (both skills install together) before
designing the run — it decides between the single-role path and
the two-agent duo cycle, and forbids hand-built fixtures.
ensign / gate), use Spacedock fixtures.
Spacedock fixtures are isolated before/after test repos. The real Spacedock
binary creates their workflow state. Do not create this state by editing files.
Before designing the run, read
`references/spacedock-duo.md` in the sibling `behavior-diff` skill's
directory (both skills install together). It chooses the single-role or
two-agent path. Create the fixtures with `make-spacedock-fixtures.sh` from
the sibling skill's bundled `scripts/` directory.

**Host note:** this variant orchestrates two parallel subagents, which
Claude Code provides. On a host without subagent dispatch (Codex), run
Expand Down
10 changes: 7 additions & 3 deletions plugin/skills/behavior-diff/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,13 @@ sonnet for claude, gpt-5.6-terra for codex; override with `--model`). Your job i

**Spacedock workflow rule?** If the changed file is a spacedock workflow
doc (the repo contains `cmd/spacedock`, or the user says spacedock / FO /
ensign / gate), read `references/spacedock-duo.md` inside this skill's
base directory before designing the run — it decides between the single-role path and
the two-agent duo cycle, and forbids hand-built fixtures.
ensign / gate), use Spacedock fixtures.
Spacedock fixtures are isolated before/after test repos. The real Spacedock
binary creates their workflow state. Do not create this state by editing files.
Before designing the run, read
`references/spacedock-duo.md` inside this skill's base directory. It chooses
the single-role or two-agent path. Create the fixtures with
`make-spacedock-fixtures.sh` from this skill's bundled `scripts/` directory.

## Steps

Expand Down
31 changes: 18 additions & 13 deletions plugin/skills/behavior-diff/references/spacedock-duo.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,25 +13,30 @@ may do in response to what the OTHER role did?
Litmus: if writing the scenario forces you to fake the other role's
output ("assume the worker has signaled…", "assume a briefing is open
and the FO is mid-review…"), it is a HANDOFF rule → run the duo cycle
(section 4). If the spacedock binary can mint the entire trigger as a
frozen, valid snapshot, it is a single-role rule → the normal skill flow
works (section 3).
(section 4). If the Spacedock binary can create the entire trigger as a
valid snapshot, it is a single-role rule → the normal skill flow works
(section 3).

Scope by RULE, not by commit: if the two shas differ by more than the
one rule under test, say which rule the scenario exercises.

## 2. Fixtures: always mint, never hand-build
## 2. Fixtures: create with Spacedock, never by hand

Use the bundled capsule script — every gotcha in it was paid for once
already (it sits in `scripts/` beside this file's `references/` folder):
Spacedock fixtures are isolated before/after repo copies. Build their workflow
state with the real Spacedock binary. Hand-built files can contain state that
Spacedock itself would never create, so the trial would measure a broken
fixture instead of the rule.

scripts/make-capsule.sh --repo <spacedock repo> \
--before <sha> --after <sha> --out <scratch>/capsule \
Use the bundled fixture builder in the sibling `scripts/` directory:

scripts/make-spacedock-fixtures.sh --repo <spacedock repo> \
--before <sha> --after <sha> \
--out <scratch>/spacedock-fixtures \
--phase base|worker-mid|briefing-open|revise-recorded

Pick the phase whose NEXT step is the decision the rule governs. Launch
no agent unless it printed `CAPSULE OK`. The source repo stays
read-only; capsules live in scratch. Known fixture traps (stale `gate
no agent unless the script printed `FIXTURES OK`. The source repo stays
read-only; fixtures live in scratch. Known fixture traps (stale `gate
validate` help, the corpus test that escapes its checkout, the red base
test, hybrid entity shapes) are in `RETRO_NOTES.md` — at the runs root
(`${BEHAVIOR_DIFF_HOME:-~/.behavior-diff}`), or in the Behavior Diff repo
Expand All @@ -41,11 +46,11 @@ when working there.

The host skill's normal flow applies, with three spacedock adjustments:

- fixtures come from the capsule (section 2), not hand-injected files;
- use the Spacedock fixtures from section 2, not hand-injected files;
- set `vocab: "spacedock"` in the run's config.json (headless runner:
pass `--vocab spacedock`) so gate/state/dispatch verbs show forks in
the deterministic flow diff without a model call;
- tell each agent the prebuilt binary is at `<capsule>/sd` and to pin
- tell each agent the prebuilt binary is at `<spacedock-fixtures>/sd` and
`GOPROXY=off` (the module cache is warm).

## 4. Handoff rule: the duo cycle
Expand All @@ -67,7 +72,7 @@ single-sample evidence, same honesty labels as behavior-diff-live.
read a README. The two variants' cycles may run in parallel; within a
cycle the FO launches only after the worker's report arrives. Every
agent prompt must include:
- work only inside <variant dir>; binary at <capsule>/sd;
- work only inside <variant dir>; binary at `<spacedock-fixtures>/sd`;
`GOPROXY=off`;
- the durable state is already real — start AT your step, do not
rebuild the world (the FO's job is its one call on the open room);
Expand Down
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
#!/usr/bin/env bash
# Mint a spacedock FO<->worker incident capsule — binary-minted, never
# hand-written (every gotcha here was learned the hard way; see
# RETRO_NOTES.md at the runs root or in the Behavior Diff repo).
# Create Spacedock before/after fixtures with workflow state built by the
# real binary. Never create that state by editing files. See RETRO_NOTES.md
# at the runs root or in the Behavior Diff repo for known fixture traps.
#
# Usage: make-capsule.sh --repo PATH --before SHA --after SHA --out DIR \
# Usage: make-spacedock-fixtures.sh --repo PATH --before SHA --after SHA \
# --out DIR \
# [--phase base|worker-mid|briefing-open|revise-recorded] \
# [--file docs/dev/README.md] [--slug native-go-status]
#
# Builds out/{before,after}: each a git-init'd copy of the repo at its sha,
# with a binary-valid state checkout walked to the requested phase. The
# capsule is only valid if the precheck at the end prints CAPSULE OK.
# Builds out/{before,after}: each a git-initialized copy of the repo at its
# commit, with a binary-valid state checkout advanced to the requested phase.
# The fixtures are valid only if the precheck prints FIXTURES OK.
# The source repo is read-only throughout.
set -euo pipefail

Expand Down Expand Up @@ -50,18 +51,18 @@ while [ $# -gt 0 ]; do
exit 0
;;
*)
echo "make-capsule: unknown argument $1" >&2
echo "make-spacedock-fixtures: unknown argument $1" >&2
exit 2
;;
esac
done
[ -n "$repo" ] && [ -n "$before" ] && [ -n "$after" ] && [ -n "$out" ] || {
echo "make-capsule: --repo, --before, --after, --out required" >&2
echo "make-spacedock-fixtures: --repo, --before, --after, --out required" >&2
exit 2
}
case "$phase" in base | worker-mid | briefing-open | revise-recorded) ;;
*)
echo "make-capsule: bad --phase $phase" >&2
echo "make-spacedock-fixtures: bad --phase $phase" >&2
exit 2
;;
esac
Expand All @@ -75,21 +76,23 @@ SD=$out/sd
(cd "$repo" && go mod download github.com/creack/pty 2>/dev/null || true)
(cd "$repo" && go build -o "$SD" ./cmd/spacedock)

mint() { # $1 = variant name, $2 = sha
create_fixture() { # $1 = variant name, $2 = sha
local dir=$out/$1
rm -rf "$dir"
mkdir -p "$dir"
git -C "$repo" archive "$2" | tar -x -C "$dir"
(cd "$dir" && git init -q -b main . && git config user.email c@apsule &&
git config user.name capsule && git add -A && git commit -qm base)
(cd "$dir" && git init -q -b main . &&
git config user.email fixture@behavior-diff.invalid &&
git config user.name fixture && git add -A && git commit -qm base)
# Gotcha 1: the state checkout must BE its own git toplevel, on branch
# spacedock-state/dev. Plain git init; `state init` needs an origin.
(
cd "$dir/docs/dev/.spacedock-state" 2>/dev/null ||
{ mkdir -p "$dir/docs/dev/.spacedock-state" &&
cd "$dir/docs/dev/.spacedock-state"; }
git init -q -b main . && git config user.email c@apsule &&
git config user.name capsule
git init -q -b main . &&
git config user.email fixture@behavior-diff.invalid &&
git config user.name fixture
[ -e README.md ] || ln -sf ../README.md README.md
git add -A && git commit -qm "state root"
git branch -M spacedock-state/dev
Expand All @@ -98,7 +101,7 @@ mint() { # $1 = variant name, $2 = sha

(
cd "$dir"
# Gotcha 2: `new` mints the sd-b32 id; stdin must begin with ---.
# Gotcha 2: `new` creates the sd-b32 id; stdin must begin with ---.
# Gotcha (first VOID run): ACs must cite real, passing tests.
"$SD" new "$slug" --workflow-dir docs/dev <<'BODY'
---
Expand All @@ -120,7 +123,7 @@ BODY
>"docs/dev/_evidence/$slug/validation-review.md"
git add "docs/dev/_evidence/$slug/validation-review.md"
git commit -qm "evidence: validation review"
# Gotcha 4: never hand-write gates: blocks — mint the room.
# Gotcha 4: never hand-write gates: blocks — create the room.
"$SD" gate prepare "$slug" --workflow-dir docs/dev \
--question "Do the cited suites establish the renderer is correct?" \
--artifact "docs/dev/_evidence/$slug/validation-review.md" \
Expand All @@ -138,10 +141,10 @@ BODY
)
}

mint before "$before"
mint after "$after"
create_fixture before "$before"
create_fixture after "$after"

# ---------- precheck: refuse to hand over a capsule that cannot work ----------
# ---------- precheck: refuse to hand over fixtures that cannot work ----------
fail=0
say() { echo "precheck: $*"; }
# variants differ by exactly the target file (+ its known symlinks)
Expand Down Expand Up @@ -192,5 +195,5 @@ if [ "$phase" != base ]; then
fi
done
fi
[ "$fail" -eq 0 ] && echo "CAPSULE OK: $out (phase $phase)" || echo "CAPSULE INVALID"
[ "$fail" -eq 0 ] && echo "FIXTURES OK: $out (phase $phase)" || echo "FIXTURES INVALID"
exit "$fail"
53 changes: 53 additions & 0 deletions tests/live-report-contract.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ here=$(cd "$(dirname "$0")" && pwd -P)
skill=$here/../plugin/skills/behavior-diff-live/SKILL.md
headless_skill=$here/../plugin/skills/behavior-diff/SKILL.md
demo_skill=$here/../.agents/skills/run-behavior-diff-demo-journey/SKILL.md
spacedock_reference=$here/../plugin/skills/behavior-diff/references/spacedock-duo.md
spacedock_fixture_script=$here/../plugin/skills/behavior-diff/scripts/make-spacedock-fixtures.sh
legacy_spacedock_fixture_script=$here/../plugin/skills/behavior-diff/scripts/make-capsule.sh
e2e_readme=$here/../e2e/README.md
nudge_script=$here/nudge-e2e.sh
renderer=$here/../plugin/skills/behavior-diff/scripts/render.py
Expand Down Expand Up @@ -36,6 +39,21 @@ require_order_after() {
*) fail "$message" ;;
esac
}
require_definition_at_first_use() {
local file=$1
local message=$2
local first_use
local definition
first_use=$(grep -nF -- 'Spacedock fixtures' "$file" |
sed -n '1s/:.*//p' || true)
definition=$(grep -nE -- \
'Spacedock fixtures.*isolated.*before/after.*test repo' "$file" |
sed -n '1s/:.*//p' || true)
if [[ -z $first_use || -z $definition ]] ||
((definition < first_use || definition > first_use + 1)); then
fail "$message"
fi
}

tmp=$(mktemp -d "${TMPDIR:-/tmp}/behavior-diff-live-contract.XXXXXX")
trap 'rm -rf "$tmp"' EXIT
Expand Down Expand Up @@ -85,6 +103,41 @@ progress() {
}

progress 'Validate manifests and live-skill reporting contract'
[[ -x $spacedock_fixture_script ]] ||
fail 'renamed Spacedock fixture builder is missing or not executable'
[[ ! -e $legacy_spacedock_fixture_script ]] ||
fail 'legacy make-capsule.sh path still exists'
require_output 'Usage: make-spacedock-fixtures.sh' \
"$spacedock_fixture_script" \
'fixture builder help still uses the legacy name'
require_output 'make-spacedock-fixtures:' "$spacedock_fixture_script" \
'fixture builder diagnostics still use the legacy name'
require_output 'FIXTURES OK' "$spacedock_fixture_script" \
'fixture builder does not produce the documented validation signal'
reject_output 'make-capsule' "$spacedock_fixture_script" \
'fixture builder still uses the legacy script name'
reject_output 'CAPSULE' "$spacedock_fixture_script" \
'fixture builder still uses the legacy validation name'
require_definition_at_first_use "$headless_skill" \
'headless skill does not define Spacedock fixtures at first use'
require_output '`make-spacedock-fixtures.sh`' "$headless_skill" \
'headless skill does not name the fixture builder'
require_definition_at_first_use "$skill" \
'live skill does not define Spacedock fixtures at first use'
require_output '`make-spacedock-fixtures.sh`' "$skill" \
'live skill does not name the fixture builder'
require_output 'Hand-built files can contain state' "$spacedock_reference" \
'Spacedock reference does not name the hand-built state risk'
require_output 'Spacedock itself would never create' "$spacedock_reference" \
'Spacedock reference does not explain why hand-built fixtures are unsafe'
require_output 'FIXTURES OK' "$spacedock_reference" \
'Spacedock reference does not name the fixture validation signal'
reject_output 'capsule' "$headless_skill" \
'headless skill still uses the unexplained capsule term'
reject_output 'capsule' "$skill" \
'live skill still uses the unexplained capsule term'
reject_output 'capsule' "$spacedock_reference" \
'Spacedock reference still uses the unexplained capsule term'
require_fixed() { grep -qF -- "$1" "$skill" || fail "$2"; }
require_output 'Run it as soon as the task is known.' "$headless_skill" \
'headless skill does not start the default run immediately'
Expand Down