feat(harness): add GitHub Copilot CLI support - #3588
Conversation
72453ef to
9434ecc
Compare
9434ecc to
2bbe77e
Compare
2bbe77e to
ca6d986
Compare
|
Speaking as Kun's firstmate: contract-class new-default (not pure opt-in). Copilot as a verified harness is opt-in when the captain runs Copilot, but this PR also (1) rewires tracked VISION: fleet-outlives-vendor aligns for the Copilot adapter itself; Authority/consent does not for the Claude hook + detection reorder. Security FYI (not a captain flag while blocked): crew Copilot launch uses Waiting on: author — complete no-mistakes |
|
Addressed the maintainer feedback and CI regressions at head 1d734e8.
The PR attestation now records test status completed. GitHub has held the fresh fork workflows before job creation, so an upstream maintainer needs to approve these runs: |
bin/fm-copilot-hook.sh's notification handler checked the snake_case field notification_type, but Copilot CLI 1.0.83 GA's real shell_completed payload uses camelCase notificationType (confirmed live: captured payload keys are cwd, message, notificationType, sessionId, timestamp, title -- no notification_type anywhere). That mismatch meant copilot_notification_has_watcher_completion returned false unconditionally, for every real notification, regardless of title, command, or shellId shape. This is very likely the same mechanism behind the notification/resume discrepancy the earlier scout reported against 1.0.83 (they tested through the -p proxy; this fix and its live verification used the real interactive -i primary path). Also relaxed the title-matcher's shellId pattern from digits-only ([0-9]+) to any non-delimiter run ([^)]+ / [^"]+): Copilot lets the model name its own async shell task's shellId freely, and a live capture showed a real session choosing a descriptive string, not a small integer. The digits-only assumption would have kept failing even with the field-name fix applied. Verified live end to end in an isolated lab: launched a real interactive Copilot primary (COPILOT_HOME isolated, GH_TOKEN from `gh auth token`), armed the watch-arm script as its own attached async task, injected an external wake via bin/fm-inbox.sh note, and confirmed the async task's completion resumed the model. With only the pre-fix code, the resumption happened through Copilot's own native shell-completion surfacing (the model noticed on its own and ran read_bash), not through this hook's enriched "FIRSTMATE WATCHER WAKE ... run the wake drain first" follow-up, because the match silently failed. Re-tested the fixed matcher directly against the real captured payload (this commit's tests) and confirmed a match. docs/supervision-protocols/copilot.md now tells the primary to name the async watch-arm task exactly "Arm Firstmate watcher", the exact phrase the fallback title matcher requires -- nothing before this told the model to produce that specific string, so the fallback path depended entirely on coincidence. tests/fm-copilot-harness.test.sh's notification fixtures all used the same wrong notification_type key as the code they were validating, so they exercised a payload shape Copilot never actually sends; updated all of them to notificationType, and changed the two "live shape" fixtures' shellId from small integers to realistic model-chosen strings so the relaxed pattern is actually exercised. Not specific to this fork's carry of PR kunchenguid#3588 beyond being part of the Copilot adapter itself; suitable to offer upstream as-is.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 53b6302f-b135-4862-8f96-3e39bbd03803
* feat(harness): verify gemini as a crewmate runtime adapter Adds Gemini CLI as a fourth dispatch target alongside claude, codex, and grok, scoped to crewmate and scout work only. Every axis was proven against gemini-cli 0.58.0 rather than inferred; docs/verification/runtime-backends.md carries the dated evidence and names what stayed unverified. Busy state is semantic, not rendered: BeforeAgent opens a turn and AfterAgent and SessionEnd close it. AfterAgent also fires on a manual interrupt, so a cancelled turn closes its own record. Three findings shaped the wiring rather than a config line: - --skip-trust and GEMINI_CLI_TRUST_WORKSPACE=true are presented by the CLI as equivalents and are not. A controlled A/B showed --skip-trust leaves project configuration unloaded, so workspace skills never load. - The worktree's .gemini/settings.json is the PROJECT's committed settings file, unlike claude's settings.local.json. Firstmate's hooks therefore go to a firstmate-owned state/<id>.gemini-settings.json reached through GEMINI_CLI_SYSTEM_SETTINGS_PATH, which also works untrusted and merges with a project's own hooks instead of replacing them. - The shipped CLI is a node bundle whose live process reports comm as MainThread, so ancestry cannot see it. GEMINI_CLI=1 is load-bearing and is tested before an inherited CLAUDECODE, and pane liveness identifies gemini from the script argument through the new bin/fm-gemini-lib.sh. Gemini is refused for secondmates: it has no primary supervision protocol. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GYdQkKfSEQrFUxtcTXZ66L * test: clear gemini's marker in launch and detection expectations Every non-gemini launch now clears GEMINI_CLI the way it already clears cursor's markers, so the two tests that pin the exact launch prefix are updated to match. The harness-detection tests that scrub foreign markers before probing ancestry scrub GEMINI_CLI too, so running the suite from inside a gemini session cannot produce a false verdict. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GYdQkKfSEQrFUxtcTXZ66L * docs: classify the gemini harness reference The documentation inventory is the single classification owner for maintained prose surfaces, and every surface must appear in it exactly once. The new harness reference is agent-runtime, matching its siblings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GYdQkKfSEQrFUxtcTXZ66L * no-mistakes(review): Narrow Gemini ancestry detection * no-mistakes(review): Restrict Gemini hooks to canonical launches * no-mistakes(document): Document Gemini adapter support boundaries * no-mistakes(ci): Fixed Gemini process identity when interpreter or script paths contain whitespace. Tmux liveness now uses NUL-delimited /proc argv on Linux, with the existing flattened ps fallback elsewhere. Added a real-process regression test. Verified with the Gemini harness test suite, full fm-lint, ShellCheck, and git diff --check. The CI and Require no-mistakes runs were action_required/attestation outcomes rather than code failures --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 53b6302f-b135-4862-8f96-3e39bbd03803
Keep ancestry-first resolution scoped to Copilot marker conflicts, adopt the shared live-test gate, and refresh the remote doctor integrity binding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 53b6302f-b135-4862-8f96-3e39bbd03803
Drive the hook-owning interactive surface in the live guard and accept the two observed watcher task titles while retaining exact completion-message validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 53b6302f-b135-4862-8f96-3e39bbd03803
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 53b6302f-b135-4862-8f96-3e39bbd03803
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 53b6302f-b135-4862-8f96-3e39bbd03803
Keep ancestry-first resolution scoped to Copilot marker conflicts, adopt the shared live-test gate, and refresh the remote doctor integrity binding. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 53b6302f-b135-4862-8f96-3e39bbd03803 # Conflicts: # .agents/skills/harness-adapters/SKILL.md # .agents/skills/harness-adapters/references/harness/copilot.md # bin/fm-harness.sh # tests/fm-copilot-harness.test.sh # tests/fm-copilot-primary-live-e2e.test.sh # tests/fm-kimi-harness.test.sh
Settle parent sessions without undocumented IDs, require successful watcher receipts for notifications, and verify session-only trust before unattended workers continue. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 53b6302f-b135-4862-8f96-3e39bbd03803
1d734e8 to
56a83de
Compare
|
Rebased and re-ran the complete no-mistakes pipeline at head 56a83de.
GitHub has again held the new fork workflows before job creation. An upstream maintainer needs to approve: |
|
Speaking as Kun's firstmate: Re-triaged after author rebase to HEAD Contract-class: new-default (unchanged). Copilot as a verified harness stays opt-in when the captain runs Copilot, and marker-first is restored for non-Copilot runtimes (ancestry authoritative only for Copilot's inherited-marker conflict). But tracked VISION: Fleet-outlives-vendor aligns for the Copilot adapter; Authority/consent does not for the Claude hook rewrite. Scripts/agents, restart, spine, one-captain, scope — otherwise fine. Waiting on CI after approval, not on the captain. No captain card until otherwise ready except the default-behavior decision. Security FYI tip-only if Copilot launch still uses broad allow flags — not waiting-captain. |
Intent
Add comprehensive GitHub Copilot CLI support to Firstmate, incorporating lessons from prior attempts, preserving repository-owned Claude configuration and existing harness behavior, keep PR 3588 conflict-free on current upstream main, and deliver it with the full no-mistakes Test phase and all required validation passing.
What Changed
.github/hooks/fm-primary.json, Copilot-specific hook adapters, worker busy-state hooks, spawn/profile support, and remote secondmate allowlisting.agentStopdecisions, publishing single-use watcher completion receipts, and keeping tracked Claude/Copilot hook files repository-owned.Risk Assessment
Testing
I ran the live Copilot primary E2E against GitHub Copilot CLI 1.0.83-3 plus focused Copilot, watcher, session-lock, tmux-liveness, and spawn-profile regressions. The live hook flow passed for session-start context, watcher-arm denial, bounded agentStop continuation, watcher wake resume, and inert unrelated notifications; the only live scenario I could not complete was fm-spawn's wrapped-path trust/config-preservation path because this host lacks a working
treehouse getimplementation for entering an isolated task worktree.FM_COPILOT_LIVE_E2E=1 bash tests/fm-copilot-primary-live-e2e.test.sh->~/.no-mistakes/evidence/01M23TWB9F4ST2AV1EQ16H9Y0B/fm-copilot-primary-live-e2e.logbin/fm-watch-arm.shas its own attached async task and have the completion wake resume Copilot so it drains and acknowledges exactly onceFM_COPILOT_LIVE_E2E=1 bash tests/fm-copilot-primary-live-e2e.test.sh->~/.no-mistakes/evidence/01M23TWB9F4ST2AV1EQ16H9Y0B/fm-copilot-primary-live-e2e.logbin/fm-wake-drain.shFM_COPILOT_LIVE_E2E=1 bash tests/fm-copilot-primary-live-e2e.test.sh->~/.no-mistakes/evidence/01M23TWB9F4ST2AV1EQ16H9Y0B/fm-copilot-primary-live-e2e.log.github/hooks/fm-primary.jsonand.claude/settings.jsonuntouched while trust clearstreehouse gettool, whichbin/fm-spawn.shrequires to move the tmux pane from the spawning project into an isolated task worktree before Copilot launch. Instal…Evidence: Copilot primary live E2E log
ok - Copilot live hooks: denial, stop continuation, watcher wake, and inert unrelated notifications (GitHub Copilot CLI 1.0.83-3.)Evidence: Copilot harness regression log
Evidence: Watch-arm regression log
Evidence: Spawn dispatch profile regression log
Evidence: Live fm-spawn blocker note
Attempted live Copilot fm-spawn validation from an isolated temp project/home with real tmux and Copilot CLI. Blocked because this host does not provide a workingtreehouse getimplementation for fm-spawn's isolated-worktree step.Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 2 issues found → auto-fixed ✅
bin/fm-bootstrap.sh- merge conflict rebasing onto refs/remotes/no-mistakes-push/feat/copilot-cli-supportdocs/remote-secondmates.md- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Review** - 2 issues found → auto-fixed ✅
bin/fm-copilot-hook.sh:66- Command-bearing watcher completions are now rejected unlessfm_copilot_watch_receipt_claimsucceeds, butbin/fm-watch-arm.shexplicitly treats receipt publication as non-fatal. A successfulbin/fm-watch-arm.shrun whose receipt write fails will therefore never inject the Copilot wake follow-up, leaving supervision stalled even though the classifier positively recognized the watcher command.bin/fm-spawn.sh:2932-copilot_trust_dialog_exactrequires the full$WTpath to appear as one literal captured line. In real tmux/herdr panes, long worktree paths commonly wrap, so the verified default trust dialog is misclassified as unsafe and first-launch Copilot spawns abort before the worker starts.🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Test** - 2 issues found → auto-fixed ✅
bin/fm-copilot-hook.sh:206- Live Copilot primary validation failed on watcher wake resume. In a realcopilotsession, the trackednotificationhook ran, received the expectedshell_completedpayload, and emitted the expected{"additionalContext":...}watcher follow-up, but Copilot never resumed to runbin/fm-wake-drain.shor the required--ack-throughacknowledgement. That leaves Firstmate supervision stalled after a successful watcher completion even though the portable hook tests pass.FM_COPILOT_LIVE_E2E=1 tests/fm-copilot-primary-live-e2e.test.sh; artifactscopilot-primary-live.logandcopilot-debug2-live.txtcopilot-hook-live.logclaude-compat-copilot-live.logharness=copilotwitheffort=maxthrough the real bootstrap entrypoint without aCREW_DISPATCHerrorbootstrap-copilot-max-live.logharness=copilot, preserving the Copilot runtime across both operationsherdrruntime or a separate SSH-reachable secondmate home to drivefm-remote-secondmate-control.sh launch/relaunchend-to-end. Provide a configured Herdr s…treehouse, and safely driving a real worker spawn would also require a dedicated disposable spawn backend/session rather than the shared gate environment. Providetreehousepl…FM_COPILOT_LIVE_E2E=1 tests/fm-copilot-primary-live-e2e.test.sh > ~/.no-mistakes/evidence/01M23TWB9F4ST2AV1EQ16H9Y0B/copilot-primary-live.log 2>&1tests/fm-copilot-harness.test.shtests/fm-watch-arm.test.shtests/fm-spawn-dispatch-profile.test.shtests/fm-remote-secondmate-control.test.shFM_HOME=<tmp> FM_PROJECTS_OVERRIDE=<tmp>/projects FM_CONFIG_OVERRIDE=<tmp>/config FM_STATE_OVERRIDE=<tmp>/state FM_DATA_OVERRIDE=<tmp>/data FM_BOOTSTRAP_DETECT_ONLY=1 FM_BOOTSTRAP_NETWORK=skip ./bin/fm-bootstrap.sh(artifact:bootstrap-copilot-max-live.log)exec -a copilot /bin/bash ./bin/fm-copilot-hook.sh notification < <payload>against an isolated real git repo fixture with real receipt files (artifact:copilot-hook-live.log)exec -a claude /bin/bash ./bin/fm-claude-compat-hook.sh fm-turnend-guard.shandexec -a copilot /bin/bash ./bin/fm-claude-compat-hook.sh fm-turnend-guard.shagainst an isolated real git repo fixture (artifact:claude-compat-copilot-live.log)tmux -L <private-socket> ... copilot -i ...manual live repro capturing pane text plus.hook-notification-payload.jsonand.hook-notification-output.json(artifacts:copilot-debug-live.txt,copilot-debug2-live.txt)🔧 Fix applied.
✅ Re-checked - no issues remain.
FM_COPILOT_LIVE_E2E=1 bash tests/fm-copilot-primary-live-e2e.test.sh->~/.no-mistakes/evidence/01M23TWB9F4ST2AV1EQ16H9Y0B/fm-copilot-primary-live-e2e.logbin/fm-watch-arm.shas its own attached async task and have the completion wake resume Copilot so it drains and acknowledges exactly onceFM_COPILOT_LIVE_E2E=1 bash tests/fm-copilot-primary-live-e2e.test.sh->~/.no-mistakes/evidence/01M23TWB9F4ST2AV1EQ16H9Y0B/fm-copilot-primary-live-e2e.logbin/fm-wake-drain.shFM_COPILOT_LIVE_E2E=1 bash tests/fm-copilot-primary-live-e2e.test.sh->~/.no-mistakes/evidence/01M23TWB9F4ST2AV1EQ16H9Y0B/fm-copilot-primary-live-e2e.log.github/hooks/fm-primary.jsonand.claude/settings.jsonuntouched while trust clearstreehouse gettool, whichbin/fm-spawn.shrequires to move the tmux pane from the spawning project into an isolated task worktree before Copilot launch. Instal…bash tests/fm-copilot-harness.test.shbash tests/fm-watch-arm.test.shbash tests/fm-session-lock-ancestry.test.shbash tests/fm-tmux-agent-liveness.test.shbash tests/fm-spawn-dispatch-profile.test.shFM_COPILOT_LIVE_E2E=1 bash tests/fm-copilot-primary-live-e2e.test.shenv -u NO_MISTAKES_GATE ... bin/fm-spawn.sh live-copilot-trust <temp-project> --scout --harness copilotinside a private tmux server with real Copilot CLI and tmux (blocked before live spawn by missingtreehouse get)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.