This is the durable method and mechanics for using the benchmark to improve GitButler's agent experience: but output, the skill bundle, and the harness that measures them. A session brief (for example an overnight prompt) should supply objectives, current state, budget and authorization, and link here for everything else. Keep dated state out of this guide.
The loop: find an evidence-grounded hypothesis, preserve and replay the triggering state, name the behavioral evidence a fix would produce, change one thing, run a controlled trial with independent review before and after, then accept, reject or retain an isolated candidate for further work. Use the wider Git/But/jj matrix to understand coverage and tool differences; use paired But trials to evaluate a proposed change.
Start from something observable: a command trace, captured command output, an error message, a usage pattern, or a contradiction inside the skill bundle. A hypothesis says what the agent did, what it saw at that moment, and what different output or guidance would have changed its next command. Hypotheses built from oracle vocabulary or fixture specifics are overfitting, not improvements.
Read inefficient successes. A passing run is not a clean run. Look at traces that used more model calls or VC commands than the native reference: help and reference detours, repeated status queries, picking at the tip and then moving a commit, tidying an empty duplicate commit, verifying the same fact twice. Product problems have hidden inside green runs more than once. Correctness, trace quality and oracle quality are three separate reviews.
Rank by avoidable work, not failure rate alone. A frequent workflow that succeeds only after extra lookups, retries or confirmation can matter more than a rare error on a rarely used surface. Weigh usage evidence and benchmark traces together. When acting on older usage data, confirm the surface still exists and behaves the same in current source before opening a cycle.
Infer the misunderstanding, and label it as inference. No adapter guarantees a reasoning transcript. agent-events.json holds normalized tool calls, results and the final message; command-output/ holds what each VC command printed; stderr and byte counts are not substitutes for a reasoning transcript. From that evidence, state what the agent appears to have believed and which output or sentence supported the belief. Then make the tool state the correcting fact at that moment, preferring CLI output at the point of use, then the skill recipe, then reference documents. Printing the exact next command usually beats orientation prose.
Attribute failures with evidence. A deterministic failure and an intermittent one can each come from the product, the harness or the environment. Check the native reference and controls, the run's run_failure_class, the guard trace, and the environment (auth expiry, PATH, rate limits) before deciding. "Always fails" does not mean harness, and "sometimes fails" does not mean agent.
Retire steering that agents ignore. Repeated non-adoption despite confirmed reads is a reason to reconsider the intervention, rather than keep strengthening the wording. One stochastic miss is not enough to declare the guidance ineffective.
Preserve a deviating run before launching anything else. Its run directory already keeps the workspace with the installed skill, but-app-data, command-output/, agent-events.json, the trace and result.json. Never reuse that directory as --out for another run: the runner deletes and recreates it.
Hashes in result.json, conditions.json and plan.json identify the recorded inputs; they do not archive them, and they do not show that an agent read or followed anything. If the binary is a local build that will be rebuilt, copy the executable and skill directory to a content-addressed location and note the hashes. Write a short replay note: task and variant, run directory, binary path and hash, skill hash, model and effort, exact commands, and what deviated.
A completed run directory contains the final state, which may differ from the state that triggered the problem. Recreate the starting fixture with the same task, variant and frozen inputs, then replay the observed command prefix up to the deviation. Alternatively use a snapshot taken before the problematic command. For any task, the runner can prepare a fresh starting state without launching an agent:
npm run pilot:agent -- --task <id> --arm but+skill --but-bin /path/to/frozen/but \
--skill-dir /path/to/frozen-skill --version-pins pins.json \
--out /path/to/new-run --prepare-only trueFor subsequent native But commands, set E2E_TEST_APP_DATA_DIR to the new run’s but-app-data directory. Keep the fixture’s remotes and other path-dependent state disposable too; a copied repository may still point at the original local remote. Use the preserved executable directly, not an old command wrapper that may point at a rebuilt binary. A native replay can establish an output or repository-state fix; proving a change in agent decisions still needs agent trials. The September printed-ID finding was confirmed with native probes on fresh fixtures.
Name the behavioral evidence before the trial. Write down what the candidate should change in the trace, for example "goes from pull and status straight to pick --below without a references read" or "no empty duplicate commit after branch update". Judge the trial primarily by whether that appears; aggregate deltas come second. For deterministic output changes, measure the bytes before and after in a fixture first. Smaller output that changes no behavior is neutral, not a win.
npm run runner:check
npm run scenario:check
npm run scenario:check-runnerThese checks exercise the harness, reject deliberately wrong outcomes, and run native solutions for all seven expansion tasks across all three tools; npm run pilot:check covers the first original task and npm run pilotN:check covers tasks 2–6. Passing a native solution proves feasibility, not that the starting state is fair or that an agent can discover the solution. Starting-state controls cover the scoped-review fixture separately.
Select Git through the launch environment's PATH; the runner resolves it once, probes that absolute file, and uses it for both fixture setup and guarded agent commands. git-binary.json and result.git_binary record its path, version and hash. Matrix and paired input fingerprints include that Git file, so changing its selection or bytes requires a new output directory.
Use explicit executable paths, versioned model IDs and explicit effort. --codex-effort-level medium or --claude-effort-level medium records the selected reasoning setting. Omitted Codex effort preserves the CLI default and is recorded as unspecified; the default isolated Claude configuration explicitly uses and records medium. The runner's model defaults live in scripts/run-pilot-agent.mjs and can drift from what any published study used, so state the models in the plan or on the command line. Configured and observed model remain separate; Codex currently does not report an observed model ID.
The runner extracts the default GitButler skill from the selected binary, which avoids pairing a newly edited source skill with an older installed CLI. --skill-dir is an explicit experimental override; its version must be the source placeholder 0.0.0 or match the binary. Every run records hashes of the selected binary and skill content.
The default version pins remain in force. To test another binary, pass --version-pins pins.json with the changed expectation, for example {"but":"but 0.5.2189"}. The profile is an explicit assertion, not a request to skip checks. Only tools required by the selected arm must be installed. Use the exact output format accepted by the runner's recorded versions.json.
Change one thing per trial. A binary-only hypothesis needs byte-identical skill content in both conditions; if the skill changes too, label the result a bundle comparison. Keep separable fixes in separate experiments so each effect stays attributable.
After the trial, check adoption in the visible tool events and captured output: did the read include the changed passage, and did the next actions follow it? Read paths and skill_reference_output_bytes help locate evidence but do not prove a particular passage was read. Implicit skill loading is not fully observed; missing visibility means unknown, not ignored. Installed hashes prove presence, not adoption. Record confirmed non-adoption as well as successes.
After any skill change, read SKILL.md, its references and the relevant CLI help or output together and look for contradictions; a stale reference sentence can override a new recipe. Examples must not model anti-patterns. A CLI replacement for guidance must deliver the fact when the agent needs it: inspect the output and test that route. Otherwise keep the guidance, move it to references, or test the deletion with a probe task.
Add controls proportionate to the change: a small unaffected task when adding general skill text, the other agent, the opposite semantic variant for review changes, and a second fixture shape when the behavior depends on topology. More repetitions on one fixture do not substitute for shape coverage.
comparison:run accepts only the seven expansion tasks (pilot-7 through pilot-13, listed in scripts/lib/scenarios.mjs). For the original six tasks, see the next section.
Copy the baseline skill from the baseline binary, then make a separate candidate directory. Change either the skill or the binary, with the corresponding skill provenance documented. Do not edit either condition while a batch runs.
/path/to/but skill install --path /path/to/experiment/baseline-skill
npm run comparison:run -- --plan /path/to/experiment/plan.json --out /path/to/experiment/runsExample plan (paths resolve relative to the plan file):
{
"task": "pilot-11-dual-source-sync",
"agent": "codex",
"model": "gpt-5.6-sol",
"effort_level": "medium",
"k": 5,
"agent_timeout_ms": 180000,
"trial_timeout_ms": 300000,
"budget_ms": 3600000,
"baseline": {
"label": "Baseline",
"but_bin": "/path/to/baseline/but",
"skill_dir": "baseline-skill"
},
"candidate": {
"label": "Candidate",
"but_bin": "/path/to/candidate/but",
"skill_dir": "candidate-skill",
"version_pins": "candidate-pins.json"
}
}The driver alternates baseline/candidate and candidate/baseline order, bounds the complete subprocess tree, preserves partial attempts, and checks source, task, Git, binary and skill hashes before and after each trial. Repeating the same command resumes untouched cells; it does not silently rerun an existing failed or interrupted attempt. The overall deadline starts with the original plan and does not reset on resume. Changed inputs require a new output directory.
Inspect report.md and ledger.json. A useful candidate preserves successful execution, state correctness and policy compliance while reducing the intended effort or time. Paired efficiency deltas include only pairs with two eligible runs and observed values. Read the traces: fewer commands can simply mean more commands batched in each shell call, and a short failed attempt is not an improvement.
For review-answer changes, also exercise the opposite semantic case: npm run pilot:agent -- --task pilot-7-scoped-review --variant equivalent --agent codex --model gpt-5.6-sol --codex-effort-level medium --arm git. The default review variant is changed; retain the variant from scenario-metadata.json and report them separately.
Treat a handful of pairs as a diagnostic experiment, not a population-level claim.
The paired driver does not support pilot-1 through pilot-6. The usable path is two matrix batches whose conditions differ only in the one change, run back to back:
npm run matrix:run -- --tasks pilot-5-squash-commits --agents codex --arms but+skill --k 5 \
--codex-model gpt-5.6-sol --codex-effort-level medium --build-but false \
--but-bin /path/to/baseline/but --skill-dir /path/to/baseline-skill \
--version-pins baseline-pins.json --batch-name squash-baseline
npm run matrix:run -- --tasks pilot-5-squash-commits --agents codex --arms but+skill --k 5 \
--codex-model gpt-5.6-sol --codex-effort-level medium --build-but false \
--but-bin /path/to/candidate/but --skill-dir /path/to/candidate-skill \
--version-pins candidate-pins.json --batch-name squash-candidatePass --skill-dir explicitly in both batches so the skill content is a deliberate input rather than whatever the binary embeds, and give each batch a pins profile matching its binary's reported version. Each batch freezes its own conditions and hashes in conditions.json. What this path lacks compared with the paired driver: no alternating order within one plan, no cross-condition input check, no pair table and no automatic pair exclusion. Compare aggregate.json and performance/report.json per batch yourself, apply the same eligibility rule (execution, state and policy all passed) to both, and say in the writeup that the comparison was sequential batches. Small alternating batches reduce time-of-day drift at the cost of more setup.
For a balanced subset use matrix:run with --tasks, --agents, --arms, --k, explicit model/effort and binary paths, --build-but false, and --rotate-arms true. Keep pilot 8 (cold adoption) in a separate batch from warm tasks; the runner refuses to mix them. --timeout-ms bounds the agent; --trial-timeout-ms bounds setup, agent and verification together. --help prints usage and exits without writing anything; --dry-run true writes the plan and conditions into the batch directory without building or running agents.
Both drivers rewrite reports as trials finish. Use live reports for progress, but make acceptance claims only after the batch stops; account explicitly for missing or interrupted attempts.
The matrix records its conditions and source, binary, explicit skill and pin-profile hashes in conditions.json. Resume rejects changed inputs before launching another trial and reuses its frozen build. Existing directories without this record are not silently upgraded; use a fresh output directory for new conditions. Keep explicit skill directories frozen when comparing tools.
The matrix retains supervisor receipts when a child times out or fails without a result, then continues. ENV_FAILURE, VERIFIER_ERROR and supervisor failures are infrastructure outcomes, excluded from tool success denominators and efficiency comparisons. The planned count remains visible so missing trials cannot masquerade as all-repetition success. agent-result.json is saved before verification, preserving completed agent work if the oracle crashes.
Do not repair old grades in place. Preserve the original run, validate a proposed fixture/guard/oracle fix with controls, and use a fresh batch for the comparison. When diagnosing a past outcome, label any re-evaluation and retain both original and corrected evidence.
Review twice, with a reviewer that does not share your context (a fresh session or a shelled-out agent). Before the trial, give it the diff and repository paths only, with no results, and ask whether the change helps real agents across repositories unlike the fixtures or pattern-matches the fixtures and oracle. Never tune toward oracle vocabulary; for example, the pilot-1 oracle accepts only four commit-message terms, and teaching them would be cheating, not guidance.
After the trial, give the reviewer the diff and the run directories and require it to recompute from result.json, ledger.json or aggregate.json rather than from your summary. Reviewers are independent, not automatically right: validate any semantic claim against source and concrete repository states before accepting or rejecting it. If no independent reviewer is available, say so and mark the change unreviewed.
Keep a change when execution, state and policy are preserved, the named behavioral evidence appears, and the control task and other agent show no regression beyond their own spread. Reject changes that fail those checks. Inconclusive candidates can remain in isolated snapshots for further work; keep them out of the accepted baseline and label the uncertainty. Preserve rejected attempts and their evidence. Record the hypothesis, evidence, conditions and hashes, the behavioral signature, and the decision in the study directory, and write a results document for anything that changes guidance or ships.
Warm and cold describe repository adoption, not model prompt caching. Each trial starts an isolated agent session; provider caching is observed where exposed and is not reset. These results therefore include discovery in a fresh agent context.
These tasks measure agents completing small, prepared version-control workflows. Much of wall time can be model/tool-call overhead while native VC execution is well under a second. They can expose discovery, output and workflow problems; they do not replace large-repository CLI microbenchmarks.
State, policy, execution, effort and time are separate dimensions. Raw Git writes, including fetch, remain blocked in But/jj arms; tool-internal Git is allowed. A policy-only failure with correct state is different from damaged history. Explicit skill-output bytes include the initial skill read and later reference reads; they are not a measurement of all context tokens. Missing counters stay unknown. Field definitions and coverage limits are in performance measurement.
The expansion was selected from observed GitButler usage and corroborating event patterns, including narrow but consequential workflows. It is not a random sample of all version-control users, and the seven tasks have no defensible population weights. Keep per-task results prominent.
For an end-to-end example, see the September expansion study: failed diagnostic attempts, a corrected comparison, a focused guidance experiment and a negative transfer probe are kept distinct.