Daniel Guirao · Independent research project · August 2026
Status: portfolio version 1.1.0 complete · v1.0.0 frozen numerical release · current status
Scale: 20,000 trajectories · 99,593 fixed-prefix rows · 25 frozen core models
Compute: frozen workflow on Kaggle CPU · no-refit supplement on local CPU · $0 paid compute
Research question: Which model-agnostic behavioral telemetry signals predict eventual coding-agent success early in a run, how much value do they add beyond coarse categorical covariates (model family, scaffold, language, and issue category), and how well do those signals generalize across agent domains?
Coding-agent evaluation usually ends with one number: whether the final patch resolves the task. AgentTrace studies the process information hidden behind that number—exploration, edits, tests, repetition, errors, recovery, and validation—using cheap structural telemetry at fixed online checkpoints.
The contribution is a leakage-safe study of incremental predictive value, calibration, low-FPR utility, and transportability. It is not a claim to introduce early failure prediction or a production-ready stopping policy.
- Telemetry adds little beyond the four static covariates at K=3–10, but pointwise paired task-cluster intervals exclude zero for several metrics at K=20–30. The precise emergence checkpoint is descriptive, not familywise-confirmed.
- At K=30, combined logistic reaches ROC-AUC 0.629 and failure PR-AUC 0.705; HGB reaches 0.644 and 0.718, a modest nonlinear advantage.
- K=30 HGB catches only 13.8% of failures near a 5% realized FPR and 22.9% near a 10% realized FPR.
- Held-out-scaffold ROC-AUC at K=10 falls to 0.547–0.576. A post-release exploratory paired contrast finds weaker scaffold-transfer ranking; its intervals are pointwise and unadjusted across supplemental comparisons.
- Grouped-label permutation and arbitrary-hash controls remain near chance (ROC-AUC 0.513 and 0.495), while a deliberately non-deployable future-information oracle reaches 0.704.
Main conclusion: pointwise evidence for moderate incremental value appears at K=20–30, while the precise emergence checkpoint remains descriptive. Low-FPR sensitivity and cross-scaffold transportability are limiting.
| Validation layer | What it established |
|---|---|
| Frozen design | Fixed K prefixes prevent final-length leakage; instance_id-grouped splits keep repeated task attempts together; calibration and thresholds use validation data only. |
| Stage 1 — Evaluation | Task-clustered intervals, calibration, subgroups, low-FPR recall, retrospective lead time, and error cases bound uncertainty and operational utility. |
| Stage 2 — Transfer | Held-out model, scaffold, and domain tests identify where the signal fails to transport. |
| Stages 3 and 5 — Interpretation | Ablation retrains without a feature family to test compensation; permutation importance scrambles a family for fixed HGB to measure current predictive dependence. Neither is causal. |
| Stage 4 — Controls | Grouped-label permutation and arbitrary hash test noise rejection; a deterministic fixture tests future-mutation invariance; a diagnostic leakage oracle tests evaluation sensitivity. The oracle is non-deployable. |
| Stage 6 — Confirmation | Interactions, confounding adjustment, nonlinearity, GEE, and multiplicity correction test pre-specified associations under alternative specifications. |
| Stage 7 — Aggregation | Ten compact tables, seven figures, and machine-readable audit metadata make the principal evidence reviewable without raw or per-fit artifacts. |
| Release supplement | Adds an exploratory K=10 hypothesis summary and pointwise clustered transfer contrasts from frozen outputs; no model was refit. |
Failure PR-AUC is prevalence-sensitive and is read alongside ROC-AUC, target prevalence, Brier score, and log loss; lower Brier score and log loss are better. Permutation importance is predictive, not causal. Retrospective lead time is descriptive context, not a deployable feature.
- Presentation notebook — an output-free walkthrough whose seven figures render directly on GitHub.
- Final research report — the complete conventional research narrative and frozen-core-to-completion workflow.
- Claim audit — principal numerical statements mapped to committed aggregate evidence.
- Study scope and deviations — what was planned, what was executed, and how each deviation limits the claims.
- Frozen evidence supplement — exploratory K=10 associations and pointwise task-cluster transfer intervals derived without refitting.
- Portfolio case study and five-minute interview walkthrough.
- Seven SVG figures and ten compact tables.
For results-only review:
git clone https://github.com/BaconMan1168/agent-trace.git
cd agent-trace
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-results.txt
jupyter lab notebooks/01_results.ipynbThe notebook reads only the committed compact aggregate release, release supplement, and status overlay. It performs no ETL, fitting, bootstrap, permutation, or new analysis. Execution is optional because its narrative and figures render on GitHub without saved outputs.
Verify all 22 committed historical and supplemental release outputs:
python -m src.reporting.verify_releaseComputational replication is a separate Kaggle CPU workflow using the
pinned Open-SWE-Traces revision,
sampling seed 20260813, and split seed 131313. Read the
compute contract,
core runbook,
frozen-boundary addendum, and
completion runbook. A clean clone
can inspect and verify the committed results. Byte-identical derivation from the
repository alone is not claimed because raw data, frozen intermediates,
predictions, fitted objects, and checkpoint ZIPs are not public.
- The corpus contains benchmark-style coding-agent trajectories, not human-in-the-loop production sessions.
- Only two model families and two scaffolds are represented.
- Parser outputs inherit the limits of the source data and deterministic event taxonomy.
- Failure PR-AUC cannot be compared across domains in isolation because failure prevalence changes.
- Behavioral associations, ablations, and permutation importance are not causal effects.
- The static baseline contains only model family, scaffold, language, and issue category; it is not a comprehensive task-difficulty model.
- Bootstrap intervals are conditional on one frozen sample and group split.
- Low-FPR recall is too limited for autonomous early termination without new target-domain validation and an explicit cost model.
The frozen core is anchored at commit 2779fa2. The core and Stages 1–7 ran on
free Kaggle CPU; the small no-refit release supplement ran locally on CPU from
frozen outputs. Total paid compute remained $0, with no paid API, LLM inference,
neural model, or locally generated trajectories. Raw data, prefix tables,
predictions, fitted models, per-fit runs, and ZIP checkpoints remain untracked.
See INDEX.md for the complete documentation map and
CITATIONS.md for research positioning and sources.
The immutable Stage 7 status JSON records four gates that were still pending when
that archive was generated. RELEASE_STATUS.json records
their later completion without rewriting the historical manifest.
The public portfolio signal is the complete research pipeline:
nested traces → deterministic event normalization → leakage-safe prefixes → vectorized telemetry → clustered inference → calibrated grouped ML → domain-shift analysis → interpretable failure analysis.