CI: benchmark PRs through lightcone-bench

[dkn16]

Adds three small workflows that hand a PR, or a merge to main, to lightcone-bench, where this repo's own eval (evals/prompt.md + evals/tasks/snae) runs through Harbor as the task lightcone-cli/snae-build. Nothing runs in this repo; each file is one API call.

What they do

• benchmark-on-pr.yml — every non-draft PR is benchmarked automatically against its head commit: on open, on each push, on reopen, and when a draft is marked ready. Fork PRs are skipped (they don't receive this repo's secrets).
• benchmark-on-comment.yml — a maintainer (OWNER / MEMBER / COLLABORATOR) comments /benchmark on a PR for the same run on demand: for a draft PR, or a re-run. Reacts 🚀 as the ack.
• benchmark-baseline.yml — every push to main dispatches the same run with save_baseline, so the bench's stored "current main" numbers track this repo.

All three pass use_skills, so the agent runs with the skills from agent-skills main installed. The skills axis is held fixed here; varying it is the agent-skills repo's own job.

What a run does

In lightcone-bench: the task's image is built with this repo pinned to the requested sha, the oracle gate runs the scripted reference solution and must score 1.0 (no LLM, so a PR that breaks the CLI contract fails here, visibly), then each agent leg runs the task and the report shows per-task rewards, the delta against the stored main baseline, and a trace analysis. Today the opencode leg runs; claude-code and codex join once their API keys are set in lightcone-bench (they skip with a notice until then). Results live in the dispatched run's job summary under lightcone-bench's Actions tab; posting them back onto the PR is a planned follow-up.

Setup (one secret)

BENCH_DISPATCH_TOKEN in this repo's Actions secrets: a fine-grained PAT (or App token) with repository access to LightconeResearch/lightcone-bench and Actions: read and write. The built-in GITHUB_TOKEN cannot dispatch into another repository.

Relation to eval.yml

eval.yml runs Claude Code on every PR inside this repo; this is the same task run by the bench, across agents, with an oracle gate and stored baselines. They don't interact. Whether the bench replaces eval.yml is a separate decision.

Testing

benchmark-on-pr.yml runs from the PR's own branch, so once the secret exists, this PR benchmarks itself on its next push. The /benchmark comment and the baseline trigger only take effect after merge: GitHub serves issue_comment and push workflows from main. The bench side was validated by dispatching it directly with this repo's current main sha, which is exactly what the senders send: oracle 1.0, opencode leg ran, report produced. The first runs will show NEW — no baseline for the skills variant until the first merge to main refreshes it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01VcKmGNUek7oWEPv8zLqo9Q

[github-actions]

✅ Eval

Metric	Value
Outputs check	success
Agent run	success
Turns	8
Tool calls	6
Cost	$0.13
Agent wall time	0m37s
Model	claude-sonnet-5-5

lc status

  mode:    direct
  sandbox: landlock (fs: declared, network: allowed)
  crate:   up to date with the outputs

  · current  baseline/best_fit        a81fe43
  · current  baseline/hubble_diagram  a81fe43
  · current  baseline/residuals       a81fe43

3 current

Confusion & pain points (Claude analysis)

Confusion & pain points

• The run was essentially clean. There were no errored tool calls, and the agent finished in 6 tool calls. The agent also never read astra.yaml field docs or the lc source; it relied on the skill and the scaffolded spec. What friction existed was small and is listed below.

• The astra CLI was invoked through uvx astra-tools@0.2.18 validate. The agent pinned a version and went through uvx, which suggests astra wasn't on PATH, or that the agent didn't trust whatever version was installed. The harness should provide astra on PATH at the version the spec targets. If it already does, the skill should say so, so agents don't guess a pin.

• The agent added license = "CC-BY-4.0" to pyproject.toml with sed, unprompted. Its summary says the choice was its own. The crate is only maintained when [project].license is set, and the agent evidently knew that (likely from the task or skill). It then picked a license on the user's behalf, which the project's own rule against asserting terms over someone's data warns against. A task that expects a crate should name the license. lc init or lc materialize could also say once, in plain text, that the crate needs [project].license.

• The agent filled the spec in with a blind str.replace Python heredoc. It never read astra.yaml back or diffed it. The edits were string-matched against the scaffold (recipe:\n command: python scripts/fit.py), so any mismatch would have silently changed nothing. Validation passed, so it worked here, but the approach is brittle. Neither astra nor lc offers a way to fill in an output's format, inputs, decisions and recipe fields programmatically. A small astra edit verb, or a documented minimal output stanza, would remove the need for this.

• There was no pre-flight in the agent's own ordering for the compute allocation. The agent had to know that lc compute launch --wait comes first and that lc materialize local takes the cluster name as its first positional argument. It got this right, but the sequence (launch, materialize, down) is easy to miss. lc materialize without a cluster should refuse with a message naming lc compute launch, and the skill or the task prompt should show the three-step sequence.

Full trace: agent-trace artifact on this run.

[github-actions]

lightcone-bench — benchmark of 36a0be2 ✅ passed: run 37077360445 — the matrix, baseline deltas and trace report are in its job summary.