You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
GitChameleon evidence run: rac-grounding arm (funded run over the scaffold) #10
The gitchameleon/ subdir scaffolds an external evidence run over GitChameleon 2.0 (arXiv:2507.12367; dataset cabbage972/GitChameleon-2.0, MIT; upstream code Apache-2.0): does grounding an agent in the recorded version-pin decision make its code target the pinned API? Arms per the DG-ADR-0001 single-variable pattern (no_grounding / rac / naive_rag); the corpus builder turns each problem into a RAC version-pin decision among deterministic distractors; the dry-run bundle assembly is built and tested offline.
Remaining (funded):
Pin the answering model and the naive_rag embedder (the scaffold refuses to run naive_rag until pinned).
Feed the dry-run bundles to the answering model; collect solutions per (example, arm).
Score all arms with the upstream harness (evaluate --solution-path …) — executable tests, deterministic; we add no scorer.
Publish per-arm pass rates with the dataset revision pin, model pin, and reproduction commands. Results are an arm comparison under our protocol (prompts do not restate the pinned version — GCB-ADR-0001), NOT comparable to the upstream leaderboard, and an unfavourable delta is published plainly.
Evidence run only — upstream scoring never enters a CI merge gate (rac-core ADR-066/ADR-097; roadmap external-benchmark-evidence).
Implementation progress is in #27: the Voyage baseline, immutable dataset/scorer pins, resumable answer generation, injection hashes, and paired-completeness checks are now built and locally validated. The issue remains open deliberately: no result exists until owner-supplied Anthropic/Voyage credentials produce all 984 answers and the pinned upstream Docker harness scores them.
Activity
docs(roadmap): link the gitchameleon scaffold ticket [roadmap:externa…
docs(roadmap): link the gitchameleon scaffold ticket [roadmap:externa…
tcballard commentedon Aug 2, 2026
Implementation progress is in #27: the Voyage baseline, immutable dataset/scorer pins, resumable answer generation, injection hashes, and paired-completeness checks are now built and locally validated. The issue remains open deliberately: no result exists until owner-supplied Anthropic/Voyage credentials produce all 984 answers and the pinned upstream Docker harness scores them.