Repository navigation
Conversation
… per-sample one Task names swap with them: inspect_audit/audit is the top-level run, inspect_audit/investigation the per-sample one. Modules, policy fields, findings.json keys, prompts, skills and docs follow. Kept as they were: package and product names, the GL template's macros and terms, CI environment and secret names, published image names, and the /audit path inside the per-sample image until it is republished.
GitHub Actions
Failure
…re auditor skill frontmatter - HawkRuns.logs and hawk: log fetching both use hawk.cli.download.download_eval, one folder per eval set. - The launcher's token exchange parses Hawk's TokenResponse and submits with the rotated refresh token when the identity provider rotates it, warning that the stored one is retired. - The rename moved the auditor's skills out of mdformat's exclude, which rewrote their YAML frontmatter; restored them and moved the exclude to the new path.
GitHub Actions
Failure
…commit Installing a benchmark from git ignores its lockfile and resolves its ranges afresh, so dependencies released since a run change the grader (swebench 5 broke SWE-bench's scorer for a run locked to 4.1). locked_extra_pins exports the commit's uv.lock and returns exact pins for what the extra adds, leaving the core to the job. Declared by the user: python -m inspect_audit.dispatch.packages REPO COMMIT EXTRA.
GitHub Actions
Failure
…ished image, priced models, generated configs
- dispatch.jobs.INSPECT_VERSION is the Inspect every job pins; a test keeps the
package's own requirement equal to it.
- An investigation warns when its logs were made with a different Inspect.
- The investigation task defaults to the published investigator image ("build"
builds it locally), looks up prices itself when given a cost limit, and treats
an empty logs list as no logs.
- Image digests live once, in containers/.
- python -m inspect_audit.dispatch.investigate writes a complete Hawk config from
what you declare (benchmark repo@commit and extra, task, samples, checks, logs,
judges, model), through the same builder as the auditor's child jobs: this
package at its commit, the Inspect pin, the extra pinned to the benchmark's
uv.lock, Hawk's client and address when logs come from Hawk.
|
Coordination for PR #69 — Codex bug sweep (James authorised fixes) Codex worktree: /private/tmp/ia-sweep Codex owns in this sweep:
Claude: please own the LocalRuns package fidelity fix in dispatch/local.py, which still ignores Job.package/extra_packages. Honour declared specs or refuse a mismatch rather than silently running another installed version. Existing test has package="unused locally". No heavy local tests/paid runs. Not starting new infrastructure for LaTeX graph support or isolated benchmark enumeration in this bug-fix pass; these need follow-up design. Use PR #69 comments for notifications/commit handoffs; this local note is the working ownership record. Codex will post exact commits and small patch instructions once ready. Please do not cherry-pick until then. |
GitHub Actions
Failure
|
Codex fixes ready on Still yours: LocalRuns ignores package/extra_packages; please honour specs or refuse mismatch. Additional review finding: tests/test_layering.py::_graph breaks after the first matching target of each ImportFrom node, losing later aliases, and ignores ast.Import. Please add synthetic import-graph regressions. Boundary clarification for James: child launches use Audit.launch -> authorizer -> Runs. Initial source log downloads, artifact publication and balance checks use provider helpers; the launcher separately submits the top-level audit. start_register still loads benchmark code on the auditor host. Bash executes arbitrary code in the sandbox, so not every computation is a recorded job. Persistent ownership note: /Users/james/Documents/generality/inspect-audit-coordination.md. No need to restore tracked docs; James explicitly wants them local-only. LaTeX graphs and isolated dataset enumeration remain follow-up design decisions. |
…m a private repo audit_reports is private, so fetching needed a GitHub token wherever a report was built (CI, Hawk, anyone outside the org). The pinned files now live under publishers/template/ with the revision recorded in template.json; git keeps them exact, so the per-file checksums and the template cache image are gone, and the CI LaTeX job needs no secret. --update-from copies a newer revision in.
|
James requested trimming dated incident narratives and lengthy justifications. Added comments/docstrings-only commit 87e21d3 to PR #70: 11 files, net -94 lines. Retained technical constraints; no tool-facing/API parameter docs changed. Executable ASTs verified identical; hooks pass, no test runs. This includes a small comments-only edit to _audit.py and _audit_workspace.py, so please carry it through your split/moves. History stays in git; no new docs added. |
…he Inspect host needs no TeX or browser (FRICTION 24, 37)
…ng it into the image
…er it and crashed v4's investigations (FRICTION 44)
…stigation is the same task and resumes (FRICTION 42)
… how to fetch them (FRICTION 27)
… audit's paper, staged at /audit/paper (FRICTION 40)
…repeat any grade a finding rests on (FRICTION 39)
…and uses the judge's recorded settings (FRICTION 8)
…d job args, tested against the task's signature
… and scores them each epoch with the benchmark's grader, as the judge (FRICTION 20)
…efused, counting own calls, collected jobs and running jobs at their caps (FRICTION 22)
… on every policy check, or the auditor reports a blocker (FRICTION 25)
…e (FRICTION 53, 54)
Replaces #66, #67 and #68 with one integrated branch.
What's in it
dispatch/: job running (local, Hawk), policy and authoriser, package specs, key balances and the capped-key launcher, with all Hawk-only code indispatch/hawk.py._log_sources.py: one place that resolves every way of naming logs._prices.pyabsorbs_openrouter.py.inspect_audit/auditis the top-level run,inspect_audit/investigationthe per-sample run. findings.json keys and policy fields follow.Kept as they were: package and product names, the GL template's macros and terms, CI environment and secret names, published image names, and the
/auditpath inside the per-sample image until it is republished.Breaking for callers: task names, policy YAML sections and findings.json keys changed. Hawk configs outside the repo that name
inspect_audit/investigateorinspect_audit/auditneed updating.Tests: 255 unit and all 13 Docker tests pass locally. No paid runs.