Consolidate dispatch, merge findings handoff, and swap role names

.copier-answers.yml

@@ -6,7 +6,7 @@ author_email: james.mann@generality.org
 author_name: James Mann
 github_owner: Generality-Labs
 license: MIT
-project_description: Benchmark validity auditing for Inspect AI evals
+project_description: Benchmark validity investigating for Inspect AI evals
 project_kind: library
 project_name: inspect_audit
 publish_to_pypi: false

.github/workflows/ci.yml

@@ -29,7 +29,7 @@ jobs:
   # One real-container lifecycle gate on PRs; the complete Docker suite on main.
   docker:
     name: Docker tests
-    # Full report tests build TeX images from cold; keep the PR step bounded below.
+    # Keep the PR lifecycle gate bounded; the full Docker suite runs on main.
     timeout-minutes: 45
     runs-on: ubuntu-latest
     steps:
@@ -43,10 +43,38 @@ jobs:
         run: |
           uv sync --locked
           uv pip install --no-deps -e tests/compatibility
+      - name: Install publishing browser
+        run: uv run python -m playwright install --with-deps chromium
       - name: PR container and child-job lifecycle gate
         timeout-minutes: 15
         if: github.event_name == 'pull_request'
-        run: uv run pytest -m docker tests/test_audit.py::test_an_audit_runs_inside_the_benchmark tests/test_lifecycle.py
+        run: uv run pytest -m docker tests/test_investigation.py::test_an_investigation_runs_inside_the_benchmark tests/test_lifecycle.py
       - name: Pytest (docker)
         if: github.event_name != 'pull_request'
         run: uv run pytest -m docker
+
+      - name: Findings handoff and GL publication
+        if: github.event_name == 'pull_request'
+        run: uv run pytest -m publishing tests/test_findings_handoff.py tests/test_render.py
+
+  latex:
+    name: LaTeX fixture
+    timeout-minutes: 20
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
+        with:
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@11f9893b081a58869d3b5fccaea48c9e9e46f990 # v8.3.2
+        with:
+          python-version: "3.13"
+      - name: Install dependencies
+        run: uv sync --locked
+      - name: Pinned LaTeX template fixture
+        env:
+          INSPECT_AUDIT_LATEX_IMAGE: inspect-audit:publishing
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y poppler-utils
+          docker build -f src/inspect_audit/containers/publishing.Dockerfile -t "$INSPECT_AUDIT_LATEX_IMAGE" src/inspect_audit/containers
+          uv run pytest -m latex tests/test_latex_publisher.py

.gitignore

@@ -1,12 +1,12 @@
-# Audit data, never in git
+# Investigation data, never in git
 cells/
 out/
-investigations/
+audits/
 reference/
 calibration/suite/
 *.eval
 dev/logs/
-dev/audit-logs/
+dev/investigation-logs/
 
 # Byte-compiled
 __pycache__/
@@ -62,9 +62,12 @@ dev/logs
 # Claude Code scratch
 .superpowers/
 
-# private working artefacts: kept in ~/Developer/inspect_ai/audit-artefacts, never in this public repo
+# private working artefacts: kept in ~/Developer/inspect_ai/investigation-artefacts, never in this public repo
 agent_artefacts/
 
 # pulled Hawk logs and bundles (scripts/hawk-artefacts.yaml -> inspect-audit-findings hawk-pull)
 artefacts/
 UPSTREAM.txt
+
+# Local documentation; publication requires James's review
+/docs/

.pre-commit-config.yaml

@@ -81,6 +81,17 @@ repos:
         # Formatting config (wrap, number, ...) lives in .mdformat.toml.
         # Prompts, templates and the investigator's skills are model-facing text;
         # keep them as written rather than reflowed.
-        exclude: ^src/inspect_audit/(prompts|templates|investigation/skills)/
+        exclude: ^src/inspect_audit/(prompts|templates|audit/skills|publishers/skills)/
         additional_dependencies:
           - mdformat-gfm # padded/aligned tables + GFM extensions (task lists, strikethrough)
+
+  # CI type-checks with basedpyright; without it here, commits that pass every
+  # hook still broke CI (FRICTION 46)
+  - repo: local
+    hooks:
+      - id: basedpyright
+        name: basedpyright
+        entry: uv run --no-sync basedpyright
+        language: system
+        types: [python]
+        pass_filenames: false

AGENTS.md

@@ -2,64 +2,77 @@
 
 Working rules for coding agents in this repository. Read this before a substantive change. Where code and this file disagree, raise it; do not quietly pick one.
 
-## What the auditor is
+## What the investigator is
 
-The audit is the benchmark's own Inspect task with the auditor in place of its solver. Inspect sets each sample up exactly as it did for the evaluated agent (their files, setup script, setup solvers, boxes, target, scorers). We add one box beside theirs for the auditor, and replace only what must be replaced.
-
-| Module         | Owns                                                                                                            |
-| -------------- | --------------------------------------------------------------------------------------------------------------- |
-| `_task.py`     | `audit_task`: declared inputs, `task_with(theirs, ...)`, every override, loud refusals                          |
-| `_compose.py`  | their compose plus the `auditor` service, merged in Python with their relative paths made absolute              |
-| `_k8s.py`      | native built-in-chart Kubernetes values plus the auditor service; preserves provider options and network policy |
-| `_auditor.py`  | the solver: stages the item and source into the auditor's box, binds tools to the live sample, runs react       |
-| `_grade.py`    | `grade`: a copy of the live state with the transcript under test, handed to their scorers                       |
-| `_reset.py`    | `reset`: force-recreate their boxes, lay files and run setup again (Docker only)                                |
-| `_logs.py`     | the declared logs: header checks, the attempt index, host-side reading tools                                    |
-| `_verdicts.py` | checks as skills, `record_verdict`, `submit`, one scorer per check                                              |
-| `_image.py`    | `view_image`: an image in a box (the auditor's or the investigator's), SVGs rasterised there                    |
-| `_source.py`   | the benchmark's code for the auditor: the task's package followed by import, data files, install specs          |
-| `_regrade.py`  | standalone concordance: Inspect's `score_async` on the declared logs, diffed against the recorded grades        |
+The investigation is the benchmark's own Inspect task with the investigator in place of its solver. Inspect sets each sample up exactly as it did for the evaluated agent (their files, setup script, setup solvers, boxes, target, scorers). We add one box beside theirs for the investigator, and replace only what must be replaced.
+
+| Module             | Owns                                                                                                                 |
+| ------------------ | -------------------------------------------------------------------------------------------------------------------- |
+| `_task.py`         | `investigation_task`: declared inputs, `task_with(theirs, ...)`, every override, loud refusals                       |
+| `_compose.py`      | their compose plus the `investigator` service, merged in Python with their relative paths made absolute              |
+| `_k8s.py`          | native built-in-chart Kubernetes values plus the investigator service; preserves provider options and network policy |
+| `_investigator.py` | the solver: stages the item and source into the investigator's box, binds tools to the live sample, runs react       |
+| `_grade.py`        | `grade`: a copy of the live state with the transcript under test, handed to their scorers                            |
+| `_reset.py`        | `reset`: force-recreate their boxes, lay files and run setup again (Docker only)                                     |
+| `_logs.py`         | the declared logs: header checks, the attempt index, host-side reading tools                                         |
+| `_verdicts.py`     | checks as skills, `record_verdict`, `submit`, one scorer per check                                                   |
+| `_image.py`        | `view_image`: an image in a box (the investigator's or the auditor's), SVGs rasterised there                         |
+| `_source.py`       | the benchmark's code for the investigator: the task's package followed by import, data files, install specs          |
+| `_regrade.py`      | standalone concordance: Inspect's `score_async` on the declared logs, diffed against the recorded grades             |
 
-## What the investigator is
+## What the auditor is
 
-One Inspect task: a react agent in its own box, launching benchmark and audit jobs. It runs wherever Inspect runs it (a Docker container locally, a pod on Hawk); only its jobs' backend differs.
+One Inspect task: a react agent in its own box, launching benchmark and investigation jobs. It runs wherever Inspect runs it (a Docker container locally, a pod on Hawk); only its jobs' backend differs.
 
-| Module                        | Owns                                                                                          |
-| ----------------------------- | --------------------------------------------------------------------------------------------- |
-| `_investigate.py`             | `investigate`: declared inputs staged on the host, the job tools, the budget tool, the task   |
-| `_runs.py`                    | a `Job` and where it runs: Hawk through `HawkClient`, or a detached local `inspect eval-set`  |
-| `_policy.py`                  | frozen launch configuration, checkpoint identity and `InvestigationAuthorizer` for child jobs |
-| `_investigation_workspace.py` | staging inputs into the box, reading the report back out, publishing to Hawk's artifacts      |
-| `_report.py`                  | findings, publication, the store's record of launched jobs, the outcome scorer                |
+| Module                | Owns                                                                                                          |
+| --------------------- | ------------------------------------------------------------------------------------------------------------- |
+| `_audit.py`           | `audit`: declared inputs staged on the host, the job tools, the budget tool, the task                         |
+| `dispatch/`           | jobs and where they run (`local`, `hawk`), `policy` and the authoriser, `packages`, `spend`, the key launcher |
+| `_log_sources.py`     | every way of naming logs (path, `hawk:`, URL) resolved to local files                                         |
+| `_audit_workspace.py` | staging inputs into the box, reading the report back out, publishing to Hawk's artifacts                      |
+| `_report.py`          | launched-job records, log provenance, operator turn-taking and outcome scoring                                |
+| `_framework.py`       | analytical definitions and rubric, without a separate judgment store                                          |
+| `_findings.py`        | editable draft, trusted submitted bundle, manifest and checkpointed phase                                     |
+| `_publishing.py`      | native dynamic tool source, publisher packages and standalone `publish` task                                  |
+| `publishers/latex.py` | default pinned GL audit template, generated structure and host/image TeX compilation                          |
+| `publishers/gl.py`    | reviewed GL components and static HTML/PDF output                                                             |
 
-Budget is the model key every call bills to (a capped OpenRouter key locally, the run's Hawk user's assigned key on Hawk); nothing here prices or reserves. Resume is Inspect's checkpointing of `/workspace`, `/inputs` and the store.
+The supported shared-budget setup routes calls through one capped OpenRouter key (the assigned Hawk user's key on Hawk). Inspect prices individual sample limits; there is no audit-wide spending ledger or fixed reporting reserve. Resume is Inspect's checkpointing of `/workspace`, `/inputs` and the store.
 
-Launch configuration is resolved once into an immutable `LaunchPolicy`. Its snapshot and SHA-256 are saved in Inspect's sample store; resume must match the original policy and investigator model. `/inputs/policy.json` is a readable copy, never authority. Child submissions pass through `InvestigationAuthorizer` and record the policy digest. The launch policy permits per-sample audit cost and concurrency limits; it does not implement a shared dollar ledger. Unsupported policy fields are refused.
+Launch configuration is resolved once into an immutable `LaunchPolicy`. Its snapshot and SHA-256 are saved in Inspect's sample store; resume must match the original policy and auditor model. `/inputs/policy.json` is a readable copy, never authority. Child submissions pass through `AuditAuthorizer` and record the policy digest. The launch policy permits per-sample investigation cost and concurrency limits; it does not implement a shared dollar ledger. Unsupported policy fields are refused.
 
-Structured YAML accepts `benchmark`, `audit`, `policy` and `execution` sections (see `examples/investigation-policy.yaml`). Restricted Hawk children require the new restricted-credentials/transcript-input API; they never receive a user refresh token. Submission authority currently remains in the investigator's host process. A separate trusted launcher and moving benchmark enumeration out of that process remain work; the policy object alone is not a process-isolation boundary.
+Structured YAML accepts `benchmark`, `investigation`, `policy` and `execution` sections (see `examples/audit-policy.yaml`). Restricted Hawk children require the new restricted-credentials/transcript-input API; they never receive a user refresh token. Submission authority currently remains in the auditor's host process. A separate trusted launcher and moving benchmark enumeration out of that process remain work; the policy object alone is not a process-isolation boundary.
 
 ## Load-bearing rules
 
-1. **Declared inputs only.** The benchmark task and its arguments, the sample ids, the checks and the logs are passed in. Nothing is inferred from log headers, the registry, the filesystem or naming conventions. If the auditor needs something it was not given, it refuses and says what to declare.
+1. **Declared inputs only.** The benchmark task and its arguments, the sample ids, the checks and the logs are passed in. Nothing is inferred from log headers, the registry, the filesystem or naming conventions. If the investigator needs something it was not given, it refuses and says what to declare.
 2. **One attempt key: (eval_id, sample_id, epoch).** The same key in two logs is an error. Never pick between inputs.
 3. **Logs must match the declared run.** Every header must record the declared task, the declared arguments (and no undeclared ones) and the installed benchmark version. A mismatch is an error, not a warning.
-4. **Their task, our solver.** Their services keep their names; their `default` stays the default. Ours is `auditor`. Never rename or rewrite their services.
-5. **Their grader runs only when the auditor calls `grade`.** The task's scorers are ours. Their scorers judge with model roles bound from the logs or declared; a scorer that judged with the evaluated model itself is refused.
-6. **Override, never inherit.** Every limit, the generate config, epochs, metrics and approval of theirs is replaced explicitly in `audit_task`.
+4. **Their task, our solver.** Their services keep their names; their `default` stays the default. Ours is `investigator`. Never rename or rewrite their services.
+5. **Their grader runs only when the investigator calls `grade`.** The task's scorers are ours. Their scorers judge with model roles bound from the logs or declared; a scorer that judged with the evaluated model itself is refused.
+6. **Override, never inherit.** Every limit, the generate config, epochs, metrics and approval of theirs is replaced explicitly in `investigation_task`.
 7. **Fail loudly.** Unsupported inputs raise with the reason. A value the source never recorded is unknown, never a default.
-8. **Private Inspect imports in the auditor live in `_reset.py` only.** Everything else uses Inspect's public API. If you need an internal, the answer is an upstream request, not an import.
-9. **Logs never enter a box.** They are read on the host. Only what the auditor asks to export is written, and only into its own box.
+8. **Private Inspect imports in the investigator live in `_reset.py` only.** Everything else uses Inspect's public API. If you need an internal, the answer is an upstream request, not an import.
+9. **Logs never enter a box.** They are read on the host. Only what the investigator asks to export is written, and only into its own box.
 
 ## Unsupported, on purpose
 
 - Benchmarks that prepare their environment inside their solver (SWE-Lancer's `chain([setup, react])`): swapping the solver skips the preparation. Not detectable; documented, not handled.
-- Selections containing different per-sample sandboxes. One shared sample sandbox is supported; its declaration is merged with the auditor service at task level.
+- Selections containing different per-sample sandboxes. One shared sample sandbox is supported; its declaration is merged with the investigator service at task level.
 - `reset` on k8s.
-- Custom Kubernetes Helm charts. Native values for the built-in chart and Kubernetes declarations using Compose are supported. Native values retain the declared network policy, so the auditor may be offline.
-- The evaluated agent's own tools. The auditor works through `benchmark_bash` and `grade`.
+- Custom Kubernetes Helm charts. Native values for the built-in chart and Kubernetes declarations using Compose are supported. Native values retain the declared network policy, so the investigator may be offline.
+- The evaluated agent's own tools. The investigator works through `benchmark_bash` and `grade`.
 
 ## Tests
 
 - Test behaviour against real Inspect, real logs and real Docker. A test that only checks our code agrees with itself measures nothing.
-- `tests/test_audit.py` runs a real audit of `tests/fixtures/toy_benchmark` with a scripted mockllm auditor. Docker tests are deselected by default: `uv run pytest -m docker`.
+- `tests/test_investigation.py` runs a real investigation of `tests/fixtures/toy_benchmark` with a scripted mockllm investigator. Docker tests are deselected by default: `uv run pytest -m docker`.
 - No model spend in tests. Paid validation runs are separate and need an explicit go.
+
+## Findings handoff
+
+The auditor can edit its draft through Bash/Python. Collected investigator records remain host-authoritative. `submit_findings` validates and freezes the bundle; publishing tools and skills are withheld until this boundary. Afterwards job tools disappear and host launch code refuses new experiments. There is no reopening. The same agent continues, or `inspect_audit/publish` starts a fresh publishing run from the saved bundle. See `docs/findings.md`. Do not introduce a second mutable report findings format into this workflow.
+
+The default latex publisher downloads and verifies the pinned audit_reports template on the host; audit uses generated analytical registry data. No template copy is vendored. Publisher validation runs before freezing findings. Agent-authored publishing input is prose only; grades, criterion assignments, counts and evidence remain in the frozen findings.
+
+The optional GL web publisher draws through pinned website components. Agent calculations may prepare data from existing evidence; agent-written drawing code is not an input. The optional Playwright dependency and Chromium browser run on the Inspect host. CI exercises actual Docker handoff and HTML/PDF publication without model spend.