Merge origin/main into feat/stage-12a-entrypoint-permissions

Brings in #765 (ruff check in CI with the first group of rule
families), #809 (policyengine 6.4.1) and #791 (redacted Stage 12
failure cause).

Conflicts:
- auth.py: kept this branch's Callable and Literal imports and sorted
  the stdlib import block the way main's newly enabled I rule expects.
- docs/generated/rules.md: regenerated with python -m scripts.rules
  render from the auto-merged registry, which holds main's enforced
  sim.quality.ruff-lint and this branch's
  sim.deploy.permission-enforcement-wired (34 of 48 rules blocking).

ruff check . passes under main's ruff.toml with ruff 0.15.22, so this
branch's own code needed no further changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Author: anth-volk

.github/workflows/pr.yml

@@ -99,6 +99,9 @@ jobs:
     - name: Run ruff format check
       run: uv run ruff format --check projects/*/src libs/*/src
 
+    - name: Run ruff check
+      run: uv run ruff check .
+
   quality-guards:
     name: Quality guards
     # Full CI runs only on the lowest unmerged pull request in a stack.

AGENTS.md

@@ -139,7 +139,7 @@ To add or change a rule, read `docs/engineering/skills/rules-registry.md`.
 - `sim.quality.every-project-tested`: Every project and library with unit tests runs them in the PR test matrix. (doc-only, not checked: follow it yourself)
 - `sim.quality.ruff-format-src`: Source code under projects/*/src and libs/*/src is formatted with ruff format. (blocking: ruff format)
 - `sim.quality.ruff-format-tests`: Test code is formatted with ruff format too; CI currently checks only src directories. (doc-only, not checked: follow it yourself)
-- `sim.quality.ruff-lint`: Python code passes ruff check with the rules selected in ruff.toml (ruff's defaults, made explicit); CI does not run ruff check yet. (advisory: ruff check (E4, E7, E9, F), not blocking)
+- `sim.quality.ruff-lint`: All Python code passes ruff check with every rule family selected in ruff.toml, with no ratchet; families are added one themed group at a time until the selection is ALL. (blocking: ruff check (E4, E7, E9, F, FLY, FURB, I, PIE, PYI, RET, RSE, UP, W))
 - `sim.quality.tests-pass`: Each project's unit tests pass in its own locked environment. (blocking: CI job `test`)
 
 ### CI workflows and scripts

changelog_entry.yaml

@@ -4,6 +4,7 @@
       - Made CLAUDE.md and the Copilot instructions thin pointers to AGENTS.md, required workflow logic longer than one line to live in tested and shellchecked scripts, and registered the tracking-issue rule
       - Added a rules registry (rules/registry.toml) with a standard-library tool that validates it, checks every rule is enforced where it claims, and renders docs/generated/rules.md and the Rules block in AGENTS.md
     changed:
+      - Ran ruff check in CI with the first group of rule families (imports, pyupgrade and other safe auto-fixes) and fixed every finding
       - Moved the multi-line `run` blocks of the simulation deployment workflows into single-line steps and tested scripts without changing their behaviour or outputs, leaving only the release-sentinel and routing-rollback blocks in the one-line `run` rule's baseline
       - Moved multi-line `run` blocks in the PR, image smoke and client publishing workflows into single-line steps and tested scripts, tested the dependency-update scripts, fixed the shellcheck finding in modal-precompute.sh, and enforced the CI-script testing and shellcheck rules with empty baselines
       - Added CODEOWNERS for the rule machinery, pinned CI's ruff to 0.15.22 with an explicit rule selection, and documented that agents never bypass branch rules
@@ -21,3 +22,4 @@
       - Bound UK constituency and local-authority weight matrices to the certified data release of the selected dataset instead of the unversioned bucket copy, and stopped runs before they start when the matrix lacks the run year or household dimension
       - Released each Stage 12 database session after its operation instead of holding pooled sessions through calculations, retried opening each Stage 12 coordinator and worker database connection that the pooler refused without ever retrying a statement or commit, and reported Stage 12 failures to telemetry before recording them in the database so a failed write cannot hide the original error
       - Accepted a null SPM provenance in Stage 12 aggregate artifacts, so UK and other reports without SPM calculations are saved, while every supplied provenance record is still validated
+      - Recorded a redacted, bounded Stage 12 failure cause naming the exception, failing stage, simulation role and message instead of only the exception class, and kept a failed worker's recorded cause on its simulation and report

docs/generated/rules.md

@@ -8,11 +8,11 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf
 
 | Tier | enforced | ratcheted | advisory | doc-only | Total |
 | --- | ---: | ---: | ---: | ---: | ---: |
-| mechanical | 32 | 1 | 1 | 14 | 48 |
+| mechanical | 33 | 1 | 0 | 14 | 48 |
 | metric | 0 | 0 | 0 | 0 | 0 |
 | judgment | 0 | 0 | 0 | 0 | 0 |
 
-**Blocking:** 33 of 48 rules.
+**Blocking:** 34 of 48 rules.
 
 ## Gaps
 
@@ -35,7 +35,7 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf
 
 ### Advisory (checked but not blocking)
 
-- `sim.quality.ruff-lint`: Python code passes ruff check with the rules selected in ruff.toml (ruff's defaults, made explicit); CI does not run ruff check yet.
+- None.
 
 ### Unproven (no proof that the checker rejects a bad example)
 
@@ -249,7 +249,7 @@ flowchart LR
 | `sim.quality.every-project-tested` | Every project and library with unit tests runs them in the PR test matrix. | mechanical | doc-only | no | nothing | — | — |
 | `sim.quality.ruff-format-src` | Source code under projects/*/src and libs/*/src is formatted with ruff format. | mechanical | enforced | yes | ruff format | `pr.yml:lint` | — |
 | `sim.quality.ruff-format-tests` | Test code is formatted with ruff format too; CI currently checks only src directories. | mechanical | doc-only | no | nothing | — | — |
-| `sim.quality.ruff-lint` | Python code passes ruff check with the rules selected in ruff.toml (ruff's defaults, made explicit); CI does not run ruff check yet. | mechanical | advisory | no | ruff check (E4, E7, E9, F) | — | — |
+| `sim.quality.ruff-lint` | All Python code passes ruff check with every rule family selected in ruff.toml, with no ratchet; families are added one themed group at a time until the selection is ALL. | mechanical | enforced | yes | ruff check (E4, E7, E9, F, FLY, FURB, I, PIE, PYI, RET, RSE, UP, W) | `pr.yml:lint` | — |
 | `sim.quality.tests-pass` | Each project's unit tests pass in its own locked environment. | mechanical | enforced | yes | CI job `test` | `pr.yml:test` | — |
 
 ```mermaid
@@ -260,18 +260,17 @@ flowchart LR
   e_ruff_format["ruff format"]
   j_CI_pr_yml_lint["CI pr.yml:lint"]
   r_sim_quality_ruff_format_tests("sim.quality.ruff-format-tests (doc-only)")
-  r_sim_quality_ruff_lint("sim.quality.ruff-lint (advisory)")
-  e_ruff_check__E4__E7__E9__F_["ruff check (E4, E7, E9, F)"]
-  local_only>"not run in CI"]
+  r_sim_quality_ruff_lint("sim.quality.ruff-lint (enforced)")
+  e_ruff_check__E4__E7__E9__F__FLY__FURB__I__PIE__PYI__RET__RSE__UP__W_["ruff check (E4, E7, E9, F, FLY, FURB, I, PIE, PYI, RET, RSE, UP, W)"]
   r_sim_quality_tests_pass("sim.quality.tests-pass (enforced)")
   e_CI_job_test["CI job test"]
   j_CI_pr_yml_test["CI pr.yml:test"]
   r_sim_quality_every_project_tested -.-> gap_unenforced
   r_sim_quality_ruff_format_src --> e_ruff_format
   e_ruff_format --> j_CI_pr_yml_lint
   r_sim_quality_ruff_format_tests -.-> gap_unenforced
-  r_sim_quality_ruff_lint --> e_ruff_check__E4__E7__E9__F_
-  e_ruff_check__E4__E7__E9__F_ -.-> local_only
+  r_sim_quality_ruff_lint --> e_ruff_check__E4__E7__E9__F__FLY__FURB__I__PIE__PYI__RET__RSE__UP__W_
+  e_ruff_check__E4__E7__E9__F__FLY__FURB__I__PIE__PYI__RET__RSE__UP__W_ --> j_CI_pr_yml_lint
   r_sim_quality_tests_pass --> e_CI_job_test
   e_CI_job_test --> j_CI_pr_yml_test
 ```

libs/policyengine-fastapi/fixtures/ping/shared.py

@@ -1,9 +1,8 @@
 """Shared fixtures for ping endpoint tests."""
 
+import pytest
 from fastapi import FastAPI
 from fastapi.testclient import TestClient
-import pytest
-
 from policyengine_fastapi import ping
 from policyengine_fastapi.health import HealthRegistry
 

libs/policyengine-fastapi/src/policyengine_fastapi/__init__.py

@@ -4,8 +4,16 @@
 
 from .observability import (
     ObservabilityConfig as ObservabilityConfig,
+)
+from .observability import (
     SimulationStage as SimulationStage,
+)
+from .observability import (
     TracerCaptureMode as TracerCaptureMode,
+)
+from .observability import (
     build_observability as build_observability,
+)
+from .observability import (
     get_observability as get_observability,
 )

libs/policyengine-fastapi/src/policyengine_fastapi/auth/__init__.py

@@ -1 +1,3 @@
 from .jwt_decoder import JWTDecoder
+
+__all__ = ["JWTDecoder"]

libs/policyengine-fastapi/src/policyengine_fastapi/auth/jwt_decoder.py

@@ -1,8 +1,8 @@
-from typing import Any
+import logging
+
+import jwt
 from fastapi import Depends, HTTPException, status
 from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
-import jwt
-import logging
 
 # Based on the auth0 article: https://auth0.com/blog/build-and-secure-fastapi-server-with-auth0/
 
@@ -35,8 +35,7 @@ def __call__(
             if self.auto_error:
                 LOG.info("Invalid auth token: Throwing 4XX error")
                 raise err
-            else:
-                LOG.info("Ignoring invalid auth token")
+            LOG.info("Ignoring invalid auth token")
         return None
 
     def _validate(self, token: HTTPAuthorizationCredentials | None):

libs/policyengine-fastapi/src/policyengine_fastapi/database.py

@@ -1,5 +1,6 @@
-from pathlib import Path
-from typing import Any, Callable, Generator
+from collections.abc import Callable, Generator
+from typing import Any
+
 from sqlalchemy import Engine, StaticPool, create_engine
 from sqlmodel import Session
 

libs/policyengine-fastapi/src/policyengine_fastapi/exit.py

@@ -4,11 +4,9 @@
 """
 
 import logging
+from collections.abc import Callable
 from contextlib import asynccontextmanager
-from functools import wraps
-from typing import Any, Callable, TypeVar, cast
-
-from fastapi import FastAPI
+from typing import Any, TypeVar
 
 log = logging.getLogger(__name__)