diff --git a/.console/log.md b/.console/log.md index b11d28dcb..21311546d 100644 --- a/.console/log.md +++ b/.console/log.md @@ -1,3 +1,26 @@ +## 2026-08-19 — correcting myself: the status context is the JOB name + +#526 said `run-name:` pins Forgejo's status context. Wrong, and I generalised +it from a failure case. A two-job workflow with `run-name: probe` produces +`multi / alpha` and `multi / beta` — `probe` appears in neither. A job with +`name: Pretty Job Name` produces `naming / Pretty Job Name`. The format is +` / ()`, **stable by default**. + +The commit-message form I saw first (`audit.yml / ci: probe the... (push)`) +only happens when a run fails *before any job starts*: with no job to name, +Forgejo falls back to the file name and run title. I read a fallback as the +rule. + +Also found while porting: Forgejo executes `.github/workflows/` as well as +`.forgejo/`, so pushing OC to Forgejo handed one local runner the entire GitHub +CI suite — 27 tasks, most failing. Operator chose: port everything to +`.forgejo/` and delete `.github/`. + +Ordering constraint that falls out: **the deletion cannot merge on GitHub.** It +removes the `audit` status branch protection requires, with enforce_admins on, +so that PR could never go green. The port must land additively first, the fleet +cuts over, and the deletion merges on Forgejo afterwards. + ## 2026-08-19 — B4 resolved: the audit context cannot match GitHub's Forgejo Actions is running: runner 6.3.1 registered against the live instance, diff --git a/.forgejo/workflows/ci.yml b/.forgejo/workflows/ci.yml new file mode 100644 index 000000000..fb44fdd23 --- /dev/null +++ b/.forgejo/workflows/ci.yml @@ -0,0 +1,332 @@ +# Ported from .github/workflows/ci.yml for the Forgejo cutover. +# +# Two deliberate differences from the GitHub original: +# +# * `on:` is pull_request only. On Forgejo a `push` trigger produces a SECOND, +# separate status context on the same head ("... (push)") alongside the +# pull_request one — duplicating every job on a single self-hosted runner and +# leaving an extra check outstanding for any gate that requires nothing +# incomplete. +# * Nothing else. The steps are byte-identical, because a gate that runs a +# different command is not the same gate. +# +# Status context format is ` / ()`, so the +# contexts these produce are listed in docs/specs/forgejo-pr-adapter.md and are +# what Forgejo branch protection must require. + +name: CI + +on: + pull_request: + branches: [main] + +jobs: + lint: + name: Lint (ruff) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install ruff + # Install the repo's own pinned toolchain rather than `ruff>=0.5`. A lint + # gate has to run the version the config was written against: `[tool.ruff]` + # here selects a deliberate rule set (BLE001 and S110 are explicitly + # DROPPED as too noisy, per the comment in pyproject), and a newer ruff + # re-enables rules that config never opted into. `ruff>=0.5` floated up to + # 0.16.1 and this job went from clean to 1996 errors — every one of them + # phantom — red-failing CI on main daily from ~2026-07-29. Taking the pin + # from [project.optional-dependencies].dev keeps one source of truth; do + # not reintroduce a version literal here. + run: pip install -e ".[dev]" + - name: Run ruff + run: ruff check . + + typecheck: + name: Type check (ty) + runs-on: ubuntu-latest + # Was advisory (continue-on-error) while 21 pre-existing diagnostics + # were unresolved. Cleared on 2026-05-07 — now blocking. + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install dependencies + run: pip install -e .[dev] + - name: Run ty + run: ty check src/ + + custodian: + name: Custodian doctor + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install dependencies + run: pip install -e .[dev] + - name: Verify .custodian.yaml is well-formed + run: custodian-doctor --strict --repo . + + license-check: + name: License headers + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Check SPDX headers + run: | + missing=$(git ls-files "*.py" | xargs grep -L "SPDX-License-Identifier" 2>/dev/null || true) + if [ -n "$missing" ]; then + echo "Missing SPDX-License-Identifier header in:" + echo "$missing" + exit 1 + fi + + test: + name: Test (pytest) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install dependencies + run: pip install -e .[dev] + - name: Run unit tests excluding slow (PR validation) + if: github.event_name == 'pull_request' + # Unit suite only — integration tests under tests/integration/ need + # live external services (SwitchBoard, Plane, Archon) and run on + # demand, not in CI. PRs exclude slow tests for quick validation. + # Coverage threshold of 85% is the design target from Stage 0. + # CI will fail until coverage improves to meet this target. + # -p no:flaky-detection — the pytest11 entry point imports the whole + # observer package at pytest startup, BEFORE coverage instrumentation, + # marking every module-level line uncovered. The plugin is opt-in by + # design; coverage jobs don't need it loaded. + run: pytest -q tests/unit -m "not slow" -p no:flaky-detection --cov=src --cov-report=html --cov-report=xml --cov-report=term-missing --cov-fail-under=85 + - name: Run full unit test suite including slow (main/merge) + if: github.event_name == 'push' + # Unit suite only — integration tests under tests/integration/ need + # live external services (SwitchBoard, Plane, Archon) and run on + # demand, not in CI. Pushes run full suite including slow tests. + # Coverage threshold of 85% is the design target from Stage 0. + # CI will fail until coverage improves to meet this target. + # -p no:flaky-detection — see PR-validation step above. + run: pytest -q tests/unit -p no:flaky-detection --cov=src --cov-report=html --cov-report=xml --cov-report=term-missing --cov-fail-under=85 + - name: Upload coverage to Codecov + if: always() + uses: codecov/codecov-action@v4 + with: + file: ./coverage.xml + flags: unittests + name: codecov-umbrella + fail_ci_if_error: false + token: ${{ secrets.CODECOV_TOKEN }} + - name: Upload coverage reports as artifacts + if: always() + uses: actions/upload-artifact@v4 + with: + name: coverage-reports-${{ github.event_name }} + path: | + coverage_html_report/ + coverage.xml + retention-days: 30 + + reviewer: + name: Reviewer state-machine tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install dependencies + run: pip install -e .[dev] + - name: Run pr_review_watcher tests + # The reviewer self-review / verdict-gate / Self-Heal Ladder state machine + # lives at tests/test_pr_review_watcher.py (repo ROOT, not tests/unit), so + # the main "Test (pytest)" job — which runs `pytest tests/unit` — never + # exercised it. This dedicated job gates the governance + self-heal code + # (the #313 regression class: never merge over a CONCERNS verdict; resolve + # concerns instead of conceding) on every PR. Standalone, no live services. + # Isolated from tests/unit so it cannot perturb the doc-accuracy suite's + # collection-count assertions. + run: pytest -q tests/test_pr_review_watcher.py -p no:flaky-detection + + reviewer-integration: + name: Reviewer integration tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install dependencies + run: pip install -e .[dev] + - name: Run reviewer integration tests + # tests/integration/reviewer exercises the verdict-consolidation + + # merge-decision state machine (happy path, CI-green gate, verdict + # override, timeout recovery, safety paths, instrumentation) end-to-end + # through pr_review_watcher._phase1. Despite living under tests/integration/, + # these are HERMETIC: the GitHub and Plane clients are fully mocked + # (tests/verdicts/conftest.py), so there are NO live services and NO + # network — they run in <1s. The blanket "integration/ needs live services, + # run on demand" policy that excludes the rest of tests/integration/ from CI + # let this suite DRIFT against the evolving self-merge gates (e.g. the + # require_branch_protection fail-closed gate added in #388). This dedicated + # job runs them on every PR so they cannot rot again. + # -p no:flaky-detection — see the "Test (pytest)" job (plugin is opt-in; + # its pytest11 entry point imports the observer package before coverage + # instrumentation). Isolated from tests/unit so it cannot perturb the + # doc-accuracy suite's collection-count assertions. + run: pytest -q tests/integration/reviewer -p no:flaky-detection + + test-rest: + name: Test (suites outside tests/unit) + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install dependencies + run: pip install -e .[dev] + - name: Run every suite the other jobs do not + # The complement of `tests/unit`. Two dedicated jobs above already gate + # single files that had drifted (tests/test_pr_review_watcher.py, + # tests/integration/reviewer) — each added after that specific suite rotted + # unnoticed. This job generalises the fix instead of waiting for the next + # one: ~1,830 tests under tests/maintenance/, tests/observer/, + # tests/verdicts/, the rest of tests/integration/, and the top-level + # tests/test_*.py had NO job at all. + # + # What that cost, concretely: + # #509 shipped a regression that broke tests/maintenance/ — found a week + # later by hand (#513). + # #521 merged a BRAND NEW failing test in tests/test_dependency_check.py. + # CI was green, the reviewer verdict was SUCCESS, and main stayed red + # until #522. + # tests/test_proposer_entrypoint.py had been failing since the board-seam + # migration weeks earlier; nothing noticed. + # + # --ignore=tests/unit keeps this isolated from the unit job, whose + # doc-accuracy suite asserts collection counts and is perturbed by running + # alongside other trees. -p no:flaky-detection for the same reason as the + # jobs above: the plugin's pytest11 entry point imports the observer package + # before coverage instrumentation. + run: pytest -q tests/ --ignore=tests/unit -p no:flaky-detection + + performance: + name: Performance regression tests + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install dependencies + run: pip install -e .[dev] + - name: Run performance regression tests + # Dedicated job surfaces timing-bound failures as a labelled CI check. + # All bounds are <50ms per scenario; actual measurements are ~0.1–0.2ms. + # A failure here means collection time regressed by >250x vs baseline. + run: pytest -q tests/unit -m "perf" --no-header -rN + + snapshot: + name: Snapshot validation + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install dependencies + run: pip install -e .[dev] + - name: Run snapshot validation tests (PR — quick, excluding slow) + if: github.event_name == 'pull_request' + # Snapshot validation uses 5-layer architecture: + # Layer 1: Schema validation (JSON ↔ Pydantic model roundtrip) + # Layer 2: Completeness validation (required signals present, min 3 non-unavailable) + # Layer 3: Consistency validation (cross-signal semantic checks) + # Layer 4: Real-world accuracy validation (snapshot vs. live tools — marked slow) + # Layer 5: Regression detection (baseline comparison — marked slow) + # + # PR runs exclude slow tests for quick feedback. + # All validation checks must pass; structural failures trigger job failure. + # Transient failures (network, timeouts) are retried up to 3 times. + run: pytest -q tests/integration/observer -m "integration and not slow" --no-header -rN + - name: Run snapshot validation tests (push — full, including slow) + if: github.event_name == 'push' + # Full validation including real-world accuracy (Layer 4) and regression detection (Layer 5). + # These layers test against actual repository state and may take longer. + run: pytest -q tests/integration/observer -m "integration" --no-header -rN + - name: Run snapshot validation tests (scheduled — full, including slow) + if: github.event_name == 'schedule' + # Scheduled runs execute comprehensive validation without time constraints. + # Full validation detects regressions in repository state snapshots, + # even when no new code changes occur. Failures trigger immediate alerts. + run: pytest -q tests/integration/observer -m "integration" --no-header -rN + - name: Upload snapshot validation reports + if: always() + uses: actions/upload-artifact@v4 + with: + name: snapshot-validation-reports-${{ github.event_name }} + path: | + tests/integration/observer/validation_reports/ + retention-days: 30 + strategy: + fail-fast: true + + flaky-test-detection: + name: Flaky test detection + runs-on: ubuntu-latest + if: github.event_name == 'push' # Run on merges (not PRs) to detect trends + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install dependencies + run: pip install -e .[dev] + - name: Run unit tests with flaky detection + # Runs all unit tests with flaky detection plugin enabled. + # Captures test outcomes, analyzes patterns, and saves metrics. + # This provides baseline data for historical trend detection. + run: pytest -q tests/unit --flaky-detection --flaky-storage=.flaky-tests -v --tb=short + - name: Aggregate flakiness history + if: always() + # Aggregates session reports from past 7 days into daily summaries. + # Computes failure rates, trends, and generates recommendations. + # Output: .flaky-tests/aggregations/YYYY-MM-DD-aggregation.json + run: | + python -c " + from pathlib import Path + from operations_center.observer import FlakyTestStorageManager, FlakyTestAggregator, FlakyTestAlertManager + storage = FlakyTestStorageManager.create_local('.flaky-tests') + agg = FlakyTestAggregator(storage) + report = agg.aggregate(days=7) + storage.save_aggregation(report) + alerts = FlakyTestAlertManager.check_alerts(report) + print('Aggregation complete: ' + str(report.flaky_test_count) + ' flaky tests') + for alert in alerts: + print(' [' + alert.severity.value.upper() + '] ' + alert.alert_type + ': ' + alert.description) + " + - name: Upload flaky test metrics + if: always() + uses: actions/upload-artifact@v4 + with: + name: flaky-test-metrics-${{ github.run_id }} + path: | + .flaky-tests/runs/ + .flaky-tests/aggregations/ + retention-days: 90 + - name: Report aggregation status + if: always() + run: | + echo "Flaky test aggregation completed" + ls -la .flaky-tests/aggregations/ 2>/dev/null || echo "No aggregation files created" + strategy: + fail-fast: false diff --git a/.forgejo/workflows/custodian-audit.yml b/.forgejo/workflows/custodian-audit.yml new file mode 100644 index 000000000..c76958d5c --- /dev/null +++ b/.forgejo/workflows/custodian-audit.yml @@ -0,0 +1,96 @@ +# Ported from .github/workflows/custodian-audit.yml for the Forgejo cutover. +# +# Two deliberate differences from the GitHub original: +# +# * `on:` is pull_request only. On Forgejo a `push` trigger produces a SECOND, +# separate status context on the same head ("... (push)") alongside the +# pull_request one — duplicating every job on a single self-hosted runner and +# leaving an extra check outstanding for any gate that requires nothing +# incomplete. +# * Nothing else. The steps are byte-identical, because a gate that runs a +# different command is not the same gate. +# +# Status context format is ` / ()`, so the +# contexts these produce are listed in docs/specs/forgejo-pr-adapter.md and are +# what Forgejo branch protection must require. + +name: custodian-audit + +on: + pull_request: + branches: [main] + +jobs: + audit: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Install Custodian + # Pinned to the SHA pyproject.toml declares (NOT a moving @main). A required + # gate must use a reproducible tool version: tracking Custodian@main let a + # mid-day upstream change (the known R1/R2 detector-id collision, #48) start + # emitting a phantom finding fleet-wide despite OC's `r1_enabled: false`, + # red-failing the audit on every repo between two PRs. Bump this SHA in + # lockstep with pyproject when intentionally adopting newer detectors. + # + # Bumped d6ba8ab -> 7a780b7 in lockstep with pyproject, as this comment + # requires. d6ba8ab predates Custodian 261bbb5, which fixed the vulture + # adapter building `vulture --min-confidence=N ` — an order + # vulture's argparse rejects (exit 2, empty stdout) that was then read as + # "no dead code". That is why this gate has reported vulture clean while + # vulture was installed: the run failed every time and the failure was + # swallowed. Same vacuous-green failure mode the step below warns about. + run: | + python -m pip install --upgrade pip + pip install "custodian[tools] @ git+https://github.com/ProtocolWarden/Custodian.git@7a780b7845337810a235111e378e78fb06361dd5" + + - name: Install repo and its pinned lint toolchain + # `.[dev]` (not plain `.`) so the adapters run OC's OWN pinned ruff/ty/vulture. + # vulture used to be `pip install vulture` in the step above — the last + # unpinned lint tool, and the same drift class as the ruff one below. It now + # comes from the dev extras with everything else. + # The reproducibility argument in the step above applies one level down: + # pinning Custodian while installing `ruff` unpinned just moves the moving + # part. It floated to 0.16.1 and this gate reported 1222 findings against a + # tree the pinned ruff (0.15.13) calls clean — the same phantom-finding + # failure the Custodian pin was added to prevent. + # + # NOT best-effort (`|| true`) any more: a failed install left the adapters + # with no ruff at all, which Custodian reports as "not installed" and skips. + # The gate then passes vacuously — worse than failing, because it looks green. + run: pip install -e ".[dev]" + + - name: Materialize boundary artifact file + # Decode the boundary disclosure artifact from the base64 CONTENT secret + # REPOGRAPH_BOUNDARY_ARTIFACT_B64 (the older *_FILE path secret cannot resolve + # on a CI runner). Graceful: skip if absent (B2 flags it if required). + env: + REPOGRAPH_BOUNDARY_ARTIFACT_B64: ${{ secrets.REPOGRAPH_BOUNDARY_ARTIFACT_B64 }} + run: | + if [ -z "${REPOGRAPH_BOUNDARY_ARTIFACT_B64:-}" ]; then + echo "REPOGRAPH_BOUNDARY_ARTIFACT_B64 not set — skipping (B2 flags if required)." + exit 0 + fi + dest="$(mktemp "${RUNNER_TEMP:-/tmp}/repograph-boundary-XXXXXX.json")" + printf '%s' "$REPOGRAPH_BOUNDARY_ARTIFACT_B64" | base64 -d > "$dest" + echo "REPOGRAPH_BOUNDARY_ARTIFACT_FILE=$dest" >> "$GITHUB_ENV" + + - name: Run Custodian audit + run: | + git config core.hooksPath .hooks + custodian-multi --repos . --fail-on-findings --no-color + + - name: D12 incomplete-integration gate (ratchet) + # D12 ("public src symbol tested but never wired into production" — the + # #313 regression class) is opt-in/off in the main audit above. This + # dedicated step enables it and fails ONLY on symbols not in the + # audit.d12_baseline ratchet (the accepted pre-existing backlog). Net: + # a NEW tested-but-unwired public symbol fails CI; the backlog is burned + # down separately (declare __all__ / wire / remove → prune the baseline). + run: | + custodian-multi --repos . --only D12,DC10 --include-deprecated --fail-on-findings --no-color diff --git a/.forgejo/workflows/eval-corpus-integrity.yml b/.forgejo/workflows/eval-corpus-integrity.yml new file mode 100644 index 000000000..10594b268 --- /dev/null +++ b/.forgejo/workflows/eval-corpus-integrity.yml @@ -0,0 +1,56 @@ +# Ported from .github/workflows/eval-corpus-integrity.yml for the Forgejo cutover. +# +# Two deliberate differences from the GitHub original: +# +# * `on:` is pull_request only. On Forgejo a `push` trigger produces a SECOND, +# separate status context on the same head ("... (push)") alongside the +# pull_request one — duplicating every job on a single self-hosted runner and +# leaving an extra check outstanding for any gate that requires nothing +# incomplete. +# * Nothing else. The steps are byte-identical, because a gate that runs a +# different command is not the same gate. +# +# Status context format is ` / ()`, so the +# contexts these produce are listed in docs/specs/forgejo-pr-adapter.md and are +# what Forgejo branch protection must require. + +# SPDX-License-Identifier: AGPL-3.0-or-later +# EVAL corpus integrity + answer-key gate (HARNESS_TRUST_HARDENING §4.2, D-OP-3). +# +# This is constitution surface #1: the required, non-bypassable check that makes +# corpus tampering structurally visible and runs the answer-key replay gate. It +# must be added to branch protection as a required status check (constitution +# surface #2) so the fleet cannot disarm it. +name: EVAL corpus integrity + +on: + pull_request: + branches: [main] + +jobs: + corpus-integrity: + name: EVAL corpus integrity + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + # Need the base ref too, to enforce the monotonic baseline floor. + fetch-depth: 0 + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install (cryptography only — verifier has no heavy deps) + run: pip install "cryptography>=42" + + - name: Extract the base ref's baseline floor (for the monotonic ratchet) + if: github.base_ref != '' + run: | + git show "origin/${{ github.base_ref }}:eval/constitution/baseline_floor.json" \ + > /tmp/base_floor.json 2>/dev/null || rm -f /tmp/base_floor.json + + - name: Verify chain + signatures + answer-key gate + monotonic floor + run: | + PYTHONPATH=src python -m operations_center.eval.verify \ + --corpus eval/corpus/ledger.jsonl \ + --constitution eval/constitution \ + $([ -f /tmp/base_floor.json ] && echo "--base-floor /tmp/base_floor.json") diff --git a/docs/specs/forgejo-pr-adapter.md b/docs/specs/forgejo-pr-adapter.md index 8cd705c1e..2fa327423 100755 --- a/docs/specs/forgejo-pr-adapter.md +++ b/docs/specs/forgejo-pr-adapter.md @@ -252,17 +252,39 @@ message changes on every push, so under the default configuration branch protection can never be satisfied — not "blocks until you rename it", but unsatisfiable in principle. -### `run-name:` makes it stable +### Correction: the middle segment is the JOB, not `run-name` -Setting `run-name: audit` pins the middle segment. Re-run with a deliberately -different commit message: +An earlier revision of this section claimed `run-name:` pins the middle +segment. **That was wrong**, and generalised from a failure case. + +A two-job workflow with `run-name: probe` produces two contexts, and `probe` +appears in neither: + +``` +multi / alpha (pull_request) +multi / beta (pull_request) +``` + +A job carrying `name: Pretty Job Name` produces: ``` -custodian-audit / audit (push) +naming / Pretty Job Name (pull_request) ``` -Stable, deterministic, and independent of the commit. That is the property -branch protection needs. +So the format is: + +``` + / () +``` + +It is **stable by default** — no `run-name:` needed, and each job gets its own +context exactly as on GitHub, just prefixed by the workflow name. + +The commit-message form in the first probe +(`audit.yml / ci: probe the status context name (push)`) appears only when a run +fails *before any job starts*: with no job to name, Forgejo falls back to the +file name and the run title. Reading a fallback as the rule is what produced the +wrong conclusion. ### Both events fire on a PR head @@ -280,7 +302,7 @@ trigger on `pull_request` only. ### Cutover configuration -* workflow: `name: custodian-audit`, `run-name: audit`, `on: pull_request` +* workflow: `name: custodian-audit`, `on: pull_request` (no `run-name:` needed) * branch protection `status_check_contexts`: `custodian-audit / audit (pull_request)` * `apply_to_admins: true` (per the earlier live finding)