ci(tooling): port the second fault-handler belt, calibrate the gate bound (BACKLOG #1304)

.github/workflows/ci.yml

@@ -1153,13 +1153,56 @@ jobs:
       #
       # `--dist loadfile` for the same reason as the engine step: these tests bind real port windows and
       # own module-scoped fixtures, so a file's tests must stay in one worker.
+      #
+      # THE SECOND FAULTHANDLER BELT, PORTED FROM THE ENGINE STEP (BACKLOG #1304). The engine
+      # `Tests (pytest)` step carries TWO watchdogs against a hung Windows leg; its derivation is there,
+      # not repeated here. This tier carried exactly ONE of them. `--timeout-method=thread` arrives
+      # repo-wide through `addopts` in pyproject.toml, so it was already live here; `PYTHONFAULTHANDLER=1`
+      # plus `-o faulthandler_timeout=` is set on the engine and web-console test steps and on nine
+      # further steps in this file, and on this one it was absent.
+      #
+      # THIS DOES NOT TURN AN UNNAMED FAILURE INTO A NAMED ONE, AND THE MEASUREMENT SAYS SO. Paired local
+      # arms on a test that blocks in `subprocess.run` with no `timeout=` of its own -- the shape 164 of
+      # this tier's 317 `subprocess` calls have, across 57 of the 138 files in tests/tooling_manifest.txt
+      # (AST count at 46ea10a78 against a control of 153 bounded calls, so a zero was reachable). WITHOUT
+      # this belt the thread method already fires and already names the frame, down to
+      # `_winapi.WaitForSingleObject`, and the `+ Timeout +` header appears twice in BOTH arms while the
+      # faulthandler header appears in ONE. Reading the pinned pytest-timeout on disk
+      # says why -- `timeout_timer` dumps from a WATCHDOG THREAD and calls `os._exit(1)`, so it never
+      # needs to interrupt the wedged main thread. Anyone porting this on the belief that the thread
+      # method cannot fire on a C-level wait is porting it for a reason that does not hold.
+      #
+      # WHAT IT ACTUALLY BUYS, which is narrower and still worth the two lines. A SECOND watchdog on an
+      # INDEPENDENT mechanism (CPython's C-level `dump_traceback_later`, not a Python `threading.Timer`),
+      # writing down a DIFFERENT path: faulthandler dumps to a raw dup'd stderr fd, while pytest-timeout
+      # writes through `config.get_terminal_writer()`. That distinction is why it is worth having on the
+      # one tier that runs `-n 4` -- `_pytest/faulthandler.py` comments that xdist monkeypatches
+      # `sys.stderr` with a non-file, and whether the terminal-writer dump survives a worker's `os._exit`
+      # and reaches the controller log is NOT established here (no xdist on the measuring box).
+      # `PYTHONFAULTHANDLER=1` additionally arms the fatal-signal handler from interpreter start, before
+      # `pytest_configure` arms pytest's own and after `pytest_unconfigure` disables it, and is inherited
+      # by Python children -- and this tier spawns more children than any other. faulthandler only DUMPS,
+      # never kills, so the outcome of every run is unchanged.
+      #
+      # 150 AGAINST THE 120 BELOW, and the ordering is the contract, not the numbers: the per-test bound
+      # must be attributed FIRST, so this sits above it, exactly as the engine legs pair 150/120 and
+      # 90/60. Both legs here share one `--timeout=120` (unlike the engine matrix, which varies it per
+      # leg), so one literal covers both and no matrix field is needed. Move one, move the other.
+      #
+      # WHAT NEITHER BELT COVERS, stated so nobody reads this as more than it is. Both arm per-ITEM, so a
+      # wedge during collection or in a session fixture reaches neither. And under `-n` the xdist
+      # CONTROLLER runs no items, so a controller left polling a dead worker is silent until the step cap
+      # -- the engine step answers THAT with a third belt (`--max-worker-restart=0`) this tier does not
+      # carry. It was deliberately not ported blind: it is a behaviour change whose zero cost was
+      # measured on the engine tier's history and on no sample from this one. See BACKLOG #1304.
       - name: Harness tests (pytest)
         timeout-minutes: 30 # see the job cap above for the derivation; must stay under it with setup
         env:
           # Same reason as the apt step above: collection imports the Qt modules whether or not this
           # job executes them, and the shipped ones construct QApplication at import on some paths.
           QT_QPA_PLATFORM: offscreen
-        run: pytest -q -n 4 --dist loadfile -m tooling --ignore-glob='*messagefoundry-webconsole*' --timeout=120 --junitxml=tooling-junit.xml
+          PYTHONFAULTHANDLER: "1"
+        run: pytest -q -n 4 --dist loadfile -m tooling --ignore-glob='*messagefoundry-webconsole*' -o faulthandler_timeout=150 --timeout=120 --junitxml=tooling-junit.xml
 
       # A marker typo, a manifest rename, or a conftest hook that silently stops firing all produce the
       # same thing: zero selected tests and a GREEN job. Deselection cannot be distinguished from success

docs/BACKLOG.md

@@ -14508,6 +14508,19 @@ point, which are the parts that must survive it.**
 > **THIS ALSO PUTS A SECOND ARM INTO THIS ITEM'S SIGNATURE, alongside the crash above.** The observable set is now: `subprocess.TimeoutExpired` on a pwsh launch, a crashed xdist worker, and a monotonicity SLO on `test_connscale_smoke`. Whether they share one cause is **still not established** and must not be inferred from their sharing a leg -- the note above says so about the crash and it applies identically here.
 > **AND THE BASE RATE, WITH ITS DENOMINATOR, because the item argues from a streak and never states one.** Over **100 CI runs on `main`** (2026-08-14T19:25Z to 2026-08-22T04:23Z, the full window the Actions API returns in one page): **68 success, 24 failure, 8 cancelled.** Of the 24 failures, `repo harness tests (windows-2025)` is named in **7**, and in **all 7** `CI gate` -- a REQUIRED context -- fails with it: runs 32549032927, 32497283563, 32483211076, 32378103644, 32293477690, 32268545492, 32206563674.
 > **THAT IS THE SEVERITY SENTENCE THIS ITEM WAS MISSING.** Not "it blocked an eight-line prose PR twice" but: this leg is the **single largest contributor to a red default branch**, and every one of its failures reddens the roll-up that gates every merge. **DELIBERATELY NOT CLAIMED:** that the other 17 failures share a cause. Four other jobs appear in that list -- `test (windows-2025)` 7 times, `test (windows-2022)` 4, `test (ubuntu-latest)` 3, plus web-console and SQL Server legs -- and **none of them was investigated here**. The 7 is a floor for this item's subject, not a partition of the 24.
+> **MORE TEST NAMES, 2026-09-03, AND THIS TIME THE LIST IS LEFT OPEN RATHER THAN RE-COUNTED.** Three occurrences tonight while the Console triaged the queue; two read directly from the job log on PR 763 and PR 771. Same file as one already named above, same bound, two different test functions.
+> ```
+> tests/test_worktree_gate_control_plane.py::test_a_bare_config_read_of_a_d0   worker gw1
+> tests/test_worktree_gate_control_plane.py::test_a_bare_config_read_of_a_d3   worker gw0
+>     pwsh -NoProfile -NonInteractive -File scripts/hooks/worktree_gate.ps1 -ReposFile <tmp>/git_repos.txt
+>     subprocess.TimeoutExpired ... timed out after 60 seconds
+> ```
+> **DO NOT RE-COUNT THIS LIST -- IT IS OPEN, AND THE NEXT READER SHOULD STOP COLLECTING NAMES.** The amendment above already recorded that an enumeration nothing re-checks is the SDS-3.6 shape, and this is the second time this one understated; a fresh total would only be the third. **The stable signature is the SCRIPT and the BOUND, not the test function:** grep a CI log for `worktree_gate.ps1` beside `TimeoutExpired`, or for `claim-reconcile.ps1`. Every `TimeoutExpired` occurrence recorded here has been one of those two scripts, while the test function has changed on every sighting.
+> **THE ONE CHECKABLE QUESTION THIS ITEM RAISED IS ANSWERED, AND THE ANSWER IS HALF.** The question was whether the tooling tier carries the engine tier's two fault-handler belts. **It carried one.** `--timeout-method=thread` reaches every leg through `addopts` in `pyproject.toml`, so it was already live here. `PYTHONFAULTHANDLER=1` plus `-o faulthandler_timeout=` was set on the engine and web-console test steps and on nine further steps in `ci.yml`, and was absent from `Harness tests (pytest)`. Ported in this change, with the ordering (150 above the step's own 120) and a guard at `tests/test_ci_faulthandler_belts.py` that goes red on each half being removed -- all three mutations verified failing before the fix was called done.
+> **AND THE ENGINE NOTE'S MECHANISM SENTENCE IS WRONG, WHICH MATTERS MORE THAN THE PORT.** `ci.yml` says the faulthandler belt "fires even when the thread-timer CANNOT interrupt a main-thread C-level wait", naming `subprocess.wait`. **Measured false**, paired local arms on a test blocking in `subprocess.run` with no `timeout=` of its own: **without** the faulthandler belt the thread method already fires and already names the frame down to `_winapi.WaitForSingleObject`, and the `+ Timeout +` header appears in **both** arms while the faulthandler header appears in **one**. Reading the pinned `pytest_timeout` on disk says why: `timeout_timer` dumps from a watchdog **thread** and calls `os._exit(1)`, so it never needs to interrupt anything. **So this tier was ALREADY instrumented for the observed signature, and the port is a second opinion rather than the thing that names the failure** -- an independent mechanism writing down a different path (a dup'd raw stderr fd, not `config.get_terminal_writer()`), which is worth having on the one tier that runs `-n 4`. The engine comment was left as it stands: its belt is still worth carrying, only its stated reason is wrong, and rewriting a contended region of another tier's derivation is not this change's business.
+> **A MECHANISM THE CRASHED-WORKER AMENDMENT ASKED FOR, READ FROM THE PINNED LIBRARY RATHER THAN INFERRED.** That amendment says "a timeout that takes the worker down with it would present exactly this way" and correctly refuses to assume it. `pytest_timeout.timeout_timer` ending in `os._exit(1)` **is** such a path: under `-n`, a worker that `os._exit`s closes its execnet channel with no `workerfinished`, which the controller reports as a crashed worker. **This establishes that the mechanism EXISTS, not that it fired.** Nothing here distinguishes it from a second cause, and the amendment's refusal stands.
+> **THE 60s BOUND IS CALIBRATED, AND CALIBRATING IT FOUND A DEFECT.** The surviving half of the not-measured list, closed at `tests/test_worktree_gate.py`. Worst case **4.6s per call** over n=127 real launches across both gate files (p50 2.1s, p99 4.2s), sequential on a box already running peer pytest sessions -- a contended sample, and still not a 4-vCPU runner, so the shape transfers and the numbers do not. **The defect: at 60 the diagnostic could never fire locally.** pytest-timeout arms in `pytest_runtest_protocol`, so its window strictly contains this one and at equal values it expires first -- measured with a paired control, 5s against `--timeout=5` gave pytest the win 2/2, 5s against `--timeout=30` gave the diagnostic the win 2/2. `addopts` carries `--timeout=60`. So the bound was live on CI's tooling leg (which overrides to 120) and **decorative on every local run**, which is exactly the silent one-platform failure its 45s sibling's comment was written to prevent. **Moved to 45**, a 9.8x margin, clearing both bounds. **Failures before this change read `after 60 seconds`.**
+> **STILL OPEN, EACH NAMING EXACTLY ONE THING.** (1) **The cause**, unchanged and unmeasurable off the runner -- not touched here. (2) **`--max-worker-restart=0`**, the engine step's third belt, **not ported**: it is a behaviour change (a worker death fails the run instead of being survived) whose zero cost was measured on the engine tier's history and on **no** sample from this one. It is the belt that would answer a controller left polling a dead worker until the 30-minute step cap, so it is the strongest remaining candidate and it needs its own measurement first. (3) **The rerun budget**, still an unwritten norm; a recommendation is in this change's PR body and the decision is the owner's.
 > Verdict: build
 > Closing-act: code