feat(logging): fail-closed application-log write guard -- detect, roll, and stop (BACKLOG #122)

docs/BACKLOG.md

@@ -2220,16 +2220,26 @@ lane; demand-gated on a first enterprise Windows/AD deployment.
 
 ## 122. Corrupted application-log detection, rollover, and connection-stop
 
-> 🔢 **Re-scored 2026-08-20 -> P2.** Value **7/10** · Difficulty **6/10** · _big bet_. No enforcement ships: the only log handlers on main are the stdout StreamHandler at logging_setup.py:427 and the syslog family, and nothing anywhere reacts to a write failure by stopping work. The owner ruling of 2026-08-11 binds this to the count-and-log invariant, which is enforcement rather than the visibility the 2/10 priced. Difficulty 6 prices a fail-closed halt across the listener and the internal routed/outbound stages plus a console-seam change, with the eight branch commits unverified and carrying a seam bump #1220 obsoleted. _(was 2/10 · 6/10.)_
+> ✅ **SHIPPED 2026-09-04 — the enforcement is built, and the halt was VERIFIED BY EXECUTION rather than by reading the branch's commit subjects.** `messagefoundry/logging_guard.py` adds `LogWriteGuard` plus `GuardedStreamHandler` / `GuardedFileHandler`: detection is a `logging.Handler.handleError` override, so one seam covers every sink with no polling, and the response is two-stage — `_roll` renames the broken file aside, opens a fresh one, records the rollover in it and re-writes the failed record; only when the REPLACEMENT also refuses does `record_unwritable` escalate. Stage 1 is bounded (`_ROLL_FLAP_WINDOW_SECONDS` / `_MAX_ROLLS_PER_WINDOW`), because a sink needing rescue every few records is a failing log rather than a transient. `[logging].file` / `file_max_bytes` / `file_backup_count` / `on_write_failure` are real `LoggingSettings` fields, and the validator refuses a `file` inside `log_dir` so the engine and NSSM can never rotate one file. The stop reaches all three tiers, `SystemStatus.log_sinks` reports per-sink health from process memory, and a `log_write_failed` alert pages through the notifier rather than through the log that broke.
+>
+> **THE PARTIAL HALT WAS REAL, and the fix for it is load-bearing — measured, not argued.** Mutating `_halt_inbound_processing` to a no-op (the pre-fix shape: listener stop plus outbound pause only) put the committed ingress row on the **outbound stage** in BOTH claim modes, while the healthy-log negative control still delivered. Mutating `_log_recovery_ok` to always-true let `restart_inbound` + `start_outbound` disarm the halt in both modes. Both mutations were reverted and the tree verified clean.
+>
+> **A THIRD HOLE OF THE SAME CLASS WAS FOUND HERE, and it was NOT on the branch.** `RegistryRunner.start` cleared `_log_write_stopped` and `_log_halted` unconditionally, so it was the one re-arm path that never asked whether the log worked. Measured before the fix, both claim modes: with both sinks unwritable BEFORE the runner was built, a committed ingress row reached `PROCESSED` and was DELIVERED while `LogWriteGuard.can_log()` read False throughout. `Engine` starts a runner on leadership acquisition and on the reload that first builds one, so it is reachable plumbing. `start` now gates the clear on `revalidate`, `_start_pooled_dispatchers` replays the halt onto the fresh internal dispatchers before they seed lanes READY, and `_unbind_for_log_failure` takes the already-bound listeners back down.
+>
+> **ADR 0162's FILE IS NOT ON `main`, and that is a ledger-gate strand rather than an oversight — the number is NOT burned.** Its claim names a worktree that no longer exists, and the branch fallback names `w3-log-write-failure`, whose history shares **no root commit** with `main` (roots `5fa6db9f` and `72bfddfa`), so no commit made there can reach a mergeable pull request. Both documented recoveries — commit from the recorded tree, or check that branch out — are unreachable from an isolated worktree, and remedy 3 (allocate a fresh number) is not licensed while the branch still exists. Prose therefore cites `ADR 0162` with no relative link, so nothing dangles; the file and its index row are the one residual. The subject to file, unallocated: the ledger gate's branch fallback assumes the recorded branch can produce a mergeable commit, and an orphan-history branch cannot.
+>
+> **What the branch's own commits turned out to be worth, and the anchor below is stale.** The branch tip is `46b3a4437` on both `origin` and locally, not the `d26d66a6` recorded below -- it moved after that note was written, and it carries eight commits rather than five. The seam bump (`ENGINE_UI_SEAM: int = 19`) was obsoleted by #1220 and is replaced by the computed digest; the PHI-inventory anti-rot gates and the reload-recovery measurement both hold and their tests pass; the stdout hair-trigger fix holds. The `docs/testing/master-test-plan/` edit was dropped, that tree having been untracked under ADR 0160 D1.
+>
+> **Re-scored 2026-08-20 -> P2.** Value **7/10** · Difficulty **6/10** · _big bet_. No enforcement ships: the only log handlers on main are the stdout StreamHandler at logging_setup.py:427 and the syslog family, and nothing anywhere reacts to a write failure by stopping work. The owner ruling of 2026-08-11 binds this to the count-and-log invariant, which is enforcement rather than the visibility the 2/10 priced. Difficulty 6 prices a fail-closed halt across the listener and the internal routed/outbound stages plus a console-seam change, with the eight branch commits unverified and carrying a seam bump #1220 obsoleted. _(was 2/10 · 6/10.)_
 >
 > **Re-scored 2026-08-03 → DEMAND-GATE.** Value 2 stands — stdout + NSSM rotation, the RFC 5425 TLS syslog forwarder (`_TlsSysLogHandler`, logging_setup.py:281) and #50's disk metering already carry log durability and visibility, so this is marginal and substantially covered. But difficulty 5 prices the wrong shape of work. D5 is "a new connector/codec behind the transport registry" — this is not a connector. logging_setup.py's module docstring (lines 3-13) records that the engine "deliberately do[es] not add file handlers here" because NSSM owns rotation, and `grep FileHandler _(was 2/10 · 5/10.)_
 > **On-trigger / demand-gate.** Numbered for tracking only — build when the trigger below fires (“demand-gate, don’t schedule”).
 > ⚠️ **AMENDED 2026-08-11 — THE OWNER RULED THIS ITEM IN, and that ruling POST-DATES the DEMAND-GATE and the 2/10 above.** The owner's words: *"we never want to process stuff if the processing cannot be logged."* That coupling **is** the count-and-log invariant CLAUDE.md marks do-not-break — every received message is persisted before the ACK and nothing is accepted-and-dropped — so the gate is discharged and the trigger is moot.
 > **The 2/10 was arrived at by conflating VISIBILITY with ENFORCEMENT.** stdout, NSSM rotation, the TLS syslog forwarder and #50's disk metering make the log *visible*; none of them makes processing *stop* when the log cannot be written. The item is the enforcement, and that is what the owner ruled in. **A guard that logs a warning and lets processing continue does not satisfy this item** — that is the specific defect to check for at review.
 > **Work exists and is UNVERIFIED.** `w3-log-write-failure` (`d26d66a6`, pushed and anchored) carries five commits whose subjects claim: the halt let the backlog keep routing so the internal stages are stopped too; a reload-recovery claim that *"was a guess"* was measured and three docs corrected; two PHI-inventory anti-rot gates found red on the branch; and a console-contract seam bump. **The lane died mid-flight on a usage limit, so none of it is verified.** It carries **ADR 0162**, whose number was independently confirmed to come from `alloc.ps1` — a real allocation record exists and `main`'s highest is 0161 — so the number is sound, but the index row should be re-checked before landing since a rebase can drop it.
 > **The claim most worth verifying by execution is the partial halt:** a halt that stops intake while routed and outbound rows keep draining would still violate the invariant this item exists to protect.
-> Verdict: demand-gate
-> Closing-act: owner-ruling
+> Verdict: build
+> Closing-act: code
 
 **Cluster:** Logging & Audit. **Priority:** P3. **Verdict:** demand-gate. **Severity (vs Corepoint):** minor.
 

docs/CONFIGURATION.md

@@ -33,7 +33,12 @@
 > `[retention].audit_days` (**reserved/keep-forever by design**), `[reference].max_staleness_seconds`,
 > `[ai].baa_attested`, and `[update_check].index_url`/`index_allowed_hosts`. The former
 > "accepted-but-ignored" keys that were never fields at all — `[delivery].outbox_workers`/`dead_letter`
-> and `[logging].file`/`max_bytes`/`backups` — now **refuse**.
+> and `[logging].max_bytes`/`backups` — now **refuse**. **`[logging].file` is no longer one of them:**
+> #122 / ADR 0162 made it a real, engine-owned field, and the two legacy spellings beside it refuse.
+> **They refuse on BOTH layers, and only one of those is the general rule.** In the file they hit the
+> unknown-key refusal above (`max_bytes` is even suggested onward as `file_max_bytes`; `backups` is
+> refused naming nothing). From **env** — where a misspelled `MEFOR_*` is otherwise dropped in
+> silence — they hit a dedicated `[logging]` validator that names the replacement for both.
 
 ## Principle — two kinds of configuration
 
@@ -674,7 +679,7 @@ Only `baa_attested` is still a forward-compat placeholder (accepted-but-ignored)
 |---|---|---|---|
 | `level` | enum | `info` | log level. `debug` can surface full message bodies / raw field values into the general log. **`serve` refuses `debug` on a `production_instance` only** (Gate #1, keyed on the production tier alone — see `[security].production_instance`). It is **not** keyed on PHI: since [ADR 0148](adr/0148-phi-default-posture-and-an-explicit-security-enforcement-level.md) a `dev`/`staging` instance also carries PHI, and one of those **will start at `debug` with nothing refusing**. Don't raise any PHI box to `debug` — the gate will not stop you. |
 | `format` | enum | `text` | stdout rendering: `text` (default) or structured `json` (one object per line). Stdlib only — no structlog |
-| `log_dir` | str | _unset_ | the directory NSSM (or another supervisor) **rotates the engine's captured stdout/stderr into**. The engine never writes log **files** itself (it logs to stdout); set this only to tell it where the supervisor parks them, and `GET /status` then **meters that directory's total bytes + filesystem free space** alongside the DB metrics (#50). Unset = stdout-only, no metering. **Metadata only** — the file contents are never read. |
+| `log_dir` | str | _unset_ | the directory NSSM (or another supervisor) **rotates the engine's captured stdout/stderr into**. The engine writes no log **file** of its own unless `file` below is set (opt-in, off by default); set this only to tell it where the supervisor parks the captured stdout, and `GET /status` then **meters that directory's total bytes + filesystem free space** alongside the DB metrics (#50). Unset = stdout-only, no metering. **Metadata only** — the file contents are never read. |
 | `forward_enabled` | bool | _derived_ | ship a copy of every record off-box to a syslog/SIEM collector (sec-offbox-log) so evidence survives a host compromise. **Default-on-when-configured (ADR 0080):** unset ⇒ on iff `forward_host` is set. Set `false` to opt out even with a host; no `forward_host` ⇒ off (stdout-only, unchanged) |
 | `forward_host` | str | — | syslog/SIEM collector host. Setting it turns forwarding on by default (above) |
 | `forward_port` | int | `514` | collector port (1–65535) |
@@ -689,7 +694,10 @@ Only `baa_attested` is still a forward-compat placeholder (accepted-but-ignored)
 | `ntp_peer` | str | — | NTP/SNTP host to compare the local clock against (**required** when `require_time_sync`) |
 | `time_sync_max_skew_seconds` | float | `2.0` | \|local − peer\| above this is "skewed" (must be > 0) |
 | `time_sync_fail_closed` | bool | `false` | **refuse to start** (instead of warn) on skew or an unreachable peer. Further opt-in; requires `require_time_sync` |
-| `file`, `max_bytes`, `backups` | str/int | — | **REFUSED** — none is a `LoggingSettings` field, and an unrecognized key now fails the start rather than loading silently. The engine logs to stdout and NSSM rotates it; `log_dir` above is how you point the engine at where it lands |
+| `file` | str | _unset_ | **opt-in application-log file the ENGINE owns end to end** (#122, ADR 0162) — it opens it, size-rotates it, and rolls it aside on a write failure. Distinct from `log_dir` above, which is where the **supervisor** parks the captured stdout: **one file, one rotation owner**, so a `file` inside `log_dir` is **refused at load** rather than left to fight NSSM. Unset (the default) = stdout-only, unchanged. A path the engine cannot open **refuses startup** — an engine that starts unable to log is the blindness this closes |
+| `file_max_bytes` | int | `50000000` | size-rotate `file` at ~50 MB (`0` = never rotate on size). Engine-side rotation, unrelated to NSSM's. The legacy planned spelling `max_bytes` is **refused at load** naming this key, rather than silently ignored -- from the file by the unknown-key refusal, and from `MEFOR_LOGGING_MAX_BYTES` by a `[logging]` validator, which is the layer the general file refusal does not reach |
+| `file_backup_count` | int | `5` | how many `file.1` … `file.N` backups to keep. The legacy planned spelling `backups` is likewise refused on both layers, though only the env one names this key: the file refusal's nearest-name hint does not reach it. The `*.broken-*` files a write failure rolls aside are **deliberately outside** this chain — they are incident evidence, and a rotation that could delete them would delete the record of the failure |
+| `on_write_failure` | enum | `stop` | **fail-closed control (#122):** when a log sink cannot be written **and** the fresh sink rolled into its place cannot be written either, stop every connection this engine **process** owns, in all three tiers — inbounds stop accepting, messages already accepted stop being routed and transformed, and outbounds pause with their queued rows **retained** (never dead-lettered). Recover by **fixing the log and then** restarting the affected connections, inbound **and** outbound (or the service): a `/config/reload` re-arms the inbounds it re-binds but deliberately never resumes a paused outbound, so on its own it moves the backlog one stage and stops. Every re-arm path is **gated on the log working again** — the engine re-checks by writing a real record to each dead sink at the moment you ask, and a restart issued against a still-unwritable log is **refused** (the connection stays halted, its listener stays down, and another `log_write_failed` names the refusal), so restarting repeatedly is not a way around the control. A first failure alone never stops anything; the roll absorbs the transient. Scope is the process because the application log is process-global and no per-connection attribution exists (ADR 0162 §4); under engine sharding that is the shard's connections. `continue` is the documented opt-out — it still rolls and still alerts, it just keeps processing with no log. The stop is announced by a `log_write_failed` alert through the notifier, a `connection_stopped` per halted connection naming the cause, and `GET /status`'s `log_sinks` block |
 
 > PHI redaction + control-char scrubbing are **always-on handler filters** (not a toggle) applied to
 > **every** sink, including the off-box forwarder ([`logging_setup.py`](../messagefoundry/logging_setup.py),