feat(sandbox): run Routers and Handlers in the subprocess sandbox by default (BACKLOG #1278)

[wshallwshall]

Closes the change named in BACKLOG #1278: [sandbox].mode now defaults to "subprocess", so a
stock serve runs Router and Handler bodies in a per-inbound worker child instead of the engine's own
address space. The isolation mode has been built and exercised since ADR 0087. Only the default was
wrong.

The item stays open for two vault-side rows a Builder cannot reach. See the last section.

First, the thing this does NOT do

It does not stop config Python executing in the engine process. The loader still runs every *.py
in the config directory in-process, as the service account, at every serve and every reload, ungated
by mode. What mode governs is where a Router's or Handler's body runs once the graph is built.
Module top level is out of its reach either way, and the safe-source DACL gate is still what covers it.

This is stated first in the setting docstring, in pipeline/sandbox.py, in CONFIGURATION.md and in
DEPLOYMENT.md, because a reader must not buy a boundary that is not there.

The samples were RUN, not inspected

The item warns in capitals that the shipped samples look compatible and that this is not the same as
having run them. So they were run, as a paired A/B on the same tree:

	bare default, no [sandbox] section at all	mode=off control
inbounds listening	9 of 13 (7 MLLP, 2 X12)	same 9
MLLP ACKs	7 of 7 AA	same
X12 interchanges accepted	2 of 2	same
File pollers (./in/fhir, ./dev-inbox/pdf)	both files consumed	same
end-to-end delivered to live receivers	OB_ACME_ADT 2, OB_EPIC_STREAM_MDM 1	identical
_sandbox_worker children	11, one per inbound with traffic	0
SandboxError occurrences	0	0

Same ACKs, same deliveries, isolation on or off — so the instrument discriminates rather than merely
reporting a green. Delivery was measured against real MLLP receivers I stood up on 2601/2641/2776,
so this is the full path (ingress, sandboxed Router, sandboxed transform, outbound delivery, downstream
ACK), not just the inbound ACK.

The winning arm declared no [sandbox] section whatsoever, which is what proves the default alone
carries it.

All 14 modules under samples/config/ loaded — 13 inbound and 12 outbound connections constructed —
and 11 inbounds executed Router and Handler bodies inside a worker child. Not exercised: the DICOM
C-STORE SCP on 11112 (needs an SCU) and the IB_RTE_RESPONSE loopback.

A counting trap, so nobody re-derives a phantom 2x

The raw process count was 22, not 11, and it looked exactly like workers spawning workers — which
would have contradicted the item's finding 2. It is not. On this box
.venv\Scripts\python.exe is a redirector that re-executes with a byte-identical command line, so
every logical Python process appears twice in Win32_Process under the same CommandLine. Proven
on the parent: one serve launch produced two processes with identical command lines, and
sys._base_executable differs from sys.executable. The real figure is one worker per inbound with
traffic, exactly as filed.

Two failures that are not mine

OB_IMMUNIZATION_BODYCRED and OB_IMMUNIZATION_REGISTRY fail to start because environments/dev.toml
carries none of the registry_* values. They appear identically in both arms, including the
mode=off control, which is the proof they are pre-existing. The engine isolates them and continues.

The smoke also needs [security].handles_real_patient_data = false, or serve --env dev refuses to
start without a store encryption key.

The capability the default costs you

db_lookup and fhir_lookup are fail-closed REFUSED inside the child. They re-enter the event
loop, which a process boundary breaks. A Handler needing either must run mode="off". That escape
is retained, supported, and tested here; it is not going away.

mode is read once at engine construction, so /config/reload does not re-read it and changing it
needs a restart.

The five findings the item named

Honoured, not re-derived. ADR 0087's numbers are cited, never re-measured.

1. The flip arms a 5-second wall cap that nothing enforced before. At mode=off run_sandboxed is
   literally fn(payload) with no timeout at all. Documented both ways round: a busy loop can no
   longer wedge intake, and a legitimately slow Handler now dead-letters where it used to finish.
   test_threat_model_doc_drift.py's label read "no Router/Handler wall cap", which the flip made
   backwards; corrected.
2. Per-inbound resource multiplier — one child process, two parent daemon threads, three parent
   pipe fds, and on Windows a job-object handle, per inbound with traffic. Now in CONFIGURATION.md.
3. One message is not one dispatch — three dispatches for a single handler with an accepts=
   predicate, 1 + 2K for fan-out to K. In the table beside the 0.19 ms / 6.2 ms figures, so the
   headline number cannot be read without its multiplier.
4. fuse_thread_hops now carries the warning that the runner hard-disables fusion underneath it
   whenever the sandbox is on. An operator reads the knob they set, not the other one.
5. Gate fidelity — DECIDED, not deferred. The item offered a choice; I took "say plainly that it
   does not." dry_run() takes no sandbox argument and always runs in-process, so a Handler calling
   db_lookup/fhir_lookup passes messagefoundry check green and then fails closed at serve,
   and wall_seconds is unenforced in the preview. Both dryrun.py and checks.py now say so
   outright. Teaching the gate to spawn a worker per inbound is its own change with its own cost;
   route_only/transform_one already take sandbox=, so the seam is there when someone wants it.
   Note checks.py's existing "previews what the default engine actually delivers" claim is scoped to
   snapshot_on_send and stays true — but it generalizes badly, so the limit sits right beside it.

The two places the old default was load-bearing

ADR 0144's rejected-alternatives rationale — the stated reason the static lint gate exists at all —
gave two grounds for refusing to rely on the sandbox alone, and "it is opt-in/off-by-default" was one
of them. That ground is gone. The rejection survives on the other, which is sufficient alone: an
address-space boundary does not catch a Handler leaking PHI into the store's own log or building SQL
inside the sanctioned db_lookup, because neither crosses an address space. Written so a reader can
see the gate still has a reason to exist.

docs/ASVS-ASSESSMENT-METHOD.md:115 used this exact cell as its worked example of rule 5, "a
working control that ships off". That premise expired. The fact is corrected; the partial verdict
is deliberately left standing — re-scoring a cell is the tracking seat's act against the vault, not a
Builder's. The table now says plainly that no live worked example of rule 5 remains, rather than
inventing a substitute, which is the choice its own 3.7.3 row already made.

Also swept

config/settings.py, pipeline/sandbox.py (module docstring, SandboxMode.OFF), pipeline/engine.py,
pipeline/wiring_runner.py, checks.py, pipeline/dryrun.py, docs/CONFIGURATION.md,
docs/DEPLOYMENT.md, docs/PHI.md, docs/adr/README.md, ADR 0087 (amended, not rewritten),
ADR 0144, ADR 0147, tests/test_threat_model_doc_drift.py, tests/test_phi_logging_inventory.py.

SandboxPolicy.mode lost its = SandboxMode.OFF — a second default free to contradict the first. All
12 construction sites already passed mode= explicitly, so requiring it was free.

tests/test_settings.py needed no edit: its round-trip arm already picks the opposite of whatever the
current default is.

Checks run vs skipped

Interpreter provenance, since a green from the primary checkout would say nothing about this change.
Run with C:\Users\Scott\Code\MessageFoundry\.venv\Scripts\python.exe from this worktree as cwd, and
verified before trusting any result:

messagefoundry.__file__ =
  C:\Users\Scott\Code\MessageFoundry\.claude\worktrees\agent-a5d66744bbc7c96bd\messagefoundry\__init__.py

That path is inside this worktree, so the tests imported the code under review.

Ran, green:

Leg	Result
test_sandbox.py	35 passed (66 s)
test_sandbox.py + _codec + _import_boundary	110 passed (81 s)
test_threat_model_doc_drift.py, test_phi_logging_inventory.py, test_settings.py, test_accepts_seam.py	192 passed, 89 skipped
test_checks.py, test_checks_gate_parity.py, test_dryrun.py, test_dryrun_snapshot_parity.py	96 passed, 1 skipped
test_doc_guards_lane.py, test_doc_ref_handle.py, test_docs_cite_no_refused_config_keys.py, test_adr_analyze.py, test_docs_security_pathways.py	323 passed
ruff check . / ruff format --check .	clean, 1241 files
mypy messagefoundry (strict)	clean, 267 source files
scripts/docs/backlog_status_check.py	OK, 664 items, each declaring exactly one status

Deliberately skipped — CI's job. The full engine suite. This is not a judgment call I made
lightly: the item's own dispatch note demands a full-suite lane, and the previous lane reverted this
exact change for want of one. The box is shared and the suite is not merely slow here, it is
unavailable — a peer measured 4 percent in 18 minutes against 28 concurrent pytest processes on 20
cores. Running it under saturation would measure the contention, not the flip. Please read the full
suite on the hosted runner before merging, along with windows-service-smoke, which never runs
locally.

Two known-spurious failures, reported by the Lander today. If tests/test_worktree_gate_control_plane.py
reds, it is not from this branch — it is failing across unrelated PRs that do not touch it while main
is green. If the web console leg on windows-2025 reds with 405 passed and zero failures, that is a
timing gate rather than a real failure.

The doc-content half of test_threat_model_doc_drift.py is inert here and in CI (the vault document
is absent from every public checkout), and it emits a ThreatModelDocUnenforced warning saying so. That
is expected, not new.

What keeps the item open

Two coupled rows in the vault-only docs/security/THREAT-MODEL.md, which no checkout can see:

1. The 15.1.3 row must be rewritten to say the Router/Handler wall cap is enforced on the
   shipped default, not off it.
2. _DANGEROUS_ROW_KEYS still anchors on "**In-process (default) or subprocess-isolated execution",
   which now names false vault text. The anchor is deliberately NOT re-picked. Choosing a
   replacement from a checkout that cannot read the document would trade a stale anchor for an
   unverifiable one and red the leg for whoever does hold the vault.

Both are marked in the test file as coupled vault-side edits, the way the pickle anchor beside them
already is.

Re-scoring signal

The prior lane's read is confirmed: difficulty 3 was wrong, 5-6 is right. The change is one default;
the verification is a full suite this box cannot run, a sweep across roughly 14 sites in code, tests and
six documents, plus two vault documents, and five findings each needing release-note prose the row did
not budget for. Value 6 still looks right.

Zero deployments (CLAUDE.md section 0), so this cost no migration and there was nobody to notify. That
removed the vacuous cost of a staged rollout; it did not lower the bar on proving the new default works,
which is what the paired A/B above is for.

🤖 Generated with Claude Code

[wshallwshall]

Reviewed: read the settings change, the regression test, the A/B evidence and the affected-docs sweep. Five commits, docs/BACKLOG.md alone in the last one, exactly as the Lander asked.

The control is what makes this landable

I briefed the item's own capitals at this Builder: the samples look compatible on inspection, and that is not the same as having run them. It ran them — and designed the arm correctly.

The winning arm declared no [sandbox] section at all. That is the whole difference between proving the mode works and proving the default carries it. An arm with mode="subprocess" set explicitly would have passed while telling you nothing about what a deploying site gets. Paired against a mode=off control on the same tree, with real MLLP receivers so the path measured is ingress through sandboxed router and transform to delivery and downstream ACK, not just an inbound ACK.

Eleven inbounds executed bodies in a child; zero SandboxError; delivery counts identical across both arms. No sample broke. The two OB_IMMUNIZATION_* start failures appear in both arms, which is what makes them pre-existing rather than an argument.

The regression test pins the default across all three objects that could contradict it — settings, service settings, and the runner policy the engine actually reads — and the dataclass assertion was falsified against a different field first, so it is known to discriminate.

The counting trap is the finding I would most want kept

The raw child count read 22, not 11, which looks like workers spawning workers and would have contradicted the item's "one child per inbound". It is the venv python.exe acting as a redirector that re-execs with a byte-identical command line, so every logical process appears twice in Win32_Process.

Proven on the parent process rather than reasoned about, and recorded in the ledger so nobody re-derives the phantom. That is the same instrument class that has bitten this repository all day — and note it would have been reported as a defect in the change, not as a measurement error.

It caught the one place the old default was load-bearing

ADR 0144's rejected-alternatives rationale used "it is off by default" as one of two grounds for not relying on the sandbox alone. That ground is now gone. The rejection survives on the other — an address-space boundary does not catch PHI reaching the store's own log, or SQL built inside db_lookup — and it is rewritten so the lint gate visibly still has a reason to exist.

A flip that quietly invalidated a rejected-alternative would have left a gate standing on a premise nobody could find. That is the compensating-control-on-a-false-premise shape, avoided.

Two judgement calls I agree with

It states plainly that dry_run() takes no sandbox argument, so a Handler calling db_lookup passes messagefoundry check green and then fails closed at serve. Saying so beats teaching the gate to spawn workers inside this item.

On the ASVS method page it corrected the fact and left the verdict alone, because re-scoring is the tracking seat's act. Correct, and the same boundary it held on the vault-only rows: it declined to re-pick an anchor from a checkout that cannot read the document, on the ground that this trades a stale anchor for an unverifiable one. That is exactly right, and it is a distinction most sessions would miss.

The risk, stated rather than buried

The full suite was not run, deliberately, because this box is saturated and would have measured contention rather than the flip. This is a behaviour change on the shipped default, so that leg plus windows-service-smoke genuinely need reading on the runner — the PR asks for both and flags the two known-spurious failures. That is the honest handling, but it is the open risk on this one.

The item correctly stays open: two coupled vault-only rows remain, and it confirms the prior lane's signal that difficulty is 5 to 6 rather than 3.

---

Labelling, and saying so. The reviewer role is retired and the required context was removed at about 23:33 UTC, so this label clears nothing and asserts nothing about independence. I commissioned this diff — I wrote the brief, so I am not independent of its scope. I read the diff, which I did not write. Not enqueued, auto-merge not armed.