Skip to content

feat(transports): per-tick ceilings on the poll sources and an association bound on the DICOM SCP (BACKLOG #1114) - #931

Open
wshallwshall wants to merge 2 commits into
mainfrom
claude/mgr-1114
Open

feat(transports): per-tick ceilings on the poll sources and an association bound on the DICOM SCP (BACKLOG #1114)#931
wshallwshall wants to merge 2 commits into
mainfrom
claude/mgr-1114

Conversation

@wshallwshall

Copy link
Copy Markdown
Collaborator

THIS PR DOES NOT MOVE THE ASVS 2.4.1 CELL AND DOES NOT CLOSE BACKLOG #1114

Stated first because it is the thing most likely to be read wrong. The item's own 2026-09-04 pass says
a builder reaching that row should not build in order to move the cell, and this PR does not. The
shipped DEFAULT_MAX_MESSAGES_PER_SECOND is untouched, the new DICOM bound ships OFF, and no gate
reads any of these keys at startup, so a default install still takes messages at an unbounded rate
on every listen intake
— the fact that decides the cell. Walking docs/ASVS-ASSESSMENT-METHOD.md
section 1 unchanged, rule 4 still gets neither limb and rule 5 still selects partial.

The two acts that could carry the cell to pass are the owner's (flipping a default, or the
refuse-to-start gate), and the closing act after that is the re-scoring seat's. The item's banner is
untouched. What this PR builds are two subjects the item lists as buildable on their own merits.

Re-measurement first, because this row has been falsified twice

Re-run at c57903c2 before any edit, by execution, on the worktree venv, each rejection paired
with a control leg that constructed the same factory WITHOUT the key:

Reading Result
MLLP / Tcp / X12 / Http + max_messages_per_second, message_burst accepted, carried into settingsthe positive control
DICOM / File / Sftp / Ftp + either key TypeError, after each constructed fine without them
DEFAULT_MAX_MESSAGES_PER_SECOND still None
_MessagePacer / _pacing_settings by module mllp.py 6, tcp.py 4, x12.py 4, http_listener.py 4, and 0 in dicom.py, file.py, remotefile.py, database.py — against a register_source control returning 2 in every one of those, including the zeros
per-tick ceiling tokens in file.py / remotefile.py 0, against a max_file_bytes control returning 9 and 17
_stop.is_set() in file.py vs remotefile.py 1 vs 2file.py checked only in the loop header, so a tick could not be interrupted
x12 / dicom in _TRANSPORTS absent; file / sftp / ftp present

The 2026-09-04 finding held in every direction I could test. One correction to its method, not
its finding: my DatabasePoll leg is uninterpretable — I passed connection= / statement= and
the real factory takes server= / poll_statement=, so the control leg failed and the probe reported
itself uninterpretable rather than absent. That row is untested by me and the 2026-09-04 reading of it
stands unchallenged. database.py is a sibling builder's file this wave in any case.

What is built

Two subjects that needed OPPOSITE defaults. Conflating them would have been the dishonest shortcut.

1. max_files_per_poll on File, Sftp, Ftp — SHIPS ON at 1000. These sources have no sender
to back-pressure: a partner writes to a directory and leaves. The excess is deferred to the next
tick, not refused
— the files stay where they are — so a number the project picked costs latency,
never a message. Sort then cut, so the ceiling takes the first N in the configured order rather than
an arbitrary subset. On RemoteFile it counts files that pass the pattern/dedup filters, not listing
entries, so a share full of non-matching names cannot starve the few that match. file.py also gained
the mid-tick stop check remotefile.py already had.

2. max_associations_per_second / association_burst on the DICOM SCP — SHIPS OFF. This one makes
a real modality wait, so the 2026-08-11 ruling applies unchanged. The unit is an association, not a
message
, and that is a property of DIMSE: pynetdicom owns the read loop, so by EVT_C_STORE the
object is already read and decoded, and pacing after decode is the shape this item's record rejects by
name. The wait sits on EVT_CONN_OPEN (before the association request is read) and the charge on
EVT_ACCEPTED, so a rejected association and a connection that never associates charge nothing.

_MessagePacer is reused, not re-implemented. But the four intakes before this one all drive it from
the single-threaded event loop, so this consumer supplies its own lock and never holds it across the
wait.

The DIMSE threading seam was measured, not assumed

A limiter that serialised the acceptor would be the denial of service it guards against. On
pynetdicom 3.0.4: EVT_CONN_OPEN runs on a per-connection thread, not the accept loop — three
concurrent connections each blocking 1.0 s there completed in 1.04 s total, at
maximum_associations 10 and again at 1. A control leg with no block and the same limit of 1 refused
all three associations too, so that refusal is pynetdicom's own concurrency behaviour and not an
effect of pacing
— separated deliberately, so the residual below does not over-claim.

Residual, in both directions

The item warns that a bound demanding a number cannot demand a sensible one, and that a too-low
bound back-pressuring a real feed is worse than unbounded intake. Both forms are real here and both
are written into the shipped docs, not just this PR body:

  • Tick ceiling, too low: a sustained drop rate above max_files_per_poll per poll_seconds grows
    the directory without bound. Draining a backlog costs one extra poll interval per ceiling's worth of
    files. It costs latency, never a message — deferral is not a drop.
  • Tick ceiling, too high / disabled: back to the unbounded tick, which is the defect.
  • Association rate, too low: a modality waits while it is already holding a max_associations
    slot, so the two must be tuned together — and a wait that outlasts the sender's ACSE timeout
    becomes an abort
    , which is the one refusal this control is otherwise careful never to make.
  • Association rate, unset (the shipped default): no bound at all, which is why the cell stays
    partial.
  • Not closed by this PR: an established DICOM association is still unbounded in the objects it may
    push (max_object_bytes / timeout_seconds bound those instead).

Choice of 1000: an order of magnitude above the throughput the pipeline itself sustains end-to-end per
interface, so on a healthy feed the pipeline — not this ceiling — stays the binding constraint.

The second commit is deliberately separate

docs/SECURITY.md's ingest row claimed "the code-first and the TOML surface both express them" across
all four paced factories. For X12 the second half is false_TRANSPORTS carries no x12 key,
so no X12 setting is expressible in connections.toml at all. The gap was already pinned in
test_ingress_message_pacing.py; what was new is that the prose generalised past its own premise.
#1114 parked this for its own diff, so it is commit 2 and touches only that sentence plus a derived
guard that flips by itself if the transport is later added.

Checks

Run on C:\Users\Scott\Code\MessageFoundry\.claude\worktrees\manager-277cbb-b1114\.venv\Scripts\python.exe
(Python 3.14.6) — the worktree venv, not bare python and not the primary checkout's venv.

Check Result
ruff check . pass
ruff format --check . pass
mypy messagefoundry (strict) pass, 268 source files
new file tests/test_poll_and_association_intake_bounds.py 39 collected, 39 passed, 0 SKIPPED
directly affected suites (DICOM x4, remotefile, connections_file, ingress pacing, security-doc, ASVS file drift, file credential/archive/ledger) 299 passed, 2 skipped
broad sweep -k "wiring or connections or file or dicom or remote or sftp or ftp or transport or connector or settings or doc_drift or asvs or security_doc" 2602 passed, 113 skipped, 0 failed (7m26s)
pre-commit (ledger gate, SPDX, leak guard, bandit, secrets, control chars) pass on both commits

Zero tests skipped in the new file — the DICOM half sits behind importorskip("pynetdicom") /
("pydicom") and every one of those ran, including 8 that drive a live pynetdicom SCP over loopback.
A skip there would have read as a pass.

Not run: the full pytest suite (targeted sweep instead), and every hosted-runner-only leg — the
SQL Server and Postgres store legs, windows-service-smoke, and the IDE leg. Somebody must read
those after I exit.
I touched no store or service code, so I expect them green, but that is an
expectation and not a measurement.

Mutation testing: 14 of 14 kill the new guards

A green test proves nothing until it is shown it can fail. Each of these was applied to the shipped
code and turned the new file red:

neuter the tick ceiling · drop the remainder instead of deferring it · cut before the sort ·
ship the ceiling OFF · remove the mid-tick stop check · neuter the RemoteFile ceiling · count listing
entries instead of matching files · flip the DICOM default to a number · remove the pre-read wait ·
charge at EVT_CONN_OPEN · pace at EVT_C_STORE (after decode) · hold the bucket lock across the
wait · make stop() stop releasing the wait · reinstate the retired SECURITY.md sentence.

Two of them survived a first draft, and the tests they exposed were rewritten rather than the
mutations dropped:

  1. test_a_paced_scp_still_establishes_every_association asserted an absolute delay of 0.15 s. Its
    own unpaced control then measured 0.25 s for six loopback associations, so the threshold was
    satisfied by the baseline and proved nothing. It is now a paired comparison — both arms in one
    test, asserting the difference. Fixing it also corrected the arithmetic: the expected delay is not
    (N - burst) / rate, because the bucket refills during the run and the last association's token is
    charged with nobody left to wait on it.
  2. test_stop_cuts_short_an_outstanding_pacing_wait set _stopping by hand, so deleting
    self._stopping.set() from stop() survived — the wait stayed interruptible with nothing left to
    interrupt it, while shutdown had silently become unbounded. It now goes through stop() itself.

Notes for whoever picks this up

  • Branch is 0 commits behind origin/main (2 ahead, 0 behind at push). No merge or rebase was
    needed or attempted.
  • No interaction with PR 918 (sibling #1109): that wires the content sniff into
    pipeline/wiring_runner.py, and this diff does not touch that file — only
    transports/{file,remotefile,dicom}.py, config/wiring.py, docs and tests.
  • docs/security/THREAT-MODEL.md is vaulted and its doc-drift guard is INERT in this checkout. It
    carries a "Resource-demanding functionality (ASVS 15.1.3)" section. This PR adds two resource bounds,
    so a seat with the vault should check whether that section needs rows for them. I could not read
    it and did not guess.
  • I did not add a reviewed label, enqueue, merge, or arm auto-merge.
  • One process note against myself: mid-task I ran git checkout -- messagefoundry/transports/dicom.py
    to undo a mutation and reverted my own uncommitted work with it. Recovered intact from a backup taken
    seconds earlier, then re-ran lint, types, the full new-file suite and all 14 mutations to confirm the
    restore. Nothing was committed in that window, but the diff is worth a second look on that file.

Ultracode: off — no system-reminder in this session's context declared it.

wshallwshall added 2 commits September 5, 2026 19:04
…each (BACKLOG #1114)

The pacer ported to raw-TCP, X12 and HTTP on 2026-09-03 and stopped there.
Re-measured at c57903c by execution, with control legs: MLLP, Tcp, X12 and
Http accept both pacing keys; DICOM, File, Sftp and Ftp each raised TypeError
on each key after constructing normally without them. Those four had no rate
control in ANY configuration.

They needed two different controls, and conflating them would have been the
dishonest shortcut.

The poll sources (File, Sftp, Ftp) have no sender to back-pressure -- a partner
writes to a directory and leaves. Their gap was an unbounded per-tick
iteration. max_files_per_poll SHIPS ON at 1000, because the excess is deferred
to the next tick rather than refused: the files stay where they are, so a
guessed number costs latency, never a message. Sort then cut, so the ceiling
takes the first N in the configured order. On RemoteFile it counts files that
pass the pattern/dedup filters, not listing entries, so non-matching names
cannot starve the matching ones. file.py also gained the mid-tick stop check
remotefile.py already had.

The DICOM SCP does have a peer to make wait, so max_associations_per_second /
association_burst SHIP OFF, for the reason the 2026-08-11 ruling gives. Its
unit is an association because pynetdicom owns the read loop: by EVT_C_STORE
the object is already read and decoded, and pacing after decode would delay a
message the count-and-log invariant has already obliged us to account for. The
wait sits on EVT_CONN_OPEN, before the association request is read; the charge
on EVT_ACCEPTED, so a rejected association charges nothing. Measured on
pynetdicom 3.0.4: EVT_CONN_OPEN runs per-connection, not on the accept loop --
three concurrent 1.0s blocks completed in 1.04s, at maximum_associations 10 and
at 1 -- so a pace delays only its own peer. _MessagePacer is reused rather than
re-implemented, but the four intakes before this one drive it from the single
event loop, so this consumer supplies its own lock and never holds it across
the wait.

This does NOT move the ASVS 2.4.1 cell and does not close the item. The ruled
off default is untouched, the DICOM bound ships off, and no gate reads any of
these keys at startup, so a default install still takes messages at an
unbounded rate on every listen intake. Rule 4 still gets neither limb and rule
5 still selects partial. Only reachability evidence changed.

Residual, stated in both directions because a knob that hides one is worse than
no knob: a sustained drop rate above max_files_per_poll per poll_seconds grows
the directory, and a too-low association rate makes a modality wait while it
already holds a max_associations slot -- a wait outlasting the sender's ACSE
timeout becomes an abort, the one refusal this control is careful never to make.

38 new tests; 14 of 14 mutations against the shipped code turn them red,
including dropping the remainder instead of deferring it, cutting before the
sort, flipping either default, pacing after decode, charging at connection open,
holding the bucket lock across the wait, and reinstating the retired
SECURITY.md sentence. Two of those mutations survived a first draft and the
tests they exposed were rewritten: one asserted an absolute pacing delay its own
control measured as the baseline, and one set the stop event by hand instead of
going through stop().
…kes, not four (BACKLOG #1114)

The ingest-plane row said both keys are parameters of MLLP(), Tcp(), X12() and
Http(), that connections.toml desugars through those same factories, and so
"the code-first and the TOML surface both express them". The last clause is
false for X12: _TRANSPORTS carries no x12 key, so NO X12 setting is expressible
in connections.toml at all, and the pacing keys are not a special case of that.

The gap itself was already measured and pinned, with its own positive control,
at test_ingress_message_pacing.py::test_x12_has_no_toml_surface_at_all_which_
is_a_separate_gap. What was missing is that the security prose generalised past
its own premise, which is the half a reader acts on.

Kept as its own commit because BACKLOG #1114 parked it as its own diff: a
security sentence deserves a change a reviewer can read alone.

The correction is guarded rather than merely made. The new assertion derives
which paced factories have a connections.toml row and requires the prose to
agree, in both directions, so adding the x12 transport later flips it by itself
instead of leaving a carve-out that has quietly become wrong. Verified it can
fail: restoring the retired sentence turns it red.
@github-actions github-actions Bot added the ci-red A required check went red. Attribute it before retrying. label Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-red A required check went red. Attribute it before retrying.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant