feat(transports): per-tick ceilings on the poll sources and an association bound on the DICOM SCP (BACKLOG #1114) - #931
Open
wshallwshall wants to merge 2 commits into
Open
feat(transports): per-tick ceilings on the poll sources and an association bound on the DICOM SCP (BACKLOG #1114)#931wshallwshall wants to merge 2 commits into
wshallwshall wants to merge 2 commits into
Conversation
added 2 commits
September 5, 2026 19:04
…each (BACKLOG #1114) The pacer ported to raw-TCP, X12 and HTTP on 2026-09-03 and stopped there. Re-measured at c57903c by execution, with control legs: MLLP, Tcp, X12 and Http accept both pacing keys; DICOM, File, Sftp and Ftp each raised TypeError on each key after constructing normally without them. Those four had no rate control in ANY configuration. They needed two different controls, and conflating them would have been the dishonest shortcut. The poll sources (File, Sftp, Ftp) have no sender to back-pressure -- a partner writes to a directory and leaves. Their gap was an unbounded per-tick iteration. max_files_per_poll SHIPS ON at 1000, because the excess is deferred to the next tick rather than refused: the files stay where they are, so a guessed number costs latency, never a message. Sort then cut, so the ceiling takes the first N in the configured order. On RemoteFile it counts files that pass the pattern/dedup filters, not listing entries, so non-matching names cannot starve the matching ones. file.py also gained the mid-tick stop check remotefile.py already had. The DICOM SCP does have a peer to make wait, so max_associations_per_second / association_burst SHIP OFF, for the reason the 2026-08-11 ruling gives. Its unit is an association because pynetdicom owns the read loop: by EVT_C_STORE the object is already read and decoded, and pacing after decode would delay a message the count-and-log invariant has already obliged us to account for. The wait sits on EVT_CONN_OPEN, before the association request is read; the charge on EVT_ACCEPTED, so a rejected association charges nothing. Measured on pynetdicom 3.0.4: EVT_CONN_OPEN runs per-connection, not on the accept loop -- three concurrent 1.0s blocks completed in 1.04s, at maximum_associations 10 and at 1 -- so a pace delays only its own peer. _MessagePacer is reused rather than re-implemented, but the four intakes before this one drive it from the single event loop, so this consumer supplies its own lock and never holds it across the wait. This does NOT move the ASVS 2.4.1 cell and does not close the item. The ruled off default is untouched, the DICOM bound ships off, and no gate reads any of these keys at startup, so a default install still takes messages at an unbounded rate on every listen intake. Rule 4 still gets neither limb and rule 5 still selects partial. Only reachability evidence changed. Residual, stated in both directions because a knob that hides one is worse than no knob: a sustained drop rate above max_files_per_poll per poll_seconds grows the directory, and a too-low association rate makes a modality wait while it already holds a max_associations slot -- a wait outlasting the sender's ACSE timeout becomes an abort, the one refusal this control is careful never to make. 38 new tests; 14 of 14 mutations against the shipped code turn them red, including dropping the remainder instead of deferring it, cutting before the sort, flipping either default, pacing after decode, charging at connection open, holding the bucket lock across the wait, and reinstating the retired SECURITY.md sentence. Two of those mutations survived a first draft and the tests they exposed were rewritten: one asserted an absolute pacing delay its own control measured as the baseline, and one set the stop event by hand instead of going through stop().
…kes, not four (BACKLOG #1114) The ingest-plane row said both keys are parameters of MLLP(), Tcp(), X12() and Http(), that connections.toml desugars through those same factories, and so "the code-first and the TOML surface both express them". The last clause is false for X12: _TRANSPORTS carries no x12 key, so NO X12 setting is expressible in connections.toml at all, and the pacing keys are not a special case of that. The gap itself was already measured and pinned, with its own positive control, at test_ingress_message_pacing.py::test_x12_has_no_toml_surface_at_all_which_ is_a_separate_gap. What was missing is that the security prose generalised past its own premise, which is the half a reader acts on. Kept as its own commit because BACKLOG #1114 parked it as its own diff: a security sentence deserves a change a reviewer can read alone. The correction is guarded rather than merely made. The new assertion derives which paced factories have a connections.toml row and requires the prose to agree, in both directions, so adding the x12 transport later flips it by itself instead of leaving a carve-out that has quietly become wrong. Verified it can fail: restoring the retired sentence turns it red.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
THIS PR DOES NOT MOVE THE ASVS 2.4.1 CELL AND DOES NOT CLOSE BACKLOG #1114
Stated first because it is the thing most likely to be read wrong. The item's own 2026-09-04 pass says
a builder reaching that row should not build in order to move the cell, and this PR does not. The
shipped
DEFAULT_MAX_MESSAGES_PER_SECONDis untouched, the new DICOM bound ships OFF, and no gatereads any of these keys at startup, so a default install still takes messages at an unbounded rate
on every listen intake — the fact that decides the cell. Walking
docs/ASVS-ASSESSMENT-METHOD.mdsection 1 unchanged, rule 4 still gets neither limb and rule 5 still selects
partial.The two acts that could carry the cell to
passare the owner's (flipping a default, or therefuse-to-start gate), and the closing act after that is the re-scoring seat's. The item's banner is
untouched. What this PR builds are two subjects the item lists as buildable on their own merits.
Re-measurement first, because this row has been falsified twice
Re-run at
c57903c2before any edit, by execution, on the worktree venv, each rejection pairedwith a control leg that constructed the same factory WITHOUT the key:
MLLP/Tcp/X12/Http+max_messages_per_second,message_burstsettings— the positive controlDICOM/File/Sftp/Ftp+ either keyTypeError, after each constructed fine without themDEFAULT_MAX_MESSAGES_PER_SECONDNone_MessagePacer/_pacing_settingsby modulemllp.py6,tcp.py4,x12.py4,http_listener.py4, and 0 indicom.py,file.py,remotefile.py,database.py— against aregister_sourcecontrol returning 2 in every one of those, including the zerosfile.py/remotefile.pymax_file_bytescontrol returning 9 and 17_stop.is_set()infile.pyvsremotefile.pyfile.pychecked only in the loop header, so a tick could not be interruptedx12/dicomin_TRANSPORTSfile/sftp/ftppresentThe 2026-09-04 finding held in every direction I could test. One correction to its method, not
its finding: my
DatabasePollleg is uninterpretable — I passedconnection=/statement=andthe real factory takes
server=/poll_statement=, so the control leg failed and the probe reporteditself uninterpretable rather than absent. That row is untested by me and the 2026-09-04 reading of it
stands unchallenged.
database.pyis a sibling builder's file this wave in any case.What is built
Two subjects that needed OPPOSITE defaults. Conflating them would have been the dishonest shortcut.
1.
max_files_per_pollonFile,Sftp,Ftp— SHIPS ON at 1000. These sources have no senderto back-pressure: a partner writes to a directory and leaves. The excess is deferred to the next
tick, not refused — the files stay where they are — so a number the project picked costs latency,
never a message. Sort then cut, so the ceiling takes the first N in the configured order rather than
an arbitrary subset. On RemoteFile it counts files that pass the pattern/dedup filters, not listing
entries, so a share full of non-matching names cannot starve the few that match.
file.pyalso gainedthe mid-tick stop check
remotefile.pyalready had.2.
max_associations_per_second/association_burston the DICOM SCP — SHIPS OFF. This one makesa real modality wait, so the 2026-08-11 ruling applies unchanged. The unit is an association, not a
message, and that is a property of DIMSE:
pynetdicomowns the read loop, so byEVT_C_STOREtheobject is already read and decoded, and pacing after decode is the shape this item's record rejects by
name. The wait sits on
EVT_CONN_OPEN(before the association request is read) and the charge onEVT_ACCEPTED, so a rejected association and a connection that never associates charge nothing._MessagePaceris reused, not re-implemented. But the four intakes before this one all drive it fromthe single-threaded event loop, so this consumer supplies its own lock and never holds it across the
wait.
The DIMSE threading seam was measured, not assumed
A limiter that serialised the acceptor would be the denial of service it guards against. On
pynetdicom 3.0.4:
EVT_CONN_OPENruns on a per-connection thread, not the accept loop — threeconcurrent connections each blocking 1.0 s there completed in 1.04 s total, at
maximum_associations10 and again at 1. A control leg with no block and the same limit of 1 refusedall three associations too, so that refusal is pynetdicom's own concurrency behaviour and not an
effect of pacing — separated deliberately, so the residual below does not over-claim.
Residual, in both directions
The item warns that a bound demanding a number cannot demand a sensible one, and that a too-low
bound back-pressuring a real feed is worse than unbounded intake. Both forms are real here and both
are written into the shipped docs, not just this PR body:
max_files_per_pollperpoll_secondsgrowsthe directory without bound. Draining a backlog costs one extra poll interval per ceiling's worth of
files. It costs latency, never a message — deferral is not a drop.
max_associationsslot, so the two must be tuned together — and a wait that outlasts the sender's ACSE timeout
becomes an abort, which is the one refusal this control is otherwise careful never to make.
partial.push (
max_object_bytes/timeout_secondsbound those instead).Choice of 1000: an order of magnitude above the throughput the pipeline itself sustains end-to-end per
interface, so on a healthy feed the pipeline — not this ceiling — stays the binding constraint.
The second commit is deliberately separate
docs/SECURITY.md's ingest row claimed "the code-first and the TOML surface both express them" acrossall four paced factories. For
X12the second half is false —_TRANSPORTScarries nox12key,so no X12 setting is expressible in
connections.tomlat all. The gap was already pinned intest_ingress_message_pacing.py; what was new is that the prose generalised past its own premise.#1114 parked this for its own diff, so it is commit 2 and touches only that sentence plus a derived
guard that flips by itself if the transport is later added.
Checks
Run on
C:\Users\Scott\Code\MessageFoundry\.claude\worktrees\manager-277cbb-b1114\.venv\Scripts\python.exe(Python 3.14.6) — the worktree venv, not bare
pythonand not the primary checkout's venv.ruff check .ruff format --check .mypy messagefoundry(strict)tests/test_poll_and_association_intake_bounds.py-k "wiring or connections or file or dicom or remote or sftp or ftp or transport or connector or settings or doc_drift or asvs or security_doc"Zero tests skipped in the new file — the DICOM half sits behind
importorskip("pynetdicom")/("pydicom")and every one of those ran, including 8 that drive a livepynetdicomSCP over loopback.A skip there would have read as a pass.
Not run: the full
pytestsuite (targeted sweep instead), and every hosted-runner-only leg — theSQL Server and Postgres store legs,
windows-service-smoke, and the IDE leg. Somebody must readthose after I exit. I touched no store or service code, so I expect them green, but that is an
expectation and not a measurement.
Mutation testing: 14 of 14 kill the new guards
A green test proves nothing until it is shown it can fail. Each of these was applied to the shipped
code and turned the new file red:
neuter the tick ceiling · drop the remainder instead of deferring it · cut before the sort ·
ship the ceiling OFF · remove the mid-tick stop check · neuter the RemoteFile ceiling · count listing
entries instead of matching files · flip the DICOM default to a number · remove the pre-read wait ·
charge at
EVT_CONN_OPEN· pace atEVT_C_STORE(after decode) · hold the bucket lock across thewait · make
stop()stop releasing the wait · reinstate the retiredSECURITY.mdsentence.Two of them survived a first draft, and the tests they exposed were rewritten rather than the
mutations dropped:
test_a_paced_scp_still_establishes_every_associationasserted an absolute delay of 0.15 s. Itsown unpaced control then measured 0.25 s for six loopback associations, so the threshold was
satisfied by the baseline and proved nothing. It is now a paired comparison — both arms in one
test, asserting the difference. Fixing it also corrected the arithmetic: the expected delay is not
(N - burst) / rate, because the bucket refills during the run and the last association's token ischarged with nobody left to wait on it.
test_stop_cuts_short_an_outstanding_pacing_waitset_stoppingby hand, so deletingself._stopping.set()fromstop()survived — the wait stayed interruptible with nothing left tointerrupt it, while shutdown had silently become unbounded. It now goes through
stop()itself.Notes for whoever picks this up
origin/main(2 ahead, 0 behind at push). No merge or rebase wasneeded or attempted.
pipeline/wiring_runner.py, and this diff does not touch that file — onlytransports/{file,remotefile,dicom}.py,config/wiring.py, docs and tests.docs/security/THREAT-MODEL.mdis vaulted and its doc-drift guard is INERT in this checkout. Itcarries a "Resource-demanding functionality (ASVS 15.1.3)" section. This PR adds two resource bounds,
so a seat with the vault should check whether that section needs rows for them. I could not read
it and did not guess.
reviewedlabel, enqueue, merge, or arm auto-merge.git checkout -- messagefoundry/transports/dicom.pyto undo a mutation and reverted my own uncommitted work with it. Recovered intact from a backup taken
seconds earlier, then re-ran lint, types, the full new-file suite and all 14 mutations to confirm the
restore. Nothing was committed in that window, but the diff is worth a second look on that file.
Ultracode: off — no
system-reminderin this session's context declared it.