feat(api): define and enforce the operator API's input-validation rules (BACKLOG #1108)

[wshallwshall]

What this does

BACKLOG #1108 asks what the operator API's input validation rules would have to be
before any documentation could define them. This decides them, makes them real, and
publishes the reference.

messagefoundry/api/validation.py is the single authority for the rules.
docs/API-INPUT-VALIDATION.md is the prose a deploying operator reads, linked from
docs/README.md under "Security reviewers". tests/test_api_input_validation.py
(24 tests) pins every pattern and ceiling, and its doc-drift check carries a
doctored-text control that must fail.

The item stays OPEN, and that is the expected outcome. Its recorded closing act
is scorecard-rescore: the ASVS Tracker re-scores the cell in the vault, then mails
the Lander for the banner flip. Two acts, two seats, neither of them a Builder's.
Nothing here touches the scorecard, and nothing here should be read as re-scoring it.

The categories that got a rule

Category	Rule
Engine-minted ids	32 lowercase hex; 64 lowercase hex for the two digest ids
Connection names	A letter, then letters, digits, _ and -, to 256 characters
Time ranges	A finite float, 0 .. 4102444800 (2100-01-01 UTC), no inf, no NaN
Free-text search	Printable, no control characters, to 512 characters
Glob patterns	No subject on this surface. See below.

Plus the values sitting beside them: role ids, permission ids, the status and
event-kind vocabularies, message type, control id, audit actor and action, one email
address, two filesystem paths, and a log level.

Measured, one instrument, before and after

The app's own generated OpenAPI schema, read at the parent commit and at the tip.
An earlier AST census counting Query(max_length=...) kwargs was retired mid-task:
it stopped answering the question the moment the constraints moved into the
annotation.

Route parameters (101 both sides)	before	after
carrying a structural rule	5	65
length bound only	25	4
unbounded strings	36	0
uncapped arrays of unbounded strings	3	0

Request-body properties (92, over 31 models)	before	after
carrying a structural rule	0	40
length bound only	61	27
unbounded strings	1	0
uncapped or unbounded collections	8	0

Two measurements decided rules a reader would otherwise have drawn wrong

The connection-name grammar is deliberately WIDER than the IDE's. #1108's own
proposed work says to adopt ide/src/connectionWizardModel.ts's
^[A-Za-z][A-Za-z0-9_]*$. Four of the 108 distinct connection names in samples/,
harness/, tests/ and messagefoundry/ carry a hyphen, and that grammar rejects
all four, so adopting it verbatim would have made four shipped connections
unreachable through the API. The rule here admits them and still excludes path
characters, whitespace, control characters and quoting metacharacters.

A lower bound of zero does not exclude infinity. ?received_from=inf parsed to
float('inf') and reached a store bind; /audit declared no bound at all, so
?since=nan was accepted too. A NaN bound is the worse of the two, because every
comparison against it is false, so the filter would return nothing rather than fail.

Three things I decided and one I corrected

The glob category has no subject on the operator API, and that is a finding. A
glob is a file-connector setting consumed at messagefoundry/transports/file.py; the
only console-to-connections.toml write seam carries flagged and direction and
nothing else. So the glob rule is a connection-configuration rule and belongs with
that surface. Saying so is a rule for the item's named category; inventing an API
rule for a data item the API does not accept would not be.

field_path keeps its one authority and gets no copy here. Its grammar is
parsing/peek.parse_path, and store/content_search.make_spec applies it eagerly at
all six acceptance points, so a malformed path is already a 4xx.

Two items keep a rule that is not a pattern, and the doc says the shape gate does
not replace it. A reload config_dir is confined by an allow-list because the
loader executes Python from it; a log level is checked against LOG_LEVELS.

A correction to the item's own text. It lists the email rule at "three acceptance
points". Only one is an operator-supplied address the API takes as an address, and
that one now carries an EmailAddress rule. UserCreateRequest.email and
UserUpdateRequest.email were NOT changed: shipped test fixtures use bob@x,
dr.who@x, j@x and a whitespace-only value, which the rule rejects. Whether a
profile address must carry a dotted domain is a real intranet-addressing question,
not a bug.

What remains, so nobody reads this as more than it is

1. The console's /ui routes declare at least 17 of their own parameter bounds for
   items this module now governs. They are hand-written copies and can drift. Two of
   them are a DIFFERENT data item: the console's received_from/received_to are
   browser datetime-local strings, not epoch numbers.
2. The console reaches several engine handlers through the in-process seam rather
   than over HTTP, and a direct call runs no request validation. Where it builds a
   request model the rules do apply, and two such sites are fixed here.
3. The engine does not enforce the connection-name rule at REGISTRATION, so a name
   authored outside the rule would be created and then be unreachable through the
   API. Nothing shipped has such a name.
4. The inbound HTTP listener, the CLI, and connections.toml are untouched.
5. The 27 request-body properties still carrying a length bound alone are passwords,
   display names, message bodies and other free text where an alphabet rule would be
   wrong rather than missing.

The data plane is untouched on purpose. EditResendRequest.raw keeps a size
bound and no alphabet rule, because an HL7 v2 body is separated by carriage returns.

Two questions I did NOT settle, because they are not mine

1. Does the verb ask only that rules be written down, or that they exist to be
   written? This build takes the harder reading and does both. That does not decide
   the method question, which #1108 raises and docs/ASVS-ASSESSMENT-METHOD.md §1.3
   bears on.
2. Does the existing HL7 and codeset documentation discharge the verb for the data
   plane? That would make this a control-plane-only cell. The data plane is not
   folded in here, and docs/API-INPUT-VALIDATION.md names both questions in its own
   text so the page cannot be mistaken for an answer to them.

[api].expose_docs was NOT flipped. The generated schema now carries every pattern,
which is a side effect and not the deliverable: a schema lists constraints and does
not say why a rule is drawn where it is, what it costs, or what it does not cover.

Callers fixed rather than worked around

messagefoundry_webconsole/routes/uploaded_logs.py built UploadResendRequest
outside its try, so a target the rule refuses escaped as a 500 instead of the
console's own refusal banner. It now records the refusal exactly as an engine 404 is
recorded, with the file id and a fixed reason and nothing caller-supplied.
routes/search.py had the same shape on the preset save.

Eight tests across four files were updated. Each had used a synthetic non-id
("missing", "nope", "abc") and asserted 404. They now assert BOTH: a
well-formed-but-absent id is 404, and a value that could not be an id is 422. A
single probe conflated the two and would have gone on passing with the id rule
removed. One test seeded a store row with user_id="ad9", which is now a
well-formed id so the AD branch it is named for is still reached.

The near-miss worth reviewing hardest

Adding an input rule can silently disarm an existing test.
test_error_banner_escapes_hostile_input was the reflected-XSS regression guard for
every rerender-with-error path in the console. It posted a hostile role id, relied
on _validate_roles echoing it into the 400 detail, and asserted the banner escaped
it. The new role-id rule refuses that value at RolesUpdateRequest, so it never
reaches _validate_roles.

Two of that test's three assertions still passed -- the 400 status and
hostile not in r.text. Only the escaped-form assertion went red, and deleting it is
the obvious fix. That would have retired the guard for every other path as an
unnoticed side effect of an unrelated change.

It is split instead. The route test pins the stronger new property, with a
well-formed-but-unknown role id as the control that the other arm still runs. The
escaping is re-armed directly against pages.user_detail_page, where no upstream
rule can disarm it, with an ordinary message as the control that the banner renders.

The same hazard applies to the other seven updated tests, and to whoever does the
console, transport-plane or CLI halves next. It is recorded in the BACKLOG note.

Checks I ran

Run in this worktree on Windows, in a python -m venv with
.[dev,harness,fhir,dicom,x12,xml,webauthn] plus the console, per ci.yml.

Check	Result
ruff check .	All checks passed!
ruff format --check .	1243 files already formatted
mypy messagefoundry messagefoundry_webconsole	Success: no issues found in 301 source files
pytest tests/test_api_input_validation.py	24 passed
pytest over the API, auth, upload, attachment, connection, alerts, apiclient and DR suites	285 passed on the first pass, then the 8 updated tests green
pytest packaging/messagefoundry-webconsole/tests + apiclient + doc guards	566 passed, 3 skipped after the two console fixes
combined regression: console + API + auth + upload + attachment + connection + alerts + apiclient + DR + doc guards	733 passed, 3 skipped, 1 failure, since fixed
pytest packaging/messagefoundry-webconsole/tests + the validation suite, after that fix	430 passed, 3 skipped
scripts/docs/link_check.py	OK: every relative link resolves (5041 links, 322 files)
scripts/docs/backlog_citation_check.py	OK (2 pre-existing advisory warnings, unrelated)
scripts/docs/backlog_status_check.py	OK — 664 backlog items, each declaring exactly one status

Not run here, and they need reading on the runner: windows-service-smoke and
anything else that only runs hosted. I did not run the full suite in one pass; the
suites named above are the ones this change reaches.

One thing I noticed and did not change

docs/README.md:3 claims "377 markdown files under docs/". find docs -name "*.md" | wc -l returns 285 before my change. The claim was already stale by a wide margin
and I could not reconstruct what population it counted, so correcting it to a number
I cannot defend seemed worse than leaving it. Worth its own item.

🤖 Generated with Claude Code