feat(api): define and enforce the operator API's input-validation rules (BACKLOG #1108)

docs/API-INPUT-VALIDATION.md

@@ -0,0 +1,140 @@
+# Operator API input validation
+
+**This page defines what the engine's operator API accepts for each kind of data item you send it.**
+It covers the control plane: the ids, connection names, time bounds and search terms an operator or a
+client sends to the engine. It does not cover message payloads. Those are the data plane, and they
+have their own documents: [HL7-VALIDATION.md](HL7-VALIDATION.md) and [CODESETS.md](CODESETS.md).
+
+The rules live in one module, [`messagefoundry/api/validation.py`](../messagefoundry/api/validation.py).
+Everything below is quoted from it, and `tests/test_api_input_validation.py` fails if this page and
+that module ever disagree.
+
+A value that breaks one of these rules gets an HTTP 422 with the field named. The engine refuses it
+before the value reaches a database query, a filesystem path, a log line or a CSV export.
+
+---
+
+## The rules
+
+| Data item | What it may be | Where you send it |
+|---|---|---|
+| Resource id | Exactly 32 lowercase hex characters | `message_id`, `file_id`, `approval_id`, `preset_id`, `user_id`, and each entry of an export `ids` list |
+| Digest id | Exactly 64 lowercase hex characters | `attachment_id`, `session_id` |
+| Custom role id | `custom:` then 32 lowercase hex characters | `role_id` on the `/roles/custom` routes |
+| Role id | A lowercase word, or a custom role id | Each entry of a `roles` list |
+| Permission id | `area:action`, lowercase letters and underscores | Each entry of a `permissions` list |
+| Connection name | A letter, then letters, digits, `_` and `-`, up to 256 characters | `{name}` in a path, and `channel_id`, `destination_name`, `to`, `source`, `connection` |
+| Time bound | A finite number from 0 up to 4102444800 (2100-01-01 UTC) | `received_from`, `received_to`, `since`, `until` |
+| Free text | Printable text, no control characters, up to 512 characters | `content`, `field_value` |
+| Vocabulary token | Letters and underscores, up to 64 characters | `status`, and each `kind` on the event routes |
+| Message type | Printable text, no control characters, up to 64 characters | `message_type` |
+| Control id | Printable text, no control characters, up to 256 characters | `control_id`, and `actor` on the audit routes |
+| HL7 field path | A three-character segment id, a field number, then optional component and subcomponent numbers, such as `PID-3` or `PID-5.1` | `field_path` |
+| Email address | One `@`, a local part with no spaces, and a dotted domain, up to 254 characters | `recipient_override` |
+
+"Printable text, no control characters" means every character except the C0 range, DEL, and the C1
+range. In practice: no NUL, no tab, no carriage return, no line feed.
+
+Some lists have a length of their own. A message export may name at most 100000 ids explicitly, the
+same ceiling as its `limit`. An events request may filter on at most 32 event kinds. A directory
+group mapping, and a counter-reset request, may carry at most 1000 entries.
+
+**Which words and codes exist is not decided here.** The role names, the permission catalog and the
+status vocabulary belong to the engine. These rules decide only the shape a value may take, so a
+value that could not be a member is refused early and one that merely does not exist gets a 404.
+
+---
+
+## Why the rules are drawn where they are
+
+**An id is minted by the engine, never typed by a person.** Every id above comes back to you in an
+earlier response. So the rule can be exact, and being exact is what makes it useful: no `.`, `/`, `\`
+or NUL survives it, which is why an id can never be read as a file path. The upload store already
+shipped this rule for one id; the module generalizes that rule rather than writing a second one.
+
+**A connection name is wider than the VS Code extension allows, on purpose.** The extension's wizard
+rejects a hyphen. Four connection names shipped in this repository contain one, so adopting the
+extension's narrower rule would make four connections unreachable through the API. The rule here
+admits them. What it still excludes earns its place: path characters, whitespace, control characters,
+and the quoting characters a value would need to carry meaning into a URL or a query.
+
+**A time bound must be finite, and that was the gap.** A lower bound of zero does not exclude
+infinity. Before this rule, `?received_from=inf` was accepted and reached a database query, and the
+audit routes accepted `?since=nan` as well. A NaN bound is the worse of the two, because every
+comparison against it is false, so the filter would return nothing rather than fail.
+
+**Free text can only be ruled on by what it must not contain.** A search term is whatever an operator
+typed to find a patient, so no alphabet rule fits it. What does fit is the control characters. These
+terms reach the search audit record, the application log and, through the audit export, a CSV file. A
+carriage return or a line feed in one of them would forge a second record in any of the three.
+
+That rule costs one capability, and this page states it rather than hiding it: a search term can no
+longer span an HL7 segment separator, because that separator is a carriage return. The console's
+search box is a single-line input, so no shipped client could send one.
+
+**Two items keep a rule that is not a pattern, and the pattern in front of them does not replace it.**
+
+1. A reload `config_dir` is confined by an allow-list, because the loader executes Python from that
+   directory. The shape rule adds the NUL a path check can be truncated by. **The allow-list is still
+   the control.**
+2. A log `level` is checked against the engine's own level names, which is why a wrong one gets a
+   400. The shape rule only keeps an arbitrary-length string out of that error message.
+
+**One rule is documented here but enforced elsewhere.** The HL7 field path grammar belongs to
+`messagefoundry.parsing.peek.parse_path`, and `messagefoundry.store.content_search.make_spec` applies
+it at every point the API accepts a `field_path`. A malformed path is already a 400. Copying that
+pattern into the API models would create a second definition of a rule that has one.
+
+---
+
+## What these rules do not cover
+
+**The data plane is untouched.** A message body reaching the engine over MLLP, a file, TCP, HTTP or a
+database poll is validated by the parsing and connector rules, not by anything on this page. The
+edit-and-resubmit body is the one place a message body arrives over the API, and it deliberately
+keeps a size bound and no alphabet rule: an HL7 v2 body is separated by carriage returns.
+
+**The web console declares its own bounds for the same items.** The console's `/ui` routes carry at
+least 17 of their own parameter declarations for items this page governs, and they are hand-written
+copies rather than references to this module. They can drift. Two of them are a different data item
+that happens to share a name: the console's `received_from` and `received_to` are `datetime-local`
+strings from a browser form, not the epoch numbers the engine API takes.
+
+**The console reaches some handlers in process, which skips this validation entirely.** The console
+is mounted inside the engine and calls a set of handler callables directly rather than over HTTP. A
+direct call runs no request validation. Where the console builds an engine request model, these rules
+do apply. Where it calls a handler with plain values, they do not.
+
+**Several hundred response fields carry no rule, and they should not.** A response field is something
+the engine emits, not something you send. It is not an input, so an unbounded response field is not a
+gap. Counting one as a gap manufactures a number that cannot be closed.
+
+**These other input surfaces are not covered here.** The engine's inbound HTTP listener, the command
+line, and the `connections.toml` file each accept operator input and each carry their own rules or
+their own absence of rules. Establishing what they should be is separate work.
+
+**The engine does not yet enforce the connection-name rule at registration.** A connection registered
+in code or in `connections.toml` under a name this page rejects would be created and then would not
+be reachable through the API. Nothing in the shipped samples, harness or tests has such a name.
+
+---
+
+## Two questions this page does not answer
+
+**Does the standard ask that rules be written down, or that they exist to be written?** This page and
+the module behind it take the harder reading and do both. Whether the softer reading would also have
+been acceptable is a method question. It is recorded in BACKLOG #1108 and is not settled here.
+
+**Does the existing HL7 and codeset documentation satisfy the same requirement for the data plane?**
+That would make this a control-plane question rather than a whole-product one. It is the second open
+question in that item, and this page deliberately does not fold the data plane into its answer.
+
+---
+
+## The generated schema is not this document
+
+The engine can produce an OpenAPI schema, and that schema does carry every pattern above. It is off
+by default, and turning it on would not be a documentation change. A schema lists types and
+constraints. It does not say why a rule is drawn where it is, what it costs, or what it does not
+cover, which is what the three sections above are for. Turning it on widens the network surface, so
+leave `[api].expose_docs` at its default unless you have a separate reason.

docs/BACKLOG.md

@@ -7528,6 +7528,24 @@ The refusal names the scheme it found. A base URL with no scheme fails here too,
 
 **Proposed work, by subject, all unallocated:** the operator-surface input-validation reference, scoped to the full input surface measured above rather than to `api/`; the engine-side connection-name grammar, adopting the rule the IDE already enforces at `ide/src/connectionWizardModel.ts:103` (note it has exactly ONE call site, `ide/src/connectionQuickInput.ts:112` -- the IDE's webview authoring path applies neither it nor the port range); the email-address validation rule at its three acceptance points, where `grep -rn EmailStr messagefoundry/` returns zero against a control of 26 `Field` uses in `auth_models.py`; the connector port-range rule the IDE and `[logging].forward_port` already enforce; and an input-rule doc-drift gate in the established house style, AST-walking `transports/` and the CLI as well as `api/` and the console routes, with a planted omission that must fire. Also owed: a record correction on the denominator, and a reconciliation note on the two readings applied to one section on one date.
 
+**2026-09-04: THE CONTROL-PLANE BUILD SHIPPED. THE ITEM STAYS OPEN, because its closing act is not this seat's to perform** -- the recorded act is `scorecard-rescore`, which the ASVS Tracker performs in the vault, and only then does a Lander flip the banner. Two seats, neither of them the builder's.
+
+**What shipped.** `messagefoundry/api/validation.py` is the single authority for the operator API's control-plane input rules, and `docs/API-INPUT-VALIDATION.md` is the reference a deploying operator reads, linked from `docs/README.md` under "Security reviewers". Rules were decided and applied for every category this item names -- engine-minted ids, connection names, glob patterns (see below), epoch time bounds, and free-text search fields -- plus the role, permission and vocabulary values sitting beside them. `tests/test_api_input_validation.py` (24 tests) pins each pattern, sweeps the shipped connection names, and carries a doc-drift check with a doctored-text control that must fail.
+
+**Measured with ONE instrument, the app's own generated OpenAPI schema, at the parent commit and at the tip.** Route parameters, 101 both sides: carrying a structural rule 5 -> 65, length-bound-only 25 -> 4, **unbounded strings 36 -> 0**, uncapped arrays of unbounded strings 3 -> 0. Request-body properties across the 31 models reachable from a route, 92 both sides: carrying a structural rule 0 -> 40, length-bound-only 61 -> 27, unbounded strings 1 -> 0, uncapped or unbounded collections 8 -> 0. The earlier AST census that counted `Query(max_length=...)` kwargs was retired mid-task because it stopped answering the question once the constraints moved into the annotation.
+
+**Two measurements decided rules that a reasonable reader would otherwise have drawn wrong.** First, the connection-name grammar is deliberately WIDER than the IDE rule this item proposes adopting: 4 of the 108 distinct connection names in `samples/`, `harness/`, `tests/` and `messagefoundry/` carry a hyphen (`FILE-OUT_ACME_ADT`, `FILE-OUT_Coverage`, `FILE-OUT_EXAMPLE_ADT`, `FILE-OUT_Test_ADT`), and `ide/src/connectionWizardModel.ts`'s `^[A-Za-z][A-Za-z0-9_]*$` rejects all four, so adopting it verbatim would have made four shipped connections unreachable through the API. Second, `ge=0` does not exclude infinity: `?received_from=inf` was accepted and reached a store bind, and `/audit` declared no bound at all, so `?since=nan` was accepted -- a NaN bound is the worse of the two, because every comparison against it is false and the filter silently returns nothing.
+
+**One correction to this item's own proposed work.** It lists the email rule at "three acceptance points". Only ONE of the three is an operator-supplied address the API accepts as an address: `AlertTestEmailRequest.recipient_override`, which now carries an `EmailAddress` rule. `UserCreateRequest.email` and `UserUpdateRequest.email` were deliberately NOT changed, and the reason is measured: `tests/` fixtures use `bob@x`, `dr.who@x`, `j@x` and a whitespace-only value, all of which the shipped rule rejects. Whether a user's profile address must carry a dotted domain is a real question about intranet addressing, not a bug, and it needs deciding before the rule moves there.
+
+**The glob category has no subject on this surface, and that is the finding rather than a skipped limb.** A glob is a file-connector setting consumed at `messagefoundry/transports/file.py:776-778` (`directory.glob`/`rglob`); the only console-to-`connections.toml` write seam is `POST /connections/{name}/flag`, which carries `flagged` and `direction` and nothing else. `grep -rniE "pattern|glob|wildcard"` over `api/models.py` returns one unrelated hit. So the glob rule is a connection-configuration rule, and enforcing it belongs with the `connections.toml` surface.
+
+**What remains, stated so nobody reads the above as more than it is.** (a) The web console's `/ui` routes declare at least 17 of their own parameter bounds for items this module now governs; they are hand-written copies, they can drift, and two of them (`received_from`/`received_to`) are a DIFFERENT data item -- browser `datetime-local` strings, not epoch numbers. (b) The console reaches several engine handlers through the in-process seam rather than over HTTP, and a direct call runs no request validation at all; where it builds a request model the rules do apply, and two such sites were fixed here. (c) The engine does not enforce the connection-name rule at REGISTRATION, so a name authored outside the rule would be created and then be unreachable through the API; nothing shipped has such a name. (d) The inbound HTTP listener, the CLI's 153 `add_argument` calls, and `connections.toml` are untouched. (e) The 27 request-body properties still carrying a length bound alone are passwords, display names, message bodies and other free text where an alphabet rule would be wrong, not missing.
+
+**Two questions this build deliberately did not settle, both of them this item's own.** Whether the verb asks only that rules be written down or that they exist to be written -- the build took the harder reading and did both, which does not decide the method question. And whether the existing HL7 and codeset documentation discharges the verb for the DATA plane, which would make this a control-plane-only cell. The data plane is untouched here on purpose; `docs/API-INPUT-VALIDATION.md` names both questions in its own text so a reader cannot mistake the page for an answer to them.
+
+**A finding that generalises past this item: ADDING AN INPUT RULE CAN SILENTLY DISARM AN EXISTING TEST, and the near-miss here is worth the paragraph.** `packaging/messagefoundry-webconsole/tests/test_webui.py::test_error_banner_escapes_hostile_input` was the reflected-XSS regression guard for **every** rerender-with-error path in the console. It worked by posting a hostile role id, relying on `_validate_roles` echoing it back into the 400 detail, and asserting the banner escaped it. The new role-id rule refuses that value at `RolesUpdateRequest`, so it never reaches `_validate_roles` and the route renders its own fixed `"invalid input"`. **Two of that test's three assertions still passed** -- the 400 status and `hostile not in r.text` -- and only the escaped-form assertion went red. Deleting it, which is the obvious "fix", would have retired the guard for every other path as an unnoticed side effect of an unrelated change. It was instead split: the route test pins the stronger new property with a well-formed-but-unknown role id as the control that the other arm still runs, and the escaping is re-armed directly against `pages.user_detail_page`, where no upstream rule can disarm it. The same hazard applies to the seven other tests this build had to update, and to whoever does the console, transport-plane or CLI halves next: a test that reaches its subject THROUGH a value an input rule now rejects stops measuring its subject while still passing.
+
 ## 1109. research an honest pass for ASVS 2.2.1 -- positive validation that does not sacrifice the tolerance the HL7 default exists to protect
 
 > 🔢 **Re-scored 2026-08-20 -> P2.** Value **7/10** · Difficulty **6/10** · _big bet_. Both limbs stand -- validation.strict ships False at config/models.py:667 and api/models.py carries no model_config across 84 models, against five extra=forbid declarations in config/models.py. Value 7 not 8: the 2.2.1 cell is graded at LEVEL 1, so rung 8's ASVS L3 Partial limb does not reach it, and an authenticated loopback-bound API silently ignoring unknown body keys is not rung 8's production blind spot with no workaround -- it is rung 7, a real gap an operator cannot close from outside the app. Difficulty 6 stands: two independent limbs, a method ruling on which clause binds an L1 requirement inside an L3 assessment, and an API-side positive-validation change that would reach the console and apiclient callers too. _(was 8/10 · 8/10.)_

docs/README.md

@@ -87,6 +87,7 @@ catalog, and **[../CHANGELOG.md](../CHANGELOG.md)**, which is authoritative for
 | [../.github/SECURITY.md](../.github/SECURITY.md) | **Vulnerability disclosure policy.** Report here. |
 | [SECURITY.md](SECURITY.md) | Authentication and RBAC — *not* the disclosure policy, despite the name. |
 | [PHI.md](PHI.md) | Where PHI can and cannot go, and what the engine guarantees. |
+| [API-INPUT-VALIDATION.md](API-INPUT-VALIDATION.md) | What the operator API accepts for each kind of data item you send it — ids, connection names, time bounds, search terms — and which input surfaces those rules do *not* reach. Control plane only; message payloads are `HL7-VALIDATION.md`. |
 | [ASVS-L2-PHASE0-CHANGES.md](ASVS-L2-PHASE0-CHANGES.md) | §4 key/crypto inventory and §5 communications inventory, both CI-drift-guarded. §1–§3 are a historical phase changelog. |
 | [Secure_Development_Standards.md](Secure_Development_Standards.md) | The standards the build process holds itself to. |
 | [SECURITY-LOOSENING.md](SECURITY-LOOSENING.md) | The inverse of a hardening guide: every `[security]` switch defaults to the protective position, and this is what moving one off it costs. |