Add opt-in Gather/Reason/Render orchestration harness

[Paul Lizer (paullizer)]

Summary

• Add an opt-in contract-v2 Gather / Reason / Render dependency graph with durable typed named results, completeness, current source authorization, and same-owner/same-conversation reuse. Analyze, Compare, and native tabular work retain results without automatically publishing files.
• Connect explicit Render tasks to the shared framework for CSV, XLSX, DOCX, PPTX, PDF, JSON, XML, YAML, Markdown, and TXT. Several files can consume one retained result; prepared-content rendering does not call a model.
• Add private per-file commits/downloads, an initial attempt plus two eligible automatic retries, separate durable manual retries, native continuation, restart recovery, and preservation of successful files across sibling failure and Stop.
• Integrate initialized services, authenticated planning/execution/history/download routes, and V2 plan/binding/output/retry UI. Both administrator switches remain default-off. Workflow integration, new media generation, external delivery, and cross-conversation memory remain deferred.
• Integrate the current paullizer-react-v2-ui through 543de8a7 (v0.261.129). Harness replies now pass chat content checks, and Auto model routing stays on standard orchestration.

Linked issue

Refs #1509. Includes all of draft #1510's foundation work: 18fc08ac directly and 9d978a9a, which is patch-identical to #1510's b43164d9. Merging this PR delivers the foundation as well, so #1510 does not need to merge separately; it is left untouched as a draft.

Target branch integration

The branch now contains the current target through 543de8a7daff0f12369d72a0fd482dd68acb2012 via two non-force merge commits. The reviewed harness history is unchanged. GitHub reports the PR as mergeable (clean), and git merge-tree against the target is conflict-free. The base branch has no protection rules, rulesets, or required checks; license/cla and enforce-branch-flow pass.

Merge	Target content	Conflicts
28b231df	Through cc649740 (v0.261.128): model catalog profiles and per-step Auto routing, configurable chat content checks and admin rechecks, the admin settings/Key Vault fix, V2 admin version status, and the ACR build fix	13 files, resolved semantically rather than by choosing a side
c79d22e8	Through 543de8a7 (v0.261.129, #1524): M365 agent stream failure persistence, mail/calendar history search, and a caller-owned stream Context	The version line and one test import block; the runtime changes auto-merged beside invocation capture

Integration decisions:

• Auto model routing stays on standard orchestration. Per-step Auto bindings are enforced by the step executor's per-step model scope, which the dependency executor does not have. With Auto selected, new plans use contract v1 even when the harness is enabled. v2 planning rejects Auto, and a v2 plan carrying Auto fails closed with model_routing_changed. Per-step model scopes for dependency tasks are a follow-up.
• Harness replies pass chat output checks before publication. A blocked reply is replaced by the safety notice and publishes no files or citations. Check-before-display withholds output until the check completes. An administrator retraction survives republication. Private check metadata never reaches stream frames or history. Run detail and list routes hide files for removed or pending replies.
• Plan records carry both contracts. Revisions, checkpoints, step events, and executor records keep model_binding beside the v2 role/inputs/outputs fields without changing existing fingerprints.
• Target publication probe fix. After a failed batch, the target's publish_message retraction probe could raise not-found. That masked typed authority uncertainty and produced a durable failed publication. A failed probe now re-raises the original error; the scheduler uncertainty tests caught this.
• Stale route extraction tests, streaming test frames, and the research-planning evaluation script were updated for the merged contracts. The second merge needed no integration code: harness agents use the non-streaming invoke_scoped_agent path, and orchestration actions load the new read-only M365 functions generically.

Recovery hardening

The earlier 403c4820 follow-up atomically persists the original v2 start/deadline/lease and admits a guarded first checkpoint binding after an early interruption. It also preserves retained maps and exact citations, refreshes active export admission, and safely initializes default deletion enrollment. 993a5730 clarifies the authority-uncertainty documentation.

Commit e8104416f12695d334e5af8fa6d110b02db63dbf completes the reviewed 14-file, version 0.261.130 reconciliation:

• Initial preparation calls the existing initialized lower store binder without importing a continuation owner, while rechecking the approved run/actor/conversation/attempt before lifecycle creation.
• Preflight, capture, and admission callbacks freeze their original owner/token/claim tuple. Mutating the original lease object cannot let cached callbacks borrow a successor claim; successful and failed callbacks both recheck ownership.
• The verified four-callback contract remains intact: real invocation preflight, configuration capture, result admission, and independent current authorization. Capture separately enforces current/actual acquisition support. No no-op hooks or capability widening were introduced.
• Typed lifecycle controls propagate before ordinary step-error mapping. Initial finalization and model-free refresh preserve retained work on recognized operational read/configuration failures. Exact opaque observation failures are kept distinct from the renderer's verified denied/held-file projections.
• Waiting Render recovery forwards the complete saved StepResult to the single public read-only resumer. Optional wait error codes and output diagnostics survive checkpointing; legacy two-field waits remain valid. Completed checkpoints use the same helper. No rendering, upload, retry, or target/sibling expiry occurs inside saved-step observation.
• Combined validation exposed a test-fixture import-lifetime defect, not a production type-check defect: a late catalog-test import retained an old transport class after offline bootstrap unloaded its module. Collection-time helper import fixes the exact failing order, with fresh normal/optimized subprocess coverage and no relaxed runtime guard.

Native producer tokens, result-store claim protocol, checkpoint wire versions, legacy fingerprints, and standalone/workflow behavior remain unchanged. Historical started records without valid timing are refused rather than assigned a fresh budget.

Release Notes & Latest Features

• New Feature
• Bug Fix
• UI Enhancement
• Breaking Change
• Internal only

Visible to users: Yes, only after explicit administrator opt-in. Both enable_chat_orchestration and enable_chat_orchestration_harness must be enabled; both default to false. Saved-contract dispatch remains independent of new-plan admission.

Latest Feature card: Not added. Screenshot: N/A. Release notes and issue comments remain deliberately untouched under the delivery boundary.

Version bump

• Application third segment: baseline 0.261.124, foundation .125, shared renderer .126, integration .127, first recovery hardening .129, reconciliation .130, target integration 0.261.131. The target independently uses .128 and .129.
• No deployer version bump required; no deployers/ files were changed.

Testing / validation

External storage/model/directory/provider I/O is doubled. Tests exercise real production modules, initialization, authorization, lifecycle fences, persistence, dependency execution, and downloads. No paid model calls, deployments, tenant permission changes, or cloud-throughput claims. Live-environment validation is performed by the maintainer and is not recorded here.

Target integration

Scope	Result
New chat-check and routing regressions	Blocked and passing replies, check-before-display pending state, preserved admin retraction, v2 Auto fail-closed, planner Auto rejection, Auto admission, and removed-reply file hiding through real HTTP detail/list routes. Harness route and chat-check suites: 162 passed in each of normal and optimized Python
40-suite harness, chat-check, and model-catalog matrix	2,332 passed +324 subtests normal; 2,333 passed +324 subtests optimized; 2 known xfails; one normal-mode cold-import timeout under parallel load passes 6/6 in isolation; 86 source hashes unchanged
All 113 orchestration suites (three chunks)	860, 2,739, and 1,508 passed after fixes. The publication-probe fix resolved 22 scheduler uncertainty failures
Target's changed suites (first merge)	405 passed +550 subtests. The same 7 failures and 4 errors occur on the pristine target (Flask test_client() with the installed Werkzeug, release-checker network probe)
Second-sync delta suites (delegation, M365, Graph, orchestration/analyze importers)	1,285 passed +184 subtests in each mode. The 77 failing IDs match across modes and match pristine 543de8a7, except one pre-existing stale-stub elicitation case and one target-only cold-import timeout
Browser suites	26 files: 394 passed in the combined run. Its conversation-context, recovery-backend, and approval-persistence failures were ordering contamination and pass in isolation (5, 9, 23). Composer 6/6 through its runner. Content-screening browser failures match pristine cc649740. The target's new chat stream banner test passes 11/11; the V2 content-check Node test passes
V2 UI	tsc -b clean; production build succeeds
Route policy and documentation	Route policies 8/8, 4/4, 2/2; coverage 7/7; source quality 6/6; regenerated inventory is current and unchanged by the second sync
Guardrails	CI-equivalent added-line BAC (78 files) and XSS (79 files) checks pass over the full PR diff. Full-file scans of target-owned legacy files show the same findings on the pristine target

Pre-integration branch validation (e8104416)

Scope	Result
Final exact nine-suite headless/HTTP/scheduler/claim/import order	1,145 passed +63 subtests in each of normal and optimized Python; 2 unchanged documented startup xfails; all 18 source hashes unchanged across both runs
Final 17-suite core/native/Render/control/capture selection	782 passed in each mode; all 21 source hashes unchanged
Previously failing fixture-order regression	Minimized catalog/scheduler order reproduced 3 passed/1 setup error before repair; exact order and fresh normal/optimized subprocess regressions now pass
Actual callbacks and ownership	Four-binding admission, actual provider preflight, stopped/stolen/retagged claims, post-effect loss, typed errors, direct binding, and immutable store views covered
Actual file recovery	Full original waiting/completed checkpoints passed once to the shared helper; typed failures preserve checkpoints; target/sibling no-expiry guards; CSV/JSON downloads verify the final retained row
Earlier version .129 joined acceptance	2,378 passed +276 subtests in each mode, same 2 baseline xfails; 49 source hashes unchanged
Original ten-format rendering/Office/legacy JSON	537 passed in each mode; two complete 30,000-row exports exceed 8 MiB and preserve original data after restart
Final documentation and legacy plan checks	Regenerated inventory unchanged/current; coverage 7/7; source quality 6/6; standalone plan schema 6/6
Final source guardrails	Full-file BAC/XSS on all 3 changed runtime modules; committed added-line checks on all 4 production paths including config; syntax/version headers and whitespace clean

Selections overlap historical coverage; these are not additive unique-test totals. The nine-suite run retains the exact ordering that previously failed rather than relying on a larger collection order to hide it.

The real HTTP acceptance starts at /plan without a fabricated run or readiness override, ignores a browser-supplied legacy contract version, and creates no files before approval. It checks the configured budget before preparation and verifies unchanged timing afterward. One retained draft produces Markdown and PDF without replay on fresh observation. Other actual pipelines cover Analyze → CSV/Markdown, Compare → JSON/Markdown, native wait → CSV/JSON, and URL Gather → JSON.

Known unchanged baseline limits:

• Two scheduler-first mixed-bootstrap cases expose NameError(enabled_required); normal web and separate scheduler entry points pass.
• The old tabular scale runner has stale AST fixtures missing existing authorization/checkpoint dependencies and an unbounded wait. No complete scale-runner pass is claimed.
• One unchanged conversation-route full-file access-checker finding remains outside these follow-up paths; edited lines pass. No suppression was added.
• Target-inherited test failures reproduce on the pristine target and are not changed here: stale AST namespaces in delegation/analyze suites (for example missing check_chat_content, M365ApprovalRequired, resolve_action_type), credential-bound msgraph suites, Flask test_client() with the installed Werkzeug, and content-screening browser assertions.
• test_orchestration_elicitation_context (stale functions_source_review stub) and the standalone test_orchestration_conversation_context runner fail identically on the pre-merge head and the target.
• Documentation checks are source checks, not a built-site claim. Existing third-party deprecation warnings remain.

Documentation

• Release notes updated — intentionally deferred.
• Feature documentation, admin guidance, user guide, chat-control references, and inventory updated.
• Fix documentation covers initialized binding, immutable claim ownership, typed read/control recovery, initial timing, deletion enrollment, and the chat-check/routing integration.

Primary references:

• docs/explanation/features/ORCHESTRATION_RENDERING_HARNESS.md
• docs/explanation/features/ORCHESTRATION_CHECKPOINT_RECOVERY.md
• docs/explanation/fixes/ORCHESTRATION_INITIAL_RESULT_BINDING_FIX.md
• docs/explanation/fixes/ORCHESTRATION_RUNTIME_BOUNDARY_HARDENING_FIX.md
• docs/explanation/fixes/ORCHESTRATION_HARNESS_CHAT_CHECKS_ROUTING_INTEGRATION_FIX.md

Security checklist

• New Flask routes retain Swagger security decorators and route-policy coverage.
• Non-admin frontend settings use sanitized projections.
• Browser runtime assets remain local; no CDN-hosted JavaScript was added.
• No secrets, private logs, or session artifacts are included.
• Consumption rechecks current ownership, source ACL/screening, producer/attempt, tombstones, and immutable claims. Full readers must be exhausted before publication; partial input cannot become complete coverage.
• Harness replies pass the configured chat output checks; blocked or removed replies expose no files, citations, or private check metadata.

Ready for review. Auto-merge is off, and no deployment or issue closure is part of this PR.