Phase 6b-1: post workflow results back into chat (server and status route)

[Paul Lizer (paullizer)]

Summary

• Workflow results now come back to the chat that asked. When a chat-orchestration plan starts one of the user's saved personal durable workflows (Phase 5 workflow_run), the plan still ends right away, but the server now posts the run's terminal outcome back into that private chat, even hours later. Each delivery generation gets exactly one post under a dated label (Results from `X` · you asked on …), one unread mark and one bell notification. The post is either a reply composed from a bounded, nonce-fenced excerpt (6a's reader and fence) or a fixed application note: failed, cancelled, skipped, saved analysis, content blocked, or status.
• When the chat can't take the post, because it was deleted, made shared, access was lost or the record expired, the server posts nothing and sends one fixed bell notice with a run link. A deleted workflow, a tombstoned runtime control or a missing control closes silently.
• For 6b-2: this adds a batched, owner-only status route, GET /api/v2/orchestration/workflow-runs/status[?conversation_id=…], and a fixed message-metadata contract (metadata.workflow_delivery). Classic Retry/Edit refuse any message a workflow run posted, with a fixed 400 {error, code}.
• Admin impact: it's all behind the existing Use Workflow Results In Chat (enable_chat_workflow_results, default Off), checked role-aware at start and again at delivery, plus the existing Run Workflows From Chat gates. With the setting off, behavior is byte-identical to the base; the off-golden test pins this.
• It ships no V2 UI. The D17 render check found no fix needed (see Message metadata for 6b-2).

Linked issue

Refs #1546
Refs #1543

Part of #1546 (Phase 6, step 6b-1, the server half) and the #1543 umbrella. 6b-2 (WorkflowRunCard, app-shell tracker, WorkflowRunningTag, V2 run deep link) stacks on this branch and builds against the status-route contract below.

Release Notes & Latest Features

• New Feature
• Bug Fix
• UI Enhancement
• Breaking Change
• Internal only

Is this visible to end users?

• Yes
• No

Is this admin-facing (Admin Settings, governance, deployment, config)?

• Yes
• No

Should this become a Latest Feature card?

• Yes
• No
• Already added

Screenshot needed for the card?

• Yes
• No
• Attached

Version bump

• application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only (0.261.226 → 0.261.227; first .218, renumbered after V2 merges, see Step 9 overlap)
• deployers/version.txt bumped, or not needed because deployers/ was not changed (not changed)

Testing / validation

Environment: Python 3.12 venv, Windows. The related-suite comparison used PYTHONPATH=application\single_app;functional_tests; the delivery suites now run without it. Base for the related-suite comparison: b5cc18ee2, then the paullizer-react-v2-ui tip and this branch's merge-base, in a clean detached worktree.

Before opening I merged origin/paullizer-react-v2-ui with git -c rerere.enabled=false merge; it was already up to date at b5cc18ee2. Step 9 (#1607), then #1612 (with #1608) and #1620 merged into V2 first, so edd748417, 35ce8ed58 and 389c9b14b merge each new tip the same way (Step 9 overlap). VERSION is 0.261.227. My last production change is 74f420870; my later commits touch only tests and docs, apart from the V2 merges and two renumber commits. The related-suite comparison ran at 766fea19d. At edd748417, after the review-survivor tests (Mutations → Review survivors), and again at the head 543f1a5a8, I reran the delivery suites four ways (table below) and the validation set below, which gave the same counts, plain and -O. At 543f1a5a8, the 22 test files V2 changed since 8a802f196 (#1608, #1612, #1620), including #1608's test_workflow_background_identity.py, pass plain and -O, except test_simplechat_conversation_messages.py (3 failures, the same on a clean 1524eeb7b: no Cosmos credential). Three UI files and test_v2_orchestration_workflow_results_input.py pass once the gitignored ui_tests/artifacts/orchestration-plan-editor build exists. After each V2 merge the UI suites need a local npm --prefix application/v2_ui run build (the bundle is gitignored); tsc -b passed. Step 9's suites and related ones (44 files) also ran on edd748417 and on a clean 8a802f196, plain and -O (Stalls under load).

Delivery suites: 13 files, 383 tests, four ways

Remove-Item Env:PYTHONPATH -ErrorAction SilentlyContinue   # each suite sets up its own sys.path
$files = Get-ChildItem functional_tests -File -Filter 'test_workflow_chat_delivery_*.py' | ForEach-Object { "functional_tests\$($_.Name)" }
python -m pytest @files -q -p no:warnings -p no:cacheprovider
python -O -m pytest @files -q -p no:warnings -p no:cacheprovider
foreach ($f in $files) { python $f }       # each file's own __main__ runner
foreach ($f in $files) { python -O $f }

Mode	Result
python -m pytest (13 files together)	383 passed in 389.88s
python -O -m pytest (13 files together)	383 passed in 431.47s
python <file>, once per file	13/13 files exit 0; 383 passed in total
python -O <file>, once per file	13/13 files exit 0; 383 passed in total
Leftover functional_tests\.workflow-result-chat-state-* directories	0

Docs

python .\scripts\build_docs_inventory.py                      # docs/_data/app_surface.yml: no diff
python .\functional_tests\test_docs_app_surface_coverage.py   # 7/7 passed
python .\functional_tests\test_docs_site_quality.py           # 6/6 passed

Related suites against the base, normal and -O

Each of 221 files ran on HEAD and on a clean, detached worktree of the base b5cc18ee2, using python [-O] -m pytest <file> -q -rfE -p no:warnings -p no:cacheprovider, 6 files at a time with a 1,200 s limit per file. The 221 files:

• the 13 new delivery suites
• 10 route-policy suites
• 4 V2 UI suites: test_v2_workflow_loops.py, test_v2_group_participant_invites.py, test_v2_notifications_bell.py and test_v2_orchestration_run_hydration.py
• 194 functional suites, including the two test: fix V2 test-harness drift (chat image refs, auto-open fixture, harness seeds) #1598 functional suites: 162 import or name a changed module; 32 cover adjacent behavior (notifications, unread state, Retry/Edit, Phase 5 workflow runs, docs)

Outcome for the 208 files that exist on both trees	Normal	-O
Pass on both	136	134
Fail identically on both: same failing tests, same pytest tail	67	67
Fail on the base only, because the base worktree has no node_modules and no built V2 SPA	4	4
Stalled under load, then clean when rerun alone (see Stalls under load)	1: test_log_injection_codeql_alerts_fix.py on HEAD	3: test_v2_chat_phase1_fixes.py and test_workflow_calendar_schedules.py on HEAD; test_public_document_publication.py on the base

Apart from the load stalls, no test fails on HEAD that passes on the base. The 67 files with identical failures are listed at the end of this section.

Route policy (normal and -O, same counts):

• route_tests/test_route_blueprint_policy_inventory.py: 11 passed (base 10). The new test pins the status route's decorator policy.
• route_tests/test_route_unauthenticated_policy_contract.py: 6 passed (base 5). The new test requires a signed-in session for the status route.
• route_tests/test_route_policy_test_coverage.py: 2 passed (base 2).

#1598 suites (normal and -O, same counts):

Suite	HEAD	Base b5cc18ee2
ui_tests/test_v2_workflow_loops.py	73 passed	73 failed: no built V2 SPA in the base worktree
functional_tests/test_group_collaboration_source_storage_fix.py	4 failed, 20 passed, 102 subtests passed. The 4 are the known non_group_invitee subtests: storage regular and group in test_rejections_do_not_read_history_or_mutate_stores and test_v1_route_preserves_rejection_statuses_without_mutation	identical
ui_tests/test_v2_group_participant_invites.py	7 passed	1 passed, 6 errors: no built V2 SPA
functional_tests/test_v2_prompt_attachment_persistence.py	84 passed	84 passed

No new seed was needed. Of these, test_v2_workflow_loops.py exercises the changed _project_runtime_run and run save.

Other V2 suites (normal and -O):

• ui_tests/test_v2_notifications_bell.py: 43 passed. The base worktree errors without the built SPA.
• functional_tests/test_v2_notifications_bell.py: 10 passed. On the base, 1 fails because its npm --prefix application/v2_ui step needs node_modules.
• ui_tests/test_v2_orchestration_run_hydration.py: 6 failed, 1 passed on both trees (pre-existing).

Stalls under load (environmental: CPython WMI hang)

A few fresh-process probes stalled, only during the comparison: 6 pytest processes across two trees, alongside another session's mutation runner. The imports test's 240 s faulthandler watchdog caught the stack: functions_authentication L8 → config L39 → functions_appinsights L12 → azure/monitor/opentelemetry/exporter/_utils.py L172 → platform.node() → _wmi.exec_query, which blocks (python/cpython#125315). The chain is byte-identical on the base, and this PR doesn't touch it. In a stall hunt, 155 of 156 cold-import probes finished, each within 21.1 s. Solo reruns were clean: test_v2_chat_phase1_fixes.py -O (5/5 runs, 10 passed each), test_workflow_calendar_schedules.py -O (5/5 runs, 512 passed each) and test_log_injection_codeql_alerts_fix.py (44 passed, normal and -O; its one failure under load was a cold-import probe through the same chain).

The 67 files that fail identically on HEAD and the base (pytest tails; each is the same normally and with `-O`)

Functional-test names drop test_ and .py. Tails: F failed, P passed, E errors, S subtests passed.

File	Tail on HEAD and the base
admin_release_notifications_registration	2F, 2P
advanced_conversation_search_matching_fix	1F, 4P
agent_stream_tool_retry_fallback	1F, 1P
ai_connection_text_consumers	12F, 16P, 89S
analyze_workflow_publication_integration	8E
approval_notification_routing_fix	3F
backend_conversations_swagger_integration	5F
background_task_distributed_locks	1F
broken_access_control_findings_fix	1F, 4P
broken_access_control_guardrails_checker	1F, 3P
chat_completion_notifications	1F, 5P
chat_early_conversation_title	1F
chat_retry_thought_tracker_init_fix	1F, 1P
chat_stream_background_execution	1F
chat_stream_compatibility_sse_syntax	1F
chat_stream_debug_logging	1F
chat_stream_heartbeat_reattach	1F
chat_stream_history_context_fix	4F, 3P
chat_stream_lifecycle_observability	1F
chat_stream_new_conversation_reattach	1F
chat_stream_retry_context_init_fix	2F
chat_stream_stop_control	1F
chat_streaming_sse_metadata_fix	1F, 2P
chat_upload_personal_workspace_handoff	2F, 5P
collaboration_conversation_summary_export_fix	2F, 1P
collaboration_message_notifications	4F
content_screening_pipeline	6F, 26P
content_screening_reviews	10F, 53P, 56S
control_center_auto_refresh_schedule	1F, 1P, 4E
conversation_fork	17F
conversations_read_ownership_authorization	9F
cosmos_throughput_background_scheduler	1F, 1P
deployer_background_tasks_app_setting	1F, 4P
document_actions_and_comparison_feature	1F
document_analysis_feature	1F
document_analysis_progress_and_limits	1F
generic_workflow_automation	2F
group_collaboration_source_storage_fix	4F, 20P, 102S
group_file_share_approval_notifications	1F, 3P
group_notification_context	1F
group_workflows_feature	1F
gunicorn_startup_support	1F
message_ordering_with_retry	2F, 1P
mixed_source_chat_search_consistency	6F, 6P
notification_polling_redirect_guard	1F
personal_conversation_followup_authorization	6F, 1P
personal_workflows_feature	1F
scope_lock_collaboration_unlock_fix	1F, 2P
shared_conversation_file_approval_fix	3F, 13P
simplechat_operation_notifications	2F
startup_scheduler_support	1F
tabular_retry_sheet_recovery	1E
tabular_tool_error_retry_and_thoughts	2F, 1P
thoughts_feature	2F, 11P
v2_chat_context_picker	1F, 9P
v2_collaboration_visual_style_fix	1F, 22P
v2_conversation_deep_link	1F, 7P
workflow_access_controls	1F
workflow_cancellation	3F, 3P
workflow_file_sync_triggers	1F
workflow_m365_rebase_integration	9F, 18P
workflow_priority_alerts	2F, 1P
workflow_runtime_integration	4F, 10P
workflow_stepped_builder	1F, 2P
workflow_task_document_actions	2F, 14P
workflow_task_result_handoff	9F, 2P
ui_tests/test_v2_orchestration_run_hydration.py	6F, 1P

Documentation

• Release notes updated, or not needed (### **(v0.261.227)**, New Features: Workflow results posted back to chat)
• Feature documentation updated, or not needed (new docs/explanation/features/CHAT_WORKFLOW_RESULT_DELIVERY.md, the full 6b-2 contract; cross-links inserted in CHAT_ORCHESTRATION_WORKFLOW_RUNS.md, CHAT_WORKFLOW_RESULTS_FOLLOW_UP.md, docs/admin/orchestration.md, docs/admin/workflow.md, docs/guides/trigger-a-workflow.md)
• Fix documentation updated, or not needed (not a fix)

Security checklist

• New Flask routes include @swagger_route(security=get_auth_security()) (the status route: @swagger_route → @login_required → @user_required → @enabled_required('allow_user_workflows') → @workflow_user_required; route-policy inventory and the unauthenticated contract are updated)
• Settings sent to non-admin frontends use sanitize_settings_for_user() (no settings are sent; the status route returns a projected row shape and a boolean available)
• Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS (no browser JS changed)
• No secrets, keys, connection strings, or local-only artifacts are included

---

Decisions

The full rationale is in the feature doc. In short:

#	Decision
D1	Stale full-document upserts. Every save of a run carrying chat_delivery goes through a guarded save (save_personal_workflow_run): an ETag replace that re-reads and keeps the stored chat_delivery and chat_invocation, so the runner's long-held run dicts can't wipe them. Runs without chat_delivery keep the plain upsert. A check-before-compose at every phase before message_created means a lost reply is found, not re-posted.
D2	Generation = the runtime control version at the terminal projection. runtime/resume (store.resume / journal_request('resume')) bumps the version and keeps the run_id. A record that is delivered/undeliverable reopens only when control.version > generation and not control.deleted (A1). tombstone() is the only other write that bumps a terminal control's version, and it sets deleted. A fresh run never reuses a generation because it has a new run_id, and the message id is assistant_workflow_delivery_{fingerprint(run_id, generation)}. Expired records never reopen.
D3	Role-aware gates without a request. Delivery is seeded only when Use Workflow Results In Chat is on for the requester at start (_chat_delivery_seed checks the cheap setting, then is_chat_workflow_results_enabled_for_user). The requester's roles are stored in chat_delivery.requester_roles. At delivery, is_user_workflows_enabled_for_user and is_chat_workflow_results_enabled_for_user are re-evaluated against those stored roles. If workflows are off for the requester, nothing is posted and one access_lost notice is sent. If results are off, the post is a fixed status note with no result text. Role revocations after start aren't seen (see Residual risks).
D4	Lineage. The delivered answer carries 6a's workflow_result descriptor and workflow_result_contexts, so every read re-authorizes it (masking), Follow up works, and later turns inherit masking.
D5	Retry/Edit refusal. Classic routes refuse any message with metadata.workflow_delivery or the assistant_workflow_delivery_ id prefix: 400 {error, code} with workflow_delivery_retry_unsupported / workflow_delivery_edit_unsupported.
D6	Expiry. The record carries expires_at: the control's deadline_at plus a 24-hour grace (DELIVERY_GRACE_SECONDS). A v1 control has no deadline_at, so compute_expires_at uses the record's creation (or reopen) time plus the run's capped deadline, or the default, plus the same grace. A run that is still non-terminal past it, or paused on deadline_exceeded, closes expired with one fixed expired notice and no post. A run that is terminal but still unpublished past it posts nothing and closes undeliverable with reason expired_before_delivery and one undeliverable notice.
D7	Mid-stream deferral. Delivery defers while the chat has a live stream, or while its newest message is a user turn less than 10 minutes old (RECENT_USER_MESSAGE_SECONDS). Fifteen minutes after the first deferral (DEFER_MAX_SECONDS) those signals are ignored and it posts. A failed newest-message read defers rather than counting as empty.
D8	Trigger + sweep. Projection reconcile signals a hint queue (non-raising). A background loop, under a distributed lock and inside the Flask app context, drains hints, and a bounded sweep is the backstop. Each record has a lease (ETag claim).
D9	Exactly once: mark before create (accepted as written). Claim → re-check → compose → mark unread (guarded, skip_if_read_since) → create message (409 = already created) → notice (deterministic key) → record delivered.
D10	Compose. 6a's reader with excerpts, a 48 KB budget stepping down 24/12/6 KB on a pre-checked prompt size, a fresh fence nonce, an untrusted-data system instruction, check_chat_content on the composed answer, and a trimmed fallback that is also safety-checked. Token usage is logged like 6a's _log_usage, plus additional_context={'workflow_result_delivery': True} and idempotency_key=f'workflow_result_delivery:{M}:{attempt}' (A4).
D11	Application-owned texts. The label, notes and notices are fixed text from closed maps. The notice preview is fixed and never contains result text.
D12	Message id and placement. A deterministic id per generation (assistant_workflow_delivery_{fingerprint(run_id, generation)}). At post time the message gets its own thread (delivery_thread_id(M)), with previous_thread_id set to the newest message's thread, and a timestamp of now, or 1 µs after the newest message if that is later. It lands at the end of the chat, not under the turn that started the run.
D13	The chat_delivery record holds codes, IDs, times, roles and model IDs, never result text.
D14	Started-run note. The Phase 5 answer promises a chat post only when every started run in the plan recorded delivery (the sidecar's chat_delivery: True, set only for a queued or running run where chat_delivery_applies holds). It then uses WORKFLOW_RUN_DELIVERY_FOLLOW_UP ("I'll post the results here when the run finishes. You can also follow it in the workflow's run history in Workflows.") or its plural form. If any started run lacks delivery, the note keeps the Phase 5 text unchanged.
D15	Status route. See the contract below.
D16	Residual risks. Listed under Residual risks below.
D17	V2 render check. No V2 change is needed.

Terminal writes

Every write that can make a chat-started run terminal, expired or reopened is covered by the trigger (projection reconcile plus signal), the sweep, or both:

#	Terminal or reopen write	Coverage
1	Runner cancellation saves and cancellation merge writes	Guarded save preserves delivery; runtime projection reconciles and signals; the sweep catches pending terminal rows after a grace period.
2	Runner failure saves (failed)	Guarded save, projection trigger and sweep backstop.
3	Runner success and partial-success saves (completed, completed_partial)	Guarded save, projection to ready with kind: result, signal and sweep.
4	Runner final exception save (failed)	Guarded save, projection and sweep.
5	Non-terminal runner progress saves	Guarded save preserves chat_delivery and chat_invocation, including the double-fault path.
6	Durable projection _project_runtime_run	Keeps stored delivery, reconciles under ETag and signals when a generation becomes ready.
7	Runtime deadline expiry on the control	The next continue projects paused + deadline_exceeded; the sweep also catches expires_at.
8	Durable cancel and reject decisions	Projection maps the terminal state to the cancelled or failed note kind and signals.
9	Scheduler continuations	Covered through durable projection.
10	Personal cancel routes	Saved through the guarded path; the sweep catches pending terminal rows.
11	runtime/resume	The control version increases; projection reopens delivered or undeliverable records when version > generation and the control isn't deleted.
12	runtime/decision approve/reject	Approve returns to running; a reject's terminal state is handled through projection.
13	resume-failed	Not a trigger. It returns 409 for durable runs, and any non-durable replacement would have no chat_invocation.
14	Workflow deletion	A deleted workflow with the run present closes silently as workflow_deleted. A deleted run has nothing to deliver.
15	MCP and group workflow saves	Out of scope: they don't carry chat_delivery. Group runs belong to Phase 7 (#1550).
16	Signal failure	Projection never fails, because the signal is non-raising; the sweep catches up later.

Leak surfaces

Surface	Handling
Delivered result message (composed or trimmed)	Posted only to the requester's personal, private, undeleted chat. Carries workflow_result + workflow_result_contexts, so every read re-authorizes it and withholds it when the run was deleted, the result changed, source access was lost, storage failed, or the chat isn't private (6a masking).
Conversation search, later turns, exports, V2 loader	Inherited from 6a's masking and lineage; test_masking_surfaces_withhold_delivered_result_and_matching_followup_after_run_deleted.
Orchestration history	Excludes messages carrying the 6a keys; notes have no result text.
Bell notices	Fixed titles and previews (Results from "X" are in your chat); metadata is IDs and codes only.
Logs	Codes, run and workflow IDs and type(exc).__name__ only; user-controlled strings go through sanitize_log_message; never content, never str(exc) (M13).
Status route	Projected single-partition query (M11/M12/M28); closed error text from a map; step_label is always null.
Step sidecar and Phase 5 note	A boolean flag; fixed note text.
chat_delivery record	Codes, IDs, times, roles and model IDs; never text. It is owner-visible in run history (residual risk below).
Conversation document	Only the unread fields and last_updated.
Model prompt	Excerpt fenced with a fresh nonce per compose (M9), plus an untrusted-data system instruction.
Error responses	Closed codes only.

Status-route contract (fixed for 6b-2)

GET /api/v2/orchestration/workflow-runs/status[?conversation_id=<id>]. It's placed after the per-answer links route in route_backend_orchestration.py.

• Decorators, in order: @bp.route(..., methods=["GET"]), @swagger_route(security=get_auth_security()), @login_required, @user_required, @enabled_required('allow_user_workflows'), @workflow_user_required.
• conversation_id: optional, at most once, trimmed, [A-Za-z0-9_-]{1,128}. Empty, repeated, too long or malformed → 400 {"error": "The conversation ID is not valid.", "code": "invalid_conversation_id"}. An invalid server-side user id or a storage/settings failure → 503 {"error": "Workflow run status isn't available right now. Try again later.", "code": "workflow_run_status_unavailable"}.
• Owner-only by construction: partition_key=<session user_id>, c.user_id = @user_id, c.trigger_source = 'chat_orchestration', IS_DEFINED(c.chat_invocation), and a projected field list (no SELECT *).
• Modes: with conversation_id, TOP 21, returning up to 20 of that chat's started runs. Without it, TOP 51, returning up to 50 non-terminal runs, open delivery records, or deliveries from the last 10 minutes. Both use ORDER BY c.chat_invocation.requested_at DESC. truncated is true when the query returned more than the limit. Every chat_invocation writer sets requested_at (test_workflow_run_chat_invocation_writer_sets_requested_at); the runs container uses the default indexing policy.
• Live reads: at most 10 live runtime reads per request, for in-flight rows, plus at most 20 cached workflow reads, one per distinct workflow and counted separately. A failed read or an exhausted cap falls back to stored values with live: false.
• available is Phase 5's workflow_run_settings_gate at request time. Rows are still returned when it's false, so an in-flight card doesn't vanish when an admin flips the setting mid-run.

{
  "available": true,
  "runs": [
    {
      "workflow_id": "wf_123",
      "workflow_scope": "personal",
      "run_id": "run_123",
      "workflow_name": "Weekly digest",
      "conversation_id": "conv_123",
      "orchestration_run_id": "orch_123",
      "step_id": "step_2",
      "requested_at": "2026-01-05T02:30:00Z",
      "status": "running",
      "phase": "running",
      "runtime_version": 3,
      "step_index": 2,
      "step_count": 5,
      "step_label": null,
      "started_at": "2026-01-05T02:30:15Z",
      "completed_at": null,
      "elapsed_seconds": 416,
      "waiting": null,
      "delivery": {
        "status": "pending",
        "generation": 3,
        "message_id": null,
        "delivered_at": null,
        "reason": null
      },
      "error": null,
      "error_code": null,
      "retry_blocked": null,
      "actions": {
        "cancel": true,
        "retry": false,
        "approve": false,
        "open_run": true
      },
      "live": true
    }
  ],
  "checked_at": "2026-01-05T02:37:11Z",
  "truncated": false
}

Key field rules (the full tables are in the feature doc, Status route contract for 6b-2):

• status / phase:

  State	status	phase
  queued	queued	running
  running, cancelling, waiting_recovery, unknown states	running	running
  Other waiting states	waiting	needs_you
  completed	completed	finished
  completed_partial	completed_partial	finished
  failed, invalid, incomplete, skipped	failed	failed
  cancelled	cancelled	cancelled
  Non-terminal with delivery expired, or paused + deadline_exceeded	expired	failed

• waiting: {reason, action, gate_id} where reason is one of approval, microsoft_365_reconnect, microsoft_365_approval, output_review, recovery, deadline_exceeded or paused, and action is approve, reconnect or open_run. A waiting_recovery row is running/running but still carries waiting: {reason: recovery, action: open_run}.

• delivery.status: one of not_applicable, pending (also covers stored ready), delivering, delivered, undeliverable or expired. message_id is set only when delivered and prefixed assistant_workflow_delivery_. reason is set only when it is a closed delivery reason code.

• runtime_version is the expected_version that runtime/resume checks (equality in journal_request). The live version is used when the live read succeeded, otherwise the run's runtime_version, otherwise the stored projection version. test_schema_one_status_runtime_version_is_resume_expected_version and test_schema_two_… feed a row's value to resume on the real store over a fake container, and both succeed.

• actions.retry is true only when the state is failed/invalid/incomplete (RESUMABLE_STATES) and retry_blocked is null. retry_blocked is checked in order: retry_unavailable (unknown eligibility, failed read, read cap exhausted, missing revision or revision error) → workflow_deleted (deleting or gone) → not_resumable (stored projection can_resume false) → deadline_exceeded (limits.deadline_at ≤ now) → workflow_definition_changed (workflow_definition_revision of the editor form ≠ run.definition_revision, computed exactly like decide_workflow_runtime) → workflow_already_running (active_run_id not in None, '' or this run_id). skipped and cancelled are never retryable. There is one test per code (test_retry_blocked_codes, test_retry_blocked_workflow_read_errors_and_retry_ok, test_skipped_and_cancelled_are_never_retryable_even_with_can_resume).

• actions.cancel: non-terminal, not expired and not cancelling. actions.approve: waiting action approve, a gate_id present, and not expired. actions.open_run: always true.

• Snapshot rule: a row is a snapshot. 6b-2's Retry must still handle a 409 from runtime/resume (workflow_definition_changed, workflow_already_running, workflow_deleting, stale_version, invalid_state, deadline_exceeded, request_conflict), plus 400, 403, 404 and 503.

• Routes 6b-2 calls: GET …/runs/<run_id>/runtime, POST …/runtime/decision, POST …/runtime/resume with {expected_version, request_id} (a client UUID), and POST …/cancel, all under /api/user/workflows/<workflow_id>. Not resume-failed: it returns 409 for durable runs.

Message metadata for 6b-2

Every posted delivery message, including fixed notes, carries:

{
  "workflow_delivery": {
    "version": 1,
    "kind": "result",
    "workflow_id": "wf_123",
    "workflow_scope": "personal",
    "run_id": "run_123",
    "generation": 3,
    "run_status": "completed",
    "orchestration_run_id": "orch_123",
    "step_id": "step_2",
    "requested_at": "2026-01-05T02:30:00Z"
  }
}

• Identify delivery messages and refuse Retry/Edit: metadata.workflow_delivery. The server refusal also checks the assistant_workflow_delivery_ id prefix. 6b-2 joins the live status row by run_id, orchestration_run_id and step_id (or delivery.message_id).
• Follow up: metadata.workflow_result, 6a's public descriptor snapshot, which is the same dict(result['descriptor']) 6a's follow-up stores, plus metadata.workflow_result_contexts. Only kinds result and analysis carry them.
• Open run: workflow_id + run_id + workflow_scope from metadata.workflow_delivery.
• Retry on a note: shown only when the live status row's actions.retry is true. The metadata alone never makes Retry available.
• Note kinds: failed, cancelled, skipped, status (gate off at delivery, or the result can't be shown) and content_blocked carry no 6a keys. analysis carries the descriptor and contexts. Each kind's text is fixed (see the feature doc, Labels and application-owned texts).
• Snapshot rule: delivered-message and note metadata are snapshots. If the workflow is later edited, deleted, resumed or blocked by another active run, the status row and route responses are the live truth.
• D17 render check (no V2 change):
  • V2 lib/workflowResults.ts parses metadata.workflow_result strictly (version, workflow_id/run_id, SHA-256, readable status). The worker stores 6a's own descriptor, so V2 accepts delivered results, hides Retry/Edit on them (messageAsksAboutWorkflowResult), and Follow up works through 6a.
  • Fixed notes have no workflow_result, so today's V2 still shows Retry/Edit on them. The server refuses with 400 {error, code}, and 6b-2 hides them using metadata.workflow_delivery.

Tests

13 new files under functional_tests/, 383 tests as pytest collects them. Each file also runs as a script through its own __main__.

File	Tests	What it pins
test_workflow_chat_delivery_contract.py	72	Pure contract helpers. Message ids and notice and token idempotency keys are deterministic per generation and attempt. Pins the dated label and the note, notice and refusal texts; the delivered-message and notice metadata shapes; the backoff table; the seed shape and expires_at; and the stored fields the save guard keeps, including cancellation state. Covers every reconcile branch: the amendment-1 reopen rule (version > generation and not deleted), silent workflow_deleted and runtime_missing closes, expired records never reopening, and capped history. Also the bounded hint queue. A terminal state makes a deferred record due at once; a merge drops an incoming chat_delivery the stored run lacks; delivery_log swallows logging failures.
test_workflow_chat_delivery_worker.py	112	The worker over fakes. One composed reply with one unread mark and one notice, and nothing more on reprocessing. Excerpt step-down, model fallbacks and the fenced fallback. Token usage mirrors 6a, and a logging failure never fails delivery. Each terminal state's fixed note. A chat that can't take the delivery, or lost workflow access, gets one notice instead. A lost create reply is found on retry, and a 409 race keeps the message already posted. Deletes before and after delivery, and a missing control, close silently. The expired notice, busy-chat deferral capped at 15 minutes, and backoff to exhaustion. The durable resume posts a failed note, then delivers the new outcome. The sweep queries nothing without its lock, stays within its time and run budgets, and closes a run with no control in one pass. An abandoned claim at the attempt cap closes without composing. A chat lost during the unread mark closes undeliverable. A workflow deleted during the read closes silently. Bad ids read nothing.
test_workflow_chat_delivery_status_route.py	48	The status route and module. Owner-only single-partition projection queries in both modes; the 10 live reads and 20 cached workflow reads; status, phase, waiting, delivery and action mapping; one case per retry_blocked code; skipped and cancelled never retryable. A row's runtime_version is accepted as expected_version by the real store's resume, in schema 1 and 2. Every chat_invocation writer sets requested_at. Fixed, non-leaking errors (workflow_run_status_unavailable). The route's sign-in and feature gates, and the exact top-level, row, delivery and actions keys, checked against the feature doc's response example. A message id needs a delivered row and the prefix; an unknown failure reason maps to failed; a foreign or group workflow document blocks retry.
test_workflow_chat_delivery_control_pins.py	35	Amendment 1, with the real WorkflowRuntimeStore over a fake container, schema 1 and 2, for every terminal state. Every public store method is classified. Every non-resume mutator leaves a terminal control's version alone. tombstone bumps it and sets deleted, after which the mutators stay pinned. resume is the only non-delete bumper, and delivery reconcile reopens only for a real resume generation.
test_workflow_chat_delivery_imports.py	31	Fresh-process cold imports, offline with the network blocked, normal and -O. The contract, worker, status module, routes and runtime load in every order after the app bootstrap and around background_tasks. The cold contract and status imports load neither config nor the runtime. Each child dumps its stacks and exits if it stalls for 240 s.
test_workflow_chat_delivery_notice_and_unread.py	25	The workflow_chat_delivery notice type matches the contract and the registry. Notices pass retry options, return the existing notice on a repeat key, and fail strictly on create errors. The guarded unread mark writes only against the ETag it read, keeps concurrent edits, and skips once the user has read since the planned time. It never writes for a missing or unavailable chat, and rejects missing ids. The read-since skip holds at the equal boundary, and require_private=False still rejects another user's chat.
test_workflow_chat_delivery_projection_and_seed.py	16	Starting a run seeds chat_delivery and the step sidecar only when the seed applies. The runtime projection keeps the stored record, reconciles it and signals the worker once. A reconcile or hint failure never blocks the runtime write. Covers resume reopening a delivered generation, a deleted control closing silently, deadline expiry, and the started-run follow-up text. Only a queued or running run with this chat's record is flagged, and a mixed pair of started runs gets the plain follow-up. The projection never copies the runner's chat_delivery. The gate-off case reaches a queued start (F).
test_workflow_chat_delivery_save_guard.py	14	The guarded run save keeps the stored delivery record and chat invocation across stale full-document writes, 412 retries, 404s and read failures; non-412 errors propagate. A non-chat run keeps the plain upsert. After a guard bypass, the worker stays exactly once and doesn't mark the chat unread again once the user has read it. The guarded replace sends the ETag it read, so a real concurrent write wins.
test_workflow_chat_delivery_loop.py	9	The background loop binds services once and throttles the sweep, never waits under one second, and runs each slice in the app context. A failure sleeps instead of spinning and logs only the error type. start_background_task_threads registers the loop, the workflow scheduler loop does no delivery work, and the real lock helpers gate the real sweep.
test_workflow_chat_delivery_placement_and_masking.py	8	In the offline app, a delivered message is placed after the newest thread and updates the conversation, and the classic and V2 loaders return it last and active. Failed notes carry no result lineage. Masking shows a delivered result and its Follow up answer while the source is intact, and withholds both once the run is deleted or the result changes. A delivery sorts 1 µs after a newer assistant message, even one stamped now.
test_workflow_chat_delivery_off_golden.py	6	With the setting absent or off, both flows (answer and run, run only) match byte for byte a golden captured on the unmodified base (0.261.216, b6172962777f).
test_workflow_chat_delivery_concurrency.py	5	In a claim race, one worker delivers and the other stays busy. A hint racing the sweep delivers once. A stale worker loses its lease after a reclaim. Two sweeps after delivery repeat nothing.
test_workflow_chat_delivery_refusals.py	2	In a fresh offline process that boots the real app, classic Retry and Edit refuse a delivered result, and the failed, cancelled and gate-off notes, with one closed 400 code before any content check or write. The delivery code wins over 6a's Follow up refusal. Another user's Retry is still refused for ownership first, and ordinary turns still replay.

Mutations

Each mutation was applied to the production code, its listed delivery tests were run under pytest (normal and -O), and the file was restored with git checkout --. M1–M22 come from plan §10, M23–M30 are coordinator amendment 3, and M5b/M7b/M11b/M14b/M17b are extras. Killing tests are shown as <name>::<test>, short for test_workflow_chat_delivery_<name>.py::<test>.

Final full pass: 35/35 killed (normal), 35/35 killed (-O), with no errors in any run. First full pass: 34/35 (normal) and 34/35 (-O); survivors then: M21.

#	Mutation	Failing tests, normal / -O	First killing test	Message
M1	The claim replace drops its ETag guard	4 / 4	concurrency::test_claim_race_allows_one_worker_to_deliver_and_the_other_to_stay_busy	two workers racing for one claim: exactly one delivers and the other stays busy; got {'worker-a': 'delivered', 'worker-b': 'delivered'}
M2	Remove check-before-compose	3 / 3	worker::test_a_create_whose_reply_was_lost_is_found_on_the_retry	a message found on the retry is never composed again
M3	The guarded run save becomes a plain upsert	8 / 8	save_guard::test_guarded_stale_save_preserves_delivery_and_sweep_stays_exactly_once	guarded save must return the stored delivered record
M4	Drop the generation from the message id	4 / 4	worker::test_a_resumed_run_delivers_its_new_outcome_after_the_failed_note	the resumed run's new outcome is posted as a second message under its own generation's ID
M5	The chat notice gets a fresh idempotency key per call	14 / 14	worker::test_delivers_a_composed_reply_once_with_one_unread_mark_and_one_notice	one chat notice goes out, keyed by the delivery's stable idempotency key
M5b	Workflow notices get a fresh idempotency key per call	14 / 14	worker::test_a_chat_that_cannot_take_the_delivery_gets_one_notice_instead[removed]	one undeliverable notice goes out, keyed by the delivery's stable idempotency key
M6	Re-mark unread unconditionally on a retry	2 / 2	worker::test_a_failed_unread_mark_retries_with_the_earlier_planned_time	a retried unread mark is skipped if the user read the chat after the first planned time
M7a	Remove the classic Retry refusal	1 / 1	refusals::test_retry_and_edit_refuse_every_message_a_workflow_run_posted	retry of the result must be refused with 400 workflow_delivery_retry_unsupported: got 404 {"error": "User message not found in thread"}
M7b	Remove the classic Edit refusal	1 / 1	refusals::test_retry_and_edit_refuse_every_message_a_workflow_run_posted	edit of the result must be refused with 400 workflow_delivery_edit_unsupported: got 400 {"error": "Only user messages can be edited"}
M8	Skip the 6a gate re-check at delivery	2 / 2	worker::test_delivers_a_composed_reply_once_with_one_unread_mark_and_one_notice	delivery re-checks the workflows gate, then the results-in-chat gate
M9	A constant fence nonce	2 / 2	worker::test_delivers_a_composed_reply_once_with_one_unread_mark_and_one_notice	the compose prompt is fenced with the fresh nonce drawn for this delivery
M10	Omit workflow_result_contexts from the delivered answer	7 / 7	worker::test_delivers_a_composed_reply_once_with_one_unread_mark_and_one_notice	the delivered answer carries workflow_result_contexts, so every read of it re-authorizes the result
M11	The per-chat status query uses SELECT *	36 / 36	status_route::test_query_shape_is_single_partition_and_projection_only_in_both_modes	status query must not use SELECT *; it projects only the allowed paths (the route failed closed with workflow_run_status_unavailable)
M11b	The all-chats status query uses SELECT *	4 / 4	status_route::test_query_shape_is_single_partition_and_projection_only_in_both_modes	status query must not use SELECT *; it projects only the allowed paths (the route failed closed with workflow_run_status_unavailable)
M12	The status query drops the partition key	39 / 39	status_route::test_query_shape_is_single_partition_and_projection_only_in_both_modes	status query must use the signed-in user's partition (the route failed closed with workflow_run_status_unavailable)
M13	The status route returns str(exc)	1 / 1	status_route::test_route_maps_status_errors_and_generic_errors_without_secret_text	expected code workflow_run_status_unavailable, got {'error': 'secret-text'}
M14	The delivery signal raises inside the projection	1 / 1	projection_and_seed::test_projection_survives_broken_hint_queue_and_direct_signal_returns_false	signaling a delivery hint must never raise, even when the hint queue is broken; it raised RuntimeError('lock unavailable')
M14b	The projection lets a reconcile error escape	1 / 1	projection_and_seed::test_projection_survives_reconcile_failure_and_still_writes_runtime	a delivery reconcile failure must never fail the runtime projection; the projection raised RuntimeError('boom')
M15	Remove the fifteen-minute deferral bound	1 / 1	worker::test_a_chat_that_stays_busy_gets_the_delivery_after_fifteen_minutes	delivery must settle on delivered within 40 attempts; it kept returning deferred x40
M16	Remove the sweep's expiry branch	1 / 1	worker::test_the_sweep_sends_the_expired_notice_for_a_run_still_running_past_its_window	the sweep must pick up a pending record whose window passed while the run kept running
M17	The sidecar always claims chat delivery	5 / 5	off_golden::test_starting_a_workflow_with_results_in_chat_off_is_byte_identical_to_the_base[answer_and_run-absent]	With Use Workflow Results In Chat absent, the message of the answer_and_run flow must match the base golden byte for byte:
M17b	The answer note always promises delivery	5 / 5	off_golden::test_starting_a_workflow_with_results_in_chat_off_is_byte_identical_to_the_base[answer_and_run-absent]	With Use Workflow Results In Chat absent, the message of the answer_and_run flow must match the base golden byte for byte:
M18	A failed newest-message read becomes an empty chat	1 / 1	worker::test_unreadable_chat_activity_backs_off_then_gives_up_with_one_notice	an unreadable chat defers and backs off, and never counts as an empty chat
M19	Skip content safety on the trimmed fallback	2 / 2	worker::test_falls_back_to_the_start_of_the_result_when_neither_model_answers	the fallback text must be checked like a reply
M20	Skip the chat ownership and privacy re-check	5 / 5	worker::test_a_chat_that_cannot_take_the_delivery_gets_one_notice_instead[another_owner]	a chat that can't take the delivery isn't marked unread
M21 ¹	The unread write drops its ETag guard	1 / 1	notice_and_unread::test_guarded_unread_write_is_conditional_on_the_etag_it_read	the unread write must be guarded by the ETag it read ('etag-conversations-1'), got (None, None)
M22	Seed chat_delivery with the 6a gate off	6 / 6	off_golden::test_starting_a_workflow_with_results_in_chat_off_is_byte_identical_to_the_base[answer_and_run-absent]	With Use Workflow Results In Chat absent, the first_write of the answer_and_run flow must match the base golden byte for byte:
M23 ᴬ	Drop the deleted check from the reopen rule	3 / 3	worker::test_a_run_deleted_after_delivery_stays_delivered[tombstone]	a run deleted after delivery stays delivered and is never reopened or posted again
M24 ᴬ	Reopen on >= instead of >	44 / 44	worker::test_delivers_a_composed_reply_once_with_one_unread_mark_and_one_notice	a delivered record stays delivered; only a resume, which moves the control past its generation, reopens it
M25 ᴬ	Treat skipped as retryable	1 / 1	status_route::test_skipped_and_cancelled_are_never_retryable_even_with_can_resume[skipped]	skipped must not be retryable
M26 ᴬ	Run the sweep without the distributed lock	2 / 2	worker::test_the_sweep_queries_nothing_without_the_lock	without the distributed lock the sweep does nothing
M27 ᴬ	Treat a 409 on message create as a failure	1 / 1	worker::test_a_create_that_loses_a_race_keeps_the_message_already_posted	a 409 on create means the message is already in the chat
M28 ᴬ	Widen the status query to cross-partition	39 / 39	status_route::test_query_shape_is_single_partition_and_projection_only_in_both_modes	status queries must not enable cross-partition reads (the route failed closed with workflow_run_status_unavailable)
M29 ᴬ	Skip check_chat_content on the composed answer	4 / 4	worker::test_delivers_a_composed_reply_once_with_one_unread_mark_and_one_notice	the composed answer is content-checked and the check is recorded on the message
M30 ᴬ	Drop token-usage logging	5 / 5	worker::test_delivers_a_composed_reply_once_with_one_unread_mark_and_one_notice	the compose's token usage is logged once, keyed by the message and the attempt

ᴬ Added by coordinator amendment 3.
¹ M21 survived the first full pass in both modes (0F/0E normal, 0F/0E -O); 70f6cfa added test_guarded_unread_write_is_conditional_on_the_etag_it_read, which kills it.

Harness. mutate.py handles one mutation at a time:

1. Refuses to start unless the target files are clean.
2. Applies the mutation as an exact-anchor text replacement, and aborts unless the anchor occurs the expected number of times.
3. Runs that mutation's listed delivery tests with python [-O] -m pytest … -q (COLUMNS=4000, 900 s limit).
4. Records the first failing test and its assertion message.
5. Restores the file with git checkout -- <file> in a finally, then confirms the tree is clean.

Coverage of the final code. The final full pass ran on the final production code. The last application/ change is 74f420870; every later commit touches only tests and docs. M1 and M15 were re-run after decb3034c reworded their kill messages.

Status-route contract. The tightened check in 766fea19d was also mutation-checked. Dropping workflow_scope from the row (functions_workflow_chat_delivery_status.py L577) is killed in both modes with "row keys changed (6b-2 contract): ['workflow_scope']".

Review survivors

The coordinator's pass (149 mutants under pytest and -O pytest, at 766fea19d) left 17 survivors and flagged one partly vacuous test (F). Commits 573bd34c8, 3cd04b4cf, 16ada4f5e, b676db476 and 447a038b3 add tests only. I reapplied each mutation at edd748417 with the same exact-anchor harness (mutate_review.py) and ran its target file: 18/18 killed in both modes, with no errors.

ID	Mutation	File	Failing tests, normal / -O	First killing test
Exhausted	_claim never applies the attempt cap (exhausted=False)	worker	1 / 1	test_an_exhausted_abandoned_claim_closes_undeliverable_without_composing
UnreadGone	_mark_unread treats a chat gone at the mark as a retry	worker	4 / 4	test_a_chat_lost_during_the_unread_mark_closes_undeliverable[removed]
X10	_undeliverable never closes silent reasons silently	worker	1 / 1	test_a_workflow_deleted_during_result_read_closes_silently
PlaceAfter	_create_message stamps now even when a newer message exists	placement_and_masking	1 / 1	test_delivered_message_is_stamped_after_newer_assistant_messages
LogSwallow	delivery_log lets a logging failure escape	contract	1 / 1	test_delivery_log_swallows_logging_failures
DueNow	reconcile keeps a pending wait on a record that became ready	contract	1 / 1	test_reconcile_pending_terminal_state_is_due_now_even_after_deferral
MergePop	merge_stored_run_fields keeps an incoming chat_delivery the store lacks	contract	1 / 1	test_merge_stored_run_fields_drops_incoming_delivery_when_store_has_none
S11	status row exposes delivery_message_id without the delivered/prefix gate	status_route	1 / 1	test_delivery_message_id_requires_delivered_status_and_prefix
S12	status error_code passes an unknown gate reason through	status_route	1 / 1	test_failed_unknown_gate_reason_maps_to_generic_failure_sentence
S15	status workflow read skips the owner/group check	status_route	1 / 1	test_retry_blocks_foreign_and_group_workflow_documents_as_deleted
StatusFilter	start flags chat_delivery on a run that is not queued/running	projection_and_seed	1 / 1	test_start_does_not_flag_completed_already_started_run_for_delivery
RunApplies	start flags chat_delivery without chat_delivery_applies	projection_and_seed	1 / 1	test_start_requires_queued_run_delivery_record_to_apply_before_flagging
M17b	note promises delivery when only some started runs deliver	projection_and_seed	1 / 1	test_workflow_run_note_uses_delivery_follow_up_texts
R2b	_project_runtime_run keeps a chat_delivery the stored run lacks	projection_and_seed	1 / 1	test_projection_drops_result_chat_delivery_when_stored_run_has_none
P3	guarded run replace drops its ETag condition	save_guard	1 / 1	test_guarded_replace_uses_read_etag_so_real_conflict_preserves_concurrent_delivery
U10	unread read-since check needs strictly later (> not >=)	notice_and_unread	1 / 1	test_guarded_unread_read_since_equal_boundary_skips_without_writing
U6	unread mark skips the owner check	notice_and_unread	1 / 1	test_guarded_unread_require_private_false_rejects_wrong_owner_without_writing
BadIds ᴱ	worker entry reads runs for empty or non-string ids (optional E)	worker	8 / 8	test_missing_ids_are_not_applicable[-run-delivery-1]

ᴱ Optional item E. M20a is equivalent (the real conversation_is_private already rejects a non-owner) and AfterCreate is defensive (every _Undeliverable raise precedes the message create); neither got a test.

F. The projection test's fake queue_run read options["chat_delivery"], so with the gate off it raised KeyError, _start returned unavailable, and the off-case checks passed whatever the code did. It now uses options.get, finalizes only when the option is present, and requires a queued start.

Guardrails

run_v2_guardrails.py ran over 1524eeb7b...543f1a5a8, the V2 tip this branch merged: 37 changed files, 12 of them application .py (2 route files), no TS/TSX.

Check	Result
broken-access-control	PASS (12 files)
xss-sinks	PASS (12 files)
swagger-routes	PASS (2 route files)
python-syntax	PASS (496 top-level single_app files compile)
malicious-pr-review	PASS, no blocker findings
v2-ts-sinks (added lines)	PASS (no TS/TSX changes)

The malicious-PR review's "Needs investigation" verdict comes from keyword heuristics: 1,421 findings, no blockers (breakdown from edd748417: 1,420).

• 507 AI/agent/workspace: the feature's own vocabulary.
• 471 security-control: try:/except, log_event(, test require/assert, # noqa imports, role normalization and skip_if_read_since. None loosens a check.
• 295 sensitive-source: mostly token-usage logging, Cosmos container names and dict or idempotency keys. The "secret" strings are test sentinels (SECRET-RUN-OUTPUT-TEXT, secret-query-text) proving run output and exception text never reach the status payload or logs. No credential is read or added.
• 102 dynamic-loading: literal getattr, fixed re.compile patterns, importlib.import_module of in-repo names, test monkeypatch.setattr.
• 36 dynamic-execution: mostly the word "empty", plus fixed-argv subprocess.run([sys.executable, …]) probes, never shell=True. The only exec (test_workflow_chat_delivery_loop.py L62–63) runs the real background_tasks.py functions, parsed with ast, in a seeded namespace, the established harness pattern.
• 9 external-connection: urlencode for the in-app notice link, tests that patch socket.connect to block the network, and the words "requests" and "telemetry" in fakes.

CodeQL / CI

CI on the head 543f1a5a8: 10 checks passed; CodeQL ran before the Python and JS/TS uploads and is being refreshed. The table is from edd748417.

Check	Result
CodeQL	success: "No new alerts in code changed by this pull request"
Analyze (python), Analyze (javascript-typescript), Analyze (actions)	success
broken-access-control-check, xss-sink-check, swagger-route-check, syntax-check	success
malicious-pr-security-review	success
enforce-branch-flow, license/cla	success

Release Notes Check runs only for PRs into Development, so it didn't run.

Earlier follow-up 0ad92d9db. It cleared three CodeQL alerts in test code from 766fea19d (two py/import-of-mutable-attribute, now unittest.mock.patch; one py/catch-base-exception, now Exception, with M1 and lease-busy still killed) and let test_workflow_chat_delivery_off_golden.py run without PYTHONPATH.

The review-survivor tests patch real modules only through monkeypatch, catch no BaseException, define no closures in loops, and store attributes only on test fakes.

Residual risks

• A third consecutive storage read failure in the double-fault guarded-save path can still wipe chat_delivery. Then nothing is posted, and the status route reports the run as not_applicable.
• Mark-before-create leaves small windows. The chat can look unread just before the message exists. A permanent create failure can leave a dangling unread mark. A chat deleted between the mark and the create can leave an orphaned message, which the existing deleted-chat handling hides.
• A crash after the unread mark but before the create can cause a second compose and possibly a second content-safety incident. The first composed text isn't persisted, by design (no result text in the record). Token usage is logged per attempt, so a real re-compose is counted, and the idempotency key prevents double-logging one compose.
• Version-1 runtime controls don't always have deadline_at. Their expiry is computed from the admitted deadline, and a run finishing after expires_at gets a notice rather than a post.
• Role revocations after start aren't seen by the headless worker, which has no live session. It uses the roles captured at start, as durable background execution already does.
• Owner-visible run history can show the raw chat_delivery record: roles, model IDs, lease IDs and codes, but never result text.
• The workflow's own alert rules still fire for the same run, so a user may get both a workflow alert and a chat post.
• Group-workflow delivery (Phase 7), waiting for a run inside a plan, and a generic unread helper are out of scope.

Deviations from the approved plan

All are documented in the feature doc's Deviations from the plan:

• Mark-before-create (D9, accepted as written by the coordinator). skip_if_read_since is kept for retries after a read, and the guarded unread mark requires all four ids.
• available: false still returns stored rows, so in-flight cards don't vanish mid-run.
• The generation source is runtime/resume (the control version), with the A1 reopen rule version > generation and not deleted. Expired records never reopen.
• The delivered bell preview is fixed non-empty text. The plan's step-goal question fallback was dropped; the worker uses the original user message or a fixed generic instruction. The prompt size is pre-checked, with the 48/24/12/6 KB budget steps. An excerpt that still doesn't fit becomes a status note.
• Attempt exhaustion (8 attempts) depends on whether a claim was made:
  • Before a claim (the runtime control can't be read): the record closes silently, as undeliverable with delivery_failed, or as delivered without a notice if the message already exists. No outcome was read, so there's nothing to tell.
  • After a claim: it closes with one best-effort notice. If the message was created, that's delivered plus the chat notice. If the kind is an expiry, it's expired plus the expired notice. Otherwise it's undeliverable with delivery_failed, plus the undeliverable notice. It never posts a second message.
• Notices use deterministic idempotency keys. The note kind is derived from the terminal run state.
• The delivery loop runs inside the Flask app context, and the worker reads workflows directly from the container.
• Refusals are {error, code} with 400. The status route reports cancelling and waiting_recovery as running, and falls back from an unknown state to the stored status, then to running. step_label is always null. The workflow-read cap is 20, separate from the 10 live reads. expired applies only to non-terminal rows. Timestamps have seconds precision. Approve needs a gate_id. An invalid user id returns 503, and an empty conversation_id= returns 400.
• Rebuilt or linked Phase 5 runs keep the older note text.
• The workflow notice link uses the classic /workflow-activity?workflowId=…&runId=…&scope=personal path, with precedent in functions_m365_runtime.py L582–589. The V2 bell shows the generic Notification label for workflow_chat_delivery until 6b-2 adds a case.
• One edited pin: functional_tests/test_orchestration_workflow_run_adapter.py (+2 lines) adds functions_settings: {is_chat_workflow_results_enabled_for_user} to the Phase 5 module's allowed lazy-import set.

Step 9 overlap

• Code: none of step 9's files are edited. Not touched: registry, functions_orchestration_workflow_context.py, functions_orchestration_execution.py, functions_orchestration_workflow_results.py, planner/schema/adapters/composition/executor/recovery/services, admin_settings_fields.py, chat-orchestration.html, workflow.html and OrchestrationRunView.tsx. In route_backend_orchestration.py, the only edits are enabled_required and workflow_user_required added to the existing from functions_settings import … line, a new import of the status module, and the new route after the links route. The functions_orchestration_workflow_context import block (L133–141) is untouched. workflow_run_settings_gate is imported in functions_workflow_chat_delivery_status.py, not the route file.
• Docs: against 8a802f196, which now holds step 9's doc rows, every doc hunk is still insertion-only (@@ -N,0 +…; 914 insertions, 0 deletions), so step 9's lines are unchanged.
  • orchestration.md: +7 after L296 (after the table) and +1 troubleshooting row after L566.
  • workflow.md: +7 after L67 and +1 after L276.
  • CHAT_WORKFLOW_RESULTS_FOLLOW_UP.md: +3 after L730.
• Merged second: Read saved workflow results from chat orchestration (workflow_results) #1607 landed first, so edd748417 merges 8a802f196 with git -c rerere.enabled=false merge. There were two conflicts. config.py keeps VERSION = "0.261.218", and release_notes.md has the .218 section on top, then step 9's whole .217 section. route_backend_orchestration.py and the three other shared docs auto-merged; against V2, the route file is +45/−1, the −1 being the from functions_settings import … line. build_docs_inventory.py regenerated docs/_data/app_surface.yml after the merge, with no diff. Then V2 chat: inline media players, image cards, agent-posted markdown, and interactive maps (0.261.224) #1612 (with Fix group uploads, Call agent saves, scheduled-run identity and GPT-6 model handling #1608) and Fix V2 New chat carrying over a busy conversation's state #1620 landed: 35ce8ed58 and 389c9b14b merge 16d4734ad and 1524eeb7b the same way (only config.py and release_notes.md conflicted; this branch's notes section stays on top, V2's are unchanged), and aa7a647b9 and 543f1a5a8 renumber this branch's 51 version lines in 21 files, .218 → .226 → .227, as V2 reached .225 and then .226. No inventory diff.
• Merge rule: whichever PR merges second merges V2 with rerere off, keeps its own VERSION, puts its release-notes section on top, and regenerates docs/_data/app_surface.yml after the merge.

Follow-ups

• Gotcha 34 beyond delivery: delivery now marks its own posts unread and sends its own notice. Other headless posters still don't, and they can reuse mark_conversation_unread_guarded. They're out of scope here.
• Phase 7 (Workflow orchestration follow-ons: group workflows, SimpleChat action parity and plan replay #1550): group-workflow delivery.
• 6c (later, optional): an in-plan wait for quick runs. Not started, and out of scope.
• 6b-2: a specific V2 bell label for workflow_chat_delivery, and hiding Retry/Edit on notes via metadata.workflow_delivery.
• Pre-existing import cycle (not introduced here, not fixed here): functions_personal_workflows → functions_document_actions → functions_document_analysis → functions_search → functions_content → functions_settings → functions_document_actions. Production is unaffected because app.py imports config and functions_settings first; the imports test bootstraps the same way. A follow-up issue is suggested.