Phase 4: orchestration proposes workflows

[Paul Lizer (paullizer)]

Refs #1547
Part of #1543

Draft until the coordinator's review passes. Version 0.261.207 (config.py .205 → .207 only).

Summary

With the new admin setting Propose Workflows From Chat (enable_chat_orchestration_workflows, off by default), a chat orchestration plan can answer a recurring request once, as a preview, and propose a personal workflow for the runs to come. "Every Monday at 8, read my email and tell me what I should focus on this week" gets this week's answer and a proposed Monday 08:00 workflow in the user's time zone.

• Planning. A new workflow_propose capability (Reason role) and workflow deliverable. The planner writes a closed Phase 2 blueprint that names the requester's agents, File Sync sources, the documents they named in this request and their existing workflows only by request-local handles. Plan validation runs the draft service's write-free checks. A blueprint that still fails after one repair is dropped with everything bound to it, and the rest of the plan runs.
• The proposal step. adapter_workflow_propose dry-runs the blueprint, then stores a small structured-v1 proposal result and a server-only sidecar (blueprint digest, handles, time zone, summary without instructions, similar workflows, expiry, reason). The card is the only surface.
• Routes. Requester-only status, accept, deny and draft routes on the producing run: GET /api/v2/orchestration/runs/<run_id>/workflow-proposals, then POST …/<proposal_id>/accept, POST …/<proposal_id>/deny and GET …/<proposal_id>/draft. Accept is idempotent (uuid5 proposal and workflow ids). Decisions use the run doc's 8-attempt etag CAS with a 120-second claim window. A denied proposal can't be accepted, proposals expire after 14 days, and a workflow the requester deleted comes back only through Create again. Accept logs the creation in the activity log, never approves Run as, and refuses URL Access.
• The V2 card. A Proposed workflow card under the answer shows the schedule, runs per month, alerts, Microsoft 365 reach and Run as, each task's runner and full instructions as plain text, and similar workflows. It offers Create & start, Create paused, Edit (the V2 editor seeded with the proposal, where Save accepts it), Deny (confirmed), Create again and Open workflow.
• Time zone. The V2 chat sends the browser's zone. Every plan path (first plan, regenerate and replan, question-card answers, plan-editor revisions, retries) keeps the validated zone. A turn that may propose a workflow writes its answer with the local-time line a calendar run uses.
• Personal File Sync in the V2 editor, including structured (v3) drafts through Phase 3a: change tracking in the V2 workflow editor #1569's authored fields. The source list says when File Sync is off for the personal workspace.
• Nothing changes with the setting off. A golden captured on the unmodified base (Commit 0) pins the planner messages, the deliverable brief and the strict-mode error, with the setting off and with it on while the capability is unavailable.

Docs: docs/explanation/features/CHAT_ORCHESTRATION_WORKFLOW_PROPOSALS.md, V2_PERSONAL_WORKFLOW_FILE_SYNC.md, docs/admin/orchestration.md, docs/reference/chat-controls.md, docs/guides/create-a-workflow.md, the v0.261.207 release notes, and the regenerated docs/_data/app_surface.yml.

Decisions made autonomously

From the approved plan

1. Reason role, not Render. Render requires file outputs, and the card is the delivery surface.
2. Shared and collaborative chats: workflow_propose is unavailable there (workflow_shared_conversation). A chat later converted to collaboration makes its proposals unavailable. The routes still check the requester.
3. Planner output stays json_object. The strict json_schema variant (a Phase 2 follow-up) is declined, and the Phase 2 schema is enforced at plan time instead.
4. Agent matching. The step carries task_actions (closed kinds). When another offered agent has every kind a task needs, a mismatched runner is repairable agent_capability_mismatch. When no offered agent has them, the plan passes, and the card shows the proposal as unavailable with no_suitable_agent.
5. Run as. A task on an agent with Microsoft 365 actions and run_as: none is repairable run_as_required, and the repair sets self.
6. One repair, then degrade. The step is dropped, the workflow deliverable becomes unavailable with workflow_draft_invalid, and the preview still runs. Plan edits are refused rather than degraded.
7. Deterministic proposal_id: uuid5(run_id:step_id) of the producing run. Retries resolve inherited stubs to it.
8. Decisions live on the producing run doc under the etag CAS. A claim older than 120 seconds can be taken over.
9. Expiry is 14 days from created_at, and it also blocks Create again. The user can ask again for a fresh, revalidated proposal.
10. Cancelled runs leave the proposal unavailable. A failed run whose proposal step completed stays actionable.
11. Runs per month uses a 30-day month (22 weekdays).
12. origin.edited: apply_workflow_origin honors a caller-supplied edited for a new workflow, computed from material fields against the stored-blueprint build.
13. Microsoft 365 calls to action are same-origin links to the existing profile and approvals pages. Accept never records approval, and there's no pre-run approval route.
14. Status is per run, since the card knows its run.
15. The workflow deliverable kind exists for the planner only while the setting is on, so messages and prompts are unchanged when it's off.
16. At the cap, the capability is unavailable at plan time (workflow_quota_reached), and accept rechecks.
17. A missing calendar timezone is filled with the validated request zone during normalization.
18. The personal File Sync source listing returns file_sync_enabled, so V2 mirrors the save gate.

Made while implementing

19. Retries. A retry child run drops recorded decisions; the decision stays on the producing run.
20. Expiry blocks only new decisions and creates. An accepted proposal still reports its workflow, and a repeat accept still returns it.
21. Deleted checkpoints. A producing run with checkpoints_deleted hides its proposals (404 proposal_not_found), and a deletion that lands during a decision returns 409 proposal_unavailable.
22. Create again creates the workflow paused.
23. Non-requesters get a non-disclosing 404 (run_not_found / proposal_not_found), not a 403.
24. Unexpected faults return 503 service_unavailable. The draft service's workflows_unavailable stays a 403.
25. Card links. Run as links to /approvals, and Connect links to /profile?tab=settings#m365-connection-status. Open workflow and the similar-workflow links use workflowProposalLink, built as agreed from Track P's WORKFLOW_LINK_PARAM. It's added to the XSS checker's approved same-origin URL builders.
26. The card mounts only when the message's capabilities_used includes workflow_propose, in a personal conversation, on an unmasked message.
27. Edit. The editor reads a 409 as a stale saved workflow, but nothing is saved yet. So the card passes an accept 409 to the editor as a 400 that says "Your draft has been retained", and the draft stays open. URL Access is also refused client-side.
28. A manual workflow's Create creates it enabled, with "It runs only when you start it from Workflows".
29. The created card shows the workflow's current name. Polling skips a tick while a request is in flight.
30. The card's Playwright test serves a static copy of the draft route's real answer for the Monday email proposal, captured from the routes suite's harness. The server side of every decision is covered by the routes and editor round-trip suites.
31. The answer step gets the local-time line whenever the turn's workflow planning is ready, not only when the plan ends up proposing.
32. The admin switch is a plain switch in chat-orchestration-capabilities-section, right after Enable Action Access.
33. Candidate documents (note 7), narrowed further. The catalog offers only the documents the user selected or #-referenced in this request (the turn's document_ids seed), still filtered by the enrichment's authority check. A document the candidate probe found by searching is never offered.
34. Test stub. Commit 5's module-level log_workflow_creation import in route_backend_orchestration.py broke every suite that imports the route through functional_tests/test_support/app_stubs.py. The shared activity-logging stub now provides it.

Coordinator review notes, as folded in

1. Full instructions on the card: plain text, collapsed by default. Only the status route returns them, and only to the requester. Playwright checks hostile instruction text renders inert.
2. Every plan path keeps the zone: one test per path, including a replan with no time_zone in the body that still uses America/New_York.
3. A failed catalog read never breaks planning: closed reason workflow_context_unavailable, logged with codes only. The queries are bounded, the build is timed, and the catalog block has a character cap that drops the lowest-priority entries first. There's one test per catalog raising.
4. Setting-off golden in Commit 0, before any planner change. captured_from stays historical.
5. Degrading drops the bindings. No capability may consume proposal (workflow_proposal_not_consumable), and a dropped step takes every binding with it. There's a test for an answer step that consumes the proposal.
6. Handles start with a letter: kind prefix, slug, -<hash6>. Tests cover "2024 Report", "日本語", "!!!" and a 200-character name.
7. Candidate documents: see decision 33.
8. Cheap status reads. m365.connected comes from the stored record only. Polling stops at the 120-second claim window and pauses while hidden.
9. CAS: the 8-attempt IfNotModified loop, with exhaustion mapped to 409 proposal_busy. It has the all-412 test and the confirm-fails-after-create test.
10. "Done when": test_orchestration_workflow_proposal_done_when.py.
11. Parallel PRs. initialDraft and onSaveOverride touch only the dialog's props and its single save call site. The plan-editing hunk is small and separate. workflowRunLink.ts, notificationLinks.ts and workflowAlertNotices.ts are unchanged.
12. The card is the only surface: no output or download chip (tested). Logs carry ids and codes only.
13. Wording. File Sync says Runs when File Sync finds changes in <source>. Manual says Create and "It runs only when you start it from Workflows".
14. Separate processes for test_orchestration_research_pre_effect.py and test_workflow_document_picker_recent_targets.py.
15. No return False tests. New tests fail with assert, self.assert* or raise.

Validation

Everything below ran at 177e0bc, except the broad sweep, which ran at 8cd926f. The two commits after the sweep change only a feature doc and the shared test stub, and the suites that import the stub were rerun at 177e0bc. The head, 5933c07, only addresses CodeQL notes, with no behavior change; the suites it touches were rerun there (see CodeQL). "Base" means V2 18dac67 (VERSION .205), run from a git archive snapshot of that commit. Each block names its runner.

pytest, one process per file, normally and with python -O (38 files)

Everything passes in both modes except the three base failures listed at the end.

• New suites (356 tests): propose capability 88, planning context 112, proposal routes 60, deliverable 26, personal File Sync sources 22, week preview 14, admin 12, setting-off golden 9, imports 6, editor round trip 5, "Done when" 2.
• Suites this branch changes or builds on: draft service 38, draft V2 round trip 19, origin provenance 67, group File Sync client parity 179, dependency allowlist 41, return-False guard 7, app-version assertion guardrails 1. Route tests: policy inventory 9, policy test coverage 2, unauthenticated contract 4.
• A1/A2 re-runs: plan-revision planner 11, references 24 (+40 subtests), routes 28/29, store 44. Also V2 plan backend 61, reference authorizer golden 2 (+74 subtests), scope authorizer 31, submission id 61, canonicalization 16, V2 plan editor references 10, V2 assist thread 14, and elicitation context 9, routes 6 and schema 5.
• Other app_stubs importers: conversation-context routes 23. ui_tests/test_v2_orchestration_plan_editor_backend.py passes 7/7.
• Note 14, in separate processes: test_orchestration_research_pre_effect.py 41, and test_workflow_document_picker_recent_targets.py 3/4 (a base failure).

Scripts (python functional_tests\<file>, __main__ result)

These files report results from __main__ (their tests return values), so they ran as scripts.

• test_orchestration_agent_selection.py 6/6, test_orchestration_run_hydration_routes.py 6/6 and test_orchestration_turn_recovery_identifiers.py 5/6. All three are identical on base.
• Docs:
  • test_docs_app_surface_coverage.py 7/7 and test_docs_site_quality.py 6/6.
  • test_docs_link_integrity.py 2/5, with the same 11 broken links as base. Head checks 1087 relative links and base 1064, so every added link resolves.
  • test_docs_release_notes_integrity.py 0/1, as on base. The generated release pages are stale: base is missing 143 releases and head 144, the extra one being v0.261.207.

Broad sweep: pytest, one process per file, normal mode (367 more files)

The sweep covered 367 more files whose names cover orchestration, workflows, V2, admin, group and public workspaces, Microsoft 365 and tabular. 287 passed. The other 80:

• 71 fail with the same failing test ids on base.
• 6 timed out at 900 s under load. Run alone:
  • external root runtime 2, output checkpoint storage 79, output configuration 181 and output metadata reads 26 pass;
  • test_app_settings_auxiliary_writers.py has 18 failures, identical on base;
  • test_tabular_row_orchestration_scale.py is identical on base, test by test. Tests 1–31, run alone: 28 pass, and Process Tabular Data #11, Add AI Search and Doc Intelligence to Admin Settings #27 and Archive Conversations Instead of Permanent Deletion #30 fail (NameErrors in the test's exec namespace). Tests 32–77, one process each at 5933c07: 38 pass, and 7 fail with the same kind of stub drift (NameErrors, a missing stub attribute, an unexpected keyword). test_phase_six_rolling_pool_pauses_dispatch_at_checkpoint_high_water never finishes on either side: it spins on one core, which is what held the whole file past the timeout. The suite reads functions_settings.py only as text, for its tabular keys, and imports none of this branch's changed modules directly.
• 2 failed under load and pass alone: test_orchestration_dependency_imports.py 11, and test_orchestration_harness_execution.py 649 (+2 xfailed).
• 1 fails on base: test_orchestration_turn_recovery_identifiers.py (see Scripts).

Playwright (pytest, one process per file): 226 tests, all pass

• New: proposal card 22, personal File Sync 8 and admin switch 7.
  • The card tests check that hostile instruction text renders inert, that Open workflow carries workflow_id=<created id>, and that proposal gets no output chip.
  • The File Sync tests include the two 3a overlap tests: a structured (v3) workflow saves file_sync, and editing sources makes one Changes-tab row whose Jump lands in File Sync.
  • The personal File Sync suite also passes 8/8 under -O.
• Coordinator re-runs: change tracking 24, calendar schedules 14, group File Sync 13, workflow editor 9, plan editor 35, plan-editor references 11, alert notices 28 and document provenance 13.
• Also: flow canvas stability 6, workflow settings errors 6 and flow authoring 30.

Node (node --test): 13 files, 188 tests, all pass

• Workflow: authoring history 38, authoring session 27, flow authoring commands 60, field drafts 15, Microsoft 365 Run as client 15 and settings client 11.
• V2 orchestration: deliverables 8, workflow deliverable 3, phase groups 4, planner display 4, approval logic 1, model routing 1 and planner model 1.

Guardrails, build and diffs

• run_v2_guardrails.py (18dac67..HEAD, 78 changed files) passes all six checks:
  • broken access control, XSS sinks (27 files), swagger routes, Python syntax and V2 TS sinks;
  • the malicious-PR review has no blockers. Its "Important" notes are keyword matches on test lines.
• V2: npm run build (tsc -b and vite build) is clean, built after the last V2 source change (the 3718cca merge). The Playwright runs used that bundle.
• Three-dot diff against 18dac67: config.py changes only .205 → .207, and release_notes.md has 35 added lines and none removed.

Base failures (identical on V2 18dac67, not changed here)

• test_orchestration_plan_revision_routes.py::PlanRevisionRouteTests::test_document_removal_is_not_lost_when_editor_opens
• test_workflow_document_picker_recent_targets.py::test_recent_target_runner_hooks
• test_broken_access_control_guardrails_checker.py::test_checker_assets_and_version_are_wired_into_repo, which expects upload-artifact@v4 while the workflow pins v7.0.1.
• test_v2_prompt_attachment_persistence.py (72 failed) and test_v2_prompt_composer_card.py (2 failed). The failing ids are identical on base; the stubbed namespace lacks names such as prepared_chat_image_references.
• test_orchestration_turn_recovery_identifiers.py 5/6 as a script (image_reference_provenance_from_seeds isn't defined).
• test_app_settings_auxiliary_writers.py (18), and the 71 sweep files above.
• test_tabular_row_orchestration_scale.py: 10 failures and one test that never finishes (see the broad sweep).
• The coordinator's known V2 failures:
  • ui_tests/test_v2_workflow_loops.py passes 72/73 (Playwright, pytest); test_revoked_record_refresh_clears_the_previous_page fails.
  • test_block_revision_assist.py passes 7/8 as a script.
  • test_workflow_priority_alerts.py needs a Cosmos key. It's one of the 71 sweep files.
  • The flow-canvas test_save_failures_retain_flow_fields…[400/409/503] passed in this run.

CodeQL

CodeQL reported 9 new alerts at 177e0bc. At 5933c07 it reports 3: the Log Injection alerts below.

Six were in code this branch adds; 5933c07 fixes them, with no behavior change:

• functions_workflow_file_sync_sources.py: a lambda that only wrapped get_user_settings (L67), and two except … pass blocks with no comment (L93, L106). Skipping there is intended: an active group or public workspace that's unset, gone or not managed by the user contributes no sources, as in the route before the helper moved out of it. The blocks now say so.
• test_personal_workflow_file_sync_sources.py L122: a lambda that only wrapped dict.
• test_v2_orchestration_workflow_proposal_card.py L32: the plan editor module was imported with both import and from … import. One from import now brings in the constants and the fixtures.
• test_workflow_settings_client.js L54: the stubbed fetch passed the request to responders that take no arguments. The tests read the recorded requests instead, so the call no longer passes it.

Rerun at 5933c07:

• pytest, normally and with -O: personal File Sync sources 22, group File Sync client parity 179, planning context 112, proposal routes 60, imports 6, "Done when" 2;
• node --test: settings client 11;
• Playwright (pytest): proposal card 22 and personal File Sync 8;
• run_v2_guardrails.py: all six checks pass.

Three are Log Injection (medium) at the existing log_event sinks in functions_appinsights.py, which this branch doesn't change: logger.exception (L515), logger.log (L531) and the fallback logger (L553). Each alert lists at least 74 user-provided sources. This branch adds 4 routes and 3 request reads, so it adds sources to existing sinks rather than new sinks. I made no change for these, because the flows are already neutralized in ways CodeQL doesn't model:

• Every new log message is a constant string. No request value is formatted into it.
• The logger call's message is the constant [SIMPLE_CHAT_LOG_EVENT]. Everything else goes in as structured properties built by _build_logger_extra. That keeps text only for allowlisted diagnostic keys, keeps a *_hash key only when it's 64 hex characters and a workflow code key only when it's a snake_case code, and reduces every other string to its length.
• sanitize_log_message replaces CR, LF and tab in the message and every property string before anything is emitted, including the console fallback.
• Where a run, conversation, turn or step id is kept for correlation, it goes through workflow_log_context, which logs only its SHA-256 hash. Other ids reach the logger properties as lengths only.

The new logging sites, and what each logs:

Site	Logs
route_backend_orchestration.py L2789 (the proposal routes' 503 path)	hashed run and conversation ids (the run id comes from the URL), stage, error type
functions_orchestration_workflow_proposals.py L215 _log, called at L395, L522, L701, L722, L759, L868, L877 and L928	hashed run and conversation ids; step, proposal and workflow ids (lengths only in the logger properties); codes, flags, mode and error types
functions_orchestration_workflows.py L127 _log, called at L296, L609, L697, L720, L744 and L759	run, conversation, step and proposal ids (lengths only in the logger properties); status and reason codes; error types
functions_orchestration_workflow_context.py L715 _log_context, called at L753, L778, L800 and L844	reason code, error type, catalog counts and build time; no catalog entries
functions_orchestration_executor.py L608	hashed run, conversation and step ids, capability id, error type
functions_orchestration_planner.py L1275	the planner's hashed correlation, validation code and rule, attempt, revision, reason codes
functions_orchestration_registry.py L290	a constant reason and the error type

This token can't read or dismiss code-scanning alerts (403), so these three need a maintainer's triage.

#1577's CodeQL check lists Log Injection at these same three sink lines, plus L419. 3b matched them to the existing alerts #2551–#2554, which are open on V2 and Development. This branch doesn't change the sink lines, so these are the same alerts, and dismissing #2551–#2554 for #1577 clears them here too. This check passes either way, because they're medium.

Follow-ups

• Route-level URL Access preparation for chat-created workflows (v1 refuses it and explains).
• The soft cap under simultaneous accepts of different proposals (documented).
• A pre-run Run as approval route, so the card can approve before the first run.
• A conversation-level status batch route, if per-run reads prove chatty.
• V2-native Microsoft 365 connection and approvals pages, replacing the classic links.
• Group blueprint builds (Workflow orchestration follow-ons: group workflows, SimpleChat action parity and plan replay #1550), and the scheduler, list-label and unknown-trigger_type gaps the kickoff listed as pre-existing.
• Pre-existing, noticed while validating, not changed here: the functions_document_actions ↔ functions_settings import cycle, the stale generated release-notes pages, and the stubbed-namespace NameErrors behind several base failures listed under Validation.