Skip to content

Read saved workflow results from chat orchestration (workflow_results) - #1607

Merged
Paul Lizer (paullizer) merged 28 commits into
microsoft:paullizer-react-v2-uifrom
paullizer:paullizer-workflow-results-orchestration-capabilit
Oct 2, 2026
Merged

Paul Lizer (paullizer) merged 28 commits into
microsoft:paullizer-react-v2-uifrom
paullizer:paullizer-workflow-results-orchestration-capabilit

Conversation

@paullizer

@paullizer Paul Lizer (paullizer) commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • New chat-orchestration capability workflow_results. In a private chat, a plan can read the stored result of one of the user's own finished personal workflow runs and answer from it. Example requests: "what did my Monday email digest workflow say this week?" or "compare yesterday's and today's run of my contract review workflow".
    • It's a read-only Gather step with static inputs: a catalog handle, a latest or completed_on: YYYY-MM-DD selector (the user's local date), and an optional closed status filter.
    • A plan can include it at most twice, and it never starts a workflow.
  • The result is used as notes, never as evidence. The excerpt reaches compose only as fenced, untrusted data with a fresh nonce. It carries no IDs or handles and is never cited.
    • A failed or cancelled run is reported by its status and local completion time only, and a run still in progress only as in progress; neither contributes output text.
    • The published answer ends with application-owned text that names the workflow and its local completion time, and says the workflow was not re-run.
  • The answer joins Phase 6a's lineage and read-time masking. It's saved with metadata.workflow_result_contexts.
    • The source result is re-authorized at compose time, at finalize, again right before publishing, and whenever a step is reused.
    • If the result changed or was deleted, the user gets a fixed failure and the saved run message is overwritten. A saved answer is masked later through 6a.
    • Runs that read a result are never offered as result aliases, and a retry or revision always reads the result again.
  • Admin impact. The capability stays dormant unless Use Workflow Results In Chat (enable_chat_workflow_results, role-aware gate) is on, together with Chat Orchestration and personal workflows.
    • The capability allowlist gains a Read workflow results option. An empty allowlist still allows every capability.
    • With the setting off, planning output is byte-identical to the base (new off golden).

Linked issue

Refs #1546

Part of #1546 (Phase 6: post workflow results back into chat and answer follow-ups from them).
Part of #1543 (Chat Orchestration Workflows roadmap, step 9).

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only. It goes from 0.261.216 to 0.261.217; the diff against V2 touches only that line.
  • deployers/version.txt bumped, or not needed because deployers/ was not changed. deployers/ is untouched.

Testing / validation

  • Environment. Python commands use the repo venv with PYTHONPATH=application/single_app;functional_tests, run from the repo root.
  • Commits. The review round on 6cf036e7b (the coordinator's mutation pass; see "Review survivors" under the kill tables) added six commits:
    • 2cf793661 is the only production edit (review item F), and it changes no behavior. It deletes three lines in _read_workflow_result_step that read the closed reason and wrote the same value back.
    • The other five add tests only:
      • aacdf8841: reads
      • f341b910a: answer
      • ad165008b: capability
      • 8bec393d2: the new time-zone suite
      • 3059ac439: the new UI test
    • Before the review round, 6cf036e7b only removed the unused imports and local that CodeQL reported as notes, and production code had last changed in eb19f38f7.
  • Pass counts. "Both ways" means one plain run and one with python -O. Every result below comes from head 3059ac439.
Command Plain python -O
The four suites this round changed, each run as python <file> and as python -m pytest -p no:cacheprovider -q -rfEs --tb=line <file>: test_orchestration_workflow_results_{reads,answer,capability,time_zone}.py All 8 runs pass: reads 53, answer 18, capability 70, time_zone 8 All 8 runs pass with the same counts
Batch B (52 files, listed below): python -m pytest -p no:cacheprovider -q -rfEs --tb=line <batch B> 2326 passed, 2 xfailed, 260 subtests passed. At 6cf036e7b it was 2271; the round adds 55 tests (reads 32→53, answer 7→18, capability 55→70, time_zone +8) 2326 passed, 2 xfailed, 260 subtests passed
python -m pytest -p no:cacheprovider -q functional_tests/test_orchestration_workflow_results_imports.py functional_tests/test_orchestration_workflow_run_imports.py functional_tests/test_orchestration_workflow_proposal_imports.py functional_tests/test_workflow_result_privacy_import_cycle.py functional_tests/test_orchestration_workflow_runs_admin.py functional_tests/route_tests/test_workflow_result_context_policy.py 108 passed 108 passed
npm --prefix application/v2_ui run build, then python -m pytest -p no:cacheprovider -q -rfEs --tb=line ui_tests/test_v2_orchestration_auto_open.py 5 passed 5 passed
npm --prefix application/v2_ui run build -- --outDir ..\..\ui_tests\artifacts\orchestration-plan-editor, then python -m pytest -p no:cacheprovider -q -rfEs --tb=line <plan-editor harness, 11 files listed below> 167 passed: the 162 from 6cf036e7b plus the new UI test's 5 167 passed
python ui_tests/test_v2_orchestration_workflow_results_input.py (the new UI test's own runner, which calls pytest.main) 5 passed 5 passed
python ui_tests/test_v2_orchestration_plan_editing.py (the file's own runner; the pytest run is under pre-existing failures) 5/5 tests passed 5/5 tests passed
python -m pytest -p no:cacheprovider -q -rfEs --tb=line <admin and chat UI, 5 files listed below> 77 passed 77 passed
node --test functional_tests/test_v2_orchestration_*.mjs functional_tests/test_workflow_results_clients.mjs (12 files) 52 tests, 52 pass, 0 fail n/a
npm --prefix application/v2_ui run typecheck (tsc -b --noEmit) Clean, no errors n/a
python scripts/build_docs_inventory.py no content change to docs/_data/app_surface.yml n/a
python functional_tests/test_docs_app_surface_coverage.py 7/7 checks passed n/a
python functional_tests/test_docs_site_quality.py 6/6 checks passed n/a
python -m pytest -p no:cacheprovider -q functional_tests/test_docs_*.py (7 files) 23 passed, 5 failed: identical on base b5cc18ee2 n/a
Guardrails: run_v2_guardrails.py <worktree> b5cc18ee2 HEAD step9 all PASS (see Guardrails) n/a

Batch B: the 52 files. All live in functional_tests/.

  • New (6): test_orchestration_workflow_results_{aliases,answer,capability,off_golden,reads,time_zone}.py
  • Regression suites fixed by test: fix V2 test-harness drift (chat image refs, auto-open fixture, harness seeds) #1598:
    • test_v2_admin_workflow_parity.py
    • test_orchestration_plan_revision_{routes,planner,references,store}.py
    • test_orchestration_turn_recovery_identifiers.py
    • test_v2_prompt_composer_card.py
  • Orchestration core:
    • test_orchestration_services.py, test_orchestration_harness_execution.py
    • test_orchestration_dependency_recovery.py, test_orchestration_recovery_safety.py, test_orchestration_dependency_allowlist.py
    • test_orchestration_single_contract_parity.py, test_orchestration_single_contract.py
    • test_orchestration_registry_contract.py, test_orchestration_executor.py, test_orchestration_deliverables.py
    • test_orchestration_composition_metadata_recovery.py
    • test_orchestration_planner_failure_diagnostics.py, test_orchestration_planner_model_setting.py
    • test_orchestration_reference_authorizer_golden.py
  • Phase 4 and 5:
    • test_orchestration_workflow_run_{answer,capability,adapter,approval_floor,effects,link_routes,planning_context}.py
    • test_orchestration_workflow_planning_context.py, test_orchestration_workflow_propose_capability.py
    • test_orchestration_workflow_proposal_{done_when,editor_round_trip,routes}.py, test_orchestration_workflow_proposals_admin.py
  • Existing goldens: test_orchestration_workflow_runs_off_golden.py, test_orchestration_workflow_setting_off_golden.py
  • Phase 6a:
    • test_workflow_result_{chat_routes,contract,followup,masking,orchestration_lineage,reader,review_paths,routes,store}.py
    • test_chat_workflow_results_admin.py

Plan-editor harness: the 11 files. All live in ui_tests/: test_v2_chat_image_uploads.py, test_v2_orchestration_{dependency_plans,generated_images,plan_editor,streaming_bubble,workflow_proposal_card,workflow_run_floor,workflow_run_links,workflow_results_input}.py, test_v2_plan_editor_references.py and test_v2_workflow_ask_ai_proposal.py. test_v2_orchestration_workflow_results_input.py is new in the review round (see "Review survivors").

Admin and chat UI: the 5 files. All live in ui_tests/: test_admin_orchestration_{workflow_runs,workflow_proposals,actions}.py, test_chat_workflow_results.py and test_v2_workflow_alert_notices.py.

Pre-existing failures, all identical on base b5cc18ee2, which I checked in a detached worktree:

  • Docs:
    • test_docs_json_gem_security_fix (1)
    • test_docs_link_integrity (3). The broken-link lists are identical on head and base, so every link this PR adds resolves.
    • test_docs_release_notes_integrity::test_generated_pages_match_source (1). The generated release-note pages were already stale on the base, and I didn't regenerate them.
  • test_workflow_cancellation.py (3).
  • Batch B's 2 xfails are test_orchestration_harness_execution.py::test_real_web_scheduler_bootstrap_import_boundaries[order1-False] and [order1-True]. That's the scheduler-first import order, which the test itself marks as a pre-existing background_tasks → app NameError. Base gives the same 2 xfails, and this PR doesn't touch that file.
  • python -m pytest ui_tests/test_v2_orchestration_plan_editing.py: 4 failed, 1 passed, both ways, with the same 4 tests failing on base.
    • The four browser tests read a module-level _PAGE that only the file's main() sets (L334–335). Under pytest they get None and return False, and the pytest reports functional tests that return False as passed #1572 guard turns that into a failure.
    • The file and ui_tests/conftest.py are unchanged from base, and the file's own runner passes 5/5 both ways (see the table).

Documentation

  • Release notes updated, or not needed. There's a new 0.261.217 section at the top of docs/explanation/release_notes.md. The 0.261.216 section and everything below it are byte-identical to V2.
  • Feature documentation updated, or not needed:
    • New docs/explanation/features/CHAT_ORCHESTRATION_WORKFLOW_RESULTS.md.
    • Updated CHAT_WORKFLOW_RESULTS_FOLLOW_UP.md, docs/guides/ask-about-workflow-results.md, docs/reference/chat-controls.md, docs/admin/orchestration.md and docs/admin/workflow.md.
  • Fix documentation updated, or not needed. Not needed; this is a feature.

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security()). There are no new routes; route_backend_orchestration.py only swaps one import and the two time-zone gates.
  • Settings sent to non-admin frontends use sanitize_settings_for_user(). No new settings reach a non-admin frontend; the capability reads server-side settings only.
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS. The only change is V2 TypeScript built into the local SPA.
  • No secrets, keys, connection strings, or local-only artifacts are included.

Merge base, head and version

Coordinator review items → code and tests

All the tests below live in functional_tests/test_orchestration_workflow_results_<file>.py.

# Item Where Covered by
1 Alias exclusion, defense in depth discover_result_aliases skips the whole run if its stored run.plan has a workflow_results step, or if any task result has a workflow_results producer. admitted_result_aliases refuses a workflow_results producer with result_reference_untrusted. aliases: test_completed_workflow_results_run_is_never_discovered_but_compose_only_run_is, test_plan_check_excludes_even_without_read_task_result_and_when_read_is_disabled, test_producer_check_excludes_workflow_results_task_even_if_plan_step_is_missing, test_partially_executed_workflow_results_run_is_not_discovered (results step done, compose not yet), test_superseded_revision_and_retry_attempts_are_excluded, test_stored_aliases_refuse_workflow_results_refs_and_clean_aliases_resolve (admitted_result_aliases / resolve_revision_result_aliases), test_retry_new_attempt_rereads_workflow_result_and_reruns_computed_steps, test_disabled_workflow_results_read_invalidates_retained_consumers_only_for_new_attempt (a new attempt that disables or removes the read never reuses the earlier compose), test_retry_child_refuses_edits_inherited_parent_changes_and_completed_run_edits
2 Catalog reachable with runs off build_workflow_planning_context now returns early only when proposals, runs and results are all off. With results on, a ready proposal context gets the workflow_results marker, and the time zone if it's missing, from _with_workflow_markers. Every other case (proposals off, at quota or context_unavailable) reads the catalog through _with_run_catalog(..., runs=..., results=..., time_zone=...). workflow_results_projection is a subset of the run catalog. capability: test_results_catalog_is_reachable_with_or_without_proposals_or_runs (R, P+R, Ru+R, P+Ru+R, with subset asserts against the run catalog), test_results_catalog_is_used_when_proposals_are_at_quota_or_unavailable, test_results_off_with_no_other_workflow_capability_reads_nothing
3 Degrade and drop The planner's degraded _normalize passes workflow_planning whenever PROPOSE, RUN or RESULTS is still in degraded_ids. drop_workflow_results(..., drop_all=context_unavailable) drops the invalid steps and writes fixed notes. capability: test_degraded_normalize_keeps_remaining_valid_results_with_plan_inputs, test_drop_workflow_results_removes_invalid_steps_and_writes_fixed_notes
4 Timestamps and queries _parse_utc explicitly handles Z, +00:00 and naive values (as UTC); unparseable rows are skipped. @lo/@hi are UTC isoformat() strings, built the same way the writers build them. See "started_at, partitioning and indexing" below. reads: test_timestamp_parsing_day_windows_and_local_date_selection (Z, +00:00, naive, a run completed exactly at local midnight, 02:30Z → the previous day in America/New_York), test_latest_skips_unparseable_rows_missing_started_at_sorts_last_and_full_bad_page_fails_closed, test_query_shapes_are_bounded_projected_partitioned_and_errors_raise
5 _finalize Compose doesn't stream (it makes one invoke), so I rely on the compose-time read plus two finalize re-authorizations. On workflow_result_changed, the published message is the fixed failure text plus non-read fixed lines only: no excerpt-derived text, and no disclosure line (workflow name or run time). The run record's at-rest message, which lease.update writes before the pre-publish re-check, is overwritten. answer: test_pre_publish_reauthorization_failure_publishes_fixed_failure_and_overwrites_at_rest, test_run_summary_and_detail_rows_are_allowlisted_against_message_summary_and_sidecar (pins _run_summary_row / _run_detail_row as allowlists without message, summary or the sidecar)
6 Thoughts Step thoughts carry fixed text only. answer: test_results_run_events_and_thought_shaped_payloads_do_not_contain_excerpt_marker checks stored thought content/detail (through a cosmos_thoughts_container stub), ThoughtTracker.add_thought, and every emitted event and frame
7 Logging User-controlled values are hashed or passed through sanitize_log_message. There's no str(exc) to clients, and no excerpt, workflow name or query text in logs. reads: test_logs_and_user_visible_failures_are_sanitized
8 Harness overlap No exec'd test-harness namespace needed a new name, and I added no seeds. I made two one-line edits to existing tests, in files #1598 didn't touch: test_orchestration_dependency_allowlist.py, which adds 'workflow_results' to the registry order list, and test_orchestration_single_contract_parity.py, which adds "workflow_results" to the retry_on_transient set. both files are in batch B
9 Base and version The base is b5cc18ee2, and VERSION is 0.261.217. The tests use assert_app_version_at_least(...), never an exact version. every new file's version test

started_at, partitioning and indexing (item 4)

  • Every personal run writer I found sets started_at as UTC isoformat() when it creates the run:

    • the runner's run record ('started_at': started_at)
    • the M365 failure path ('started_at': now)
    • functions_workflow_runtime (control.get("created_at") or _now())

    The helpers are functions_workflow_runner._utc_now_iso, functions_workflow_runtime._now and functions_workflow_runtime_store._now, all UTC.

  • Personal runs are partitioned by /user_id (config.py) with default indexing. Each query passes partition_key=user_id and orders by a single property (c.started_at DESC or c.completed_at DESC), so no composite index is needed.

  • Queries:

    • latest: SELECT TOP 5 c.id, c.workflow_id, c.status, c.started_at, c.completed_at FROM c WHERE c.user_id=@u AND c.workflow_id=@w AND ARRAY_CONTAINS(@statuses, c.status) ORDER BY c.started_at DESC.
    • The in-progress probe: TOP 1 … NOT ARRAY_CONTAINS(@finished, c.status).
    • completed_on: SELECT TOP 20 … AND c.completed_at >= @lo AND c.completed_at < @hi ORDER BY c.completed_at DESC. The window is the local day's exact UTC bounds, computed with DST-aware zoneinfo and widened by one minute on each side. An exact local-date filter is applied after parsing, and the date range is today − 366 days to today.
    • Storage errors raise and become the retryable workflow_results_unavailable; they're never reported as "no runs".
    • A full page of unusable rows fails closed as unavailable rather than guessing past TOP.
  • What latest does with a doc missing started_at. Under the default indexing policy, Cosmos DB's ORDER BY returns such a doc and sorts the undefined value lowest, so it comes last under DESC. Older emulator builds exclude it instead. Either way, it never beats a run that has started_at.

    • It's still read when it's the only usable row, and then without the "a newer run is in progress" line.
    • eb19f38f7 made the reads fixture genuinely omit started_at, and it pins this lone-row case.

Kill tables (every mutation fails at least one test, plain and under -O)

Planned mutations (M1–M20):

# Mutation Killed by
M1 Handle bound from an upstream step capability test_results_workflow_handle_must_be_static_and_not_bound
M2 Privacy re-check skipped reads test_runtime_access_chain_fails_closed_before_query_or_result_read
M3 Storage error treated as "no runs" reads test_query_storage_errors_fail_retryably_or_safely_not_as_no_runs, test_storage_backed_query_run_errors_raise_instead_of_looking_empty
M4 Wrong zone or local date (02:30Z, America/New_York) reads test_timestamp_parsing_day_windows_and_local_date_selection
M5 IDs or handles in the notes reads test_completed_run_reads_notes_context_and_uses_budget_without_evidence
M6 Evidence instead of notes (sources or a grounded origin) reads test_completed_run_reads_notes_context_and_uses_budget_without_evidence
M7 Bare setting key, or no role gate capability test_results_gates_are_role_aware_and_fail_closed
M8 Proposal instructions on a results-only turn capability test_planner_keeps_proposal_instructions_off_for_a_results_only_turn
M9 TOP dropped reads test_query_shapes_are_bounded_projected_partitioned_and_errors_raise
M10 Window dropped reads test_query_shapes_are_bounded_projected_partitioned_and_errors_raise
M11 Lineage missing from the metadata answer test_successful_answer_uses_plural_lineage_metadata_and_safe_disclosure
M12 Singular workflow_result used instead of workflow_result_contexts answer test_successful_answer_uses_plural_lineage_metadata_and_safe_disclosure
M13 No re-authorization at finalize answer test_pre_publish_reauthorization_failure_publishes_fixed_failure_and_overwrites_at_rest
M14 Reused step without re-authorization reads test_rebuild_reauthorizes_context_and_fails_closed_for_changed_or_malformed_sidecar
M15 Run and results allowed for the same handle capability test_duplicate_limit_and_same_plan_run_rules
M16 render_file able to bind the output capability test_only_compose_may_consume_workflow_results
M17 Step summary containing excerpt text reads test_step_summary_and_events_never_include_excerpt_text
M18 Alias discovery not excluding the run the aliases tests (item 1)
M19 Missing sidecar not failing closed reads test_rebuild_reauthorizes_context_and_fails_closed_for_changed_or_malformed_sidecar
M20 time_zone not plumbed when results are on capability test_planner_offers_results_only_when_ready_and_projects_time_zone

Coordinator mutations:

Mutation Killed by
Drop the producer alias check only aliases test_producer_check_excludes_workflow_results_task_even_if_plan_step_is_missing
Drop the plan-step alias check only aliases test_plan_check_excludes_even_without_read_task_result_and_when_read_is_disabled, test_partially_executed_workflow_results_run_is_not_discovered
Results catalog unreachable with runs off capability test_results_catalog_is_reachable_with_or_without_proposals_or_runs, test_results_catalog_is_used_when_proposals_are_at_quota_or_unavailable
Disclosure line kept on the failure path answer test_pre_publish_reauthorization_failure_publishes_fixed_failure_and_overwrites_at_rest
Re-authorization skipped on reuse only reads test_rebuild_reauthorizes_context_and_fails_closed_for_changed_or_malformed_sidecar, answer test_sidecar_stays_server_only_and_restore_reauthorizes_or_fails_closed
Status enum widened capability test_status_filter_is_a_closed_enum
Handle accepted from depends_on capability test_results_workflow_handle_must_be_static_and_not_bound
Sidecar leaked into task_results answer test_sidecar_stays_server_only_and_restore_reauthorizes_or_fails_closed
Excerpt in a step summary reads test_step_summary_and_events_never_include_excerpt_text
Nonce reused reads test_compose_defends_against_nonce_reuse_by_requesting_another_nonce

Import-lifecycle and /steps mutations:

Mutation Killed by
M-a: eager module-level import of the 6a reader test_orchestration_workflow_results_imports.py
M-b: the schema imports the results module at module level test_orchestration_workflow_results_imports.py
M-c: adapter registry swap test_orchestration_workflow_results_imports.py
M-d: renamed lazy attribute test_orchestration_workflow_results_imports.py
public_step_record allowlist widened with workflow_results answer test_sidecar_stays_server_only_and_restore_reauthorizes_or_fails_closed
public_step_record allowlist widened with arguments (would serve the handle) answer test_sidecar_stays_server_only_and_restore_reauthorizes_or_fails_closed

About the import tests:

  • The 26 tests load the module in fresh normal and optimized interpreters, in ten application import orders: web bootstrap, settings → results → routes, the scheduler, both orders with the schema, with the workflow context and with the 6a follow-up module, and through the adapters registry.
  • Only Cosmos DB is doubled, and every network socket is blocked.
  • The 6a reader and follow-up modules are pinned as lazy runtime dependencies.
  • A static check confirms that the schema and the orchestration callers reach the results module only from function bodies.

Review survivors. These are the 30 mutants that survived the coordinator's pass at 6cf036e7b. The review round added tests only; no production behavior changed. I reapplied each mutant locally and ran the suites both ways, and every one is killed in both modes.

Survivor What the mutant did Killed by
T08 workflow_time_zone_configured used the proposals gate only capability test_time_zone_is_needed_for_proposals_or_results_but_not_runs_only
W02, E07 A new turn (route) and execution fell back to the proposals-only gate time_zone test_results_only_planning_stores_local_zone_and_runs_with_it
W01 _validate_retry_context zone fell back to the proposals-only gate time_zone test_results_only_retry_validation_keeps_saved_zone_and_workflow_context
E08 _workflow_results_time_zone dropped the record-zone fallback time_zone test_saved_workflow_result_note_uses_record_zone_when_context_zone_is_absent
E09 Note written with time_zone=None time_zone test_saved_workflow_result_note_uses_planning_zone_for_local_midnight, …_uses_context_zone_when_planning_zone_is_absent, …_uses_record_zone_when_context_zone_is_absent
K04 Compose passed None instead of the planning zone time_zone test_compose_fence_uses_planning_zone_before_context_zone
K05 Compose passed neither zone time_zone test_compose_fence_uses_planning_zone_before_context_zone, test_compose_fence_uses_context_zone_when_planning_zone_is_absent
K01, K02 Compose never recognized a results reader; the fenced-notes policy was never appended answer test_compose_sends_only_fenced_workflow_result_notes_and_policy_to_model
K03 The compose workflow_results_error branch was dropped answer test_compose_reread_failures_keep_specific_workflow_result_failure_codes
R07b Run-time _select_completed_on window check dropped reads test_completed_on_rejects_dates_outside_runtime_window_without_query
R18 A point-read Cosmos error returned None reads test_point_reads_raise_retryable_failure_for_cosmos_errors_and_not_found_is_none
R19 partition_key=None in _query_runs reads test_storage_backed_query_uses_user_partition_key
R22 The day query accepted rows from another local day reads test_completed_on_skips_margin_rows_from_the_wrong_local_day
R24 len(rows) >= 20 became > 20 reads test_completed_on_full_bad_page_fails_closed_but_short_bad_page_is_no_match
R25 probe_started < chosen_started reads test_latest_marks_only_later_in_progress_runs_as_newer
R35 Rebuild ignored closed reads test_rebuild_returns_none_when_access_has_closed_since_original_read
R36 Reader storage-unavailable became an UNAVAILABLE outcome reads test_reader_storage_unavailable_fails_step_retryably
R38 saved_inputs no longer forced ANALYSIS_ONLY reads test_saved_inputs_force_analysis_only_without_retained_context_or_excerpt
R39 The post-read recheck() was removed reads test_workflow_results_step_rechecks_cancellation_after_read_before_persisting
R42 Lineage kept duplicates answer test_duplicate_workflow_result_reads_save_one_lineage_context
R43 Lineage included outcomes that weren't READ answer test_non_read_workflow_result_outcomes_complete_without_lineage, test_mixed_read_and_no_match_workflow_results_save_only_read_lineage
R44 Lineage included steps that didn't complete answer test_failed_partial_workflow_results_input_does_not_create_lineage_or_changed_failure
S05 workflow_results_unavailable removed from the transient codes answer test_workflow_results_unavailable_is_transient_and_results_step_retries_once
E04b The waiting exception for the results note was removed answer test_waiting_run_with_prepared_answer_keeps_saved_workflow_results_note
T06 Results were offered to a user with zero saved workflows capability test_empty_results_catalog_is_unavailable_and_not_offered
P01 The planner's workflow_results_notes pop was removed capability test_the_model_cannot_write_why_a_workflow_result_was_not_read
D02, D03 _workflow_results_reason planner-truth text capability test_unavailable_results_are_explained_with_closed_text. It covers the five closed reasons, plus workflow_results_disabled, an unknown reason (feature_disabled) and a missing reason; the last three all read "turned off for this deployment".

Notes on the review round:

  • Rerun set. W01, W02, E07, E08, E09, K04 and K05 are killed only by the new test_orchestration_workflow_results_time_zone.py. The mutant rerun set should become CAP + READS + ANSWER + ALIASES + GOLDEN + TIME_ZONE.
  • UI wiring (item E). This isn't a mutant row. ui_tests/test_v2_orchestration_workflow_results_input.py drives OrchestrationRunView.tsx through the harness entry. I also applied three local TSX mutations and the test caught each one, after which the TSX was restored:
    1. dropping the results-argument filter
    2. looking names up in planInputWorkflows instead of planInputWorkflowResults
    3. showing the results <dl> on workflow_run steps
  • Item F. I removed the three-line self-assignment of closed['reason'] in _read_workflow_result_step. It was a no-op, so behavior is unchanged.

Leak surfaces

Surface Handling Covered by
Compose prompt Fenced as untrusted data with a fresh nonce for each read (fence_workflow_result / new_fence_nonce). The system message lists every nonce; no IDs or handles. reads test_compose_inputs_refence_each_read_with_fresh_nonces_and_no_marker_outside_fences, test_compose_defends_against_nonce_reuse_by_requesting_another_nonce, test_compose_rejects_changed_or_unauthorized_result_without_exposing_excerpt
Retained task result Context only (workflow_id, run_id, result_sha256), sources=[], origin='generated'. The excerpt is read only inside compose, through read_workflow_result(..., expected_sha256=..., excerpt_budget_bytes=24 KiB). reads test_completed_run_reads_notes_context_and_uses_budget_without_evidence; answer sidecar test
result['workflow_results'] sidecar Server-only. It's rebuilt and re-authorized on restore and never copied into task_results. answer test_sidecar_stays_server_only_and_restore_reauthorizes_or_fails_closed
Run list and detail rows _run_summary_row / _run_detail_row are allowlists; they never serve the run's message, summary, task_results or the sidecar. answer test_run_summary_and_detail_rows_are_allowlisted_against_message_summary_and_sidecar
/steps (public_step_record) Allowlist: the sidecar and arguments are dropped, and the summary is fixed text. answer sidecar test (added in 1c1c40edf)
public_execution_fields Fixed text only. answer sidecar test
Step SSE, progress frames and run_done Allowlisted fields. run_done is emitted only after the pre-publish re-check. answer events test and pre-publish test
Thoughts, stored and tracked Fixed text only. answer events test
Run record at rest (message, summary, task_results) Unserved. The failure path overwrites message. answer route-rows test and pre-publish test
Published chat answer 6a lineage through metadata.workflow_result_contexts, plus read-time masking when the source changes or is deleted. answer test_successful_answer_uses_plural_lineage_metadata_and_safe_disclosure, test_stored_answer_with_workflow_result_contexts_is_masked_when_source_changes_or_is_deleted; 6a suites
Result aliases Results-bearing runs are excluded from discovery and refused at admission. aliases tests (item 1)
Retry, revision and resume A new attempt re-reads and re-runs consumers; the same attempt re-authorizes on reuse. aliases retry tests, reads rebuild test, answer restore
Downstream binding and /outputs Transitive taint closure: only compose may bind to or depend on a results step or anything downstream of one, and final_response may select only that compose. capability test_only_compose_may_consume_workflow_results, test_compose_may_consume_results_and_final_response_may_select_that_compose
Conversation search, chat history, sharing, saved analyses 6a's existing message_uses_workflow_result handling, unchanged, which also covers workflow_result_contexts. 6a suites (test_workflow_result_*, re-run unchanged in batch B)
Orchestration history Unchanged 6a behavior: the history reader (functions_orchestration_context) leaves out messages with workflow_result / workflow_result_contexts. 6a test_workflow_result_orchestration_lineage.py (batch B)
Logs and user-visible failures Sanitized or hashed; no excerpt, name or query text; no str(exc). reads test_logs_and_user_visible_failures_are_sanitized

Decisions made autonomously

  • latest is the newest finished run that matches the status filter, ordered by started_at.
    • When a newer run is still in progress, the notes add a one-line mention.
    • The outcome is in_progress only when no finished run matches but one is running.
  • completed_on is the user's local date, from today − 366 days to today, in the planner's stored time zone.
    • The schema always checks the format. It checks the range when a planning context is present, and the adapter checks it again.
  • The status filter is a closed enum:
    • completed → completed and completed_partial
    • failed → failed, invalid and incomplete
    • cancelled → cancelled and skipped
  • A full TOP page fails closed as unavailable rather than silently missing rows.
  • Excerpt budget: 24 KiB per step, so 48 KiB at most with 2 steps. There's no step-down yet (follow-up).
  • Saved Analyze runs (analysis_only / saved_inputs) are never used in a plan. The reader identifies them, the step discards their text and retains no context, and the step gets the closed analysis_only outcome. That outcome tells the user to choose Ask in chat on the run in its workflow's run history.
  • Group workflows (group_id) are rejected; this capability is personal-only.
  • The same-plan rule rejects run and results steps for the same handle, and it also counts disabled run steps. After the correction round, the step is dropped with a fixed note, following Phase 5's pattern.
  • The taint closure is transitive, and only compose may consume.
    • knowledge_basis handling is unchanged, because results are notes, not sources.
  • Compose's system message lists every nonce when a plan has two reads.
  • Recovery: a new attempt (retry or revision) always re-runs an enabled results step and its consumers. Only a same-attempt restart reuses, and then it re-authorizes.
  • A lineage or re-authorization failure is the non-retryable workflow_result_changed, raised from None.
  • Answer text:
    • The disclosure line appears only for a read.
    • The results note is left out of a reply for a plan that's waiting for approval or was cancelled, unless that reply carries the composed answer.
    • The fence labels status as "Run status:".
  • Time zone: the turn's time zone is stored when proposals or results are configured (workflow_time_zone_configured), so output stays byte-identical when results are off.
  • Admin and V2 UI:
    • The admin option label is "Read workflow results".
    • The V2 RunView shows the selector in words and names the workflow from plan inputs, never by handle. There's no run link yet (follow-up).
  • Docs and release notes:
    • The docs make no Retry/Edit claim for this step.
    • I didn't regenerate the release-note pages; they were already stale on the base.
    • There's no Latest Feature card.
  • Admin tests: the planned edits to test_orchestration_workflow_runs_admin.py and test_chat_workflow_results_admin.py weren't needed, and both pass unchanged. The new option's position and help text are pinned by capability test_admin_capability_option_sits_after_workflow_run_with_the_expected_help.

Overlap with 6b-1

  • 6b-1 (paullizer-phase-6b-1-workflow-results-to-chat, .218) adds a status route after the links route in route_backend_orchestration.py, plus names on the from functions_settings import … line.
  • This PR makes three edits to route_backend_orchestration.py:
    • In the functions_orchestration_workflow_context import block, it adds workflow_time_zone_configured and drops the now-unused workflow_proposals_configured.
    • It switches the two time-zone gates (the retry context and the new turn's time_zone) to workflow_time_zone_configured.
    • It leaves the functions_settings import line and the links route untouched.
  • Whichever PR merges second merges V2 with rerere off and keeps its own VERSION and release-notes section on top.

Guardrails

run_v2_guardrails.py <worktree> b5cc18ee2 HEAD step9, run at head 3059ac439. Its own summary: "40 changed files; app .py 15, xss surface 21, route .py 15, v2 ts/tsx 4".

  • broken-access-control: PASS, 15 files.
  • xss-sinks: PASS, 21 files.
  • swagger-routes: PASS, 1 changed route file.
  • python-syntax: PASS, 492 top-level single_app .py files compile.
  • v2-ts-sinks: PASS, no sink patterns in the added lines of 4 .ts/.tsx files.
  • malicious-pr-review: PASS, "No blocker findings detected in reviewed scope."
    • The full report (mpr-review-step9.md) lists 1,326 keyword-marker findings across 30 files: 1,173 Important and 153 Moderate, all "Needs investigation" heuristics, none a blocker.
    • Where they fall:
      • 554 are in the off golden JSON.
      • 521 are in the other test files.
      • 90 are in the docs.
      • 95 are in the new results module.
      • 66 are in 12 wired production files: the orchestration modules, route_backend_orchestration.py and the admin workflow pane.
    • The runner prints only the last 12 lines of output, so its log shows just 8 annotations. At this head they all point at the new UI test:
      • L278–298 are the harness fixture's conversation keys.
      • L301 is the "Approve and run the plan" button.
      • L363 is the test name containing "hidden".
    • The previous body said the log's 8 warnings "all point at test data". That described only that tail, not the whole report.

CodeQL

  • At 1c1c40edf: the CodeQL check passed (success) with "8 new alerts", all note-level. None was a security alert.
    • functions_orchestration_workflow_results.py: an unused import of WORKFLOW_RESULTS_SELECTOR_LATEST, and an unused local name in _check_access.
    • test_orchestration_workflow_results_capability.py: unused imports (datetime, timezone, AGENT_SETTINGS, _build_proposal_context, BOTH, BOTH_SETTINGS, NOW, PRIVATE, RUN_SETTINGS, USER_INFO).
    • test_orchestration_workflow_results_reads.py L96: a stub method's first parameter was named self_inner instead of self.
  • Fixed in 6cf036e7b, which removes all eight. Neither production edit changes behavior:
    • _display_name is pure, so the dropped name = _display_name(None, entry) line had no effect.
    • Nothing reads WORKFLOW_RESULTS_SELECTOR_LATEST through this module. It's still defined and used in functions_orchestration_registry.py.
  • The whole pattern, not just the flagged lines. At 6cf036e7b I ran pyflakes over all 23 changed .py files and compared the output with base. The only new hits left were:
    • the shared fixture imports (modules, wf), which the existing suites also import into test modules
    • the reads test's refresh, which defaults to None and is redefined only in its shared_after_refresh scenario
  • At 6cf036e7b: CodeQL passed with "No new alerts in code changed by this pull request" (0 annotations), and every other check passed.
  • Rerun at 3059ac439. I reran the same comparison, with line numbers stripped, over the 25 changed .py files. Every new hit is a pytest fixture imported into a test module and then named as a test or fixture parameter:
    • modules, wf, real_http_harness and results_module in the functional suites
    • editor_assets, editor_browser and connect_options in the new UI test. editor_browser requests connect_options, and 14 existing UI tests import the same three under # noqa: F401.
    • The reads test's refresh hit is gone.
  • At 3059ac439: CodeQL passed with "No new alerts in code changed by this pull request" (0 annotations), and all 11 checks passed.

Follow-ups (out of scope)

  • Step-down excerpt budgets in compose.
  • A run link from the results notes and RunView.
  • Reading structured (v3) results and saved Analyze runs in plans (6b).
  • Group workflows.
  • Relaxing the orchestration history exclusion.
  • A Latest Feature card.

Paul Lizer (paullizer) and others added 21 commits October 1, 2026 15:39
Registers the workflow_results capability (dormant unless Use Workflow
Results In Chat is on), extends the workflow planning context so a
results-only turn gets the catalog, time zone and request local time, and
captures the off golden on unmodified 86f3f36. VERSION 0.261.217.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…lizer-workflow-results-orchestration-capabilit

# Conflicts:
#	application/single_app/config.py
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adds the results instructions, repair hint, results projection, server-only
notes pop and the degrade/drop branch (including workflow_planning for the
degraded _normalize when results remains).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Deliverables facts and a recipe appear only while enable_chat_workflow_results
is on; the adapter wrapper lazily loads the results module.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…sult aliases

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…tings

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…lizer-workflow-results-orchestration-capabilit
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…e and logging

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…and thought leak surfaces

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…nd new-attempt re-reads

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
No result card exists before 6b, so the reason now points to Ask in chat in the run history.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reads test now removes started_at from its missing-started row, keeps it last as Cosmos DB orders an undefined sort property under DESC, and pins that such a row is still read when it is the only usable one, without a newer-run-in-progress mention.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread application/single_app/functions_orchestration_workflow_results.py Fixed
Comment thread application/single_app/functions_orchestration_workflow_results.py Fixed
Comment thread functional_tests/test_orchestration_workflow_results_capability.py Fixed
Comment thread functional_tests/test_orchestration_workflow_results_capability.py Fixed
Comment thread functional_tests/test_orchestration_workflow_results_capability.py Fixed
Comment thread functional_tests/test_orchestration_workflow_results_capability.py Fixed
Comment thread functional_tests/test_orchestration_workflow_results_capability.py Fixed
Comment thread functional_tests/test_orchestration_workflow_results_reads.py Fixed
Paul Lizer (paullizer) and others added 7 commits October 2, 2026 04:48
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The closed reason was read and written back unchanged. Readers already fall back through WORKFLOW_RESULTS_REASON_TEXT.get(...), so behavior is unchanged.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
… R36 R38 R39

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…4 S05

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…Review view

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paullizer
Paul Lizer (paullizer) marked this pull request as ready for review October 2, 2026 17:29
@paullizer
Paul Lizer (paullizer) merged commit 8a802f1 into microsoft:paullizer-react-v2-ui Oct 2, 2026
11 checks passed
Paul Lizer (paullizer) added a commit to paullizer/simplechat that referenced this pull request Oct 2, 2026
Step 9 (microsoft#1607, 0.261.217) merged first, so this branch merges V2 second with
rerere disabled, as agreed:
- config.py keeps VERSION 0.261.218.
- release_notes.md keeps the 0.261.218 section on top, followed by the whole
  0.261.217 section.
- docs/_data/app_surface.yml was regenerated after the merge with
  scripts/build_docs_inventory.py and has no diff.
- route_backend_orchestration.py, docs/admin/workflow.md, docs/admin/orchestration.md
  and CHAT_WORKFLOW_RESULTS_FOLLOW_UP.md merged cleanly; all 6b-1 doc lines remain.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Paul Lizer (paullizer) added a commit that referenced this pull request Oct 2, 2026
…call-agent

Resolves the conflicts with #1607. Both branches started from 0.261.216 and used 0.261.217. VERSION stays 0.261.221, and the release notes combine both 0.261.217 entries under one heading.
Paul Lizer (paullizer) added a commit that referenced this pull request Oct 2, 2026
…eact-v2-inline-media

Brings in #1608 and the base (#1607). VERSION stays 0.261.224. The group upload test keeps #1608's version of the shared CodeQL autofix.
Paul Lizer (paullizer) added a commit to paullizer/simplechat that referenced this pull request Oct 3, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Paul Lizer (paullizer) added a commit that referenced this pull request Oct 3, 2026
…lows-capability

Update roadmap status: #1607 and 6b-1 done, 6b-2 and 7a started
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants