Fix orchestration runs failing on settings saves and unexplained file failures

[Paul Lizer (paullizer)]

Summary

Fixes two V2 orchestration bugs reported from a production deployment and traced through its App Service console logs and Application Insights. A plan waiting on a long tabular analysis replied "The request could not be completed. Saved step inputs changed." while the analysis was still running, and the plan panel kept spinning on Waiting for results with no retry. Separately, a Word file built from action results failed with only "This file could not be created." and offered no retry.

• Background settings saves no longer fail waiting runs. Saved run checkpoints were bound to a fingerprint of the whole settings document, including Cosmos metadata (_etag, _ts, _settings_revision) and runtime state that background tasks write. In the failing run, the Cosmos throughput autoscaler saved settings under 4 seconds before the scheduler continued the waiting run, so the continuation raised recovery_changed; the analysis itself finished about 2 seconds later. All three recovery_changed failures in the previous 14 days came within 30 minutes of a settings save, two of them within 6 seconds. The fingerprint now covers configuration only (new functions_settings_runtime_state.py registry). An admin configuration change still invalidates saved work.
• A run that ends while a step is waiting says so, and can be retried. Run-level failures never updated execution_steps, so the panel spun forever and recovery refused retry with result_not_ready. Recovery now ignores waiting steps of a terminal run, and the V2 run view shows them as Not finished with their deliverables Not delivered. A retry also no longer restores the previous attempt's pending wait from a reused checkpoint snapshot, which made retries fail with recovery_changed too.
• Files built from web, research, agent or action results publish with the user's sign-in. Since 0.261.209, publication rechecked a file's sources through a fresh service built on the render worker thread, which has no request context and so no roles. The external-lineage recheck was refused (external_identity_session_unavailable) and surfaced as a generic output_access_denied. Publication now reuses the rendering service running the attempt through a ContextVar scope (render_attempt_scope). A file that still can't be checked without a session fails with output_sign_in_required and says so, and other failed files now carry a specific reason instead of the generic message.
• Failed files can be retried from the plan. Per-file retries render in the background scheduler, which has no session, so they can't fix a sign-in failure. When a file failed for good and no file is still being prepared, the recovery notice now offers Retry from failed step, which renders the plan's files again without repeating completed steps.

Worth a careful look

• The runtime-state registry lists known storage and background-writer keys. Any key it doesn't list stays in the fingerprint, so an unknown key still fails closed. An AST test fails if a known background settings writer starts saving a key the registry doesn't cover.
• The attempt scope is a ContextVar, so it never crosses threads, and _bound_service only uses it when the user and conversation match the binding's producer.
• Runs saved before 0.261.303 carry whole-document bindings, so retrying one after the upgrade is refused; the user asks again. This is documented in the fix doc.
• Not addressed here (follow-ups): a native analysis job keeps running after its run fails terminally (its result is unused); runtime monitor state still lives in the settings document; and the background output reconcile can mark a completed external-lineage file unavailable when it rechecks without a session (pre-existing).

Linked issue

N/A. No issue was filed; both bugs were reported from production and investigated from telemetry.

Release Notes & Latest Features

• New Feature
• Bug Fix
• UI Enhancement
• Breaking Change
• Internal only

Is this visible to end users?

• Yes
• No

Is this admin-facing (Admin Settings, governance, deployment, config)?

• Yes
• No

Should this become a Latest Feature card?

• Yes
• No
• Already added

Screenshot needed for the card?

• Yes
• No
• Attached

Version bump

• application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only (0.261.301 -> 0.261.303; the base released 0.261.302 while this PR was open)
• deployers/version.txt bumped, or not needed because deployers/ was not changed

Testing / validation

Run from the repo root with PYTHONPATH=application/single_app;functional_tests and PYTHONIOENCODING=utf-8, in a venv with pyOpenSSL 26.4.0 (this machine's global Python pairs cryptography 50 with pyOpenSSL 25.3.0, which breaks import OpenSSL through semantic_kernel/aiortc). Unless noted, results are on this branch's HEAD after merging the latest paullizer-react-v2-ui (merge commit 53c6f65).

• python -m pytest -q functional_tests/test_orchestration_settings_runtime_state_binding.py functional_tests/test_orchestration_terminal_wait_retry.py functional_tests/test_orchestration_file_sign_in_publication.py functional_tests/test_orchestration_settings_upgrade.py functional_tests/test_orchestration_output_lifecycle.py functional_tests/test_docs_app_surface_coverage.py functional_tests/test_session_store_merge.py functional_tests/test_m365_connections.py functional_tests/test_m365_routes.py: 367 passed, 1 failed, pre-existing. The new regression tests fail with the fixes stashed. The failure, test_m365_routes.py::test_real_notification_helper_only_targets_subject_and_deduplicates, also fails on a clean checkout of the base branch.
• python functional_tests/test_docs_site_quality.py: passed (6/6 checks).
• npm --prefix application/v2_ui run typecheck: passed.
• npm --prefix application/v2_ui run build -- --outDir ../../ui_tests/artifacts/orchestration-plan-editor: passed.
• python -m pytest -q ui_tests/test_v2_orchestration_recovery.py ui_tests/test_v2_drawer_generated_files.py -m ui: passed (43 passed in Chromium, including 3 new tests).
• Before the merge: python -m pytest -q functional_tests/test_xss_guardrails_checker.py functional_tests/test_broken_access_control_guardrails_checker.py: 15 passed, 1 failed, pre-existing. test_checker_assets_and_version_are_wired_into_repo expects actions/upload-artifact@v4 in .github/workflows/broken-access-control-full-scan.yml, which the base branch now pins to a SHA of actions/upload-artifact v7.0.1. This PR doesn't touch .github/.
• Full orchestration suite, before the merge (175 functional_tests/test_orchestration*.py files with --continue-on-collection-errors): 6932 passed, 76 failed, none caused by this change. 61 failures are in four files that fail identically on the base branch: test_orchestration_document_merge_capability.py (4), test_orchestration_plan_revision_references.py (24), test_orchestration_plan_revision_routes.py (29) and test_orchestration_services.py (4). The other 15 are in test_orchestration_failure_telemetry.py and test_orchestration_workflow_propose_capability.py, which pass when run on their own. Not re-run after the merge; the merged base changes don't touch orchestration code.
• Not run: a manual end-to-end check against a deployed app; the fix hasn't been deployed yet.

Documentation

• Release notes updated, or not needed (new v0.261.303 section with two bug fixes, above the base's v0.261.302 section)
• Feature documentation updated, or not needed (ORCHESTRATION_CHECKPOINT_RECOVERY.md, ORCHESTRATION_DELIVERABLES.md, plus docs/reference/chat-controls.md, two orchestration guides and troubleshooting rows in docs/admin/orchestration.md)
• Fix documentation updated, or not needed (ORCHESTRATION_SETTINGS_WRITE_RECOVERY_FIX.md, ORCHESTRATION_FILE_SIGN_IN_PUBLICATION_FIX.md)

Security checklist

• New Flask routes include @swagger_route(security=get_auth_security()) (no new routes)
• Settings sent to non-admin frontends use sanitize_settings_for_user() (nothing new is sent to the browser)
• Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS
• No secrets, keys, connection strings, or local-only artifacts are included