Skip to content

Fix orchestrated web search and deep research in V2 chat - #1582

Merged
Paul Lizer (paullizer) merged 5 commits into
microsoft:paullizer-react-v2-uifrom
paullizer:paullizer-orchestration-retained-result-failure
Sep 30, 2026
Merged

Paul Lizer (paullizer) merged 5 commits into
microsoft:paullizer-react-v2-uifrom
paullizer:paullizer-orchestration-retained-result-failure

Conversation

@paullizer

Copy link
Copy Markdown
Contributor

Summary

In V2 chat, every orchestrated web search, linked-page read and deep research step failed with "A required retained result is unavailable or changed." This PR makes them work without any new permission, deployer change or setting.

  • Root causes. Orchestration looked up each user's app roles in Microsoft Graph on every external-source access. That needs the Directory.Read.All application permission, which deployments don't grant; classic chat never makes this call. The test deployment's Application Insights showed the Graph GET /users/{id} call returning 403 on every run. Separately, deep research refused to start whenever query or link planning was on, which is the default.
  • Session roles, like classic chat. Orchestration now trusts the app roles in the signed-in session. They are captured on the request thread when the run is prepared and carried to the worker thread that executes the plan. Conversation ownership, user settings and Control Center access restrictions are still rechecked on every access. Nothing calls Graph and roles are never persisted.
  • Deep research planners admitted. The research profile gate is removed. Query and link planners run within the existing Knowledge limits, each planner request still follows a fresh attestation of the planner model, and a planner setting change during a run stops the step.
  • Clear message when no one is signed in. If the background scheduler finishes a run (for example after an app restart, or while a large spreadsheet computation is pending), there is no session. External-source steps then fail closed with a new external_session_required message asking the user to send the request again. Step, saved-wait, finalization and content-preparation failure events log the refusal code as sc_authority_reason.

Reviewer notes

  • The first two commits (11a6dcb1b, 500639ab3) were an earlier approach: grant Directory.Read.All, add it to the deployers, and warn admins when it is missing. 8a804e45a reverts them and adds the fix above. Please review the net diff; a squash merge is recommended. The net diff does not touch deployers/ or admin templates.
  • Trade-off: a role removed in Entra ID takes effect at the user's next session refresh, the same as classic chat.
  • A Directory.Read.All grant made only for orchestration can be revoked.

Linked issue

Refs #1509

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only
  • deployers/version.txt bumped, or not needed because deployers/ was not changed

This branch uses 0.261.205. The target branch is now at 0.261.208, so config.py and docs/explanation/release_notes.md conflict, and the version and its release notes section need renumbering (for example to 0.261.209) before merge. All code and other doc files merge cleanly.

Testing / validation

  • python -m pytest on the nine changed orchestration test files (external_identity, external_bootstrap, external_error_markers, external_preflight_adapter, research_capture, research_pre_effect, dependency_runtime, source_authority_runtime, image_references): 674 passed, 92 subtests passed.
  • Mutation check: 31 deliberate code breaks are each caught by at least one test. They include trusting a missing session, skipping the role check, reading exception text, and dropping a planner switch from the compared settings.
  • Full orchestration suite: 134 of 141 files pass. Five failures also fail on the base commit; two fail only under parallel load and pass when run alone.
  • python .\functional_tests\test_docs_app_surface_coverage.py (7/7) and python .\functional_tests\test_docs_site_quality.py (6/6).
  • Not yet verified on a live deployment with this change.

Documentation

  • Release notes updated, or not needed
  • Feature documentation updated, or not needed
  • Fix documentation updated, or not needed

New fix doc: docs/explanation/fixes/ORCHESTRATION_SESSION_IDENTITY_FIX.md. Updated: ORCHESTRATION_EXTERNAL_SOURCE_ACCESS.md, ORCHESTRATION_CHECKPOINT_RECOVERY.md, ORCHESTRATION_GATHER_REASON_RENDER.md, ANALYZE_RESULTS.md, docs/admin/orchestration.md and docs/reference/logging-tags.md.

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security()) (N/A: no new routes)
  • Settings sent to non-admin frontends use sanitize_settings_for_user() (N/A: no settings sent to the browser)
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS (N/A: no browser JavaScript changes)
  • No secrets, keys, connection strings, or local-only artifacts are included

Orchestrated web search, linked-page, deep research, agent and action steps
reread the user's app roles from Microsoft Graph with the application's own
identity. Without the Directory.Read.All application permission Graph answers
403, which surfaced as "A required retained result is unavailable or changed."

- Report application-side directory refusals with their own reasons and the
  directory_access_unavailable step code, telling users which source could not
  be verified and to contact their administrator.
- Warn administrators in Admin Settings (Chat Orchestration and Web Search) and
  with a once-per-day admin notification, with an admin-only check route.
- Request and consent to Directory.Read.All in the Bicep/azd, Azure CLI and
  Terraform deployers (deployer version 1.0.33).
- Add tests, fix documentation, admin docs and release notes (0.261.204).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Render the real Chat Orchestration and Web Search panes and run the
directory access module in Chromium against a simulated check endpoint.
Cover each warning state, Check again (in progress, success and a failed
retry), a failed first check, server values shown as text, the See how to
fix it card link, and the warnings at desktop and 390px widths.

21 deliberate breakages of the module and the two templates were each
caught. Document the UI test in the fix doc and release notes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Orchestrated web search, linked-page and deep research steps failed with
"A required retained result is unavailable or changed." Two causes:

- Orchestration read each user's app roles from Microsoft Graph on every
  call, which needs Directory.Read.All. Classic chat never does this.
- Deep research refused to start whenever query or link planning was on,
  which is the default.

Changes:

- Revert the unreleased Directory.Read.All grant, deployer, admin warning
  and notification work (11a6dcb, 500639a).
- Replace the Graph reader with SessionExternalIdentityReader, which trusts
  the signed-in session's roles like classic chat and still rechecks
  conversation ownership, settings and access restrictions on every call.
- Remove the research planner profile gate; planners run after attestation
  and a planner setting change during a run stops the step.
- Keep the refusal's stable code as authority_reason and map a missing
  signed-in session to the new external_session_required failure, which
  asks the user to send the request again. Failure events log the code as
  sc_authority_reason.
- Update tests, feature docs, logging tags, admin docs, release notes and
  add ORCHESTRATION_SESSION_IDENTITY_FIX.md. Version 0.261.205.

Refs microsoft#1509

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve the two conflicts with the base branch. The base took 0.261.205
through 0.261.208 for other changes, so this fix moves above them:

- config.py: VERSION 0.261.209 (base was 0.261.208).
- release_notes.md: keep every base section and put this fix's entry on
  top as v0.261.209.

Refs microsoft#1509

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The base branch used 0.261.205 for the workflow flow canvas fix, so this
fix now ships in 0.261.209. Move its module version headers,
implemented-in markers, test version assertions, fix doc, feature docs,
admin troubleshooting row and logging tag note to 0.261.209. The base's
own 0.261.205 references are unchanged.

Refs microsoft#1509

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread application/single_app/functions_source_review.py
Comment thread functional_tests/test_orchestration_research_pre_effect.py
@paullizer
Paul Lizer (paullizer) merged commit b0ed6ea into microsoft:paullizer-react-v2-ui Sep 30, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants