Skip to content

Fix orchestrated SharePoint and OneDrive actions that never read files - #1712

Merged
Paul Lizer (paullizer) merged 6 commits into
paullizer-react-v2-uifrom
paullizer-investigate-orchestrator-m365-sharepoint
Oct 8, 2026
Merged

Paul Lizer (paullizer) merged 6 commits into
paullizer-react-v2-uifrom
paullizer-investigate-orchestrator-m365-sharepoint

Conversation

@paullizer

@paullizer Paul Lizer (paullizer) commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

With Orchestrate on, a plan step that used the Microsoft 365 SharePoint action never searched SharePoint. Asked the same question in chat, the m365 agent found the document. In App Insights for the reported run, the step's search_files call was refused with model_context_unavailable (sc_resource=spo) before any Microsoft Graph request, and the step still reported Completed.

  • Root cause: SharePoint and OneDrive file functions limit what they return to the model's remaining context. They read that limit from a model token budget, which chat binds through the selected agent's continuation journal. An orchestration "Use an action" step calls its model without an agent, so nothing bound a budget. The 0.261.238 fix gave these steps a Microsoft 365 execution context but no model budget. Email and Calendar don't use a budget, which is why those steps worked.
  • Fix: a SharePoint or OneDrive action step now builds its model's ModelTokenBudget from the same endpoint and model resolution as its chat service, mirroring build_agent_model_budget. An auto function-invocation filter binds that budget around each function call. The same filter offers the step's own model to retained-file analysis, for that request only. analyze_file therefore completes after the user approves deeper analysis and retries. Before, it failed every retry with m365_unavailable.
  • Clear failures instead of "nothing found": model-limit and conversation-evidence refusals now stop the step with the new m365_model_limits_required and m365_evidence_unavailable failures. Their messages say what to do next. A model without verified token limits is caught before the step's model or Graph is called. Bounded-coverage outcomes, such as a full context window, file limits, nothing found and throttling, are still reported as findings.
  • User impact: orchestrated answers can use SharePoint and OneDrive documents again. No UI change was needed: V2 already shows the server's message for any failure code.

Worth a careful look:

  • The step sends no max-output setting, so the budget sets no request limit and reserves the model's documented output limit. For gpt-5.6-terra that is a 1,050,000-token context and 128,000-token output.
  • get_m365_analysis_agent() is replaced by get_m365_analysis_model(). It returns either the chat agent's model, through the journal, or the step-bound model. Either one must match the request, data user and conversation. The step binding is a ContextVar, set and reset around each function call.
  • _build_action_model() keeps its signature. Its model resolution and capture-configuration code moved into separate helpers, so file steps resolve the endpoint once.
  • The branch merges paullizer-react-v2-ui (most recently at 0.261.299), with conflicts only in config.py and the release notes each time, and renumbers this fix to 0.261.300.

Linked issue

N/A. No issue was filed; the root cause was confirmed from App Insights telemetry, described above.

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only. It is now 0.261.300, after merging the base at 0.261.299.
  • deployers/version.txt bumped, or not needed because deployers/ was not changed

Testing / validation

I ran the Python tests in a fresh virtual environment built from application/single_app/requirements.txt plus pytest.

  • python -m pytest functional_tests/test_orchestration_m365_file_actions.py -q: passed, 7 tests (new). These run the real action runner, step scope, Microsoft 365 runtime, approval service, SharePoint plugin and file-runtime wiring. Graph, storage and the model are doubled, and network access is blocked. Each of four deliberate breaks makes at least one test fail: removing the step filter (the shipped bug), the analysis-model binding, the evidence refusal codes, or the up-front limits check.
  • Before the base merge: python -m pytest functional_tests/test_orchestration_m365_file_actions.py functional_tests/test_orchestration_m365_actions.py functional_tests/test_orchestration_action_runtime.py functional_tests/test_m365_runtime_adapters.py functional_tests/test_m365_file_analysis.py functional_tests/test_m365_agent_continuation.py functional_tests/test_m365_file_runtime.py functional_tests/test_m365_provider_core.py functional_tests/test_m365_catalog_budget_integration.py functional_tests/test_orchestration_external_metadata.py functional_tests/test_orchestration_external_pre_effect.py functional_tests/test_orchestration_dependency_runtime.py functional_tests/test_orchestration_source_authority_runtime.py functional_tests/test_orchestration_workflow_run_adapter.py functional_tests/test_pytest_return_false_guard.py -q: passed, 768 tests.
  • Before the base merge: python -m pytest functional_tests/test_m365_agent_streaming.py functional_tests/test_m365_chat_action_cards.py functional_tests/test_orchestration_catalog_auto_execution.py functional_tests/test_orchestration_external_configuration_capture.py functional_tests/test_orchestration_external_preflight_adapter.py functional_tests/test_orchestration_visual_outputs.py functional_tests/test_workflow_m365_rebase_integration.py -q: passed, 409 tests.
  • After the latest base merge (at 0.261.299) and renumber: python -m pytest functional_tests/test_orchestration_m365_file_actions.py functional_tests/test_orchestration_m365_actions.py functional_tests/test_orchestration_action_runtime.py functional_tests/test_m365_runtime_adapters.py functional_tests/test_m365_file_analysis.py functional_tests/test_m365_agent_continuation.py functional_tests/test_m365_file_runtime.py functional_tests/test_m365_provider_core.py functional_tests/test_m365_catalog_budget_integration.py functional_tests/test_docs_site_quality.py functional_tests/test_docs_app_surface_coverage.py -q: passed, 288 tests.
  • python scripts/check_broken_access_control.py --full-file application/single_app/functions_orchestration_actions.py application/single_app/functions_orchestration_m365.py application/single_app/functions_m365_agent_continuation.py application/single_app/functions_m365_analysis_runtime.py application/single_app/functions_orchestration_schema.py: passed for 5 files.
  • python scripts/check_xss_sinks.py on the same 5 files: passed for 5 files.
  • python scripts/build_docs_inventory.py: no change to the inventory.
  • Not run: ui_tests/test_v2_orchestration_m365_recovery.py, which needs a V2 build and Chromium. V2 is unchanged, and that test supplies its own failure payloads.
  • Not yet validated in a deployed app. After deploying, ask a SharePoint question with Orchestrate on and confirm the step returns document excerpts.

Documentation

  • Release notes updated, or not needed
  • Feature documentation updated, or not needed. A feature doc isn't needed for a bug fix; I updated the admin guide's "Microsoft 365 actions in plans" section and the [ORCHESTRATION_M365] row in the logging-tags reference.
  • Fix documentation updated, or not needed (docs/explanation/fixes/ORCHESTRATION_M365_FILE_ACTION_EVIDENCE_FIX.md)

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security()) (no route changes)
  • Settings sent to non-admin frontends use sanitize_settings_for_user() (no settings payload changes)
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS (no frontend changes)
  • No secrets, keys, connection strings, or local-only artifacts are included

The model budget holds no secrets. The step-bound analysis model serves only its own request, data user and conversation, and adds no data or model access beyond what the step already uses.

Paul Lizer (paullizer) and others added 4 commits October 7, 2026 22:28
An orchestration "Use an action" step calls its model without a chat agent,
so SharePoint and OneDrive file functions had no model token budget and
refused every call with model_context_unavailable before reaching Microsoft
Graph. The step still completed, and the answer said the documents couldn't
be searched. Production telemetry showed the refusal for search_files on spo
at 2026-10-07 21:40:39Z; the m365 agent found the same file in chat.

- Bind the step model's ModelTokenBudget around each file function call, and
  offer the step's own model to retained-file analysis, so analyze_file
  completes after the user approves deeper analysis and retries.
- Check the model's token limits before the step's model or Graph is called.
- Stop the step with m365_model_limits_required or m365_evidence_unavailable
  instead of reporting application failures as findings.
- Version 0.261.294, fix doc, admin and logging-tag docs, release notes, and
  functional tests for the real runner, step scope and file runtime.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A retry runs the plan's approved step model, and an Auto plan whose model
limits changed is refused with "Review a new plan". So after an admin sets
the model's token limits in Model Endpoints, the user asks again.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…lizer-investigate-orchestrator-m365-sharepoint

# Conflicts:
#	application/single_app/config.py
#	docs/explanation/release_notes.md
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread functional_tests/test_orchestration_m365_file_actions.py Fixed
Paul Lizer (paullizer) and others added 2 commits October 8, 2026 06:53
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve the config.py and release notes conflicts, and renumber this fix
from 0.261.297 to 0.261.300: the base now uses 0.261.297 through 0.261.299
for other changes. Only this fix's references were renumbered.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paullizer
Paul Lizer (paullizer) merged commit 59ca957 into paullizer-react-v2-ui Oct 8, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants