Skip to content

Fix Ask AI agent creation and add debug-gated diagnostics - #1760

Merged
Paul Lizer (paullizer) merged 4 commits into
paullizer-react-v2-uifrom
paullizer-ask-ai-diagnostics
Oct 13, 2026
Merged

Paul Lizer (paullizer) merged 4 commits into
paullizer-react-v2-uifrom
paullizer-ask-ai-diagnostics

Conversation

@paullizer

Copy link
Copy Markdown
Contributor

Summary

  • Fix the live-reproduced SQL-agent Ask AI failure: the model used unprefixed action fields such as displayName instead of /displayName and supplied blank connection details. Clarify the creation prompt's exact paths and distinguish them from parent-agent set paths; omit unknown connection details or ask for clarification instead of producing invalid placeholders.
  • Add bounded, content-free diagnostics for agent, action, and workflow Ask AI, including per-attempt rejection rules, correction outcomes, provider timing/status, JSON-mode fallback, and hashed submission correlation. New detailed events reach configured Application Insights and console logging only while the existing Enable Debug Logging setting is enabled.
  • Preserve strict validation, atomic proposals, authorization, deadlines, and the single correction retry. Add regressions and troubleshooting documentation, and bump the application to 0.261.329. No deployment, restart, SQL connection, or stored Azure setting change was made.

Linked issue

N/A. The user chose to continue without creating an issue.

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

No new admin setting or UI control is introduced. Release notes remain unchanged because approval to update them was unavailable.

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only
  • deployers/version.txt bumped, or not needed because deployers/ was not changed

Application version: 0.261.328 -> 0.261.329. No deployer files changed.

Testing / validation

  • Backend: 407 passed, 1 known baseline check deselected. Run in a session-isolated Python environment; production dependency manifests are unchanged.

    python -m pytest .\functional_tests\test_editor_assist_create_item_prompt.py .\functional_tests\test_ask_ai_debug_diagnostics.py .\functional_tests\test_editor_assist_core.py .\functional_tests\test_workflow_assist_limits_runtime.py .\functional_tests\test_workflow_assist_scenarios.py .\functional_tests\test_workflow_assist_operations_security.py .\functional_tests\route_tests\test_editor_assist_policy.py .\functional_tests\route_tests\test_workflow_assist_policy.py -q --tb=short -k 'not test_the_v2_bootstrap_reports_the_assistant_flag_from_the_gate'
  • Real-bundle browser coverage: 11 passed. PLAYWRIGHT_SERVICE_URL was empty and PYTHONPATH included ui_tests\fixtures.

    python -m pytest .\ui_tests\test_v2_agent_action_ask_ai.py -q --tb=short
  • npm --prefix .\application\v2_ui run build: passed, with the existing large-chunk warning. Generated assets are not included in this PR.

  • python .\functional_tests\test_docs_app_surface_coverage.py: 7/7 checks passed, committed inventory current.

  • python .\functional_tests\test_docs_site_quality.py: 6/6 checks passed.

  • git -c core.whitespace=blank-at-eol,blank-at-eof,space-before-tab,cr-at-eol --no-pager diff --check: passed.

  • Manual live-model reproduction: read-only Azure configuration identified the configured model; a local production-bundle fixture generated the real 12-field new-agent request with synthetic workspace choices. Both pre-fix runs returned 502. Six post-fix runs succeeded: four unfinished SQL drafts and two useful clarifications. The final two explicitly checked for no invented connection details, credential fields, or read-write SQL setting. No raw model responses or credentials were retained. The original historical response remains unavailable; this is fresh reproduction evidence, not a reconstruction of that response.

  • Known pre-existing failures: the deselected bootstrap AST assertion expects the old unwrapped gate expression rather than the existing measure() wrapper. Earlier unfiltered coverage reported this failure. python -m pytest .\functional_tests\test_logging_tag_standardization.py .\functional_tests\test_log_event_call_contract.py -q --tb=short reported 1 failed, 1 passed: legacy mixed-case tags fail normalization, while the logging call contract passes. Those unrelated tags are unchanged; new diagnostic tags use the normalized format.

Documentation

  • Release notes updated, or not needed
  • Feature documentation updated, or not needed
  • Fix documentation updated, or not needed

Updated the agent/action and workflow Ask AI feature references and logging-tag inventory. Added docs/explanation/fixes/ASK_AI_OPERATION_DIAGNOSTICS_FIX.md with the reproduced contract failure, privacy boundaries, validation evidence, and bounded single-line KQL examples. Release-note approval remains outstanding.

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security())
  • Settings sent to non-admin frontends use sanitize_settings_for_user()
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS
  • No secrets, keys, connection strings, or local-only artifacts are included

No routes, settings-to-browser flows, or browser assets were added. Diagnostic properties use closed categorical allowlists and bounded measurements; prompts, drafts, replies, credentials, raw identity IDs, arbitrary field names, and provider exception text are excluded. Debug-off absence is covered in both logging sinks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment thread application/single_app/functions_editor_assist.py Fixed
Replace the invalidated workflow reader source-only review with real SDK regressions and align the M365 route harness with the new diagnostic logger adapter.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Resolve the config.py-only conflict at version 0.261.330. The merged branch passes all 85 targeted regressions and has zero Cosmos inventory review gaps; record the completed 704-test contract run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paullizer
Paul Lizer (paullizer) merged commit 8f52ea8 into paullizer-react-v2-ui Oct 13, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants