Data Management history 503 is undiagnosable: provider status code and message are discarded #1275

Summary

GET /api/admin/data-management/backups returns 503 and the Backup Inventory panel shows the generic "Data Management history could not be loaded. Please try again later or review application logs."

The underlying provider error is caught but its status code and message are discarded, so the logs cannot identify the cause. Six rounds of investigation across App Service console logs, App Insights, Cosmos Maintenance, and Cosmos diagnostic logs failed to determine why — because the one component that had the full error in hand threw it away.

Impact

• Backup Inventory and Job History panels are unusable when this fires.
• No data loss, but the operator has no actionable path — the message literally says "review application logs" and the logs contain nothing useful.
• Every failure mode collapses to one indistinguishable 503.

Observed behavior

Browser console:

/api/admin/data-management/backups?status=available&scheduled=all&page_size=25
  Failed to load resource: the server responded with a status of 503 (Service Unavailable)

App Service console log — this is the entire diagnostic output:

[DEBUG][ERROR][Log] [DATA_MANAGEMENT] Data Management history could not be loaded. --
 {'history_list': 'backups', 'reason': 'history_provider_unavailable',
  'maintenance_required': False, 'error_type': 'CosmosHttpResponseError'}

error_type is the exception class name only. No status code, no provider message.

What has been ruled out

Remaining candidates are throttling (429) or another non-400 CosmosHttpResponseError. data_management_jobs was observed oscillating 1,000→5,000 RU on container_utilization_above_threshold, and Admin Settings fires a burst of Cosmos-heavy admin calls on load.

Root cause of the diagnosis failure

_data_management_history_unavailable_response in route_backend_data_management.py logs only type(original_error).__name__. _raise_data_management_history_unavailable in functions_data_management.py discards the provider status and message entirely when constructing DataManagementHistoryUnavailableError.

Additionally:

• _is_data_management_history_index_error requires status 400 and the literal text "composite index". Any wording drift routes to the generic message.
• Throttling has no dedicated classification, so a 429 produces "review application logs" instead of "retry shortly".
• _query_data_management_history_items has no retry of its own beyond the SDK default.

Expected behavior

• The provider status code and sanitized message reach operator logs.
• Throttled reads retry briefly and, if still failing, report retryable busy guidance rather than a generic error.
• Index errors are recognized across reasonable provider wording variations.

Proposed fix

1. Carry provider_status_code and provider_message on DataManagementHistoryUnavailableError and log both. Provider detail stays out of the browser payload.
2. Add throttle classification (429/503, "request rate is large") with a distinct retryable message.
3. Bounded retry on the history query for throttles and transient transport errors.
4. Loosen the index classifier to also match ORDER BY + "does not have a corresponding" / "not served".

This does not require knowing the root cause and makes the next occurrence self-diagnosing.

Environment

• SimpleChat 0.250.219
• Azure Cosmos DB, container-level autoscale
• Azure App Service, 3 instances