Track completed full and partial backups created by Data Management jobs.
-
diff --git a/docs/explanation/fixes/BACKUP_INVENTORY_GROUP_BY_AGGREGATE_FIX.md b/docs/explanation/fixes/BACKUP_INVENTORY_GROUP_BY_AGGREGATE_FIX.md
new file mode 100644
index 000000000..70c21c09e
--- /dev/null
+++ b/docs/explanation/fixes/BACKUP_INVENTORY_GROUP_BY_AGGREGATE_FIX.md
@@ -0,0 +1,147 @@
+# Backup Inventory GROUP BY Aggregate Fix
+
+**Fixed in version: 0.260.002**
+
+## Issue Description
+
+The Backup Inventory panel in Admin Settings → Data Management always failed to load. The
+UI reported `Backup history is temporarily unavailable. Please try again.` and every summary
+tile rendered zeros:
+
+- Available backups: `0`
+- Full backups: `0`
+- Partial backups: `0`
+
+This happened even when backups had completed successfully, and even when the job list itself
+loaded correctly. The `GET /api/admin/data-management/backups` route returned `503`.
+
+## Root Cause
+
+`_get_data_management_backup_global_summary()` computed its counts with a single grouped
+aggregate:
+
+```sql
+SELECT c.backup_type, c.status, COUNT(1) AS count FROM c
+WHERE c.type = @type AND c.operation = @operation
+GROUP BY c.backup_type, c.status
+```
+
+The `azure-cosmos` Python client does not advertise support for `GroupBy` combined with a
+**non-VALUE aggregate** (`COUNT(1) AS count`). During query plan negotiation the service
+rejects the request:
+
+```
+(BadRequest) {"code":"BadRequest","message":"Query contains the following features,
+which the calling client does not support:\nNone GroupBy NonValueAggregate ..."}
+```
+
+The `BadRequest` propagated out of `_query_data_management_history_items`, was classified as
+a provider failure, and surfaced to the admin as a generic `503`.
+
+This was never a throttling, indexing, or capacity problem. Cosmos Maintenance consistently
+reported the composite index as **Aligned / 0 missing / 7 containers checked**, and the
+`400` entries with `requestCharge = 0` and sub-millisecond duration visible in
+`AzureDiagnostics` are the normal, benign cross-partition query-plan negotiation.
+
+Because the aggregate was part of the summary from the moment the feature shipped, Backup
+Inventory had never worked in any deployment.
+
+### Why tests did not catch it
+
+`FakeHistoryContainer` in `functional_tests/test_data_management_history_pagination.py`
+implemented `GROUP BY c.backup_type, c.status` in Python and returned grouped rows. The test
+double supported a query shape the real client cannot serve, so the suite passed against a
+query that always failed in production.
+
+## Technical Details
+
+### Files Modified
+
+| File | Change |
+|------|--------|
+| `application/single_app/functions_data_management.py` | Replaced the grouped aggregate with bounded `SELECT VALUE COUNT(1)` queries; added `_count_data_management_backups()` |
+| `application/single_app/templates/admin_settings.html` | Added retention cleanup tooltip and expandable help panel |
+| `functional_tests/test_data_management_history_pagination.py` | Fake now rejects `GROUP BY` / non-VALUE aggregates like the real client; added coverage |
+| `application/single_app/config.py` | Version bump |
+
+### Code Changes
+
+A new helper issues a single supported scalar aggregate:
+
+```python
+def _count_data_management_backups(extra_clauses=None, extra_parameters=None):
+ """Count backups with a VALUE aggregate; the Python client cannot serve GROUP BY here."""
+ clauses = ["c.type = @type", "c.operation = @operation"]
+ parameters = [...]
+ clauses.extend(extra_clauses or [])
+ parameters.extend(extra_parameters or [])
+ rows = _query_data_management_history_items(
+ query=f"SELECT VALUE COUNT(1) FROM c WHERE {' AND '.join(clauses)}",
+ parameters=parameters,
+ )
+ return _safe_int(rows[0], default=0, minimum=0) if rows else 0
+```
+
+The summary now issues six bounded counts — `total`, `available`, `running`, `failed`,
+`full`, and `partial`. `SELECT VALUE COUNT(1)` is fully supported by the Python client and is
+already used elsewhere in this module. The two `SELECT TOP 1 *` queries that resolve
+`latest_full` and `latest_partial` were already valid and are unchanged.
+
+This deliberately avoids the alternative of projecting every backup row and aggregating on the
+client, which would grow unbounded as backup history accumulates. Each count is served by
+Cosmos and returns a single scalar.
+
+### Retention Cleanup Guidance
+
+Admins had no way to tell what **Run Retention Cleanup** did, and a run that deleted nothing
+looked like a failure. The button now carries a hover tooltip, and an `(i)` toggle expands an
+inline panel explaining that cleanup:
+
+- permanently deletes backups older than the configured retention period, plus their stored artifacts;
+- only considers backups in a finished state;
+- protects the most recent successful full backup when **Keep latest full backup** is enabled;
+- deletes at most 25 backups per run;
+- also runs automatically on the configured schedule.
+
+The panel calls out that `found no expired backups to delete` means every backup is still
+inside the retention window, which is expected rather than an error.
+
+## Validation
+
+### Test Results
+
+`functional_tests/test_data_management_history_pagination.py` — **14 passed**.
+
+New and updated coverage:
+
+- `test_backup_summary_avoids_unsupported_group_by_aggregates` asserts no emitted history query
+ contains `GROUP BY` or `COUNT(1) AS`, and that exactly six `SELECT VALUE COUNT(1)` queries are issued.
+- `test_retention_cleanup_button_explains_what_it_does` asserts the tooltip, the `(i)` toggle,
+ the collapse target, and the explanatory copy are present.
+- `FakeHistoryContainer` now raises the real `BadRequest ... GroupBy NonValueAggregate` error
+ instead of emulating `GROUP BY`.
+
+### Regression Probe
+
+The fix was neutralized by restoring a grouped aggregate. Two tests failed with the exact
+production error:
+
+```
+RuntimeError: (BadRequest) Query contains the following features, which the calling client
+does not support: GroupBy NonValueAggregate.
+```
+
+- `test_backup_summary_is_global_and_page_independent`
+- `test_backup_summary_avoids_unsupported_group_by_aggregates`
+
+The fix was then restored and all 14 tests passed again, confirming the tests fail for the
+right reason.
+
+### Before / After
+
+| Behavior | Before | After |
+|----------|--------|-------|
+| `GET /api/admin/data-management/backups` | `503` | `200` |
+| Backup Inventory tiles | Always `0` | Actual counts |
+| Backup Inventory table | Error banner | Backup rows |
+| Retention cleanup purpose | Undocumented in UI | Tooltip + expandable help |
diff --git a/docs/explanation/release_notes.md b/docs/explanation/release_notes.md
index f8db95546..332b082bf 100644
--- a/docs/explanation/release_notes.md
+++ b/docs/explanation/release_notes.md
@@ -2,6 +2,25 @@
For feature-focused and fix-focused drill-downs by version, see [Features by Version](/explanation/features/) and [Fixes by Version](/explanation/fixes/).
+### **(v0.260.002)**
+
+#### Bug Fixes
+
+* **Backup Inventory No Longer Fails To Load**
+ * Fixed the Backup Inventory panel in Admin Settings → Data Management always returning `503` and showing `0` for available, full, and partial backups.
+ * The global summary used a Cosmos `GROUP BY` with a non-VALUE aggregate (`COUNT(1) AS count`), a combination the `azure-cosmos` Python client does not support. Cosmos rejected the query during plan negotiation with `BadRequest ... GroupBy NonValueAggregate`.
+ * Counts are now computed with bounded, fully supported `SELECT VALUE COUNT(1)` queries, so the panel renders real numbers without loading backup history into memory.
+ * This was not a throttling or indexing problem; the composite index was already aligned. Backup Inventory had been broken since the summary shipped.
+ * (Ref: `functions_data_management.py`, `_get_data_management_backup_global_summary`, `_count_data_management_backups`, `/api/admin/data-management/backups`)
+
+#### User Interface Enhancements
+
+* **Run Retention Cleanup Now Explains Itself**
+ * Added a hover tooltip and an `(i)` toggle that expands inline guidance next to the **Run Retention Cleanup** button.
+ * Documents that cleanup permanently deletes backups past the retention period along with their artifacts, skips jobs that are still running, honors **Keep latest full backup**, and deletes at most 25 backups per run.
+ * Clarifies that "found no expired backups to delete" means every backup is still inside the retention window, which is expected rather than a failure.
+ * (Ref: `admin_settings.html`, backup retention cleanup, Data Management)
+
### **(v0.260.001)**
v0.260.001 consolidates all work released after v0.250.001 into one major release note, spanning 117 incremental patch builds. This rollup highlights the major feature, UI, reliability, security, and operations themes while preserving the full per-build history in the Detailed Change Log at the end of this section.
diff --git a/functional_tests/test_data_management_history_pagination.py b/functional_tests/test_data_management_history_pagination.py
index e7d8813b2..9233477e4 100644
--- a/functional_tests/test_data_management_history_pagination.py
+++ b/functional_tests/test_data_management_history_pagination.py
@@ -65,19 +65,16 @@ def query_items(
for item in self.documents
if self._matches(item, query, parameter_map)
]
- if "GROUP BY c.backup_type, c.status" in query:
- grouped = {}
- for item in matching:
- key = (item.get("backup_type"), item.get("status"))
- grouped[key] = grouped.get(key, 0) + 1
- return [
- {
- "backup_type": backup_type,
- "status": status,
- "count": count,
- }
- for (backup_type, status), count in grouped.items()
- ]
+ if "GROUP BY" in query.upper() or "COUNT(1) AS" in query.upper():
+ # The azure-cosmos Python client does not advertise GroupBy or
+ # NonValueAggregate support, so Cosmos rejects these during query
+ # plan negotiation. Mirror that here so the fake cannot mask it.
+ raise RuntimeError(
+ "(BadRequest) Query contains the following features, which the "
+ "calling client does not support: GroupBy NonValueAggregate."
+ )
+ if "SELECT VALUE COUNT(1)" in query.upper():
+ return [len(matching)]
matching.sort(
key=lambda item: (item.get("created_at", ""), item.get("id", "")),
@@ -99,6 +96,10 @@ def _matches(item, query, parameters):
return False
if "@status" in parameters and item.get("status") != parameters["@status"]:
return False
+ if "@running" in parameters and item.get("status") != parameters["@running"]:
+ return False
+ if "@failed" in parameters and item.get("status") != parameters["@failed"]:
+ return False
if (
"@completed" in parameters
and "@completed_with_warnings" in parameters
@@ -499,6 +500,29 @@ def test_backup_summary_is_global_and_page_independent(monkeypatch):
assert result["summary"]["latest_partial"]["id"] == "partial-new"
+def test_backup_summary_avoids_unsupported_group_by_aggregates(monkeypatch):
+ """Keep summary counts on VALUE aggregates the Cosmos Python client can serve."""
+ jobs = [
+ build_job("full-new", "2026-07-30T12:00:00+00:00"),
+ build_job("partial-new", "2026-07-30T11:00:00+00:00", backup_type="partial"),
+ ]
+ container = FakeHistoryContainer(jobs)
+ module = load_data_management_module(monkeypatch, container)
+
+ module.get_data_management_backup_summary(limit=5)
+
+ emitted = [entry["query"].upper() for entry in container.queries]
+ assert emitted, "Backup summary should emit at least one history query."
+ for query in emitted:
+ assert "GROUP BY" not in query, f"Unsupported GROUP BY in history query: {query}"
+ assert "COUNT(1) AS" not in query, f"Unsupported non-VALUE aggregate: {query}"
+
+ count_queries = [query for query in emitted if "COUNT(1)" in query]
+ assert len(count_queries) == 6, "Summary should use one VALUE count per bucket."
+ for query in count_queries:
+ assert "SELECT VALUE COUNT(1)" in query
+
+
def test_history_provider_index_errors_are_actionable(monkeypatch):
"""Convert missing Cosmos composite-index failures into admin-safe guidance."""
container = FailingHistoryContainer()
@@ -795,3 +819,23 @@ def test_deployers_apply_the_data_management_history_index():
assert compare_simplechat_versions(deployer_version, "1.0.24") >= 0, (
f"Deployer version must include the history index change, got {deployer_version}"
)
+
+
+def test_retention_cleanup_button_explains_what_it_does():
+ """Give admins hover and expandable guidance before deleting expired backups."""
+ template = (
+ APP_ROOT / "templates" / "admin_settings.html"
+ ).read_text(encoding="utf-8")
+
+ cleanup_button_start = template.index("data-management-run-retention-cleanup-btn")
+ cleanup_button = template[cleanup_button_start:cleanup_button_start + 600]
+ assert 'data-bs-toggle="tooltip"' in cleanup_button
+ assert "retention period" in cleanup_button
+
+ assert 'id="data-management-retention-cleanup-help"' in template
+ assert 'data-bs-target="#data-management-retention-cleanup-help"' in template
+ assert 'aria-label="What does Run Retention Cleanup do?"' in template
+ assert "bi bi-info-circle" in template
+ assert "Keep latest full backup" in template
+ assert "at most 25 backups" in template
+ assert "found no expired backups to delete" in template