Admin settings can revert across workers after successful saves #1477

Issue

Admin settings can appear to revert after a successful save in a scaled-out, containerized App Service deployment. Reloads can alternate between the old and saved settings without another admin save. Some code paths can also overwrite the persisted settings with an older snapshot.

Steps to Reproduce

1. Use multiple Python workers or app instances sharing settings storage.
2. Load Admin Settings so more than one worker has read the existing settings.
3. Change and save a setting.
4. Reload through different workers. Observe old -> saved -> old values before local version TTLs expire.
5. For the lost-update case, have a stale worker update unrelated metadata (for example, the release-check timestamp). Its full-document upsert can revert the setting in Cosmos.

Expected Behavior

A completed save should be visible to other workers without waiting for local settings cache expiry. Unrelated settings writes, startup initialization, and recovery from Redis failures must not restore older values. Failed or unconfirmed saves must not be reported as successful.

Confirmed Code Causes

• Worker-local settings snapshots reuse a cached shared version for 15 seconds, even with Redis enabled.
• update_settings reads through that cache and upserts the entire document without an ETag precondition.
• Settings reads/migrations and auxiliary metadata writers can persist stale documents.
• Worker/scheduler startup publishes an earlier snapshot to Redis without coordinated versioning.
• Redis outage fallback advances a separate Cosmos version while Redis retains its old payload/version.
• The 302 redirect occurs after synchronous save/cache calls return; adding a delay is not the root-cause fix.

These paths were reproduced offline with independent worker modules and shared fake services. They establish code defects, but do not identify which path occurred on the reported live deployment.

Agreed Behavior and Local Implementation

Implemented locally on branch fix/admin-settings-consistency, version 0.261.025; not committed, pushed, or deployed yet.

• Remove worker-local app settings payload/version caches. Read shared Redis on each lookup, or Cosmos directly when Redis is disabled.
• Reject settings saves when configured Redis is unavailable; allow Cosmos read fallback. Do not return an old worker snapshot when both services fail.
• Use ETag-conditional Cosmos writes, partial updates, and a document revision.
• Publish settings, revision, and the Cosmos session token together using a single Redis state key and atomic compare-and-set.
• Persist a pending marker before a database write so interrupted publication cannot expose the previous Redis payload. Repair abandoned publication after its 30-second lease via a conditional Cosmos write.
• Remove startup snapshot publication and add an admin form revision check.
• Make auxiliary writers use intended field updates and report failed saves accurately.
• Complete the logging-settings re-entrancy guard so cache failure logging cannot recurse.

Acceptance Criteria

• Remove the 15-second worker-local settings staleness window without changing unrelated caches.
• Prevent stale snapshots and concurrent metadata updates from overwriting unrelated saved settings.
• Prevent delayed startup or expired writers from publishing older cache contents.
• Reject writes before database mutation when configured Redis is unavailable.
• Surface an unconfirmed result if Redis fails after a database write; never imply a guaranteed rollback.
• Cover Redis failure/recovery, no-Redis reads, migrations, creation, partial writes, stale forms, and logging re-entrancy with deterministic tests.
• Commit/review the implementation and deploy all participating workers and scheduler together.
• Validate repeated saves/reloads across the scaled-out deployment and confirm instance/process/revision behavior.

Validation

• Combined targeted tests: 190 passed, 1 skipped. The skipped test requires an authenticated browser environment.
• Syntax checked for 27 changed/new Python files; existing added-line XSS/access-control checks reported zero findings; whitespace checks clean.
• Standalone logging/merge regressions and documentation site quality passed.
• Documentation coverage has an unrelated pre-existing inventory mismatch in the chat upload label (xsd); left unchanged.

Notes

• Cosmos fallback remains subject to Session consistency; removing local cache does not create global strong consistency during outages.
• Coordinated rollout is required: old workers do not participate in the new publication protocol. Redis connection/enablement changes require coordinated worker restarts.
• Relevant files: app_settings_store.py, app_settings_cache.py, functions_settings.py, functions_appinsights.py, route_frontend_admin_settings.py, worker/scheduler startup, auxiliary settings writers.
• Tests: functional_tests/test_app_settings_store_consistency.py, functional_tests/test_app_settings_auxiliary_writers.py, ui_tests/test_admin_settings_save_consistency.py.
• Documentation: docs/admin/scale.md; release notes for v0.261.025.
• Priority: P1. Estimated size: L (shared storage protocol, multiple writer surfaces, and concurrency/failure validation).