Skip to content

Make the rate limiting (429) response message admin-configurable with markdown support #1354

Closed

Description

@paullizer

Summary

Make the message users see when they are rate limited (HTTP 429) admin-configurable, with markdown support, surfaced consistently across every 429 the application returns. Add the control to a new Rate Limiting tab in the Security group of Admin Settings.

User Value

In APIM-fronted deployments admins deliberately throttle traffic. SimpleChat already retries 429s with backoff, which handles the transient cases well, but retries eventually run out. When they do, the user currently gets no indication that throttling is what happened, no guidance on whether to wait, and no admin-authored context.

The chat experience is the worst offender: an exhausted 429 never becomes a real 429. It falls through to the generic streaming error, so the user sees:

Something went wrong while streaming the response. Please try again.

That is indistinguishable from a genuine failure, and it gives the admin no way to tell their users what is actually going on.

Current Behavior

Every 429 surface is hardcoded and unrelated to the others:

Surface Current response
route_backend_chats.py streaming Retries exhausted 429s, then emits the generic CLIENT_SAFE_STREAM_ERROR_MESSAGE. Never actually returns a 429.
route_backend_tts.py "Service temporarily unavailable due to high load. Please try again."
swagger_wrapper.py "Too many requests for swagger.json. Please wait before trying again."
route_inbound_mcp.py JSON-RPC -32029, "Inbound MCP tool rate limit exceeded."

There is no global Flask 429 error handler and no frontend 429 handling anywhere.

Proposed Behavior

  • Two new settings: enable_custom_rate_limit_message (boolean, default off) and rate_limit_message (markdown). Turning the toggle off, or leaving the message blank, falls back to a built-in default so users always get something sane.
  • A single resolver in functions_settings.py that every 429 surface calls, so the fallback logic is not reimplemented per surface.
  • A global @app.errorhandler(429) that negotiates content: JSON for API/XHR callers, server-rendered HTML for browser navigations.
  • Chat classifies an exhausted provider/APIM 429 and returns the configured message with a rate_limited flag, instead of the generic stream error.
  • Markdown reuses what already exists: markdown_filter (markdown2 + bleach) server-side, and the globally loaded local marked + DOMPurify client-side. No new dependency and no CDN asset, per the local browser assets rule.
  • A new Rate Limiting tab in the Security group holds the toggle, the markdown editor, and a live preview.

Acceptance Criteria

  • enable_custom_rate_limit_message and rate_limit_message exist as settings, persist through Admin Settings, and survive sanitize_settings_for_user().
  • Toggle off, or a blank message, falls back to the built-in default.
  • A global 429 error handler returns the configured message as JSON for API callers and rendered HTML for browser navigations.
  • An exhausted 429 in chat streaming shows the configured message in the error banner, rendered as markdown, instead of the generic streaming error.
  • TTS, Swagger spec, and inbound MCP 429s all use the configured message. Inbound MCP keeps its structured rate-limit data payload so clients can still back off correctly.
  • Markdown is sanitized on both sides. Non-rate-limit stream errors keep their existing plain-text rendering.
  • New Rate Limiting tab appears in the Security group with a live markdown preview.
  • docs/admin/security.md documents the tab with a {#rate-limiting} anchor, docs/_data/features.yml claims the new capability key, and docs/_data/app_surface.yml is regenerated.
  • Functional test coverage added, and the docs coverage/quality tests pass.

Notes

  • Scope is limited to what a user is told when a 429 happens. Adding new SimpleChat-enforced rate limits is explicitly out of scope.
  • Closest existing precedent is access_denied_message (Security → Access & Roles), which is plain text with nl2br rather than markdown.
  • Application version is being locked at 0.261.001 for this work.

Activity

  1. paullizer commented on Aug 25, 2026

    @paullizer
    CollaboratorAuthor

    Implementation is complete and open for review in #1357.

    What landed

    • functions_rate_limit.py resolves the message in one place; enable_custom_rate_limit_message and rate_limit_message ship as settings and survive sanitize_settings_for_user().
    • Global @app.errorhandler(429) with content negotiation, plus a plain-text fallback if the page shell fails to render.
    • Chat classifies an exhausted throttle and shows the rendered Markdown in a dedicated banner instead of the generic streaming error. This was the main gap.
    • TTS, Swagger spec endpoints, and inbound MCP share the message. Inbound MCP keeps its structured limit/window/reset data so clients still back off correctly.
    • New Security -> Rate Limiting admin tab.

    Validation

    All acceptance criteria are covered. test_rate_limit_message_configuration.py passes 10/10, and docs coverage (7/7), docs quality (6/6), sidebar parity (4/4), XSS guardrails (4/4), and both route policy suites pass. The 429 handler was additionally exercised against a live Flask test client to confirm JSON/HTML negotiation, Markdown rendering and sanitization, and the toggle-off fallback.

    Known limitation

    Edge-enforced limits (Front Door WAF, or an APIM policy that rejects before the request reaches the app) cannot be customized, because the request never arrives. Documented in docs/explanation/features/RATE_LIMIT_RESPONSE_MESSAGE.md.

    Version is locked at 0.261.001 as requested, not patch-bumped.

  2. moved this from Pending Evaluation to Released in Simple Chat Roadmapon Aug 26, 2026
  3. added a commit that references this issue on Aug 26, 2026
    d0f896d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

P2Priority 2: important, scheduled after P0/P1enhancementNew feature or request

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions