feat(llm): graceful provider failover with ordered chain (#39)

[psalmuel3]

Summary

Adds graceful LLM provider failover: an ordered, configurable provider chain in
which a provider that is still failing after its own retries are exhausted hands the
request to the next provider, while auth/validation/policy errors fail fast and a stream
that has already emitted text is never replayed.

The change is built on the existing provider layer — LLMProvider, RetryingProvider,
is_transient_error, the registry, and provider_config — rather than beside it. Each
chain entry is a RetryingProvider, so retries stay inside one provider; only an
exhausted provider triggers failover.

Closes #39

Design

Layering

FailoverProvider([RetryingProvider(openai), RetryingProvider(anthropic), ...])

• app/services/providers/failover.py (new)
  • FailoverProvider — tries each provider in order for chat() and stream().
  • ProviderFailoverError — raised when every provider failed; carries the
    per-attempt diagnostics. It is deliberately not a transient error, so wrapping
    the chain in another retry layer would fail fast instead of replaying the chain.
  • ProviderAttempt / diagnostics() — one immutable record per attempt plus a
    machine-readable {"request_id", "chain", "served_provider", "partial_provider", "failed", "attempts"} view.
  • build_failover_chain(names, build=..., wrap=RetryingProvider) — resolves names into
    providers and wraps each in its own retry layer. build/wrap are injected so call
    sites keep their existing test seams.
• app/services/providers/retry.py — adds classify_error() and stable category
  constants (rate_limit, unavailable, authentication, configuration, response,
  provider, unknown). Transient = rate_limit + unavailable, reusing the existing
  is_transient_error semantics.
• app/services/provider_config.py — extends the existing config with
  resolve_provider_chain(), routing_rules(), default_provider_chain() and
  provider_chain_status().
• app/chat/routes.py / app/chat/api.py — build the chain at the real call sites and
  use the chain-aware pre-flight status.

Configuration (backwards compatible)

• LLM_PROVIDER_CHAIN — comma-separated ordered chain, e.g. openai,anthropic.
• LLM_ROUTING_RULES — optional route=provider,provider;... rules, e.g.
  vision=openai,anthropic;cheap=mock. A request may pass "route": "vision"; the
  conversation's explicit provider is always tried first, then the route/default chain.
• Unset (the default) resolves to exactly [LLM_PROVIDER], so a single-provider
  setup behaves precisely as before. A single-provider chain even re-raises the original
  provider error unchanged rather than wrapping it.

Mapping to the acceptance criteria

Criterion	Where
Ordered list + optional routing rules	LLM_PROVIDER_CHAIN, LLM_ROUTING_RULES, resolve_provider_chain()
On failure after retries, try the next provider	FailoverProvider.chat/stream after RetryingProvider gives up
Serving provider reported (logs/tests, not user-facing)	served_provider + diagnostics() + chat.provider_attempt audit event
All providers fail → clear error via SSE / JSON	ProviderFailoverError is a ProviderError, so the existing SSE error event and JSON 502 paths surface it
Tests verify ordering + all-failed	tests/test_llm_failover.py

Mapping to @Triumph1701's guidance

• Logical request ID across the chain — FailoverProvider.request_id is the HTTP
  request's correlation id when available, otherwise a generated id; every attempt event
  for one request shares it (test_one_event_per_attempt_with_a_shared_request_id).
• Each attempt is a separate event — one chat.provider_attempt record per provider
  with attempt, provider, status, error, error_category, transient, stream,
  served, partial, latency_ms and the chain.
• Classify retryable transport/429/5xx separately from auth/validation/policy —
  classify_error() labels each attempt; only is_transient_error failures fail over,
  and test_non_transient_fails_fast_without_failover proves an auth error stops the
  chain without touching the next provider.
• Stop once partial output exists — once a stream has yielded a chunk the attempt
  owns the answer: the error is surfaced (partial=True, partial_provider set) and no
  other provider is tried, because a second provider would duplicate rendered text
  (test_partial_stream_is_not_replayed_on_failure). There are no replayable tool side
  effects in this codebase.
• Effective provider in diagnostics so an all-failed path stays explainable —
  diagnostics() reports served_provider (or None), the chain, and the per-attempt
  outcome; ProviderFailoverError embeds the request id and each provider's category and
  message.

Tests

../venv/bin/python -m pytest -q

Full-suite result: 16 failed, 1920 passed in 738.48s. All 16 failures are pre-existing and
unrelated: they are every test in tests/test_chat_markdown_sanitize.py, which shells out to
Node to evaluate app/static/js/chat_markdown.js and fails with md.renderMarkdown is not a function — a checked-in-asset/Node-version issue in this checkout, untouched by this PR. The
targeted suites for the provider layer, retries, chat SSE, settings, GitHub context and audit
logging all pass.

New coverage in tests/test_llm_failover.py (30 tests):

• failover ordering (chat + stream) and that the serving provider is reported;
• retries exhaust before failover (the flaky provider is called twice, the fallback once);
• transient vs non-transient classification — auth/malformed errors do not fail over;
• all-providers-failed raising ProviderFailoverError with both names, categories and the
  request id, plus the SSE error event at the route level;
• the no-replay-after-partial-output invariant;
• one chat.provider_attempt event per attempt sharing one logical request id;
• chain/routing configuration, including the single-provider backwards-compatible default;
• an end-to-end SSE test where the first provider fails and the second serves.

ruff and black are not installed in the provided virtualenv (nothing was installed);
python -m py_compile is clean and all touched files respect the configured 100-char
line length.

Caveats / limitations

• The conversation's model is passed to every provider in the chain. A fallback that
  does not support that model raises a non-transient ProviderResponseError, which
  correctly fails fast rather than silently switching models.
• Providers are configured at the environment level; per-user failover chains are out of
  scope here.
• Chain members that are not usable for the current user (no environment/stored key) are
  filtered out before the call, so a keyed fallback can serve even when the primary has
  no key; provider_chain_status mirrors that so the pre-flight check agrees.

[psalmuel3]

@Ayinkx Implemented the graceful provider failover requested in #39, following @Triumph1701's guidance.

• FailoverProvider (app/services/providers/failover.py) wraps an ordered chain where each entry is its own RetryingProvider, so retries stay inside one provider and only an exhausted provider hands over.
• Only transient failures fail over (ProviderRateLimitError / ProviderUnavailableError); authentication, validation and configuration errors fail fast so a bad key is never papered over by another vendor.
• Partial output is never replayed: once a stream has emitted a chunk that attempt owns the answer and the error is surfaced instead of failed over, mirroring the existing retry-layer invariant.
• One logical request id spans the chain with one chat.provider_attempt event per attempt, and diagnostics() reports the effective serving provider plus the per-attempt outcome, so an all-failed path stays explainable.
• When every provider fails, the chain raises a clear error surfaced through the existing SSE error event / JSON error path.
• The chain is configurable via LLM_PROVIDER_CHAIN and optional LLM_ROUTING_RULES; leaving it unset behaves exactly as before.

Verification: 30 new tests in tests/test_llm_failover.py pass (failover ordering, transient-vs-terminal classification, the all-failed path, and the no-replay-after-partial-output invariant), and the provider/retry/chat suites stay green. The 16 test_chat_markdown_sanitize.py failures are pre-existing and reproduce on main.

Could you review when you get a chance? Thanks!

[psalmuel3]

@Ayinkx Implemented the graceful provider failover requested in #39, following @Triumph1701's guidance.

• FailoverProvider (app/services/providers/failover.py) wraps an ordered chain where each entry is its own RetryingProvider, so retries stay inside one provider and only an exhausted provider hands over.
• Only transient failures fail over (ProviderRateLimitError / ProviderUnavailableError); authentication, validation and configuration errors fail fast so a bad key is never papered over by another vendor.
• Partial output is never replayed: once a stream has emitted a chunk that attempt owns the answer and the error is surfaced instead of failed over, mirroring the existing retry-layer invariant.
• One logical request id spans the chain with one chat.provider_attempt event per attempt, and diagnostics() reports the effective serving provider plus the per-attempt outcome, so an all-failed path stays explainable.
• When every provider fails, the chain raises a clear error surfaced through the existing SSE error event / JSON error path.
• The chain is configurable via LLM_PROVIDER_CHAIN and optional LLM_ROUTING_RULES; leaving it unset behaves exactly as before.

Verification: 30 new tests in tests/test_llm_failover.py pass (failover ordering, transient-vs-terminal classification, the all-failed path, and the no-replay-after-partial-output invariant), and the provider/retry/chat suites stay green. The 16 test_chat_markdown_sanitize.py failures are pre-existing and reproduce on main.

Could you review when you get a chance? Thanks!