Skip to content

fix(auth): transient Redis timeout on the token-blacklist check returns 500 to the client #660

Description

@ericfitz

Summary

When the token-blacklist lookup in cmd/server/jwt_auth.go cannot reach Redis, the request fails
with HTTP 500 {"error": "server_error", "error_description": "Authentication service error"}.
A brief Redis blip therefore surfaces to the caller as a 500 on an otherwise valid, authenticated
request. Per the Zero-500 policy this needs a better answer than 500.

Evidence

Observed during an integration run on 2026-08-01. TestDiagramCRUD/Setup_CreateThreatModel failed;
its POST /threat_models hung ~6s and returned 500:

diagram_crud_test.go:65: Expected status code 201, got 500
Response body: {"error":"server_error","error_description":"Authentication service error"}

Server log (logs/tmi-test-server.log:11054):

level=ERROR msg="Failed to check token blacklist: failed to check token blacklist:
failed to check token blacklist: i/o timeout" request_id=2e790e19-... client_ip=192.168.65.1

Four such timeouts occurred in that run, on distinct request ids. The identical POST /threat_models succeeded seconds later in the same test, confirming the condition is transient
rather than a broken configuration.

Raised at cmd/server/jwt_auth.go:505-508:

logger.Error("Failed to check token blacklist: %v", err)
return &AuthError{
    Code:        "server_error",
    Description: "Authentication service error",
}

Impact

  • Test flakiness: this is a second, distinct integration-test flake alongside test(integration): TestIdentityLink flakes on identity_already_bound - 4-digit suffix from UnixNano%10000 collides across runs #653. Here the
    whole of TestDiagramCRUD cascades to failure, because the setup step's threat model id comes
    back empty and every later URL degenerates to /threat_models//diagrams//.... That makes the
    failure look far worse and far less obviously environmental than it is, which costs triage time.
  • Production: a Redis hiccup becomes a 500 on any authenticated request. Failing closed is the
    right call for a blacklist check (treating an unknown token as un-revoked would be worse), but
    500 is the wrong code for it and there is no retry.

Suggested direction

  1. Retry briefly before failing. The blacklist check is a fast point lookup; one short retry
    would absorb the overwhelming majority of transient timeouts.
  2. Return 503, not 500, when the failure is a dependency timeout rather than a logic error, with
    Retry-After. Note this requires adding 503 to the affected operations in
    api-schema/tmi-openapi.json first, per the Documented-Status-Code Policy.
  3. Consider whether a Redis outage should fail closed on every request or only for tokens issued
    before the last known-good blacklist sync. Fail-closed is defensible; it should just be a
    deliberate, documented choice.

Acceptance Criteria

  • Transient Redis timeouts no longer produce a 500 on authenticated requests
  • Whatever code replaces it is documented in the OpenAPI spec for the affected operations
  • Integration tests stop cascading when a setup step fails to return an id

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions