Skip to content

fix(realtime): authenticate websocket with connection tickets and in-band refresh - #45404

Open
villebro wants to merge 7 commits into
apache:masterfrom
villebro:fix/realtime-ws-tickets
Open

villebro wants to merge 7 commits into
apache:masterfrom
villebro:fix/realtime-ws-tickets

Conversation

@villebro

@villebro villebro commented Oct 17, 2026 •

Copy link
Copy Markdown
Member

SUMMARY

An idle browser tab loses its realtime socket and never gets it back on its own. The websocket credential was an httponly cookie re-minted only as a side effect of other HTTP requests, the client's keepalive was anchored to socket-open time rather than cookie-mint time, and the reconnect path never refreshed the cookie. After a websocket server restart, Redis blip, or laptop sleep, every retry failed with "JWT not present" until the user happened to make an HTTP request. The server also destroyed unauthenticated upgrades with no HTTP status, so the client could not tell an auth failure from a network failure.

This replaces the cookie with short-lived connection tickets and in-band re-authentication. The client always connects with a credential it has just obtained, knows its lifetime, and refreshes it on the open socket without reconnecting.

Backend: new GET /api/v1/realtime/token/ (gated by can_read on Realtime, 401 without a session, 403 without the permission) returning {"token": "<jwt>", "expires_in": <seconds>} using the existing get_realtime_principal() and mint_channel_token(), with Cache-Control: no-store, private so a shared cache can never store or replay a ticket (embedded guests authenticate by header, not cookie). The cookie path (register_ws_channel_cookie, _should_remint) and the WEBSOCKET_JWT_COOKIE_NAME/SECURE/SAMESITE/DOMAIN config keys are removed; WEBSOCKET_JWT_SECRET and WEBSOCKET_JWT_EXPIRATION_SECONDS stay. The now-unused WEBSOCKET_JWT_EXPIRATION_SECONDS bootstrap entry is dropped because the client reads expires_in from the server instead.

Websocket server: the handshake keeps the Origin allowlist and subscriber-health check, no longer reads cookies, and answers refusals with real statuses (403 for origin, 503 for unhealthy). A new connection starts pending_auth, registered on no channel, and is closed with 4401 if no valid auth frame ({"type": "auth", "token": "...", "tab_id": "..."}) arrives within 10s. Pending sockets count toward maxTotalConnections, and a new maxPendingConnections (MAX_PENDING_CONNECTIONS) can cap them separately; an over-limit upgrade is closed with 1013. If the Redis subscriber has become unhealthy by the time the first auth frame arrives, the socket is closed with 1012 and nothing is registered. The first valid frame registers the principal and tab channels and replies realtime.auth with expires_in. A later frame refreshes the expiry; naming a different channel closes the socket with 4403, and a changed tab_id rebinds the tab channel. Expired sockets are closed with 4401 instead of terminated. Inbound frames are capped at 16 KiB, and realtime.auth acks go through the same bounded send path (maxSocketBufferBytes) as every other outbound message. The jwtCookieName option and the cookie dependency are removed.

Frontend (realtime.ts): every connect (initial and reconnect) fetches a ticket, opens the socket, sends auth, and reports open only after the realtime.auth acknowledgement (abandoned and retried after 10s). A 401 or 403 from the ticket endpoint enters a new terminal unauthorized state with no connect loop; the request passes ignoreUnauthorized, so a refused ticket (e.g. an anonymous visitor under PUBLIC_ROLE_LIKE) never triggers the client's login redirect or guest handler. Other ticket errors (404 from an older web server during a rolling upgrade, 5xx, network) are transient and retried with backoff. A ticket fetch, ack, refresh timer or close event from a superseded connection attempt is ignored. The ticket is refreshed on the open socket at 0.75 x expires_in with backoff on transient failures; close 4401 on a previously authenticated socket reconnects immediately, other closes keep the exponential backoff. KEEPALIVE_FRACTION, COOKIE_REFRESH_ENDPOINT, and the keepalive teardown are removed, and tab_id moved from the connect URL into the auth frame, so a tab-id reassignment no longer reconnects. The keepalive open reason is gone since refreshes never reopen the socket; when the server acknowledges a changed tab id on the open socket, the client emits a single rebind open event so asyncEvent runs one status catch-up. realtime.ts exposes getRealtimeState(), and asyncEvent.ts settles chart requests promptly while the state is unauthorized or unhealthy, both for waiters pending at the transition and for requests registered afterwards. The embedded entry reconnects realtime once the guest client is configured, since the module-load attempt runs before a guest token exists.

Security notes: the token is now readable by page JavaScript, which only lets the holder receive this principal's realtime nudges (opaque ids and task statuses); script in the page can already call the REST API as the user. Tokens never appear in URLs, so proxies do not log them. A refresh cannot switch principal on a live socket. Revocation stays bounded by the token lifetime: after logout the ticket endpoint returns 401, no refresh happens, and the server closes the socket at expiry. SECURITY.md describes an open connection as authorized by the most recent ticket presented on it. The Origin allowlist stays as defense in depth. This also removes the requirement that the websocket server share a browser-visible host with the web application.

Docs, SECURITY.md and UPDATING.md are updated, including the removed config keys. UPDATING.md asks operators to upgrade the web and websocket servers together and reload open tabs: tabs still running the old cookie-based client are refused by the new server and retry until reloaded. The websocket README notes that the ticket lifetime no longer bounds connection lifetime, since clients refresh on the open socket, so draining a replica needs explicit socket closes or pod termination. The superset-websocket/utils/client-ws-app dev tool is updated to send an auth frame instead of setting a cookie. Operator follow-up: stop rendering WEBSOCKET_JWT_COOKIE_NAME and passing JWT_COOKIE_NAME to the websocket server.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A, no visual change.

TESTING INSTRUCTIONS

Unit tests: pytest tests/unit_tests/websocket, npm test in superset-websocket (includes spec/auth.test.ts, which drives the real ws upgrade, auth, refresh, 4401, and 4403 paths over in-memory duplex streams), and npx jest src/middleware/realtime.test.ts src/middleware/asyncEvent.test.ts src/embedded/index.test.tsx src/views/CRUD/hooks.test.tsx in superset-frontend.

Manual, with WEBSOCKET_ENABLE = True and WEBSOCKET_JWT_EXPIRATION_SECONDS = 60:

  1. Open a dashboard, make no further requests, and restart the websocket server at about 70% of the token lifetime. The socket recovers and stays connected well past the original expiry.
  2. Keep one socket open for more than 3x the token lifetime. There are zero reconnects and a single long-lived 101.
  3. Block the client's network until after expiry, then restore it. The first reconnect attempt succeeds.
  4. Log out in another tab. This tab's socket closes within one token lifetime and makes no further connect attempts.
  5. Send a refresh frame carrying another principal's token. The socket closes with 4403.
  6. With PUBLIC_ROLE_LIKE = "Gamma", open a public dashboard while logged out. The page is not redirected to login, and no socket is opened.
  7. Open an embedded dashboard. The socket connects once the guest token is delivered.

ADDITIONAL INFORMATION

  • Has associated issue:
  • Required feature flags: none (applies when WEBSOCKET_ENABLE is on)
  • Changes UI
  • Includes DB Migration (follow approval process in SIP-59)
    • Migration is atomic, supports rollback & is backwards-compatible
    • Confirm DB migration upgrade and downgrade tested
    • Runtime estimates and downtime expectations provided
  • Introduces new feature or API
  • Removes existing feature or API

Fixes #45493

@github-actions github-actions Bot added api Related to the REST API doc Namespace | Anything related to documentation dependencies:npm labels Oct 17, 2026
@netlify

netlify Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for superset-docs-preview ready!

Name Link
🔨 Latest commit 88ea23d
🔍 Latest deploy log https://app.netlify.com/projects/superset-docs-preview/deploys/6ae4ed52d331573ef0c66923
😎 Deploy Preview https://deploy-preview-45404--superset-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

Comment on lines +829 to +834
const authTimer = setTimeout(() => {
if (socketId === undefined) {
statsd.increment('ws_auth_timeout');
ws.close(AUTH_CLOSE_CODE, 'authentication timeout');
}
}, AUTH_TIMEOUT_MS);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Because the upgrade is accepted before this timer and the limit is checked only after authentication, an unauthenticated client can bypass configured connection caps for 10 seconds.

Assessment: 🟠 Major · 🔁 Occurrence: Rarely · 🏷️ Security

Use CodeAnt Skill Fix in Cursor Fix in VSCode Claude

Prompt for AI Agent 🤖
This is a comment left during a code review.

**Path:** superset-websocket/src/index.ts
**Line:** 829:834
**Comment:**
	*Security: Because the upgrade is accepted before this timer and the limit is checked only after authentication, an unauthenticated client can bypass configured connection caps for 10 seconds.

Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise.
Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix
👍 | 👎

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed—the connection limits are only evaluated inside authenticate, so sockets that never send an auth frame are uncounted for the full 10s auth window (plus the close handshake), and anonymous clients can hold many upgraded sockets regardless of maxTotalConnections. Before this change an unauthenticated upgrade was rejected immediately. Should pending sockets count toward a total or separate pre-auth cap?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pending sockets should count toward maxTotalConnections; that setting is the global resource cap, while maxConnectionsPerChannel cannot be evaluated until authentication identifies a principal. Reserve one slot when accepting the connection and keep it reserved when the socket becomes authenticated; release it on close or timeout.

A separate, lower maxPendingAuthConnections limit could further protect against unauthenticated clients monopolizing the total slots, but it should supplement—not replace—the total cap. This closes the bypass while keeping the existing total-connection limit meaningful. Add tests for rejecting at capacity and releasing a pending slot after timeout/close.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid. Upgraded-but-unauthenticated sockets now count toward maxTotalConnections (capacity is reserved at upgrade and released on auth, close, or auth timeout), and a new maxPendingConnections (MAX_PENDING_CONNECTIONS) can cap them separately. Over-limit upgrades are closed immediately with 1013. Tests in spec/auth.test.ts. Fixed in f273ec7.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same fix as above: pending sockets are counted and capped, see f273ec7.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of b8eaf29.

The upgrade handler now reserves a pending-connection slot and rejects upgrades when configured connection limits or the pending-connection cap are reached. The reservation is released on close, authentication, or timeout.

If that's not right, unresolve this thread and CodeAnt will leave it open.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of 4089b91.

The upgrade handler now reserves a pending-connection slot and rejects upgrades when configured connection limits or the pending-connection cap are reached. The reservation is released on close, authentication, or timeout.

If that's not right, unresolve this thread and CodeAnt will leave it open.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of 0d4a54e.

The upgrade handler checks connection limits and the pending-connection cap before accepting a pending reservation. It increments pendingConnections and releases the reservation on close, timeout, or authentication.

If that's not right, unresolve this thread and CodeAnt will leave it open.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of 88ea23d.

The upgrade handler checks connection limits and the pending-connection cap before accepting a pending reservation. It increments pendingConnections and releases the reservation on close, timeout, or authentication.

If that's not right, unresolve this thread and CodeAnt will leave it open.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of 2d0d1a8.

The upgrade handler checks connection limits and the pending-connection cap before accepting a pending reservation. It increments pendingConnections and releases the reservation on close, timeout, or authentication.

If that's not right, unresolve this thread and CodeAnt will leave it open.

@codeant-ai-for-open-source

codeant-ai-for-open-source Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

CodeAnt PR Risk: Medium Risk

  • The PR needs attention before merging because pending WebSocket connections bypass the configured global connection cap.
  • In wsConnection, limits are checked only after authentication, so unauthenticated clients can exceed maxTotalConnections during the 10-second auth window.
  • Reserve a global slot when accepting a connection and release it on close or timeout.

Assessed commit: 71f312635fc0

@codecov

codecov Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.12644% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.74%. Comparing base (aa7b2bc) to head (2d0d1a8).

Files with missing lines Patch % Lines
superset-frontend/src/middleware/realtime.ts 96.35% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #45404      +/-   ##
==========================================
+ Coverage   84.73%   84.74%   +0.01%     
==========================================
  Files        3069     3070       +1     
  Lines      205781   205859      +78     
  Branches    48391    48415      +24     
==========================================
+ Hits       174366   174459      +93     
+ Misses      28122    28114       -8     
+ Partials     3293     3286       -7     
Flag Coverage Δ
hive 34.51% <0.00%> (+<0.01%) ⬆️
javascript 81.00% <96.55%> (+0.02%) ⬆️
mysql 52.61% <0.00%> (+0.01%) ⬆️
postgres 52.61% <0.00%> (+<0.01%) ⬆️
presto 36.23% <0.00%> (+<0.01%) ⬆️
python 87.77% <100.00%> (+<0.01%) ⬆️
sqlite 52.38% <0.00%> (+0.01%) ⬆️
unit 82.53% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread superset-websocket/src/index.ts Resolved
Comment thread UPDATING.md Resolved
Comment thread superset-websocket/README.md Resolved
@villebro

Copy link
Copy Markdown
Member Author

Addressed the latest review round in the follow-up commits.

Connection state is now the single source of truth: realtime.ts exposes getRealtimeState(), and asyncEvent.ts settles a chart request that registers while the state is unauthorized or unhealthy immediately, in addition to abandoning waiters that existed at the transition. Tests cover a waiter registered after each state.

Stale connection results are now guarded by a per-attempt identity on top of the generation, so a ticket fetch, auth ack, refresh timer or close event from an earlier connection cannot affect a newer one. New tests cover a disconnect during a refresh, a reconnect or reconfigure while a ticket fetch is in flight, and a delayed realtime.auth ack (plus close and message) from an obsolete socket.

Ticket endpoint errors other than 401/403, including 404 from an older web server during a rolling upgrade, 5xx and network errors, are treated as transient: the client backs off and retries and never enters unauthorized. A test covers each case and recovery once the endpoint appears.

UPDATING.md and the websocket README now state that after logout or session revocation an open socket stays authenticated until its current ticket expires, because the next refresh fails.

@rusackas rusackas left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The connection-cap bypass is fixed and tested (f273ec7), good.

The other three threads are still open with no reply yet, and two of them look serious enough to block on. Verified the public-role one myself: getRealtimeTicket calls SupersetClient.get({ endpoint: TICKET_ENDPOINT }) with no custom unauthorizedHandler, and the client's default unauthorized handling does navigate to /login?next=... on a 401. So a deployment with PUBLIC_ROLE_LIKE = "Gamma" (a real, documented pattern for public dashboards, not a contrived config) would bounce anonymous visitors to the login screen on every page load once this ships. The embedded guest-token race looks just as real. asyncEvent.ts's init() fires the ticket request at module load, before setupGuestClient() has a chance to run, and enterUnauthorized() being terminal means the later guest token never gets a retry.

That's two ways this could break public/embedded dashboards in production, not edge cases. Holding off on approving until those get a response.

Comment thread superset-websocket/src/index.ts Resolved
…band refresh

Replace the host-scoped httponly JWT cookie with a short-lived ticket fetched from GET /api/v1/realtime/token/ and sent as the first frame of the socket. The client refreshes the ticket on the open socket at 75% of its lifetime, so an idle tab no longer loses its socket at cookie expiry and never tears down to re-authenticate.

The websocket server answers handshake refusals with real HTTP statuses, holds new connections in a pending_auth state, and closes with 4401 (expired or invalid ticket) or 4403 (principal switch on refresh). The client reconnects immediately on 4401 and stops on 401/403 from the ticket endpoint (new unauthorized state).

Removes WEBSOCKET_JWT_COOKIE_NAME/SECURE/SAMESITE/DOMAIN and the websocket server's jwtCookieName.
…ber, reconcile on rebind and unauthorized

Count upgraded-but-unauthenticated sockets toward maxTotalConnections and add maxPendingConnections. Refuse the first auth frame when the Redis subscriber is unhealthy. Raise a rebind open event when the server acknowledges a changed tab id so waiters run one catch-up, and settle async waiters on the terminal unauthorized state. Correct the upgrade note and the README on connection lifetime.
…ale attempts

asyncEvent settles a waiter registered while the realtime state is unauthorized or unhealthy. Each connection attempt gets an identity so a stale ticket fetch cannot open a socket. Document that an open socket stays authenticated until its ticket expires after revocation.
…after the guest token, bound auth acks

- The ticket request passes ignoreUnauthorized, so a refused ticket (e.g. an
  anonymous visitor under PUBLIC_ROLE_LIKE) ends in the 'unauthorized' state
  instead of running the client's login redirect or guest handler.
- The embedded page reconnects realtime once the guest client is configured,
  since the module-load attempt runs before a guest token exists.
- The websocket server sends realtime.auth acks through the bounded send path,
  so replayed auth frames hit maxSocketBufferBytes like any other message.
@villebro
villebro force-pushed the fix/realtime-ws-tickets branch from 4089b91 to e76a1f9 Compare October 29, 2026 23:10
@villebro

Copy link
Copy Markdown
Member Author

Rebased onto master and addressed the open threads in e76a1f9: the ticket request no longer triggers the unauthorized handler (public-role redirect), the embedded page reconnects realtime once the guest client is configured, and auth acks go through the bounded send path. @rusackas both blocking cases you verified are covered, with tests.

Comment thread superset/websocket/api.py Resolved
Comment thread superset-websocket/spec/auth.test.ts Resolved
Comment thread SECURITY.md Resolved
villebro and others added 2 commits October 30, 2026 12:17
…sts, update SECURITY.md

- The ticket endpoint sets Cache-Control: no-store, private. A guest
  authenticates by header, not cookie, so a shared cache must not store or
  replay a bearer ticket.
- auth.test.ts closes both ends of each connection and waits for the pending
  reservations to be released before resetting server state, so a late close
  can't decrement the freshly reset count.
- SECURITY.md describes connections as authorized by the most recent ticket
  presented on the socket, not by a JWT accepted at upgrade.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api Related to the REST API dependencies:npm doc Namespace | Anything related to documentation size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Idle tabs lose the realtime websocket and never reconnect: JWT cookie expires (7.0.0rc2)

3 participants