Skip to content

Master super user access token grows with every realm and breaks websocket connections #3566

Open

Description

@richturner

Problem

On instances with many realms, the access token of a master realm super user grows with every realm until requests that carry it are rejected. The manager UI then fails to connect its websocket and shows the generic Event provider error (eventPhase: 0), even though the manager itself is healthy.

The UI passes the access token in the websocket URL (ui/component/core/src/event.ts, ?Realm=…&Authorization=Bearer <jwt>), because browsers cannot set headers on a WebSocket. HAProxy's default tune.bufsize of 16 KB (minus tune.maxrewrite) limits the request head to roughly 15 KB:

Token in URL Response
≤ 12 KB 101 Switching Protocols
~15 KB 431 Request Header Fields Too Large (from HAProxy)
≥ 16.5 KB 414 URI Too Long (from HAProxy)

On the affected instance, a master admin token with 23 realms had a payload of ~14.5 KB (base64), about 15 KB in total. REST calls send the same token in the Authorization header and are close to the same limit.

Cause

  • In the master realm, Keycloak creates a <realm>-realm management client for every realm, and the master admin role is a composite of all of their roles (view-realm, manage-users, manage-organizations, … 21 roles each since Keycloak 26.7).
  • The openremote client is created with Keycloak's default Full scope allowed (ManagerKeycloakIdentityProvider.generateOpenRemoteClientRepresentation() does not set it), so every effective role ends up in resource_access, and every <realm>-realm client is added to aud.
  • Each additional realm adds about 600 bytes to the token.

The manager does not need these roles:

  • It only reads realm_access roles (admin, restricted_user, see AuthContext) and the openremote client roles.
  • Realm and user management uses the manager's own Keycloak admin client, not the user's token.
  • The Keycloak admin console uses its own security-admin-console client, so it is unaffected.

Without the <realm>-realm entries, the same token's payload is ~1.6 KB.

Proposed fix

  1. In generateOpenRemoteClientRepresentation(), set fullScopeAllowed(false) and add explicit scope mappings to the openremote client for:
    • realm roles: admin, restricted_user, offline_access
    • client roles: all openremote client roles (read:* and write:* roles, read, write)
  2. Make sure realm creation (createRealm and the realm setup path) applies the same scope mappings, so new realms get the trimmed client too.
  3. Migrate existing instances (see below).
  4. Optional hardening, as a separate follow-up: stop sending the token in the websocket URL, for example by authenticating with the first frame after onopen, or with a short-lived connection ticket. ClientEventService already prefers the Authorization header over the query parameter. This removes the URL length limit altogether.

Migration of existing instances (Flyway Java migration)

Existing openremote clients are not touched by generateOpenRemoteClientRepresentation(), so existing instances need a one-off migration. There is a precedent for this: V20260715_1443_1__BackfillNotificationRealmUsers extends AbstractKeycloakMigration and uses the Keycloak Admin API (openKeycloak(), isKeycloakDeployment()) rather than the Keycloak schema.

Suggested manager/src/main/java/org/openremote/manager/setup/database/V<yyyyMMdd>_<HHmm>__RestrictOpenRemoteClientScope.java, extending AbstractKeycloakMigration. It would:

  • Skip if !isKeycloakDeployment().
  • For each realm returned by keycloak.realms().findAll():
    • find the openremote client (clients().findByClientId(KEYCLOAK_CLIENT_ID)), and skip it if missing
    • if fullScopeAllowed is true:
      • add realm-level scope mappings for admin, restricted_user and offline_access, where those roles exist in the realm (clients().get(id).getScopeMappings().realmLevel().add(…))
      • add client-level scope mappings for all roles of the openremote client (getScopeMappings().clientLevel(openremoteClientId).add(…))
      • only then set fullScopeAllowed(false) and update() the client
  • Be idempotent. Only change clients that still have full scope, so re-runs and clients an admin has already customised are left alone.
  • Log a summary of how many clients were updated in how many realms.

Notes:

  • Use the Admin API, not SQL against the Keycloak tables (client.full_scope_allowed, scope_mapping). Keycloak caches clients and roles in Infinispan, and its schema is not a stable interface. SQL changes would not take effect until Keycloak restarts and could break on Keycloak upgrades.
  • Add the scope mappings before turning off full scope, so a partial failure never leaves users with tokens that lack the openremote roles.
  • Users only get the smaller token after their next token refresh or login, so no forced logout is needed.
  • Name the class following the repository's Flyway convention (V<date>_<time>__…, UTC).

Workarounds for affected instances

  • Proxy: add tune.bufsize 65536 to the global section of haproxy.cfg. This was verified: a websocket upgrade with a 30 KB URL then returns 101.
  • Keycloak: in each realm, go to Clients → openremote → Client scopes → openremote-dedicated → Scope, turn off Full scope allowed, and assign the realm roles admin, offline_access and restricted_user, plus all openremote client roles.

Acceptance criteria

  • A master super user's access token no longer contains <realm>-realm entries in resource_access or aud, and its size does not grow with the number of realms.
  • Super user, regular user, restricted user and service user access is unchanged. This includes offline and refresh tokens and cross-realm super user access (TokenVerifierImpl).
  • New realms and existing instances (via the migration) both end up with a restricted openremote client scope.
  • Spock coverage: a test that creates several realms and asserts the master admin token's resource_access and aud contain only openremote/account, plus a test of the migration's idempotency.

Activity

  1. added a commit that references this issue on Nov 20, 2026
    9b5e821
  2. wborn commented on Nov 20, 2026

    @wborn
    Member

    I opened PR #3603 to fix the master superuser token growth.

    Why the proposed scope change didn’t work

    Keycloak expands composite roles when it evaluates the client’s allowed scope. Explicitly allowing the master admin role therefore also allows its inherited <realm>-realm management roles. Those roles still appear in resource_access, and Audience Resolve still adds their client IDs to aud. In the Keycloak 26.7.5 test, the proposed configuration produced a 14,442-byte JWT with 24 realms. Keycloak scope resolution

    What the PR changes

    The PR configures built-in protocol mappers on master/openremote. They preserve the existing realm roles, including admin, emit client roles only for openremote and account, and add the account audience only for users with account roles. Clean setup and the Flyway migration use the same configuration. The replacement mappers are verified before the shared roles scope is detached from this client.

    In the isolated test, the compact admin JWT stayed at 1,358 bytes with 1, 24, and 29 realms. The PR also adds a regression test with real Keycloak tokens that checks token size after adding realms and verifies that rerunning the migration preserves the required claims.

    Could you review the PR, especially the mapper configuration and its compatibility with existing deployments?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

BugError or issue in the system

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions