Repository navigation
Description
Problem
On instances with many realms, the access token of a master realm super user grows with every realm until requests that carry it are rejected. The manager UI then fails to connect its websocket and shows the generic Event provider error (eventPhase: 0), even though the manager itself is healthy.
The UI passes the access token in the websocket URL (ui/component/core/src/event.ts, ?Realm=…&Authorization=Bearer <jwt>), because browsers cannot set headers on a WebSocket. HAProxy's default tune.bufsize of 16 KB (minus tune.maxrewrite) limits the request head to roughly 15 KB:
| Token in URL | Response |
|---|---|
| ≤ 12 KB | 101 Switching Protocols |
| ~15 KB | 431 Request Header Fields Too Large (from HAProxy) |
| ≥ 16.5 KB | 414 URI Too Long (from HAProxy) |
On the affected instance, a master admin token with 23 realms had a payload of ~14.5 KB (base64), about 15 KB in total. REST calls send the same token in the Authorization header and are close to the same limit.
Cause
- In the master realm, Keycloak creates a
<realm>-realmmanagement client for every realm, and the masteradminrole is a composite of all of their roles (view-realm,manage-users,manage-organizations, … 21 roles each since Keycloak 26.7). - The
openremoteclient is created with Keycloak's default Full scope allowed (ManagerKeycloakIdentityProvider.generateOpenRemoteClientRepresentation()does not set it), so every effective role ends up inresource_access, and every<realm>-realmclient is added toaud. - Each additional realm adds about 600 bytes to the token.
The manager does not need these roles:
- It only reads
realm_accessroles (admin,restricted_user, seeAuthContext) and theopenremoteclient roles. - Realm and user management uses the manager's own Keycloak admin client, not the user's token.
- The Keycloak admin console uses its own
security-admin-consoleclient, so it is unaffected.
Without the <realm>-realm entries, the same token's payload is ~1.6 KB.
Proposed fix
- In
generateOpenRemoteClientRepresentation(), setfullScopeAllowed(false)and add explicit scope mappings to theopenremoteclient for:- realm roles:
admin,restricted_user,offline_access - client roles: all
openremoteclient roles (read:*andwrite:*roles,read,write)
- realm roles:
- Make sure realm creation (
createRealmand the realm setup path) applies the same scope mappings, so new realms get the trimmed client too. - Migrate existing instances (see below).
- Optional hardening, as a separate follow-up: stop sending the token in the websocket URL, for example by authenticating with the first frame after
onopen, or with a short-lived connection ticket.ClientEventServicealready prefers theAuthorizationheader over the query parameter. This removes the URL length limit altogether.
Migration of existing instances (Flyway Java migration)
Existing openremote clients are not touched by generateOpenRemoteClientRepresentation(), so existing instances need a one-off migration. There is a precedent for this: V20260715_1443_1__BackfillNotificationRealmUsers extends AbstractKeycloakMigration and uses the Keycloak Admin API (openKeycloak(), isKeycloakDeployment()) rather than the Keycloak schema.
Suggested manager/src/main/java/org/openremote/manager/setup/database/V<yyyyMMdd>_<HHmm>__RestrictOpenRemoteClientScope.java, extending AbstractKeycloakMigration. It would:
- Skip if
!isKeycloakDeployment(). - For each realm returned by
keycloak.realms().findAll():- find the
openremoteclient (clients().findByClientId(KEYCLOAK_CLIENT_ID)), and skip it if missing - if
fullScopeAllowedistrue:- add realm-level scope mappings for
admin,restricted_userandoffline_access, where those roles exist in the realm (clients().get(id).getScopeMappings().realmLevel().add(…)) - add client-level scope mappings for all roles of the
openremoteclient (getScopeMappings().clientLevel(openremoteClientId).add(…)) - only then set
fullScopeAllowed(false)andupdate()the client
- add realm-level scope mappings for
- find the
- Be idempotent. Only change clients that still have full scope, so re-runs and clients an admin has already customised are left alone.
- Log a summary of how many clients were updated in how many realms.
Notes:
- Use the Admin API, not SQL against the Keycloak tables (
client.full_scope_allowed,scope_mapping). Keycloak caches clients and roles in Infinispan, and its schema is not a stable interface. SQL changes would not take effect until Keycloak restarts and could break on Keycloak upgrades. - Add the scope mappings before turning off full scope, so a partial failure never leaves users with tokens that lack the
openremoteroles. - Users only get the smaller token after their next token refresh or login, so no forced logout is needed.
- Name the class following the repository's Flyway convention (
V<date>_<time>__…, UTC).
Workarounds for affected instances
- Proxy: add
tune.bufsize 65536to theglobalsection ofhaproxy.cfg. This was verified: a websocket upgrade with a 30 KB URL then returns101. - Keycloak: in each realm, go to Clients →
openremote→ Client scopes →openremote-dedicated→ Scope, turn off Full scope allowed, and assign the realm rolesadmin,offline_accessandrestricted_user, plus allopenremoteclient roles.
Acceptance criteria
- A master super user's access token no longer contains
<realm>-realmentries inresource_accessoraud, and its size does not grow with the number of realms. - Super user, regular user, restricted user and service user access is unchanged. This includes offline and refresh tokens and cross-realm super user access (
TokenVerifierImpl). - New realms and existing instances (via the migration) both end up with a restricted
openremoteclient scope. - Spock coverage: a test that creates several realms and asserts the master admin token's
resource_accessandaudcontain onlyopenremote/account, plus a test of the migration's idempotency.
Activity
Prevent master access tokens from growing with realm count
wborn commentedon Nov 20, 2026
I opened PR #3603 to fix the master superuser token growth.
Why the proposed scope change didn’t work
Keycloak expands composite roles when it evaluates the client’s allowed scope. Explicitly allowing the master
adminrole therefore also allows its inherited<realm>-realmmanagement roles. Those roles still appear inresource_access, and Audience Resolve still adds their client IDs toaud. In the Keycloak 26.7.5 test, the proposed configuration produced a 14,442-byte JWT with 24 realms. Keycloak scope resolutionWhat the PR changes
The PR configures built-in protocol mappers on
master/openremote. They preserve the existing realm roles, includingadmin, emit client roles only foropenremoteandaccount, and add theaccountaudience only for users with account roles. Clean setup and the Flyway migration use the same configuration. The replacement mappers are verified before the sharedrolesscope is detached from this client.In the isolated test, the compact admin JWT stayed at 1,358 bytes with 1, 24, and 29 realms. The PR also adds a regression test with real Keycloak tokens that checks token size after adding realms and verifies that rerunning the migration preserves the required claims.
Could you review the PR, especially the mapper configuration and its compatibility with existing deployments?