Skip to content

Commit 322b721

Browse files
committed
fix(credentials): harden keyless sign-in for AWS and Databricks
Saving a keyless Databricks connection now refuses a federation policy that trusts every Sim subject, as AWS roles already were. Keyless stays off when the issuer isn't https on the default port, and an unpinned issuer is logged. Docs cover a zero-downtime key rotation, the issuer requirements, finding the service principal ID, and troubleshooting.
1 parent 54af5ec commit 322b721

7 files changed

Lines changed: 203 additions & 30 deletions

File tree

‎apps/docs/content/docs/integrations/aws-credentials.mdx‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,8 +58,21 @@ To rotate a key, create the new key in AWS, reconnect the credential with it, co
5858
Keyless roles need two things on a self-hosted deployment:
5959

6060
- `WORKLOAD_IDENTITY_SIGNING_KEY`: a PKCS#8 PEM RSA private key that signs Sim's tokens.
61-
- An issuer AWS can reach over the public internet. It defaults to `<NEXT_PUBLIC_APP_URL>/oidc/workload`; set `WORKLOAD_IDENTITY_ISSUER` to pin it. Trust policies reference this URL, so never change it once roles trust it.
61+
- An issuer AWS can reach over the public internet: an `https` URL on the default port, served by a certificate from a public CA. It defaults to `<NEXT_PUBLIC_APP_URL>/oidc/workload`; set `WORKLOAD_IDENTITY_ISSUER` to pin it. Trust policies reference this URL, so never change it once roles trust it. Sim keeps keyless off when the issuer isn't `https` or names a port, because AWS can't use it.
6262

63-
To rotate the signing key, move the old key to `WORKLOAD_IDENTITY_PREVIOUS_KEY` and set a new `WORKLOAD_IDENTITY_SIGNING_KEY`. Both keys stay published until you remove the old one.
63+
Generate a key with `openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048`.
64+
65+
To rotate the signing key without failed runs:
66+
67+
1. Set the new key as `WORKLOAD_IDENTITY_PREVIOUS_KEY` and redeploy. Sim publishes it but keeps signing with the current key.
68+
2. Wait at least 5 minutes, so every cached copy of the published keys includes the new one.
69+
3. Swap them: set the new key as `WORKLOAD_IDENTITY_SIGNING_KEY` and the old key as `WORKLOAD_IDENTITY_PREVIOUS_KEY`, and redeploy.
70+
4. After 5 more minutes, remove `WORKLOAD_IDENTITY_PREVIOUS_KEY`.
6471

6572
Without a signing key, the keyless option is unavailable and access keys work as usual.
73+
74+
## Troubleshooting
75+
76+
- **`InvalidIdentityToken` or "No OpenIDConnect provider found":** the IAM identity provider's URL or audience doesn't match the dialog exactly, or AWS can't reach the issuer. Open `<issuer>/.well-known/openid-configuration` from outside your network; it must load over `https`.
77+
- **`AccessDenied` when saving:** the role's trust policy doesn't trust this credential's subject. Paste the trust policy from the dialog again; if you chose **Issue a new subject**, update the policy to the new one.
78+
- **"This role trusts every Sim subject":** the trust policy is missing the `sub` condition. Use the policy from the dialog.

‎apps/docs/content/docs/integrations/databricks-credentials.mdx‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,15 +29,20 @@ Every option resolves on Sim's servers at run time; tokens never appear in block
2929
In Sim, add a **Databricks connection**, choose **Service principal (keyless, via Sim OIDC)**, and enter the workspace host and the service principal's **application ID**.
3030
</Step>
3131
<Step>
32-
A Databricks account admin runs the **Federation policy** command from the dialog with the service principal's numeric ID. The policy trusts Sim's issuer, the Sim audience, and this connection's subject only.
32+
A Databricks account admin runs the **Federation policy** command from the dialog with the service principal's numeric ID (find it with `databricks account service-principals list --filter 'applicationId eq "<application ID>"'`, or on the service principal's page in the account console). The policy trusts Sim's issuer, the Sim audience, and this connection's subject only.
3333
</Step>
3434
<Step>
35-
Make sure the service principal can access the workspace and the resources your workflows use, then save. Sim verifies the connection by exchanging a token and reading the service principal's identity.
35+
Make sure the service principal can access the workspace and the resources your workflows use, then save. Sim verifies the connection by exchanging a token and reading the service principal's identity, and refuses a policy that would also trust a different subject, since every Sim workspace shares the issuer and audience.
3636
</Step>
3737
</Steps>
3838

3939
Reconnecting a keyless connection keeps its subject by default, so the federation policy keeps working.
4040

4141
## Self-hosted deployments
4242

43-
Keyless connections need `WORKLOAD_IDENTITY_SIGNING_KEY` set and an issuer Databricks can reach over the public internet. See [AWS credentials](/integrations/aws-credentials#self-hosted-deployments) for the same settings.
43+
Keyless connections need `WORKLOAD_IDENTITY_SIGNING_KEY` set and an `https` issuer Databricks can reach over the public internet. See [AWS credentials](/integrations/aws-credentials#self-hosted-deployments) for the same settings and for rotating the signing key.
44+
45+
## Troubleshooting
46+
47+
- **"Databricks rejected the token request":** the federation policy's issuer, audience, or subject doesn't match the dialog, or the policy is on a different service principal than the application ID you entered.
48+
- **"This service principal's federation policy trusts every Sim subject":** the policy has no `subject`. Recreate it with the command from the dialog.

‎apps/docs/content/docs/platform/self-hosting/environment-variables.mdx‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -167,10 +167,10 @@ Saved AWS and Databricks credentials can trust Sim through OpenID Connect instea
167167
| Variable | Description |
168168
| --- | --- |
169169
| `WORKLOAD_IDENTITY_SIGNING_KEY` | PKCS#8 PEM RSA private key that signs Sim's workload-identity tokens: `openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048` |
170-
| `WORKLOAD_IDENTITY_ISSUER` | Issuer URL clouds trust. Defaults to `<NEXT_PUBLIC_APP_URL>/oidc/workload` and must be reachable from the public internet. Never change it once trust policies reference it. |
171-
| `WORKLOAD_IDENTITY_PREVIOUS_KEY` | During rotation, the retiring key (private or public PEM), still published so tokens it signed keep verifying |
170+
| `WORKLOAD_IDENTITY_ISSUER` | Issuer URL clouds trust. Defaults to `<NEXT_PUBLIC_APP_URL>/oidc/workload`; must be `https` on the default port and reachable from the public internet, or keyless stays off. Never change it once trust policies reference it. |
171+
| `WORKLOAD_IDENTITY_PREVIOUS_KEY` | A second published key (private or public PEM) that never signs: the incoming key before a rotation, or the retiring key after it |
172172

173-
See [AWS credentials](/integrations/aws-credentials#self-hosted-deployments).
173+
See [AWS credentials](/integrations/aws-credentials#self-hosted-deployments) for setup and the rotation steps.
174174

175175
## Access control
176176

‎apps/sim/lib/credentials/databricks-service-account.server.test.ts‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,9 @@ import { encryptionMock, encryptionMockFns } from '@sim/testing/mocks/encryption
88
import { beforeEach, describe, expect, it, vi } from 'vitest'
99

1010
vi.mock('@/lib/core/security/encryption', () => encryptionMock)
11+
vi.mock('@/lib/credentials/workload-identity/issuer', () => ({
12+
mintWorkloadIdentityToken: async ({ subject }: { subject: string }) => `token-for:${subject}`,
13+
}))
1114

1215
import { credential } from '@sim/db/schema'
1316
import {
@@ -56,6 +59,76 @@ describe('saving a Databricks connection', () => {
5659
})
5760
})
5861

62+
describe('saving a keyless Databricks connection', () => {
63+
const HOST = 'dbc-1234abcd-5678.cloud.databricks.com'
64+
const CLIENT_ID = '00000000-0000-0000-0000-000000000000'
65+
const SUBJECT = 'sim:workspace:ws-1:credential:abcdef0123456789'
66+
const fetch = vi.fn()
67+
const config = { authType: 'oidc_federation', host: HOST, clientId: CLIENT_ID }
68+
const reconnectOwner = {
69+
...owner,
70+
storedSecret: { type: 'databricks_credential_v1', config: { ...config, subject: SUBJECT } },
71+
}
72+
73+
/** A service principal whose federation policy pins `subject` to {@link SUBJECT}, or trusts any subject. */
74+
function principalTrusting(pinsSubject: boolean, probeStatus = 401) {
75+
fetch.mockImplementation(async (url: string, init: RequestInit) => {
76+
if (url.endsWith('/oidc/v1/token')) {
77+
const body = new URLSearchParams(String(init.body))
78+
const trusted = !pinsSubject || body.get('subject_token') === `token-for:${SUBJECT}`
79+
return trusted
80+
? jsonResponse({ access_token: 'dbx-token' })
81+
: jsonResponse({ error: 'invalid_grant' }, probeStatus)
82+
}
83+
return jsonResponse({ userName: 'sim-workflows' })
84+
})
85+
}
86+
87+
beforeEach(() => {
88+
fetch.mockReset()
89+
vi.stubGlobal('fetch', fetch)
90+
encryptionMockFns.mockEncryptSecret.mockReset()
91+
encryptionMockFns.mockEncryptSecret.mockResolvedValue({ encrypted: 'ciphertext', iv: 'iv' })
92+
})
93+
94+
it('exchanges this credential’s token as the service principal and saves', async () => {
95+
principalTrusting(true)
96+
97+
await expect(
98+
verifyAndEncryptDatabricksCredential(JSON.stringify(config), reconnectOwner)
99+
).resolves.toMatchObject({ identity: 'sim-workflows' })
100+
const exchange = fetch.mock.calls
101+
.map(([url, init]) => ({ url, body: new URLSearchParams(String(init?.body)) }))
102+
.find(({ body }) => body.get('subject_token') === `token-for:${SUBJECT}`)
103+
expect(exchange?.url).toBe(`https://${HOST}/oidc/v1/token`)
104+
expect(Object.fromEntries(exchange?.body ?? [])).toEqual({
105+
client_id: CLIENT_ID,
106+
subject_token: `token-for:${SUBJECT}`,
107+
subject_token_type: 'urn:ietf:params:oauth:token-type:jwt',
108+
grant_type: 'urn:ietf:params:oauth:grant-type:token-exchange',
109+
scope: 'all-apis',
110+
})
111+
})
112+
113+
it('refuses a federation policy that trusts any Sim subject, so no other workspace can use it', async () => {
114+
principalTrusting(false)
115+
116+
await expect(
117+
verifyAndEncryptDatabricksCredential(JSON.stringify(config), reconnectOwner)
118+
).rejects.toThrow(/every Sim subject/)
119+
expect(encryptionMockFns.mockEncryptSecret).not.toHaveBeenCalled()
120+
})
121+
122+
it('does not save when the any-subject check cannot be completed', async () => {
123+
principalTrusting(true, 503)
124+
125+
await expect(
126+
verifyAndEncryptDatabricksCredential(JSON.stringify(config), reconnectOwner)
127+
).rejects.toThrow(/could not confirm/)
128+
expect(encryptionMockFns.mockEncryptSecret).not.toHaveBeenCalled()
129+
})
130+
})
131+
59132
describe('getting a Databricks token for a run', () => {
60133
const fetch = vi.fn()
61134

‎apps/sim/lib/credentials/databricks-service-account.server.ts‎

Lines changed: 70 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ import {
1717
} from '@/lib/credentials/workload-identity/issuer'
1818
import {
1919
bindWorkloadIdentitySubject,
20+
newWorkloadIdentitySubject,
2021
type WorkloadIdentityCredentialOwner,
2122
WorkloadIdentitySubjectError,
2223
} from '@/lib/credentials/workload-identity/subject'
@@ -223,6 +224,66 @@ function bindSubject(
223224
}
224225
}
225226

227+
/** Proves a configuration by calling the workspace's SCIM `Me` with the resolved token. */
228+
async function verifyWorkspaceIdentity(
229+
config: DatabricksCredentialConfig,
230+
owner: WorkloadIdentityOwner
231+
): Promise<string> {
232+
try {
233+
const token = await resolveDatabricksAccessToken(config, owner)
234+
const response = await fetch(`${workspaceOrigin(config.host)}/api/2.0/preview/scim/v2/Me`, {
235+
headers: { Authorization: `Bearer ${token}`, Accept: 'application/json' },
236+
redirect: 'error',
237+
signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
238+
})
239+
if (!response.ok) throw new Error(`workspace returned HTTP ${response.status}`)
240+
const me = (await response.json().catch(() => ({}))) as {
241+
userName?: unknown
242+
displayName?: unknown
243+
}
244+
return (
245+
(typeof me.userName === 'string' && me.userName) ||
246+
(typeof me.displayName === 'string' && me.displayName) ||
247+
config.host
248+
)
249+
} catch (error) {
250+
throw new DatabricksCredentialError(
251+
`Could not verify the Databricks credential: ${getErrorMessage(error, 'Unknown error')}`
252+
)
253+
}
254+
}
255+
256+
/**
257+
* Sim signs tokens for any service principal a member names, and every Sim
258+
* tenant shares the issuer and audience, so a federation policy that doesn't
259+
* pin `subject` would let any Sim workspace act as the principal. Exchanging a
260+
* subject no credential holds must be refused before the credential is saved.
261+
*/
262+
async function assertPolicyPinsSubject(
263+
config: Extract<DatabricksCredentialConfig, { authType: 'oidc_federation' }>,
264+
owner: WorkloadIdentityOwner
265+
): Promise<void> {
266+
try {
267+
await resolveDatabricksAccessToken(
268+
{ ...config, subject: newWorkloadIdentitySubject(owner) },
269+
owner,
270+
AbortSignal.timeout(REQUEST_TIMEOUT_MS)
271+
)
272+
} catch (error) {
273+
const refused =
274+
error instanceof DatabricksCredentialError &&
275+
error.status !== undefined &&
276+
providerFailureCode(error.status) === 'invalid_credentials'
277+
if (refused) return
278+
throw new DatabricksCredentialError(
279+
`Sim could not confirm the federation policy limits this service principal to this credential: ${getErrorMessage(error, 'Unknown error')}. Try again.`
280+
)
281+
}
282+
throw new DatabricksCredentialError(
283+
`This service principal's federation policy trusts every Sim subject. Use the policy shown in the dialog, whose subject is ${config.subject}, so only this credential can use it.`
284+
)
285+
}
286+
226287
/**
227288
* Validates the submitted configuration, proves it by calling the workspace's
228289
* SCIM `Me` endpoint with the resolved token, and returns the encrypted blob
@@ -247,28 +308,15 @@ export async function verifyAndEncryptDatabricksCredential(
247308
config = submitted
248309
}
249310

250-
let identity: string
251-
try {
252-
const token = await resolveDatabricksAccessToken(config, owner)
253-
const response = await fetch(`${workspaceOrigin(config.host)}/api/2.0/preview/scim/v2/Me`, {
254-
headers: { Authorization: `Bearer ${token}`, Accept: 'application/json' },
255-
redirect: 'error',
256-
signal: AbortSignal.timeout(REQUEST_TIMEOUT_MS),
257-
})
258-
if (!response.ok) throw new Error(`workspace returned HTTP ${response.status}`)
259-
const me = (await response.json().catch(() => ({}))) as {
260-
userName?: unknown
261-
displayName?: unknown
262-
}
263-
identity =
264-
(typeof me.userName === 'string' && me.userName) ||
265-
(typeof me.displayName === 'string' && me.displayName) ||
266-
config.host
267-
} catch (error) {
268-
throw new DatabricksCredentialError(
269-
`Could not verify the Databricks credential: ${getErrorMessage(error, 'Unknown error')}`
270-
)
271-
}
311+
// The subject-pin probe is independent of the verification, so both run at
312+
// once; a verification failure still reports first.
313+
const [verified, pinned] = await Promise.allSettled([
314+
verifyWorkspaceIdentity(config, owner),
315+
config.authType === 'oidc_federation' ? assertPolicyPinsSubject(config, owner) : undefined,
316+
])
317+
if (verified.status === 'rejected') throw verified.reason
318+
if (pinned.status === 'rejected') throw pinned.reason
319+
const identity = verified.value
272320

273321
const { encrypted } = await encryptSecret(
274322
JSON.stringify({ type: DATABRICKS_CREDENTIAL_SECRET_TYPE, config })

‎apps/sim/lib/credentials/workload-identity/issuer.test.ts‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ async function loadIssuer() {
2626
describe('workload identity issuer', () => {
2727
beforeEach(() => {
2828
setEnv('WORKLOAD_IDENTITY_SIGNING_KEY', rsaPem())
29+
setEnv('WORKLOAD_IDENTITY_ISSUER', 'https://sim.example.com/oidc/workload')
2930
})
3031

3132
afterEach(() => {
@@ -101,6 +102,20 @@ describe('workload identity issuer', () => {
101102
expect((await workloadIdentityJwksResponse()).status).toBe(404)
102103
})
103104

105+
it.each(['http://sim.example.com/oidc/workload', 'https://sim.example.com:3000/oidc/workload'])(
106+
'stays off for an issuer no cloud can fetch (%s)',
107+
async (url) => {
108+
setEnv('WORKLOAD_IDENTITY_ISSUER', url)
109+
const issuer = await loadIssuer()
110+
const { workloadIdentityDiscoveryResponse } = await import(
111+
'@/lib/credentials/workload-identity/responses'
112+
)
113+
114+
expect(await issuer.isWorkloadIdentityConfigured()).toBe(false)
115+
expect((await workloadIdentityDiscoveryResponse()).status).toBe(404)
116+
}
117+
)
118+
104119
it('keeps signing with the current key when the retiring key is malformed', async () => {
105120
setEnv('WORKLOAD_IDENTITY_PREVIOUS_KEY', 'not a PEM key')
106121
const issuer = await loadIssuer()

‎apps/sim/lib/credentials/workload-identity/issuer.ts‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,24 @@ async function loadPreviousPublicJwk(): Promise<JWK | null> {
7070
}
7171
}
7272

73+
/**
74+
* AWS and Databricks fetch discovery and keys from the issuer over https on the
75+
* default port, so any other issuer can never be trusted; keyless stays off
76+
* rather than advertising an issuer no relying party accepts.
77+
*/
78+
function assertPublicIssuer(issuer: string): void {
79+
const url = new URL(issuer)
80+
if (url.protocol !== 'https:' || url.port) {
81+
throw new Error('Workload identity issuer must be an https URL without a port')
82+
}
83+
if (!env.WORKLOAD_IDENTITY_ISSUER?.trim()) {
84+
logger.warn(
85+
'Workload identity issuer follows the app URL; set WORKLOAD_IDENTITY_ISSUER so customer trust policies survive a URL change',
86+
{ issuer }
87+
)
88+
}
89+
}
90+
7391
let signingMaterial: Promise<SigningMaterial> | undefined
7492

7593
/**
@@ -82,6 +100,7 @@ function loadSigningMaterial(): Promise<SigningMaterial> {
82100
if (!signingKey) return Promise.reject(new WorkloadIdentityNotConfiguredError())
83101
if (!signingMaterial) {
84102
signingMaterial = (async () => {
103+
assertPublicIssuer(getWorkloadIdentityIssuer())
85104
const privateKey = createPrivateKey(normalizePem(signingKey))
86105
const current = await toPublicJwk(privateKey)
87106
const keys = [current]

0 commit comments

Comments
 (0)