[CXH-2028] - Support workload identity federation - Databricks Connector

README.md

@@ -18,9 +18,10 @@ right top corner that will open a dropdown menu with the account ID along other
 options.
 
 Another requirement is to have valid credentials to run the connector with. The
-connector authenticates with the OAuth client credentials flow, using an
-account-level service principal that works across the account and every
-workspace it has access to.
+connector authenticates as an account-level service principal that works across
+the account and every workspace it has access to. It can prove it is that
+service principal in two ways: with a client secret, which is the default, or
+with workload identity federation. See [Authentication](#authentication).
 
 To set this up, create a service principal and add an OAuth secret (client ID
 and secret) to it. You can do that by going to the user management tab and
@@ -84,13 +85,53 @@ deployment name, or numeric workspace ID, matched case-insensitively.
 
 ## Authentication
 
-OAuth is the only authentication method: an account-level service principal's
-client ID and secret, supplied through `--databricks-client-id` and
-`--databricks-client-secret`. There is no personal access token (PAT) or
+The connector always authenticates as an account-level service principal, named
+by `--databricks-client-id`. There is no personal access token (PAT) or
 workspace-token option, and no username/password option.
 
-OAuth requires a reachable account API. If the account API check fails at
-startup, the connector fails validation rather than falling back to a
+Exactly one of the following credentials must be supplied. Passing more than one
+is a configuration error.
+
+### Client secret (default)
+
+Pass the service principal's OAuth secret to `--databricks-client-secret` (or
+`BATON_DATABRICKS_CLIENT_SECRET`). The connector runs the OAuth client
+credentials flow against the account token endpoint.
+
+### Workload identity federation
+
+Instead of a long-lived secret, the connector can present an externally issued
+JWT that identifies the workload, such as a SPIFFE JWT-SVID or a Kubernetes
+projected service account token, and exchange it for a short-lived Databricks
+OAuth token ([RFC 8693](https://datatracker.ietf.org/doc/html/rfc8693)).
+
+Create a [service principal federation policy](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-federation-exchange)
+on the same service principal, trusting the issuer, subject and audience of that
+JWT. Then select the method and point the connector at the token:
+
+```
+--auth-method workload-identity-file \
+  --databricks-oidc-token-file /var/run/secrets/token
+```
+
+`--auth-method` (or `BATON_AUTH_METHOD`) is required here. Without it the
+connector validates against the default method and asks for a client secret.
+
+The file is read again on every token refresh, so whatever rotates it controls
+the lifetime and the connector needs no restart.
+
+This method runs the connector yourself: on the command line or in service mode.
+A ConductorOne-hosted connector has no file system you control, so the option is
+not offered there. Hosted support needs ConductorOne to issue the JWT, which is
+tracked in CXE-1557.
+
+Nothing else changes: the exchange happens at the same account token endpoint,
+and the resulting token is used for both account and workspace APIs.
+
+### Validation
+
+Either method requires a reachable account API. If the account API check fails
+at startup, the connector fails validation rather than falling back to a
 workspace-only sync, even when `--workspaces` is set.
 
 To instead exclude specific workspaces from the sync, pass them to the
@@ -101,7 +142,7 @@ ID. Excluded workspaces and their roles are skipped entirely.
 
 ## Group provisioning
 
-Account groups are provisioned through the OAuth client ID and secret flow.
+Account groups are provisioned as the authenticated service principal.
 [Here](https://docs.databricks.com/aws/en/admin/users-groups/groups#:~:text=Types%20of%20groups%20in%20Databricks,permissions%20to%20identity%20federated%20workspaces.)
 are the different types of groups in Databricks.
 
@@ -156,6 +197,7 @@ Flags:
       --databricks-client-id string                      required: The Databricks service principal's client ID used to connect to the Databricks Account and Workspace API ($BATON_DATABRICKS_CLIENT_ID)
       --databricks-client-secret string                  required: The Databricks service principal's client secret used to connect to the Databricks Account and Workspace API ($BATON_DATABRICKS_CLIENT_SECRET)
       --databricks-exclude-workspaces strings            Workspaces to exclude from sync, identified by workspace name, deployment name, or numeric workspace ID. Mutually exclusive with workspaces. ($BATON_DATABRICKS_EXCLUDE_WORKSPACES)
+      --databricks-oidc-token-file string                required: Path to a file holding an externally issued JWT identifying the workload. The file is read again on every token refresh, so an external rotation is picked up without a restart. ($BATON_DATABRICKS_OIDC_TOKEN_FILE)
       --external-resource-c1z string                     The path to the c1z file to sync external baton resources with ($BATON_EXTERNAL_RESOURCE_C1Z)
       --external-resource-entitlement-id-filter string   The entitlement that external users, groups must have access to sync external baton resources ($BATON_EXTERNAL_RESOURCE_ENTITLEMENT_ID_FILTER)
       --external-resource-traits strings                 Resource type traits (e.g. "user", "group", "app") to sync and match from the external resource c1z. When unset the matcher falls back to user and group; passing this flag replaces the full set rather than adding to it. ($BATON_EXTERNAL_RESOURCE_TRAITS)

config_schema.json

@@ -164,5 +164,23 @@
   ],
   "displayName": "Databricks",
   "helpUrl": "/docs/baton/databricks",
-  "iconUrl": "/static/app-icons/databricks.svg"
+  "iconUrl": "/static/app-icons/databricks.svg",
+  "fieldGroups": [
+    {
+      "name": "client-secret",
+      "displayName": "Client Secret",
+      "helpText": "Authenticate with the Databricks service principal's OAuth client ID and client secret.",
+      "fields": [
+        "account-hostname",
+        "account-id",
+        "databricks-client-id",
+        "hostname",
+        "workspaces",
+        "base-url",
+        "databricks-exclude-workspaces",
+        "databricks-client-secret"
+      ],
+      "default": true
+    }
+  ]
 }

docs/connector.mdx

@@ -29,7 +29,16 @@ Revoking the workspace **member** entitlement removes the principal's entire wor
 
 ## Authentication methods
 
-The connector authenticates with **OAuth** — an account-level service principal's client ID and secret. This is the only authentication method the connector supports.
+The connector authenticates as an account-level service principal, named by its client ID. It proves it is that service principal in one of two ways:
+
+- **OAuth client secret** (default): the service principal's client ID and secret.
+- **Workload identity federation**: the connector presents an externally issued JWT that identifies the workload, such as a SPIFFE JWT-SVID or a Kubernetes projected service account token, and Databricks exchanges it for a short-lived token. No long-lived secret is stored, and the Databricks administrator controls access through a service principal federation policy. This fits a deployment where the platform running the connector already issues it a workload identity token.
+
+The authentication method selects the credential. There is no personal access token (PAT), workspace token, or username and password option.
+
+<Warning>
+Workload identity federation is available when you run the connector yourself, on the command line or in service mode. The connector reads the JWT from a file, and a ConductorOne-hosted connector has no file system you control, so the option is not offered there. Select it with `--auth-method workload-identity-file`.
+</Warning>
 
 ## Gather Databricks credentials
 
@@ -55,7 +64,7 @@ A user with the **Account admin** role in each Databricks workspace you want to
 
 ### Generate Databricks credentials
 
-The Databricks connector authenticates with OAuth, which syncs info from all Databricks workspaces the service principal can access.
+The Databricks connector authenticates as a service principal, which syncs info from all Databricks workspaces that service principal can access.
 
 <Steps>
   <Step>
@@ -72,6 +81,24 @@ The Databricks connector authenticates with OAuth, which syncs info from all Dat
 - OAuth client ID
 - OAuth client secret
 
+### Use workload identity federation instead of a secret
+
+You can skip the OAuth secret and have the connector present an externally issued JWT instead. The connector must be able to read that JWT where it runs, so this suits a deployment whose platform injects a workload identity token.
+
+<Steps>
+  <Step>
+  Create the service principal as described above, but do not generate an OAuth secret.
+  </Step>
+  <Step>
+  Follow the [Databricks OAuth token federation documentation](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-federation-exchange) to add a service principal federation policy that trusts the issuer, subject and audience of the JWT your workload receives.
+  </Step>
+  <Step>
+  Make the JWT available to the connector, ideally as a file that your platform rotates.
+  </Step>
+</Steps>
+
+**Done.** In this case the credentials are the account ID, the OAuth client ID, and the JWT, in place of the OAuth client secret.
+
 Next, move on to the instructions for your chosen setup method.
 
 ## Configure the Databricks connector
@@ -204,6 +231,18 @@ stringData:
   BATON_DATABRICKS_CLIENT_ID: <OAuth client ID>
   BATON_DATABRICKS_CLIENT_SECRET: <OAuth client secret>
 
+  # Alternative to BATON_DATABRICKS_CLIENT_SECRET: workload identity federation.
+  # Both variables below are required together. Without BATON_AUTH_METHOD the
+  # connector validates against the client secret method and fails on the
+  # missing secret.
+  # Point the file at a projected service account token, or any other externally
+  # issued JWT your federation policy trusts. It is read again on every token
+  # refresh, so a rotated token needs no restart.
+  # Nothing creates this file for you: uncomment the matching volume and
+  # volumeMount in the Deployment below, or mount the JWT there yourself.
+  # BATON_AUTH_METHOD: workload-identity-file
+  # BATON_DATABRICKS_OIDC_TOKEN_FILE: /var/run/secrets/databricks/token
+
   # Optional: limit the sync to specific workspaces
   # (workspace name, deployment name, or numeric ID).
   # Mutually exclusive with BATON_DATABRICKS_EXCLUDE_WORKSPACES — set one or the other, never both.
@@ -223,7 +262,7 @@ stringData:
 </Warning>
 
 <Note>
-OAuth requires a reachable account API. If the account API check fails at startup, the connector fails validation instead of falling back to a workspace-only sync, even when `BATON_WORKSPACES` is set.
+Both credential options require a reachable account API. If the account API check fails at startup, the connector fails validation instead of falling back to a workspace-only sync, even when `BATON_WORKSPACES` is set.
 </Note>
 
 See the connector's README or run `--help` to see all available configuration flags and environment variables.
@@ -259,6 +298,21 @@ spec:
         envFrom:
         - secretRef:
             name: baton-databricks-secrets
+        # Only for BATON_DATABRICKS_OIDC_TOKEN_FILE: mount the JWT at the path the
+        # secret points to. Leave this out when you use the client secret.
+        # volumeMounts:
+        # - name: databricks-oidc-token
+        #   mountPath: /var/run/secrets/databricks
+        #   readOnly: true
+      # volumes:
+      # - name: databricks-oidc-token
+      #   projected:
+      #     sources:
+      #     - serviceAccountToken:
+      #         path: token
+      #         # Must match the audience of the Databricks federation policy.
+      #         audience: <federation policy audience>
+      #         expirationSeconds: 3600
 ```
 
 ### Step 3: Deploy the connector

docs/docs-info.md

@@ -17,28 +17,31 @@ While developing the connector, please fill out this form. This information is n
 
 1. What credentials or information are needed to set up the connector? (For example, API key, client ID and secret, domain, etc.)
 
-   > The connector requires a Databricks account ID and OAuth credentials:
+   > The connector requires a Databricks account ID and a service principal OAuth client ID. It syncs the account and every workspace that service principal can access. The service principal is then authenticated in one of two ways:
    >
-   > - **OAuth**: a service principal OAuth client ID and client secret. Syncs the account and every workspace the service principal can access. This is the connector's only authentication method.
+   > - **OAuth client secret** (default): the service principal's OAuth client secret.
+   > - **Workload identity federation**: an externally issued JWT identifying the workload, such as a SPIFFE JWT-SVID or a Kubernetes projected service account token, read from a file. The connector exchanges it for a short-lived Databricks token, so no long-lived secret is stored. This requires a service principal federation policy in Databricks. It is available only when you run the connector yourself, on the command line or in service mode, and is selected with `--auth-method workload-identity-file`.
+   >
+   > The authentication method selects the credential.
    >
    > Google Cloud Platform and Azure Databricks customers also provide the account hostname and hostname.
 
 2. For each item in the list above:
-
-   * How does a user create or look up that credential or info? Please include links to (non-gated) documentation, screenshots (of the UI or of gated docs), or a video of the process.
+   - How does a user create or look up that credential or info? Please include links to (non-gated) documentation, screenshots (of the UI or of gated docs), or a video of the process.
 
      > - **Account ID**: in the Databricks account console, open the menu next to your username in the upper-right corner; the account ID is shown there.
      > - **OAuth client ID and secret**: follow the [Databricks OAuth (M2M) documentation](https://docs.databricks.com/en/dev-tools/auth/oauth-m2m.html) to create a service principal and generate an OAuth secret.
+     > - **Workload identity federation**: follow the [Databricks OAuth token federation documentation](https://docs.databricks.com/aws/en/dev-tools/auth/oauth-federation-exchange) to add a federation policy to that same service principal. The JWT itself comes from the workload's own platform, not from Databricks.
      > - **Deployment name**: the subdomain in the workspace URL (not the numeric workspace ID).
 
-   * Does the credential need any specific scopes or permissions? If so, list them here.
+   - Does the credential need any specific scopes or permissions? If so, list them here.
 
      > The credential must have admin access to each resource it reads or writes: account-admin on the Databricks account (for account-level sync and provisioning) and workspace-admin on each workspace being synced.
 
-   * If applicable: Is the list of scopes or permissions different to sync (read) versus provision (read-write)? If so, list the difference here.
+   - If applicable: Is the list of scopes or permissions different to sync (read) versus provision (read-write)? If so, list the difference here.
 
      > No separate scopes: Databricks admin access covers both read (sync) and read-write (provision).
 
-   * What level of access or permissions does the user need in order to create the credentials? (For example, must be a super administrator, must have access to the admin console, etc.)
+   - What level of access or permissions does the user need in order to create the credentials? (For example, must be a super administrator, must have access to the admin console, etc.)
 
      > Account admin access to the Databricks account console (to create the service principal and OAuth secret), and workspace admin on each workspace being synced.

pkg/config/config.go

@@ -4,6 +4,13 @@ import (
 	"github.com/conductorone/baton-sdk/pkg/field"
 )
 
+// Auth method names. The connector branches on the selected one, so the group
+// definitions and that branch read the same constant.
+const (
+	ClientSecretAuthMethod         = "client-secret"
+	WorkloadIdentityFileAuthMethod = "workload-identity-file"
+)
+
 var (
 	AccountIdField = field.StringField(
 		"account-id",
@@ -24,6 +31,14 @@ var (
 		field.WithRequired(true),
 		field.WithDisplayName("OAuth2 Client Secret"),
 	)
+	DatabricksOIDCTokenFileField = field.StringField(
+		"databricks-oidc-token-file",
+		field.WithDescription("Path to a file holding an externally issued JWT identifying the workload. "+
+			"The file is read again on every token refresh, so an external rotation is picked up without a restart."),
+		field.WithDisplayName("Workload Identity Token File"),
+		field.WithRequired(true),
+		field.WithExportTarget(field.ExportTargetCLIOnly),
+	)
 	WorkspacesField = field.StringSliceField(
 		"workspaces",
 		field.WithDescription(
@@ -54,11 +69,46 @@ var (
 		field.WithDescription("Workspaces to exclude from sync, identified by workspace name, deployment name, or numeric workspace ID. Mutually exclusive with workspaces."),
 		field.WithDisplayName("Exclude Workspaces"),
 	)
+	// commonFields apply whichever way the connector authenticates. Validation
+	// skips any field the selected group does not list, so every group repeats
+	// them: account-id is required and would stop being checked otherwise.
+	commonFields = []field.SchemaField{
+		AccountHostnameField,
+		AccountIdField,
+		DatabricksClientIdField,
+		HostnameField,
+		WorkspacesField,
+		BaseURLField,
+		ExcludeWorkspacesField,
+	}
+
+	// The client secret is the default. The token file exchanges an externally
+	// issued JWT, and is read again on every exchange, so a rotated file keeps
+	// working without a restart.
+	fieldGroups = []field.SchemaFieldGroup{
+		{
+			Name:        ClientSecretAuthMethod,
+			DisplayName: "Client Secret",
+			HelpText:    "Authenticate with the Databricks service principal's OAuth client ID and client secret.",
+			Default:     true,
+			Fields:      append(append([]field.SchemaField{}, commonFields...), DatabricksClientSecretField),
+		},
+		{
+			Name:        WorkloadIdentityFileAuthMethod,
+			DisplayName: "Workload Identity (token file)",
+			HelpText: "Exchange an externally issued JWT for a Databricks token. " +
+				"The JWT is read from a file on every exchange, so whatever writes that file controls the lifetime.",
+			ExportTarget: field.ExportTargetCLIOnly,
+			Fields:       append(append([]field.SchemaField{}, commonFields...), DatabricksOIDCTokenFileField),
+		},
+	}
+
 	configFields = []field.SchemaField{
 		AccountHostnameField,
 		AccountIdField,
 		DatabricksClientIdField,
 		DatabricksClientSecretField,
+		DatabricksOIDCTokenFileField,
 		HostnameField,
 		WorkspacesField,
 		BaseURLField,
@@ -72,6 +122,7 @@ var Config = field.NewConfiguration(
 	field.WithConnectorDisplayName("Databricks"),
 	field.WithHelpUrl("/docs/baton/databricks"),
 	field.WithIconUrl("/static/app-icons/databricks.svg"),
+	field.WithFieldGroups(fieldGroups),
 	field.WithConstraints(
 		field.FieldsMutuallyExclusive(WorkspacesField, ExcludeWorkspacesField),
 	),