[CXH-2720] - Add Unity Catalog client, access-path routing, and metastore sync - Databricks Connector

.github/workflows/ci.yaml

@@ -8,6 +8,12 @@ on:
 jobs:
   test:
     runs-on: ubuntu-latest
+    # This job grants and revokes one hardcoded group membership on the shared
+    # Databricks tenant, so two PRs running it at once read each other's writes.
+    # The group is deliberately not per-ref: it serializes across every branch.
+    concurrency:
+      group: databricks-sync-test
+      cancel-in-progress: false
     env:
       BATON_LOG_LEVEL: debug
       # Add any environment variables needed to run baton-bitbucket-datacenter

README.md

@@ -5,7 +5,7 @@
 `baton-databricks` is a connector for Databricks built using the
 [Baton SDK](https://github.com/conductorone/baton-sdk). It communicates with the
 Databricks API, to sync data about Databricks identities (users, groups and
-service principals), roles and workspaces.
+service principals), roles, workspaces and Unity Catalog metastores.
 
 Check out [Baton](https://github.com/conductorone/baton) to learn more about the project in general.
 
@@ -30,6 +30,11 @@ the Generate secret button. You can use this secret to authenticate across all
 workspaces that service principal has access to. This requires admin access to
 the Databricks account and each workspace you want to sync.
 
+Syncing Unity Catalog needs more than account admin on its own: the service
+principal also has to be able to read the privileges recorded on each metastore.
+See [Unity Catalog permissions](#unity-catalog-permissions) for the exact
+requirement.
+
 # Using Azure Databricks
 
 To work with Azure Databricks, you need to provide the hostname flag.
@@ -76,12 +81,21 @@ baton resources
 - Service Principals
 - Users
 - Roles
+- Metastores
 
 The connector fetches all resources from the account and every workspace the
 service principal can access. To limit the scope, pass a comma-separated list of
 workspaces to the `--workspaces` flag. Each entry can be a workspace name,
 deployment name, or numeric workspace ID, matched case-insensitively.
 
+`--workspaces` also scopes which metastores sync: only a metastore attached to a
+listed workspace is synced, so narrowing the list removes the other metastores
+and their grants from ConductorOne. A list that names no workspace on the
+account is rejected during credential validation, before any sync runs.
+
+Metastores are Unity Catalog securables and are opt-in: the type stays off until
+it is enabled for the connector in ConductorOne.
+
 ## Authentication
 
 OAuth is the only authentication method: an account-level service principal's
@@ -97,7 +111,9 @@ To instead exclude specific workspaces from the sync, pass them to the
 `--databricks-exclude-workspaces` flag (or the
 `BATON_DATABRICKS_EXCLUDE_WORKSPACES` environment variable) as a comma-separated
 list. Each entry can be a workspace name, deployment name, or numeric workspace
-ID. Excluded workspaces and their roles are skipped entirely.
+ID. Excluded workspaces and their roles are skipped entirely. A metastore
+attached only to excluded workspaces is not synced, and its grants are removed
+from ConductorOne.
 
 ## Incremental sync
 
@@ -188,6 +204,55 @@ membership from a group removes the group's entire workspace assignment,
 including admin access if the group is a workspace admin, so its members lose
 it.
 
+## Unity Catalog
+
+A metastore is the top of the Databricks Unity Catalog hierarchy. The resource
+tree the connector syncs is:
+
+```
+account
+├── workspace
+│   └── role
+├── user
+├── group
+├── service principal
+└── metastore
+```
+
+A metastore exposes its privileges as entitlements (`CREATE_CATALOG`,
+`READ_METADATA`, `USE_SHARE` and so on) plus a non-grantable `owner`. A
+privilege held by a group is expandable, so ConductorOne resolves the group's
+members as holders.
+
+`MANAGE` and `ALL_PRIVILEGES` are not offered: Databricks rejects both at the
+metastore level.
+
+A metastore lives on the account plane, but its privileges are read through a
+workspace attached to it. The connector picks that workspace itself: the first
+running one in scope where it can read every privilege, which it proves by
+owning the metastore or by holding Admin directly on that workspace. Grant and
+Revoke resolve the same route from the metastore id.
+
+A metastore syncs through a running workspace in scope. The sync fails, naming
+the metastore, while an attached workspace in scope is still provisioning, or
+while the metastore's workspace list cannot be read.
+
+## Unity Catalog provisioning
+
+A metastore privilege can be granted to and revoked from a user, group or
+service principal. The `owner` entitlement is read-only: granting or revoking it
+fails with an `InvalidArgument` error.
+
+## Unity Catalog permissions
+
+The service principal must own the metastore, or be assigned **Admin** directly
+on at least one running workspace attached to it. Admin inherited only through a
+group is not enough. Otherwise the sync fails naming the metastore.
+
+The connector requests the `all-apis` OAuth scope. Each resource type declares
+the scopes, roles and privileges it needs, so `./baton-databricks capabilities`
+reports them per type.
+
 # Contributing, Support and Issues
 
 We started Baton because we were tired of taking screenshots and manually