[CXH-3600] - Sync metastore privileges held by the workspace admins group - Databricks Connector

README.md

@@ -239,6 +239,9 @@ A metastore exposes its privileges as entitlements (`CREATE_CATALOG`,
 privilege held by a group is expandable, so ConductorOne resolves the group's
 members as holders.
 
+Metastore privileges held by the built-in workspace admins group are synced as
+read-only grants. Databricks does not allow granting them back after revocation.
+
 `MANAGE` and `ALL_PRIVILEGES` are not offered: Databricks rejects both at the
 metastore level.
 

docs/connector.mdx

@@ -46,6 +46,7 @@ The Metastores resource type is turned off by default. Turn it on from the conne
 **Notes on Unity Catalog access in C1:**
 
 - Ownership is shown but cannot be provisioned. An owner that is not a Databricks account identity, such as a system account or a workspace admins group, is not shown at all.
+- Metastore privileges held by a workspace's built-in admins group are shown as read-only grants when that workspace is in the sync scope. Databricks does not allow a revoked privilege to be granted back to this system group, and the group's own membership cannot be changed from the workspace. Manage both in Databricks.
 - **MANAGE** and **ALL_PRIVILEGES** are not offered as entitlements, because Databricks rejects both at the metastore level.
 - Unity Catalog tags are not synced.
 </Note>

docs/docs-info.md

@@ -55,7 +55,9 @@ While developing the connector, please fill out this form. This information is n
 - **Direct grants only.** Databricks inherits privileges downward. The connector reports the assignments stored on the object. The inheritance endpoint does not return principal IDs.
 - **Direct Admin, or the sync fails.** Admin inherited through a group does not count. Databricks then returns HTTP 200 with only the privileges the caller can see. The sync fails instead of reporting that list, so ConductorOne keeps the grants it already has.
 - **Ownership is read-only.** It is shown and cannot be granted or revoked. An owner that is not an account identity (`System user`, `_workspace_admins_…`) is not shown.
+- **Workspace admins discovery.** A RUNNING workspace declares its `admins` group as a child; the metastore resolver skips the same workspaces, so both sides agree on which ones carry the group. The group is named `admins (<workspace name>)`, because Databricks gives every workspace a group with the same name and one metastore can be shared by several of them. The unqualified name stays on the resource profile. The workspace syncer reads no Unity Catalog endpoint, so a tenant without Unity Catalog syncs its workspaces as before. The group's members are emitted only when the account holds them, because workspace-local identities are not synced. Its membership is read-only: an identity-federated workspace answers a membership PATCH with 400 and says the group is managed in the account, so the entitlement is not offered as provisionable. A privilege row whose workspace is still provisioning fails the sync before SCIM, so the missing group is not treated as revoked access. A row whose workspace is cancelling is skipped, because that workspace never becomes RUNNING.
+- **Workspace admins privileges are read-only.** Metastore assignments to `_workspace_admins_<workspace_name>_<workspace_id>` resolve to the existing workspace-scoped `admins` group, including its membership expansion. The alias's numeric workspace ID selects the workspace; the permissions read route need not be that workspace. Allowlisted and excluded workspaces apply. A present `principal_id` must match that workspace's `admins` group; only a name-only assignment looks up `admins` by name. These grants are immutable and Grant/Revoke reject workspace-local groups: Databricks permits revoking these system grants but does not allow granting them back. See [Workspace admin privileges](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges/admin-privileges#workspace-admin-privileges-when-workspaces-are-enabled-for-unity-catalog-automatically).
 - **No `MANAGE` or `ALL_PRIVILEGES` on a metastore.** Databricks rejects both, so they are not offered.
-- **Read through one running workspace.** Privileges are not served on the account plane. The connector reads each metastore through the first running workspace in scope where it can see every privilege. With no running workspace in scope, the metastore is not synced and ConductorOne removes it. A workspace that is still provisioning fails the sync until it is `RUNNING`. An unreadable assignment list fails the sync and ConductorOne keeps the metastore.
+- **Read through one running workspace.** Privileges are not served on the account plane. The connector reads each metastore through the first running workspace in scope where it can see every privilege. With no running workspace in scope, the metastore is not synced and ConductorOne removes it. A workspace that is still provisioning fails the sync until it is `RUNNING`. A workspace being deleted does the same, and the error names it and its status: pruning the metastore while the deletion runs would report it as gone and delete its grants, and the wait ends on its own when the account stops listing the workspace. An unreadable assignment list fails the sync and ConductorOne keeps the metastore.
 - **`workspaces` scopes metastores.** An allowlist that names no workspace on the account fails at credential validation, before any sync.
 - **Not synced yet:** catalogs, schemas, tables, volumes, functions, registered models, external locations, storage credentials, connections, shares, recipients, and tags.

pkg/connector/connector.go

@@ -29,16 +29,22 @@ type Databricks struct {
 
 // ResourceSyncers returns a ResourceSyncerV2 for each resource type that should be synced from the upstream service.
 func (d *Databricks) ResourceSyncers(_ context.Context) []connectorbuilder.ResourceSyncerV2 {
+	// Both sides read one condition so they agree on which workspaces carry the group.
+	syncWorkspaceAdmins := d.willSyncResourceType(metastoreResourceType.Id) &&
+		d.willSyncResourceType(groupResourceType.Id) &&
+		d.willSyncResourceType(workspaceResourceType.Id)
+
 	return []connectorbuilder.ResourceSyncerV2{
 		newAccountBuilder(d.client),
 		newGroupBuilder(d.client),
 		newServicePrincipalBuilder(d.client),
 		newUserBuilder(d.client),
-		newWorkspaceBuilder(d.client, d.workspaces),
+		newWorkspaceBuilder(d.client, d.workspaces, syncWorkspaceAdmins),
 		newRoleBuilder(d.client),
 		newMetastoreBuilder(d.client, d.clientID, configuredWorkspaceSet(d.workspaces), syncPrincipals{
 			groups:            d.willSyncResourceType(groupResourceType.Id),
 			servicePrincipals: d.willSyncResourceType(servicePrincipalResourceType.Id),
+			workspaceGroups:   syncWorkspaceAdmins,
 		}),
 	}
 }

pkg/connector/group_patch_test.go

@@ -302,7 +302,7 @@ func testWorkspaceGroup(t *testing.T) *v2.Resource {
 	r, err := groupResource(context.Background(), &databricks.Group{
 		BaseResponse: databricks.BaseResponse{ID: "g-target"},
 		DisplayName:  "Target Group",
-	}, parent)
+	}, parent, "")
 	if err != nil {
 		t.Fatalf("groupResource: %v", err)
 	}

pkg/connector/groups.go

@@ -40,7 +40,17 @@ func groupResourceId(_ context.Context, groupId string, parentResourceId *v2.Res
 	return strings.Join([]string{parentResourceId.ResourceType, parentResourceId.Resource, groupResourceType.Id, groupId}, "/")
 }
 
-func groupResource(ctx context.Context, group *databricks.Group, parent *v2.ResourceId) (*v2.Resource, error) {
+// workspaceAdminsGroupName is the name Databricks gives the built-in admins
+// group in every workspace. The group is addressed by this name, not by an id,
+// so the listing filter and the alias check must agree on it.
+const workspaceAdminsGroupName = "admins"
+
+// groupResource builds a group resource. workspaceName qualifies the display name of a
+// workspace-scoped group: Databricks names the built-in group "admins" in every
+// workspace, so a metastore shared by several of them would otherwise list each holder
+// of the same privileges under one indistinguishable name. It is empty for an account
+// group. The unqualified name stays in the profile.
+func groupResource(ctx context.Context, group *databricks.Group, parent *v2.ResourceId, workspaceName string) (*v2.Resource, error) {
 	profile := map[string]any{
 		"display_name": group.DisplayName,
 		"group_id":     group.ID,
@@ -59,8 +69,13 @@ func groupResource(ctx context.Context, group *databricks.Group, parent *v2.Reso
 	// under workspaces stays unique for match_baton_id.
 	options = append(options, rs.WithAnnotation(&v2.RawId{Id: groupId}))
 
+	displayName := group.DisplayName
+	if workspaceName != "" {
+		displayName = fmt.Sprintf("%s (%s)", group.DisplayName, workspaceName)
+	}
+
 	resource, err := rs.NewGroupResource(
-		group.DisplayName,
+		displayName,
 		groupResourceType,
 		groupId,
 		nil,
@@ -74,16 +89,40 @@ func groupResource(ctx context.Context, group *databricks.Group, parent *v2.Reso
 	return resource, nil
 }
 
-// List returns all the groups from the database as resource objects.
-// Groups include a GroupTrait because they are the 'shape' of a standard group.
+// workspaceDisplayName returns the name of the workspace a group hangs off, or an empty
+// string for an account group. GetWorkspace reuses the account workspace listing, which
+// the workspace syncer has already made in this sync, so it answers from the cache.
+func (g *groupBuilder) workspaceDisplayName(ctx context.Context, deploymentName string, annos *annotations.Annotations) (string, error) {
+	if deploymentName == "" {
+		return "", nil
+	}
+
+	workspace, rateLimitData, err := g.client.GetWorkspace(ctx, deploymentName)
+	if rateLimitData != nil {
+		annos.WithRateLimiting(rateLimitData)
+	}
+	if err != nil {
+		return "", fmt.Errorf("databricks-connector: failed to read the workspace %s a group hangs off: %w", deploymentName, err)
+	}
+
+	return workspace.Name, nil
+}
+
+// List returns groups as resource objects, with a GroupTrait because they are the
+// 'shape' of a standard group. Under the account it returns every group. Under a
+// workspace parent it returns only the built-in admins group, on a single page:
+// the other workspace-local groups are not synced, and the privileges this one
+// holds on a metastore are what the workspace parent exists for.
 func (g *groupBuilder) List(ctx context.Context, parentResourceID *v2.ResourceId, attr rs.SyncOpAttrs) ([]*v2.Resource, *rs.SyncOpResults, error) {
 	if parentResourceID == nil {
 		return nil, nil, nil
 	}
 
 	workspaceId := ""
+	vars := []databricks.Vars{databricks.NewGroupAttrVars()}
 	if parentResourceID.ResourceType == workspaceResourceType.Id {
 		workspaceId = parentResourceID.Resource
+		vars = append(vars, databricks.NewFilterVars(`displayName eq "`+workspaceAdminsGroupName+`"`))
 	}
 
 	bag, page, err := parsePageToken(attr.PageToken.Token, &v2.ResourceId{ResourceType: groupResourceType.Id})
@@ -94,32 +133,41 @@ func (g *groupBuilder) List(ctx context.Context, parentResourceID *v2.ResourceId
 	groups, total, _, err := g.client.ListGroups(
 		ctx,
 		workspaceId,
-		databricks.NewPaginationVars(page, ResourcesPageSize),
-		databricks.NewGroupAttrVars(),
+		append(vars, databricks.NewPaginationVars(page, ResourcesPageSize))...,
 	)
 	if err != nil {
 		return nil, nil, fmt.Errorf("databricks-connector: failed to list groups: %w", err)
 	}
 
+	annos := annotations.Annotations{}
+	workspaceName, err := g.workspaceDisplayName(ctx, workspaceId, &annos)
+	if err != nil {
+		return nil, &rs.SyncOpResults{Annotations: annos}, err
+	}
+
 	var rv []*v2.Resource
 	for _, group := range groups {
 		gCopy := group
 
-		gr, err := groupResource(ctx, &gCopy, parentResourceID)
+		gr, err := groupResource(ctx, &gCopy, parentResourceID, workspaceName)
 		if err != nil {
 			return nil, nil, err
 		}
 
 		rv = append(rv, gr)
 	}
 
-	token := prepareNextToken(page, len(groups), total)
+	// The workspace listing is filtered to one group, so the total is not a page count.
+	token := ""
+	if workspaceId == "" {
+		token = prepareNextToken(page, len(groups), total)
+	}
 	nextPage, err := bag.NextToken(token)
 	if err != nil {
 		return nil, nil, fmt.Errorf("databricks-connector: failed to create next page token: %w", err)
 	}
 
-	return rv, &rs.SyncOpResults{NextPageToken: nextPage}, nil
+	return rv, &rs.SyncOpResults{NextPageToken: nextPage, Annotations: annos}, nil
 }
 
 // Entitlements return all entitlements relevant to the group.
@@ -136,12 +184,20 @@ func (g *groupBuilder) Entitlements(ctx context.Context, resource *v2.Resource,
 
 	// membership entitlement - for group members
 	memberAssignmentOptions := []ent.EntitlementOption{
-		ent.WithGrantableTo(userResourceType, groupResourceType, servicePrincipalResourceType),
 		ent.WithDisplayName(fmt.Sprintf("%s %s", resource.DisplayName, groupMemberEntitlement)),
 		ent.WithDescription(fmt.Sprintf("%s %s in Databricks", resource.DisplayName, groupMemberEntitlement)),
 	}
+	// An identity-federated workspace rejects a membership PATCH with 400, so offering
+	// the entitlement as grantable would only produce failing requests.
+	if workspaceId == "" {
+		memberAssignmentOptions = append(memberAssignmentOptions,
+			ent.WithGrantableTo(userResourceType, groupResourceType, servicePrincipalResourceType))
+	}
 
 	rv = append(rv, ent.NewAssignmentEntitlement(resource, groupMemberEntitlement, memberAssignmentOptions...))
+	if workspaceId != "" {
+		return rv, nil, nil
+	}
 
 	_, groupId, err := parseResourceId(resource.Id.Resource)
 	if err != nil {
@@ -168,6 +224,53 @@ func (g *groupBuilder) Entitlements(ctx context.Context, resource *v2.Resource,
 	return rv, nil, nil
 }
 
+// splitMemberRef splits a SCIM member ref such as "Users/123" into its type and id.
+func splitMemberRef(ref string) (string, string, error) {
+	memberType, memberID, found := strings.Cut(ref, "/")
+	if !found || memberType == "" || memberID == "" || strings.Contains(memberID, "/") {
+		return "", "", fmt.Errorf("databricks-connector: invalid member format of %s", ref)
+	}
+
+	return memberType, memberID, nil
+}
+
+// accountHeld reports, per member type, which member ids the account holds. It resolves the
+// whole page in one request per type, and a member the account does not hold is absent.
+func (g *groupBuilder) accountHeld(ctx context.Context, members []databricks.Member) (map[string]map[string]struct{}, *v2.RateLimitDescription, error) {
+	idsByType := make(map[string][]string)
+	for _, member := range members {
+		memberType, memberID, err := splitMemberRef(member.Ref)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		switch memberType {
+		case "Users", "Groups", "ServicePrincipals":
+			idsByType[memberType] = append(idsByType[memberType], memberID)
+		default:
+			return nil, nil, fmt.Errorf("databricks-connector: invalid member type: %s", memberType)
+		}
+	}
+
+	held := make(map[string]map[string]struct{}, len(idsByType))
+	var rateLimitData *v2.RateLimitDescription
+
+	for memberType, ids := range idsByType {
+		heldIDs, typeRateLimitData, err := g.client.FindAccountHeldIDs(ctx, memberType, ids)
+		if typeRateLimitData != nil {
+			rateLimitData = typeRateLimitData
+		}
+
+		if err != nil {
+			return nil, rateLimitData, fmt.Errorf("databricks-connector: failed to look up %s on the account: %w", memberType, err)
+		}
+
+		held[memberType] = heldIDs
+	}
+
+	return held, rateLimitData, nil
+}
+
 // Grants return all grants relevant to the group.
 // Databricks Groups have membership and role permissions grants (granting identity resource some permission to this specific group, e.g. group manager).
 func (g *groupBuilder) Grants(ctx context.Context, resource *v2.Resource, attr rs.SyncOpAttrs) ([]*v2.Grant, *rs.SyncOpResults, error) {
@@ -216,22 +319,49 @@ func (g *groupBuilder) Grants(ctx context.Context, resource *v2.Resource, attr r
 		zap.String("cursor", cursor),
 	)
 
+	// Only the account syncs members, so one it does not hold is workspace-local and has no
+	// resource to point a grant at.
+	var held map[string]map[string]struct{}
+	if isWorkspaceGroup {
+		heldByType, rateLimitData, err := g.accountHeld(ctx, members)
+		if rateLimitData != nil {
+			annos.WithRateLimiting(rateLimitData)
+		}
+		if err != nil {
+			return nil, nil, err
+		}
+
+		held = heldByType
+	}
+
 	for _, member := range members {
 		// member.Ref contains the type and ID separated by "/", e.g., "Users/123" or "Groups/456"
-		pp := strings.Split(member.Ref, "/")
-		if len(pp) != 2 {
-			return nil, nil, fmt.Errorf("databricks-connector: invalid member format of %s", member.Ref)
+		memberType, memberID, err := splitMemberRef(member.Ref)
+		if err != nil {
+			return nil, nil, err
 		}
 
-		memberType, memberID := pp[0], pp[1]
 		var resourceId *v2.ResourceId
 		var anns []protoreflect.ProtoMessage
 
+		if isWorkspaceGroup {
+			if _, ok := held[memberType][memberID]; !ok {
+				l.Debug("databricks-connector: skipping a workspace-local member the account does not hold",
+					zap.String("group", resource.Id.Resource),
+					zap.String("member", member.Ref))
+				continue
+			}
+		}
+
 		switch memberType {
 		case "Users":
 			resourceId = &v2.ResourceId{ResourceType: userResourceType.Id, Resource: memberID}
 		case "Groups":
-			rid, expandAnnotation, err := groupGrantExpansion(ctx, memberID, parentId)
+			memberParent := parentId
+			if isWorkspaceGroup {
+				memberParent = &v2.ResourceId{ResourceType: accountResourceType.Id, Resource: g.client.GetAccountId()}
+			}
+			rid, expandAnnotation, err := groupGrantExpansion(ctx, memberID, memberParent)
 			if err != nil {
 				return rv, nil, err
 			}
@@ -252,7 +382,7 @@ func (g *groupBuilder) Grants(ctx context.Context, resource *v2.Resource, attr r
 	}
 
 	// role permissions grants are only emitted with the first page of members
-	if cursor != "" {
+	if cursor != "" || isWorkspaceGroup {
 		return rv, &rs.SyncOpResults{NextPageToken: nextPage, Annotations: annos}, nil
 	}
 
@@ -310,9 +440,14 @@ func (g *groupBuilder) Get(ctx context.Context, resourceId *v2.ResourceId, paren
 		return nil, annos, fmt.Errorf("databricks-connector: failed to get group %s: %w", groupId.Resource, err)
 	}
 
-	resource, err := groupResource(ctx, group, parentResourceId)
+	workspaceName, err := g.workspaceDisplayName(ctx, workspaceId, &annos)
 	if err != nil {
-		return nil, nil, err
+		return nil, annos, err
+	}
+
+	resource, err := groupResource(ctx, group, parentResourceId, workspaceName)
+	if err != nil {
+		return nil, annos, err
 	}
 
 	ctxzap.Extract(ctx).Debug("databricks-connector: Get: refreshed group",
@@ -488,7 +623,6 @@ func (g *groupBuilder) Revoke(ctx context.Context, grant *v2.Grant) (annotations
 	if err != nil {
 		return nil, fmt.Errorf("databricks-connector: failed to parse entitlement resource id: %w", err)
 	}
-
 	if parentResourceId == nil {
 		return nil, fmt.Errorf("databricks-connector: parent resource id not found")
 	}

pkg/connector/groups_test.go

@@ -50,7 +50,7 @@ func testTargetGroupResource(t *testing.T, nativeGroupID string) *v2.Resource {
 		BaseResponse: databricks.BaseResponse{ID: nativeGroupID},
 		DisplayName:  "Target Group",
 	}
-	r, err := groupResource(context.Background(), group, parent)
+	r, err := groupResource(context.Background(), group, parent, "")
 	if err != nil {
 		t.Fatalf("groupResource: %v", err)
 	}
@@ -439,7 +439,7 @@ func newTestGroupBuilder(t *testing.T, members *[]databricks.Member, ruleSetCall
 func testAccountGroup(t *testing.T) *v2.Resource {
 	t.Helper()
 	parent := &v2.ResourceId{ResourceType: accountResourceType.Id, Resource: testAccountId}
-	g, err := groupResource(context.Background(), &databricks.Group{ID: "g1", DisplayName: "all users"}, parent)
+	g, err := groupResource(context.Background(), &databricks.Group{ID: "g1", DisplayName: "all users"}, parent, "")
 	if err != nil {
 		t.Fatalf("groupResource: %v", err)
 	}