Skip to content

Add experimental RoaringLookupResources to the Materialize API - #175

Merged
josephschorr merged 1 commit into
authzed:mainfrom
josephschorr:experimental-roaring-lookup-resources
Sep 16, 2026
Merged

josephschorr merged 1 commit into
authzed:mainfrom
josephschorr:experimental-roaring-lookup-resources

Conversation

@josephschorr

Copy link
Copy Markdown
Member

What this adds

ExperimentalRoaringLookupResources, a new experimental RPC in authzed.api.materialize.v0: given a resource type, a permission, and a subject, it returns a roaring64 bitmap of the IDs of every resource of that type on which the subject has the permission.

The bitmap is serialized in the RoaringFormatSpec 64-bit portable format, which is what OpenSearch consumes directly via a terms query with "value_type": "bitmap" on long fields (once Base64-encoded), and what Java's Roaring64NavigableMap.deserializePortable() reads. Integrators can attach the permission set to each search query instead of replicating access data into the search index and keeping it in sync.

API shape

  • Request: a standard Consistency block, resource_object_type, permission, and a SubjectReference — one subject per request.
  • Response: the serialized bitmap, its cardinality, and the ZedToken (at_revision) at which the lookup was performed, so callers can thread the same snapshot through subsequent requests.

Constraints, documented on the RPC

  • Every resource object ID of the requested type must parse as a canonical-decimal uint64. Any that does not fails the call with FAILED_PRECONDITION rather than returning a partial bitmap — a bitmap that is quietly too small is a wrong authorization answer. Canonical matters: document:007 and document:7 are distinct objects that both parse to 7.
  • When a permission relates a type to itself (e.g. group#member looked up for a group#member subject), the subject is one of its own resources, so its object ID is held to the same rule.
  • The whole bitmap returns in one unary message. Dense IDs are nearly free, but sparse IDs cost close to 10 bytes each, so large sparse results can exceed the default 4 MiB client-side gRPC receive limit; the doc comment tells callers how to raise it. No server-side result limit is applied.

Naming

Follows the existing convention in this package (ExperimentalCountRelationshipsByFilter in RelationshipsService): the RPC and its messages carry the Experimental prefix; the API is subject to change or removal.

buf lint, buf format, and buf build are clean.

@josephschorr
josephschorr force-pushed the experimental-roaring-lookup-resources branch 2 times, most recently from 22115ca to b495f75 Compare September 15, 2026 17:28
Adds an experimental API returning the resources of a given type on which
a subject has a given permission, as a roaring64 bitmap in the
RoaringFormatSpec 64-bit portable format. That format is consumed directly
by OpenSearch's bitmap terms filtering (value_type: bitmap on long
fields), letting integrators send a compact permission set per query
instead of replicating access data into the search index.

The IDs in the bitmap are the resource object IDs exactly as they appear
in the relationships; no surrogate ID is introduced. For the API to be
usable, every resource object ID of the requested type must be a
canonical decimal integer of at most 44 bits (<= 17592186044415). Any ID
violating that fails the call with FAILED_PRECONDITION rather than
returning a partial bitmap. The response carries the bitmap's cardinality
and the ZedToken at which the lookup was performed.
@josephschorr
josephschorr force-pushed the experimental-roaring-lookup-resources branch from b495f75 to 11008f5 Compare September 15, 2026 17:30

@vroldanbet vroldanbet left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tstirrat15 tstirrat15 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@josephschorr
josephschorr merged commit 3270b08 into authzed:main Sep 16, 2026
3 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants