control-plane: Snapshot authorization model and retryable staleness error (1/7) - #3341
Closed
bbartman wants to merge 11 commits into
Closed
control-plane: Snapshot authorization model and retryable staleness error (1/7)#3341bbartman wants to merge 11 commits into
bbartman wants to merge 11 commits into
Conversation
This was referenced Aug 11, 2026
Closed
…able staleness error Adds validation::Error::AuthorizationSnapshotStale and is_authz_snapshot_stale() as the retryable signal that an authorization denial was evaluated against a Snapshot which may not yet reflect a committed grant. Adds the in-process authorization surface to Snapshot: the three-way Authorization enum (Authorized / Denied / Stale) with ok_or_stale(), user_authorization / role_authorization over the snapshot's grants, spec_capabilities() for error-message rendering, and STALE_RETRY_WAKE. Also renames App::snapshot to App::snapshot_watch, ahead of follow-ups where executors pin per-poll Snapshot tokens distinct from the watch itself. This PR is purely additive; nothing consumes the new surface yet. Split 1 of 4 from #3155.
bbartman
force-pushed
the
bmb/2781-stack-1-snapshot-authz
branch
from
August 11, 2026 18:23
cf94b28 to
96ea0b5
Compare
bbartman
changed the title
Contributor
Author
`build:rocksdb` passed PORTABLE=0, which RocksDB reads as `-march=native`, so librocksdb.a -- and every binary linking it -- took its instruction floor from whichever machine compiled it. One master build drew an AVX-512 runner and produced a flowctl that SIGILLs on any host without AVX-512, breaking most estuary/connectors CI jobs. The same library goes into the reactor and control-plane images, three of which deploy automatically from Platform Build. This commit pins haswell on x86_64, the level we had already been shipping: reactor:v0.6.13-11-gd9006db953c has zero AVX-512 instructions under RocksDB symbols, and the same source line compiles to vinserti128 there and vinserti64x2 after. Note: we should avoid x86-64-v3, which lacks the PCLMUL that RocksDB's crc32c fast path needs, and PORTABLE=1, which sets no -march at all. aarch64 adds no -march under either setting, so it is unchanged.
The default was raised from 5m to 20m in d09cb56 (June 2025), but three descriptions of it were left behind: the ShardTemplate doc comment, its generated entry in flow.schema.json, and the hand-written table in the Configuring Task Shards page. No behavior change. The 20 minute default in assemble is intentional and unchanged; only the text describing it is corrected. flow.schema.json is edited in place rather than regenerated, because the released flowctl predates syncSchedule and regenerating with it would drop valid schema. The edited string is byte-identical to what an in-tree regeneration emits.
…eld (#3355) BYOC customers enrolled in AWS's Migration Acceleration Program need a cost-allocation tag (e.g. map-migrated) applied to every AWS resource of their data-plane, including hosts est-dry-dock replaces over time via Pulumi's provider default_tags. data-plane-controller re-serializes data_planes.config into the Pulumi stack config with no serde catch-all, so a field added only on the est-dry-dock/Pydantic side would be silently dropped before Pulumi ever sees it. This adds the Rust side of that field so it survives the round trip; it must ship and deploy before the est-dry-dock change.
…d gotcha (#3272) The firstWriteWins/lastWriteWins page linked ./#reduction-guarantees (itself) instead of ../#reduction-guarantees (the reduction-strategies overview page, where that heading actually lives). With the only broken anchor site-wide now fixed, flip onBrokenAnchors from warn to throw so future broken anchors fail CI instead of shipping silently. Also documents the pre-existing hubspot-real-time redirect collision: it only reproduces on macOS's case-insensitive filesystem and doesn't affect the Linux-based production build, so it's a local-dev note rather than a bug fix.
…3220) Google is removing gsutil from the default gcloud CLI bundle after March 2027; gcloud storage is the supported replacement.
…3233) * docs: clarify sync schedule connector scope and backfill heuristic Sync schedules are used by warehouse connectors AND file/object-store connectors (S3/GCS/Azure Blob file sinks, Iceberg), not just warehouses. Transactional databases (Postgres, MySQL, SQL Server) and streaming/API destinations do not use them. Also document how a connector distinguishes catching-up from backfilling (recent transaction sizes) so the delay-skip behavior during backfills is explicit. * modify language on upload interval connectors --------- Co-authored-by: aeluce <emily@estuary.dev>
…able staleness error Adds validation::Error::AuthorizationSnapshotStale and is_authz_snapshot_stale() as the retryable signal that an authorization denial was evaluated against a Snapshot which may not yet reflect a committed grant. Adds the in-process authorization surface to Snapshot: the three-way Authorization enum (Authorized / Denied / Stale) with ok_or_stale(), user_authorization / role_authorization over the snapshot's grants, spec_capabilities() for error-message rendering, and STALE_RETRY_WAKE. Also renames App::snapshot to App::snapshot_watch, ahead of follow-ups where executors pin per-poll Snapshot tokens distinct from the watch itself. This PR is purely additive; nothing consumes the new surface yet. Split 1 of 4 from #3155.
…flow into bmb/2781-stack-1-snapshot-authz
|
🚀 Preview deployed to https://docs.estuary.dev/pr-preview/pr-3341/ |
7 tasks
Contributor
|
Docs have moved, and this PR straddles the split. The code belongs here; the
Suggested split:
Two notes on the non-doc
Flagging it because the doc half would otherwise merge into a tree nobody Context: migration announcement. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Split 1 of 4 of #3155 (issue #2781): the foundation layer for moving catalog
authorization out of SQL and onto the in-memory authorization
Snapshot.validation: addsError::AuthorizationSnapshotStale { catalog_name }—the retryable signal that an authorization denial was evaluated against a
Snapshot which may not yet reflect a committed grant — and
is_authz_snapshot_stale()to classify it afteranyhowwrapping.control-plane-apiSnapshot authorization model (server/snapshot.rs):Authorizationenum (Authorized/Denied/Stale) withok_or_stale(), which converts a provisional denial into the retryableerror above.
Snapshot::resolve_authorization(),user_authorization(),role_authorization()— grant evaluation against the pinned Snapshot,classified by whether the Snapshot postdates the operation's freshness
anchor (
taken_after, allowingTEMPORAL_SKEW).Snapshot::spec_capabilities()— in-process replacement for theSQL-computed
spec_capabilitiescolumn, used to render "Available grantsare:" error messages.
STALE_RETRY_WAKE, the re-poll interval used by later splits.App::snapshot→App::snapshot_watch, ahead of follow-upswhere executors pin per-poll
Snapshottokens distinct from the watch.This PR is purely additive: nothing consumes the new surface yet.
Tests
Unit tests in
server/snapshot.rs(no DB required):test_resolve_authorization— the three-way classification, including thetemporal-skew boundary.
test_spec_capabilities— insta snapshots of capability rendering againstSnapshot::build_fixture.test_taken_after_allows_for_temporal_skew.Stack