Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ submitqueue/ # repo root (Go module github.com/uber/submi
│ ├── extension/ # Stovepipe-specific extension contracts and implementations
│ └── core/ # Stovepipe-internal queue contracts and shared infrastructure
├── runway/ # Runway domain (single service — the domain *is* the service)
│ └── controller/ # Runway service controllers (consumes the merge queues; no gateway/orchestrator split)
│ └── controller/ # Runway service controllers (consumes the land queues; no gateway/orchestrator split)
├── tool/ # Development and CI tooling
├── service/ # Runnable server/client wiring (entry points + Docker Compose)
│ ├── submitqueue/ # Runnable SubmitQueue servers/clients + Docker Compose
Expand Down Expand Up @@ -113,7 +113,7 @@ func (c *Controller) Process(ctx context.Context, delivery consumer.Delivery) er

Controllers receive `consumer.Delivery` (a subset interface without Ack/Nack) to enforce separation of business logic from queue mechanics. `delivery.Hold(delayMs)` requests delayed redelivery without consuming retry budget; the controller must then return `nil`.

**Queue payloads: IDs within a boundary, full payloads across one.** When producer and consumer share a store (for example SubmitQueue's `build`→`buildsignal` flow), put only the entity **ID** on the queue and reload from storage (the store is the source of truth, messages stay small, redelivery is idempotent). Reloading from storage is what makes the publish ordering load-bearing — see "Persist before you publish" above. When a queue **crosses a service boundary** (for example SubmitQueue validation or merge handing work to Runway), publish the **full payload** the consumer needs, and have the **client own the correlation ID** so it can match the asynchronous result back to the work it is tracking. The queue's **owner defines the wire contract and topic keys** (in its own domain package); the other side imports them.
**Queue payloads: IDs within a boundary, full payloads across one.** When producer and consumer share a store (same service — e.g. `build`→`buildsignal`, `validate`→`landconflict`), put only the entity **ID** on the queue and reload from storage (the store is the source of truth, messages stay small, redelivery is idempotent). Reloading from storage is what makes the publish ordering load-bearing — see "Persist before you publish" above. When a queue **crosses a service boundary** (the consumer cannot read the producer's store — e.g. orchestrator→runway), publish the **full payload** the consumer needs, and have the **client own the correlation ID** so it can match the async result back to the work it is tracking. The queue's **owner defines the wire contract and topic keys** (in its own domain package); the other side imports them.

### Entities

Expand Down Expand Up @@ -163,7 +163,7 @@ When in doubt, ask: *"If the next implementation were DynamoDB / Kafka / Bigtabl
Paths follow the directory layout: shared packages live under `platform/` at the repo root; domain code nests under `submitqueue/`, `stovepipe/`, and other domain folders.

- RPC Controllers: `github.com/uber/submitqueue/{domain}/{service}/controller` (e.g. `.../submitqueue/gateway/controller`; single-service domains drop the `{service}` segment, e.g. `.../runway/controller`)
- Queue Controllers: `github.com/uber/submitqueue/{domain}/{service}/controller/{step}` (single-service: `.../runway/controller/{step}`, e.g. `.../runway/controller/merge`)
- Queue Controllers: `github.com/uber/submitqueue/{domain}/{service}/controller/{step}` (single-service: `.../runway/controller/{step}`, e.g. `.../runway/controller/land`)
- Proto (generated): `github.com/uber/submitqueue/api/{domain}/{service}/protopb` (single-service: `.../api/{domain}/protopb`, e.g. `.../api/runway/protopb`)
- Queue contracts: external `github.com/uber/submitqueue/api/{domain}/messagequeue`; internal `github.com/uber/submitqueue/{domain}/core/messagequeue`
- Domain entities: `github.com/uber/submitqueue/{domain}/entity` (e.g. `.../submitqueue/entity`)
Expand Down Expand Up @@ -198,7 +198,7 @@ To add a new `.proto` to a service, drop it in the service's `api/{domain}/{serv

New queue contracts are defined in **proto3** (`.proto` under `proto/`, generated Go in `protopb/` as the binding) and serialized as **protobuf JSON** (protojson) so the queue keeps storing self-describing JSON. Location follows audience: external/cross-domain contracts go under `api/{domain}/messagequeue/`; internal contracts (used only within the owning domain) go under `{domain}/core/messagequeue/`. Bazel `visibility` enforces the split — internal targets are domain-scoped, `api/` targets are public.

For proto-backed contracts, the message types are generated and the contract package adds generic `protojson` glue — `Marshal(m)` / `Unmarshal[T](b, m)` — owning the wire conventions: `UseProtoNames` (snake_case fields), UPPER_SNAKE enum values, int64-as-string, and unknown fields discarded on read (additive evolution). The topic key(s) carrying a message are declared on the message via the `topic_keys` proto option — a `google.protobuf.MessageOptions` extension defined in `api/base/messagequeue`. A topic key is a stable logical name, not a concrete wire topic; each implementer maps it to its backend's topic name, and a `TopicKeys(msg)` reflection helper reads the option back. It is contract metadata, not the hot path — publish/consume still routes on `consumer.TopicKey` + `TopicRegistry`. The contract package owns both halves: the proto payload and the `TopicKey` constants for its topic keys. A contract test round-trips the payloads and asserts every topic key is bound to exactly one message. Shared field types (`Change`, `Strategy`) are shared protos under `api/base/{change,mergestrategy}`. `api/runway/messagequeue/` and `stovepipe/core/messagequeue/` are current examples.
The message types are generated; the contract package adds only generic `protojson` glue — `Marshal(m)` / `Unmarshal[T](b, m)` — owning the wire conventions: `UseProtoNames` (snake_case fields), UPPER_SNAKE enum values, int64-as-string, unknown fields discarded on read (additive evolution). The topic key(s) carrying a message are declared on the message via the `topic_keys` proto option — a `google.protobuf.MessageOptions` extension defined in `api/base/messagequeue`. A topic key is a stable logical name, not a concrete wire topic; each implementer maps it to its backend's topic name, and a `TopicKeys(msg)` reflection helper reads the option back. It is contract metadata, not the hot path — publish/consume still routes on `consumer.TopicKey` + `TopicRegistry`. The contract package owns both halves: the proto payload and the `TopicKey` constants for its topic keys. A contract test round-trips the payloads and asserts every topic key is bound to exactly one message. Shared field types (`Change`, `Strategy`) are shared protos under `api/base/{change,landstrategy}`. `api/runway/messagequeue/` is the reference example.

SubmitQueue's internal pipeline predates the proto-backed convention. It continues to serialize domain entities with `encoding/json` and declares its logical keys in `submitqueue/core/topickey/`. Do not convert or mix these wire formats incidentally; treat migration as an explicit compatibility change.

Expand Down Expand Up @@ -376,6 +376,6 @@ Errors are classified by origin (user vs infra) and retryability. The framework
**Key rules:**
1. **Non-retryable by default** — a plain `fmt.Errorf(...)` is non-retryable. Retryability is opted into explicitly, but that decision is almost always made by a classifier, not a controller (see rule 4).
2. **Infra by default** — any error not wrapped with `NewUserError` is infra. There is no `NewInfraError`.
3. **Extensions return plain errors** — extension interfaces (`MergeChecker`, `Storage`, `Publisher`) return standard `error` values with their own domain sentinels (e.g. `storage.ErrNotFound`). They do NOT classify errors as user or infra.
3. **Extensions return plain errors** — extension interfaces (`LandChecker`, `Storage`, `Publisher`) return standard `error` values with their own domain sentinels (e.g. `storage.ErrNotFound`). They do NOT classify errors as user or infra.
4. **Classifiers do the bulk of classification; controllers override only with knowledge a classifier lacks** — primary pipeline consumers compose per-backend classifiers into `errs.NewClassifierProcessor(...)`; the processor runs once per chain in the consumer and decides retryability from the raw error. So the common case is a controller returning the raw error (`fmt.Errorf("...: %w", err)`) and letting the classifier verdict stand. Reserve an explicit `errs.New*Error` wrap for the rare case where the controller knows something the classifier cannot infer from the error value alone (e.g. `storage.ErrNotFound` meaning "user asked for a missing resource" *in this call site*). Do **not** wrap a failure as retryable just because replaying it is convenient (e.g. a failed queue publish) — that turns permanent failures into infinite retries instead of dead-lettering. DLQ reconciliation consumers use `errs.AlwaysRetryableProcessor` instead. See [platform/errs/README.md](platform/errs/README.md).
5. **Error chain works end-to-end** — extensions wrap custom errors, controllers wrap with `errs.New*Error`, and `errors.Is`/`errors.As` walks the full chain.
22 changes: 11 additions & 11 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ GOIMPORTS_VERSION ?= v0.33.0
# (the out_dir convention in tool/proto/BUILD.bazel) and copied back here. A
# package may hold multiple .proto files (e.g. an RPC contract plus messagequeue
# contracts); all generated stubs land in the same protopb/ dir.
PROTO_PACKAGES = api/base/change api/base/hook api/base/mergestrategy api/base/messagequeue api/runway/messagequeue api/runway api/submitqueue/gateway api/submitqueue/orchestrator api/stovepipe stovepipe/core/messagequeue
PROTO_PACKAGES = api/base/change api/base/hook api/base/landstrategy api/base/messagequeue api/runway/messagequeue api/runway api/submitqueue/gateway api/submitqueue/orchestrator api/stovepipe stovepipe/core/messagequeue

# Set REPO_ROOT for docker-compose
export REPO_ROOT := $(shell pwd)
Expand All @@ -46,7 +46,7 @@ export REPO_ROOT := $(shell pwd)
# path, so adding a provider is mostly adding a directory — see
# service/submitqueue/demo/provider/README.md.
#
# fake a change is a URI; nothing merges anywhere. Needs nothing.
# fake a change is a URI; nothing lands anywhere. Needs nothing.
# git branches in a bare repository on disk; real fetch, cherry-pick, push.
# github real pull requests. Needs a repository and GITHUB_TOKEN.
PROVIDER ?= fake
Expand All @@ -60,7 +60,7 @@ PROVIDER_COMPOSE_FILE_git = service/submitqueue/docker-compose.git.yml
PROVIDER_COMPOSE_FILE_github = service/submitqueue/docker-compose.provider.yml
PROVIDER_COMPOSE_FILE = $(PROVIDER_COMPOSE_FILE_$(PROVIDER))

# Where PROVIDER=git keeps the bare repository it merges into. Outside the
# Where PROVIDER=git keeps the bare repository it lands into. Outside the
# repository, so a demo leaves nothing in a checkout, and bind-mounted rather
# than kept in a volume so `git log` on the host can show what landed.
#
Expand Down Expand Up @@ -118,7 +118,7 @@ endef
#
# The two have to agree. A change minted for one provider is meaningless to a
# stack wired to another: fake changes point at no repository, so a stack
# running the git merger rejects every one of them as a commit it cannot find,
# running the git lander rejects every one of them as a commit it cannot find,
# and fifty requests fail identically for a reason that is nowhere in the error.
# The stack knows which provider it has — it is mounted at /etc/submitqueue —
# so a run that was not told otherwise asks it rather than guessing.
Expand Down Expand Up @@ -263,10 +263,10 @@ demo-requests: ## Create N changes, enqueue each as it is created, and watch (PR
deps: tidy-go ## Download and tidy Go dependencies
@echo "Dependencies installed!"

e2e-git-test: ## Run the hermetic git E2E (real merger against a bare repo; no credentials)
e2e-git-test: ## Run the hermetic git E2E (real lander against a bare repo; no credentials)
@echo "Running hermetic git end-to-end tests..."
@$(BAZEL) test //test/e2e/submitqueue:go_default_test --test_output=errors \
--test_filter='TestGitMergeE2E'
--test_filter='TestGitLandE2E'

e2e-test: ## Run end-to-end tests (hermetic; Bazel builds all inputs; runs in parallel)
@echo "Running end-to-end tests (parallel)..."
Expand Down Expand Up @@ -403,7 +403,7 @@ local-init-submitqueue-schemas: ## Manually apply all database schemas
@echo "✅ All schemas applied successfully"

local-init-runway-queue-schema: ## Apply queue schema only (mysql-queue) for Runway compose stacks
@echo "Applying queue schema to mysql-queue (Runway; consumes the merge queues)..."
@echo "Applying queue schema to mysql-queue (Runway; consumes the land queues)..."
@for file in platform/extension/messagequeue/mysql/schema/*.sql; do \
echo " - Applying $$(basename $$file)..."; \
docker exec -i $(RUNWAY_LOCAL_PROJECT)-mysql-queue-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
Expand Down Expand Up @@ -494,8 +494,8 @@ local-submitqueue-restart: build-all-linux ## Restart all services (rebuild and

local-submitqueue-start: build-all-linux ## Start full stack (PROVIDER=fake|git|github; github needs GITHUB_TOKEN)
@echo "Starting full stack against provider '$(PROVIDER)' ($(SQ_PROVIDER_CONFIG_DIR))..."
@test -f "$(SQ_PROVIDER_CONFIG_DIR)/merge.yaml" \
|| { echo "No such provider '$(PROVIDER)': $(SQ_PROVIDER_CONFIG_DIR)/merge.yaml not found"; exit 2; }
@test -f "$(SQ_PROVIDER_CONFIG_DIR)/land.yaml" \
|| { echo "No such provider '$(PROVIDER)': $(SQ_PROVIDER_CONFIG_DIR)/land.yaml not found"; exit 2; }
@test -n "$(PROVIDER_COMPOSE_FILE)" \
|| { echo "Provider '$(PROVIDER)' has no compose overlay; add PROVIDER_COMPOSE_FILE_$(PROVIDER) to the Makefile"; exit 2; }
@if [ "$(PROVIDER)" = "git" ]; then \
Expand All @@ -519,7 +519,7 @@ local-submitqueue-start: build-all-linux ## Start full stack (PROVIDER=fake|git|
@echo ""
@echo "Gateway gRPC port: $$(docker port $(SUBMITQUEUE_LOCAL_PROJECT)-gateway-service-1 8080 2>/dev/null | cut -d: -f2 || echo 'unknown')"
@if [ "$(PROVIDER)" = "git" ]; then \
echo "Merge target: $(SQ_GIT_SANDBOX_DIR)/sandbox.git"; \
echo "Land target: $(SQ_GIT_SANDBOX_DIR)/sandbox.git"; \
fi
@echo ""
@echo "Generate traffic with:"
Expand Down Expand Up @@ -579,7 +579,7 @@ local-stovepipe-stop: ## Stop the Stovepipe service

mocks: ## Generate mock files using mockgen
@echo "Generating mocks..."
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/consumergate/... ./platform/extension/hook/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/conflict/... ./submitqueue/extension/speculation/... ./submitqueue/extension/validator/... ./platform/consumer/... ./stovepipe/core/requestlog/... ./stovepipe/extension/storage/... ./stovepipe/extension/sourcecontrol/...
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/consumergate/... ./platform/extension/hook/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/landchecker/... ./runway/extension/lander/... ./submitqueue/extension/conflict/... ./submitqueue/extension/speculation/... ./submitqueue/extension/validator/... ./platform/consumer/... ./stovepipe/core/requestlog/... ./stovepipe/extension/storage/... ./stovepipe/extension/sourcecontrol/...
@echo "Mocks generated successfully!"

proto: ## Generate protobuf files from .proto definitions
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Slack](https://img.shields.io/badge/Slack-join%20the%20community-4A154B?logo=slack&logoColor=white)](https://join.slack.com/t/submitqueue/shared_invite/zt-46gkqj682-7zcQphxm2pYqkjDo9lbmYA)

SubmitQueue is a high-performance speculative merge queue that keeps your trunk consistently green at scale. Rather than validating changes one at a time, SubmitQueue speculatively rebases and validates multiple changes in parallel against predicted future states of HEAD. When validations pass, changes land automatically. When they fail, SubmitQueue isolates the offending change and retries the rest — all without human intervention.
SubmitQueue is a high-performance speculative submission queue that keeps your trunk consistently green at scale. Rather than validating changes one at a time, SubmitQueue speculatively rebases and validates multiple changes in parallel against predicted future states of HEAD. When validations pass, changes land automatically. When they fail, SubmitQueue isolates the offending change and retries the rest — all without human intervention.

Designed for large monorepos and fast-moving teams where concurrent changes can introduce subtle conflicts and destabilize builds.

Expand Down
2 changes: 1 addition & 1 deletion api/base/hook/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ A domain with no versioned entities (Runway holds no durable state of its own) p

## Payload

Shaped per `type` by the domain that publishes it, add-only, and documented by that domain. It must carry the subject's id, and it must carry any fact recorded nowhere else — merge step outcomes, build failure detail — because for those the event is the only durable record.
Shaped per `type` by the domain that publishes it, add-only, and documented by that domain. It must carry the subject's id, and it must carry any fact recorded nowhere else — land step outcomes, build failure detail — because for those the event is the only durable record.

It must **not** be an entity snapshot. A snapshot is stale the moment it is redelivered, it competes with the store as a source of truth, and it drags a domain's schema into a contract shared by every domain. Hooks resolve entities from their stores.

Expand Down
4 changes: 2 additions & 2 deletions api/base/hook/hook_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,9 @@ func TestHookEventRoundTrip(t *testing.T) {
}),
},
"unversioned": {
Id: "runway/merge.completed/queue-a-42/msg-9/0",
Id: "runway/land.completed/queue-a-42/msg-9/0",
Source: "runway",
Type: "merge.completed",
Type: "land.completed",
TimestampMs: 1722800012345,
Payload: mustStruct(t, map[string]any{"request_id": "queue-a/42"}),
},
Expand Down
9 changes: 4 additions & 5 deletions api/base/hook/protopb/hook.pb.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading