Repository navigation
Conversation
Validate new public sources synchronously while preserving operator bans, source revision fencing, admission limits and private-network protection. Persist failed checks with a one-minute retry cooldown.
Reject plaintext sources and redirect downgrades, including persisted insecure hosts, while retaining explicitly gated acceptance fixtures.
Allow operators to bootstrap and serve candidate repository listings from approved sources, with safeguards for credentials and service shutdown.
Add collection dir service
Persist and retry source lifecycle delivery across outages and restarts, with revision fences preventing stale requests from reviving sources. Use the same durable handoff for public and console admission.
Specify Jetstream ownership, replay and policy boundaries, lifecycle signals and honest history guarantees without changing raw Relay feeds.
Preserve policy and provenance across ingestion and recovery, publish only committed events, and expose explicit cursor and lifecycle semantics. Keep ordinary subscriptions and raw Relay streams compatible.
Keep scoped sync replacements intact through bootstrap merge and compaction, and prevent stale inactive-account syncs from stopping acquisition. Reject unavailable legacy replay before streaming and drain prepared batch writes even when a later append fails.
Stop completed recovery receipts from retaining cancellation watchers while preserving retry and shutdown cancellation behavior.
Keep filtered and superseded input from appearing durably appended to callers while preserving accepted event and import validation behavior.
Avoid inverted replay intervals when a collection is removed and re-enabled without intervening events, while retaining genuine gaps.
Return an explicit unavailable response instead of misclassifying an unready policy gate as a persistence failure, without changing policy.
Measure complete uncompressed frames before compression and skip oversized events without interrupting replay or live cursor progress.
Send a bounded terminal error and log replay coordinates when archive reads fail, while keeping cancellation quiet and cursor progress intact.
Preserve optional metadata and null or empty disclosure fields through generated decoding and round trips without changing ordinary frames.
Match the example to the Jetstream event type actually emitted so consumers do not dispatch selected commits as raw Relay events.
Distinguish the outer message from its typed payload and use the emitted info discriminator and policy notice name throughout the contract.
Limit timestamp translation and clamping promises to ordinary clients, and describe selected sequence validation before streaming starts.
Separate upstream coordinates from selected witnessed timestamps and zero-valued nested sequences, directing clients to outer coordinates.
Separate persistence responsibilities while preserving atomic commits, conflict ordering, and legacy recovery validation with regression tests.
The T20 backlog wait watched the Relay source cursor for any advance past -1. Bootstrap has already advanced that cursor, so the wait returned on its first poll and guarded nothing: a lagging Relay could replay too few events to overflow the verify queue and report a false failure. Wait instead until the Relay's revision for the backlog account equals the PDS head. The Relay persists an outbound event before advancing an account's revision, so equality proves the whole backlog is replayable. A timeout reports both revisions, which separates a fixture that never connected from ingestion lag. Also dispatch the t20-recovery assertion before the stream-shaped argument check, which rejected its single state path before the assertion could run. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…tream-repository-recovery # Conflicts: # tests/acceptance/driver/driver.mjs
A recovery request for a DID already owned at another source, PDS or policy coordinate, or one beyond the active-DID capacity, returned 500 persistence_error although nothing was persisted. It now returns the documented 409 state_conflict. The separate sentinel stays, so the coalescer keeps retrying ownership conflicts. A failed latest-revision discovery recorded only the DID. The job-level warning needs a repository attempt failure, and the job usually ends passes later without attempting discovery again, so the cause was never reported. Each seeding pass now emits one warning with the job ID, stage, failed-DID count and the bounded cause of the most recent failure. Document that a booking for a source that is not admitted is rejected without retry, and how to recover those DIDs after admitting it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The acceptance README and the scenario's changeset still said T20 was expected to fail because the recovery path did not exist. This branch adds that path, so both statements were wrong for the release that carries them. Also list T20 with the other run commands and describe its Relay barrier. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
The T20 assertion checked that a recovery job is not published as whole-source coverage only when the coverage response held a row for the source. A missing response, an unrecognised shape or no matching row skipped both checks, so the scenario could pass without that evidence. The source is admitted before the backlog is written, so its row must exist: each of those cases now fails. Split the assertion into one function per piece of evidence. The single function was already over the complexity limit and the new checks would have raised it further. This work was assisted by a machine
…tream-repository-recovery This work was assisted by a machine
Combine Dependabot PRs #84-#89. The Go image bumps are applied to every service Dockerfile at 1.27.3, and the module toolchains move with them, so CI tests on the same compiler the images ship with. The go language line stays at 1.26 to keep Go 1.26 GODEBUG defaults. The Postgres driver 1.6 changes (wrapped translated errors, AutoMigrate default and index handling, DSN timezone) need no code changes: the Relay compares only gorm.ErrRecordNotFound directly, sets no DSN timezone, and its migrations issue no DDL when the new driver runs against a schema created by 1.5.7. jetstream/go.mod also records golang.org/x/time as a direct dependency, as go mod tidy requires. This work was assisted by a machine
…2026-11 Build with Go 1.27.3 and update Go dependencies
…-recovery Recover events shed by the verify queue instead of losing them
Resolve conflicts with main's selected-stream attribution and policy adoption work: - control/handler.go: keep main's sourceRevision and policyRevision job fields alongside this branch's recoveryDidCount. - jobs/jobs.go: adopt main's cancelPolicyJobs refactor and carry over clearing RecoveryFollowupDIDs when a policy change cancels a job. - orchestrator/bootstrap.go and jetstreamd/runtime.go: keep main's attribution and policy-gate wiring and add this branch's DroppedEvents recovery sink. Main made receipt delivery asynchronous, so the recovery receipt test now passes a WaitGroup to flushReceipts and waits for delivery before asserting that no receipt was sent. This work was assisted by a machine
- Report a DID completed at an older revision as ready rather than complete when the listed revision is newer. A retry keeps CompletedRepos while restarting the inventory, and a partial rediscovery never prunes the stale completion, so the runner stopped with "invalid repository retry state". - Retry failed identity lookups in the recovery coalescer for a bounded 60 flushes instead of dropping the DID's recovery intent. An identity without a PDS or an unsupported DID method stays terminal. - Let a full 4,096-DID recovery scope through the Jetstream control API and the administration operation route, and bound each DID at 512 characters so the count and byte limits agree. The larger administration limit applies only to the operation route, after authentication. - Describe the change as current-state repair rather than event recovery, and make the rerun test detect repeated DID discovery. This work was assisted by a machine
…rflow-recovery Prove when a replayed firehose backlog silently loses events
Say which license and copyright cover each part of the repository
…itzer-with-hanken-grotesk-and Replace Switzer with Hanken Grotesk
prepare for release v0.10.0
The Changesets v2 migration set push-git-tags to false, and scripts/release.mjs only tags the runner's checkout, so the v0.10.0 release run created and validated its tag without publishing it. Push the tag after validation, refusing a tag that does not resolve to the checked-out release commit. A plain push never moves an existing tag. This work was assisted by a machine
Deploy v0.10.0 to Staging
A single HTTP 502/503 from the relay during reconnect made atmos end the upstream iterator, and the live consumer returned nil, leaving Jetstream running without ingestion for hours. On restart the stale cursor replayed a large backlog that overflowed the per-DID verify queue (TECH-883). - Retry transient dial rejections (408, 425, 429, 5xx) with backoff and log each successful upstream connection. - Return ErrUpstreamStreamEnded when the iterator ends while the consumer is still running, so the process exits instead of stalling silently. - Raise verify parallelism to 128 (per-DID queue 256) and pace upstream reads at 500 frames/s by default; both are configurable. - Summarise repeated verification failure log lines per DID and class. This work was assisted by a machine
- Mark the changeset minor: it adds three configuration options. - Assert that redials after transient rejections resume from the persisted cursor. - Add hypercerts markers to the remaining upstream-file hunks and drop a pass-through classifier wrapper. This work was assisted by a machine
…-live-ingest-silently-stops-after-a fix(jetstream): keep live ingest running across relay restarts
…d-tag Push the validated release tag from the Release workflow
prepare for release v0.10.1
Deploy v0.10.1 to Staging
Contributor
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the
 |
Ashex
marked this pull request as draft
November 12, 2026 14:51
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes the staged candidate for v0.10.1 (
dev@edf3d859, from tagv0.10.1/b6b64228) to production. Merging starts the promotion workflow, which deploys the exactrelay,jetstreamandadministrationimage digests already validated on staging. Nothing is rebuilt.Production was last promoted on 2026-10-23 (#78), so this covers v0.9.0 through v0.10.1. The merge tree is byte-for-byte identical to
dev(723d98f7), as the promotion workflow requires.Before merging: production prerequisites
Production is not ready to deploy this release yet. Railway's production environment needs environment variables set first, and the services below need configuring. Don't merge until every item is done:
Railway environment variables. Set the variables this release requires on each production service before merging. They are listed per service in
docs/railway.mdunder "Required variables per environment", and the items below call out the ones tied to specific changes.Rainbow and collectiondir: staged in Railway production, not yet committed. The staged change mirrors staging and deploys nothing until it is committed.
collectiondirservice: branchproduction,cmd/collectiondir/Dockerfile, the staging start command, healthcheck/_health, a 50 GB volume at/data,COLLECTIONS_API_LISTEN,COLLECTIONS_METRICS_LISTEN,COLLECTIONS_UPSTREAM=ws://relay.railway.internal:2470andPORT=2510.RAINBOW_COLLECTIONDIR_HOST=http://collectiondir.railway.internal:2510.COLLECTIONS_ADMIN_TOKEN(production-only, at least 32 random bytes) on collectiondir in the Railway dashboard before committing. collectiondir refuses to start without it.productionbranch only contains collectiondir after this merge; committing earlier would fail its build.Relay control link (Clean up configuration of Administration and relay integration for jeystream #97). Relay has
RELAY_CONTROL_ADDR=[::]:2472andHC_RELAY_CONTROL_SECRET. Jetstream has the sameHC_RELAY_CONTROL_SECRETandJETSTREAM_RELAY_CONTROL_URL=http://relay.railway.internal:2472. Administration hasRELAY_CONTROL_URLandJETSTREAM_CONTROL_URL. The full table is indocs/railway.mdunder "Required variables per environment". Without these, collection changes are rejected withpolicy_mirror_unavailable.HTTPS-only PDS sources (Validate public PDS admission synchronously with durable safeguards #79). HTTP and WS sources are now rejected, even for administrators.
RELAY_ALLOW_INSECURE_HOSTS/--allow-insecure-hostsnow fails startup. Make sure production has no plaintext or private-network sources and doesn't set that flag.Public PDS admission (Validate public PDS admission synchronously with durable safeguards #79). Public
requestCrawlcan now validate and admit a new PDS, subject to the daily new-host limit. Confirm this is intended for production.What changes
Jetstream
JETSTREAM_LIVE_VERIFY_PARALLELISM,JETSTREAM_LIVE_UPSTREAM_READ_RATE,JETSTREAM_LIVE_UPSTREAM_READ_BURST.Relay
Administration
Build and release
Staging evidence (v0.10.1, deployed 14:11–14:18 UTC today)
SUCCESS.upstream dial rejected; retrying, and reconnected two seconds later at the same cursor (197219). There were no verify-queue drops or verification failure floods./xrpc/_healthreturns ok. Jetstream/subscribereplayed 409 retained-collection events from the previous four hours.After merge
Confirm each production Railway deployment reaches
SUCCESS. Then check that Jetstream logsupstream connectedand its/statusupstream cursor follows the Relay. Finally, change a collection through Administration to prove the Relay control link (#97).🤖 Generated with Claude Code