Repository navigation
Phase 2: production reliability, resilience and observability - #22
Merged
Merged
Conversation
cargo-deny and cargo-audit on dev (after PR #21) reported: - RUSTSEC-2026-0382 rustls-webpki: certificate verification may accept distrusted intermediates → 0.103.16 - RUSTSEC-2026-0381 webpki-roots: bundled roots trust distrusted emSign intermediates → 1.0.11 - RUSTSEC-2026-0386 tokio: AsyncFd constructors violate I/O safety (unsound) → 1.54.1 Lockfile-only; `cargo deny check` and `cargo audit` pass.
… at start The Docker smoke job on dev failed in all-in-one mode: pg_ctl could not start because /var/run/postgresql existed only as a by-product of the package install at build time. CI's BuildKit builder (and any runtime that mounts a fresh tmpfs on /run) does not keep it. Reproduced locally with `--tmpfs /run`. The entrypoint now creates the directory on every start. Non-root (Phase 1 follow-up), preserving both deployment modes: - The server runs as `checkgate` (UID/GID 10001) via setpriv; the bundled PostgreSQL and Redis run as their package users. The entrypoint starts as root only in all-in-one mode, to prepare volumes (existing root-owned Redis volumes are re-owned), and drops privileges for every process. - External mode also runs with `--user 10001:10001` (Kubernetes runAsNonRoot), in which case nothing runs as root. All-in-one mode started as non-root is refused with a message pointing at external mode. - su -c string interpolation replaced by setpriv with argv; a `psql | grep -q` check that could fail under pipefail replaced. - HEALTHCHECK probes /health/live; POSTGRES_* defaults live in the entrypoint instead of image ENV (no credential baked into the image); builder apt uses --no-install-recommends. docker-smoke.sh now runs all-in-one with a fresh /run tmpfs and checks the UID of every service process, external mode under --user 10001:10001, the refusal of non-root all-in-one, /health/live and /health/ready, and the dashboard's security headers. An in-place upgrade from the 28eb233 image on the same volumes keeps data and login.
TRUSTED_PROXY_HEADER=forwarded reads the `for=` parameters of the RFC 7239 Forwarded header from trusted proxies, with the same right-to-left walk as X-Forwarded-For. Only one header is ever honoured: a proxy that sets one usually passes the other through from the client unchanged, so reading both would let clients choose their address. The default stays x-forwarded-for. Quoted values, IPv6 in brackets, ports, and parameter order are handled. Obfuscated identifiers (`_hidden`), `unknown`, a missing `for=`, or a value broken by a comma inside quotes end the walk at the nearest trusted proxy, as a malformed X-Forwarded-For hop already does — a broken header can merge clients into a proxy's budget, never mint a fresh identity.
…and SSE admission control Recovery (server/src/sync.rs) - A server that died between committing a change and publishing it left every other server's SSE clients on the old state indefinitely: Redis Pub/Sub is at-most-once and the only holder of the event (and of the RESYNC retry) was the dead process. Each server now keeps an index of the newest revision it relayed per flag, seeded from PostgreSQL, and every SYNC_RECONCILE_INTERVAL_SECS (default 30, ±20 % jitter) compares it with the committed revisions of all flags and tombstones in one snapshot. A committed version still unrelayed SYNC_RECONCILE_GRACE_SECS (default 10) after it was first noticed makes only that environment's clients re-bootstrap. Suspects are judged by the version first noticed, so busy flags do not false-alarm; events from pre-revision servers adopt the committed revision (rolling upgrades). Also repairs changes made in SQL. - A Redis subscription that stays open but delivers nothing was never noticed. The subscriber now pings every REDIS_PING_INTERVAL_SECS (default 10, timeout <= 5 s) and reconnects + resyncs on failure. Subscribed RESP2 connections answer ["pong", ""], so any reply counts. - Events are parsed once per instance (Arc<SyncEvent> broadcast) instead of once per SSE connection. REDIS_CHANNEL selects the channel. - Migration 28: archive/unarchive keeps the flag's revision. It publishes no event and changes nothing SDKs evaluate; a new revision would read as a lost change. Lifecycle (server/src/lifecycle.rs, main.rs) - Graceful shutdown never finished while an SSE stream was open, so every termination with a connected SDK ended in SIGKILL. SIGTERM now fails readiness, waits SHUTDOWN_DRAIN_DELAY_SECS, closes every stream, cancels the subscriber, reconciler, scheduler, resync retry and limiter eviction, bounds in-flight requests by SHUTDOWN_GRACE_PERIOD_SECS (default 20) and closes the pool. - GET /health/live (never dependency-bound) and GET /health/ready (PostgreSQL required, Redis optional via READY_REQUIRES_REDIS, 503 while draining). /health unchanged. Startup subscribes, then seeds the index, then listens. Pool acquire timeout 10 s. Admission control and backpressure (server/src/stream.rs) - SSE_MAX_CONNECTIONS (default 10 000) and SSE_BOOTSTRAP_CONCURRENCY slots (default half the pool) shared by bootstraps and /flags/snapshot, with a bounded wait; refusals answer 503 with a jittered Retry-After. - Resynced streams close at a random point within SSE_RESYNC_SPREAD_MS (default 5 s) instead of all at once. - TCP_USER_TIMEOUT_SECS (default 60, Linux): a client that stops reading leaves its stream unpolled, so no application timer can fire for it; the kernel now aborts such connections. - The connection guard is created before the stream, so a response dropped unpolled no longer leaks its health-tracking entry. Metrics (server/src/metrics.rs): Prometheus/OpenMetrics on METRICS_ADDR (off by default, never on PORT); route templates, status classes and small enums only as labels; enumerated series pre-created at zero. Static files: the dashboard fallback was added after the layers, so it bypassed the security headers; it now gets them, with a CSP that allows the web fonts index.html loads. Dashboard files stay rate-limit-exempt.
…uites Real server processes against one real PostgreSQL and Redis, real HTTP writes and real SSE streams held by simulated SDK clients that follow the shipped SDKs' revision, staging and reconnect/poll-fallback rules (common/sdk.rs). Faults come from per-instance TCP proxies (common/chaos.rs: refuse, blackhole, strand existing connections, sever) and process kills; each cluster (common/cluster.rs) gets its own database and Redis channel, so the suites run in parallel. Every scenario asserts convergence to the state committed in PostgreSQL within a bounded deadline, records the measured recovery window, and prints/saves instance logs on failure (CHECKGATE_TEST_ARTIFACTS). reliability.rs covers propagation, concurrent writes, update/delete races, cluster-wide and single-instance Redis outages, a silently dead subscription, kill/restart, a publisher killed between commit and publish, a commit with no event, slow and dead readers, bootstraps racing writes, dropped client links and poll fallback, and environment isolation. Against 28eb233 (CHECKGATE_TEST_SERVER_BIN), the silently dead subscription, the publisher crash, the eventless commit and the dead reader never converged. lifecycle.rs covers readiness against database and Redis loss, SIGTERM drain and exit, metrics validity and label hygiene, SSE capacity refusals, dashboard security headers, and that archival and segment edits cause no false resyncs while a real loss resyncs only its environment. The shared harness retries a cluster instance on a fresh port when another process takes its port between free_port() and the bind.
…measured baseline - bench/ (new, unpublished crate): Criterion benchmarks of checkgate-core — boolean lookup at 10/100/1 000/10 000 flags, missing flag, 1 and 10 targeting rules, rollout, weighted multivariate, prerequisite chains, concurrent readers with and without a writer replacing flags, and one SDK event apply — plus an allocation profile from a counting allocator. Kept out of core/ because the React Native packager copies core's manifest without benches/, which broke the packed SDK's native build (found by verify-android-native.sh). Dockerfile copies the new member. - server/examples/loadgen.rs: starts N release servers, then measures SSE ramp-up, held-connection RSS/CPU, write-to-visible propagation on one monotonic clock (no clock skew), fan-out spread, snapshot throughput and REST latency, and reads server-side histogram means from /metrics. - docs/performance: methodology, baseline (i5-9400F; two repetitions; raw JSON and Criterion output in results/), including a synchronous_commit=off run showing commit fsync is most of a write's latency.
… extended workflow - server-integration: also runs tests/lifecycle.rs and tests/reliability.rs (bounded deadlines, ~2 min), uploads instance logs and measured recovery windows as an artifact, has a 30-minute timeout, and pins the PostgreSQL (16.15) and Redis (7.4.11) service images by digest. - reliability.yml (nightly and on demand): repeats the fault-injection suites N times with a recovery-window summary, runs the control-plane load profiles, and runs the Criterion benchmarks, uploading all results. Kept off pull requests so ordinary CI stays fast.
…ment - docs/operations/reliability.md: how changes propagate, every loss mode and its recovery path, the stated guarantee (bounded staleness once PostgreSQL is reachable; not exactly-once), and the test matrix. - docs/operations/observability.md: metrics reference, how to tell healthy, Redis-down, database-down, lagging, resyncing and capacity states apart, alert rules, health endpoints, stable log events. - docs/operations/failure-recovery.md: lifecycle, Kubernetes probes and grace periods, Docker, load-balancer settings (no stickiness needed), capacity limits and overload behaviour, troubleshooting. - Self-hosting and API reference: new settings, health endpoints, 503 refusals, Forwarded header, per-instance rate-limit note; sidebar entries. - CHANGELOG and the Phase 2 engineering report.
akosidencio
marked this pull request as ready for review
October 29, 2026 09:31
akosidencio
added a commit
that referenced
this pull request
Oct 29, 2026
…bility Phase 2: production reliability, resilience and observability
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of the roadmap: multi-instance synchronization correctness, failure recovery, observability, lifecycle, benchmarks. Also fixes the two jobs failing on
devafter #21 (Supply chain, Docker image smoke tests). Full engineering report:PHASE2-PRODUCTION-RELIABILITY.md.Confirmed defects fixed
Each was reproduced against
28eb233first, using a new multi-instance fault-injection suite (real PostgreSQL, real Redis, real server processes, simulated SDK clients, per-instance TCP fault proxies):TCP_USER_TIMEOUT(Linux)/runGuarantee now stated and tested: bounded staleness once PostgreSQL is reachable (≈ 51 s worst case with defaults; milliseconds normally). Not exactly-once.
Added
/health/live,/health/ready(PostgreSQL required, Redis optional, 503 while draining);/healthunchanged.METRICS_ADDRlistener (off by default), low-cardinality labels only.SSE_MAX_CONNECTIONS, bounded bootstrap slots, 503 + jitteredRetry-After), resync spread to avoid reconnect storms.Forwarded(TRUSTED_PROXY_HEADER=forwarded).bench/) and a control-plane load generator; baseline indocs/performance/.docs/operations/(reliability, observability, deployment/recovery).Compatibility
SSE event contract and REST API unchanged (new endpoints only). New
503refusals are handled by the SDKs' existing backoff. Migration 28: archiving no longer bumps a flag's revision. Existing all-in-one volumes upgrade in place (verified).Validation (local)
fmt, workspace clippy
-D warnings,cargo test --workspace --all-featureswith integration env (incl. reliability 14, lifecycle 8), Node/Web/RN/CLI/edge/SSR tests, Node SDK live-SSE e2e, Docker smoke, cargo-deny, cargo-audit, Trivy (0 critical), Miri, RN Android native (4 ABIs), wasm32 build + Node-hosted smoke, docs build.Not run locally: Flutter tests (no toolchain here), browser-hosted wasm tests (none exist in the repo).
Deferred, with reasons in the report (§16)
Redis-backed distributed rate limits, SDK-key-aware rate limits, cross-instance SDK health aggregation.