Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e2ebdb2
test: migrate fault transport to owned Groupnet buffers
coconutbird Oct 6, 2026
8d5cc2f
feat(sync): managed Groupnet links, explicit gossip neighbors, and sp…
coconutbird Nov 12, 2026
06c11e5
build: pin groupnet to 30407c0bb057a7f12bf1520397f7398b38f97103
coconutbird Nov 12, 2026
0dcc0aa
refactor(sync): follow Groupnet's lease module, fallible publication …
coconutbird Nov 12, 2026
5f6bd1b
refactor(sync): typed seed resolution, recovery config errors and the…
coconutbird Nov 12, 2026
06642fa
refactor(sync): port the fleet bootstrap adapter to Groupnet's typed …
coconutbird Nov 12, 2026
cbb77c1
refactor(sync): satisfy the adapter lints and rustfmt after the Group…
coconutbird Nov 12, 2026
44f59b0
test: port unit and integration fixtures to the Groupnet 30407c0 API
coconutbird Nov 12, 2026
143ef43
test(sync): write the malformed feed head under Groupnet's default fe…
coconutbird Nov 12, 2026
7ba792a
docs: name the single gossip cadence and the new sync submodules
coconutbird Nov 12, 2026
6b577d6
refactor(sync): keep RecoveryConfigRejected in sync::config, not the …
coconutbird Nov 12, 2026
b4665c0
Merge remote-tracking branch 'origin/main' into feat/groupnet-main
coconutbird Nov 12, 2026
0943992
ci: run the test gate on pull requests and run every unit test
coconutbird Nov 12, 2026
4427328
feat(sync): fence strong writes on every peer before the origin call
coconutbird Nov 12, 2026
1039bec
style(sync): rustfmt the pre-write fence
coconutbird Nov 12, 2026
5418463
test(sync): cover the pre-write fence's crash points, stall and lapse
coconutbird Nov 12, 2026
263ded0
test(strong_rw): report acknowledged write latency
coconutbird Nov 12, 2026
c35e1b4
docs: describe the strong pre-write fence and its cost
coconutbird Nov 12, 2026
f2eb517
test(coherence): count the fence and outcome events of each strong write
coconutbird Nov 12, 2026
56a1a7c
docs: count feed events, and describe pending fences across fleet boo…
coconutbird Nov 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,18 @@ on:
- ".dockerignore"
- "deploy/**"
- ".github/workflows/build.yml"
# Pull requests run the test gate only; the image push and the Fleet bump stay on main.
pull_request:
paths:
- "src/**"
- "tests/**"
- "scripts/**"
- "Cargo.toml"
- "Cargo.lock"
- "Dockerfile"
- ".dockerignore"
- "deploy/**"
- ".github/workflows/build.yml"
workflow_dispatch:

permissions:
Expand All @@ -30,7 +42,8 @@ permissions:

jobs:
# Locked Rust lint/test/docs gate plus Helm lint/render. Run the real MinIO tests
# locally. CI runs unit tests and the integration modules that do not need an origin.
# locally. CI runs every unit test and the integration modules that do not need an
# origin. The unit tests run in their own step: a test-name filter would skip them.
test:
runs-on: ubuntu-latest
permissions:
Expand All @@ -44,7 +57,8 @@ jobs:
- uses: Swatinem/rust-cache@v2
- run: cargo fmt --all --check
- run: cargo clippy --locked --workspace --all-targets --all-features
- run: "cargo test --locked --workspace --all-features --lib --bins --test it -- metrics_endpoint:: tier_cache:: startup_readiness::"
- run: cargo test --locked --workspace --all-features --lib --bins
- run: "cargo test --locked --workspace --all-features --test it -- metrics_endpoint:: tier_cache:: startup_readiness::"
- run: cargo doc --locked --workspace --all-features --no-deps
env:
RUSTDOCFLAGS: -D warnings
Expand All @@ -68,6 +82,7 @@ jobs:
# reuse across builds. provenance/sbom off: the attestation stages turn the export
# into a manifest list and break registry cache reuse (build-push-action#820).
server:
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
permissions:
contents: read
Expand Down Expand Up @@ -104,6 +119,7 @@ jobs:
# The tag never moves backwards: an older run that finishes after a newer one keeps
# the newer tag.
bump:
if: github.event_name != 'pull_request'
needs: [test, server]
runs-on: ubuntu-latest
permissions:
Expand Down
21 changes: 17 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,17 +46,21 @@ src/
mod.rs # cache link point: module docs and declarations only
proxy.rs # cache types and core tier/index helpers
service.rs # the S3 service implementation
mutation.rs # write-through paths: mutation tails, and unknown outcomes fenced and reconciled
mutation.rs # write-through paths: the strong pre-write fence, mutation tails, and unknown outcomes fenced and reconciled
tests.rs # cache unit tests
index.rs # in-memory LIST key index
index/pending.rs # peers' announced writes, fenced until settled
tier.rs # hot (moka heap) / warm (mmap disk) tiered object-body cache
sync/
mod.rs # sync link point: module docs and declarations only
coherence.rs # gossip write-feed and coherence core
config.rs # gossip env/config parsing and node construction
coherence/apply.rs # the apply loop: peer writes into the index, acks, gaps and lapses
coherence/fence.rs # the writer's pre-write fence: announce, wait, settle or abandon
config.rs # gossip env/config parsing and managed node construction
volatile.rs # S3 origin facts/effects for Groupnet's bounded recovery driver
wire.rs # write-event and session-token codec
tests.rs # coherence unit tests
tests/fixtures.rs # managed in-memory nodes and wired pairs for those tests
metrics.rs # counters, warm-residency gauges, and periodic stats logging
tests/it/ # one integration-test binary; each module is a suite (links the library crate)
deploy/helm/s3cache/ # Helm chart
Expand Down Expand Up @@ -163,6 +167,11 @@ documentation-only edit does not require an image build.
| `S3CACHE_STATS_SECS` | `60` | Stats log interval |
| `S3CACHE_METRICS_LISTEN` | empty (disabled) | Bind address for the Prometheus text endpoint (`GET /metrics`) |

Gossip uses one managed network: `S3CACHE_GOSSIP_SEEDS` explicitly admits UDP routing
neighbors on both endpoints; DNS and advertisements refresh addresses only. The chart
lists every ordinal. Startup awaits link binding, and planned shutdown closes the
links and resolver after mutation drain and feed seal.

`GET /ready` on the metrics listener is the pod readiness probe
(`CachingProxy::probe_ready`): ready once this node's index is complete, or while
no live (`Alive`/`Suspect`) peer declares the `s3cache:indexed` gossip capability or
Expand Down Expand Up @@ -199,8 +208,12 @@ completion of the initial index and coherence warm-up.

The default `strong` mode is **lease-backed**: a node may answer a LIST, a GET, or an
authoritative 404 from its own state only while it holds an unexpired *serve-lease* its
peers granted (groupnet's `consistency::lease`, tier T3), and a write ends when every
lease-holder has either applied the invalidation (one ack round) or had its lease lapse
peers granted (groupnet's `consistency::lease`, tier T3). A write is **fenced** before
its origin call: it publishes `IndexOp::Pending` and waits until every lease-holder has
applied it or lapsed (`WriteSync::fence`), and each peer answers the key from the origin
until the outcome event settles it (`index::pending`); a writer's next life hands
anything it left open to the origin reconciliation. A write ends when every
lease-holder has either applied the outcome (one ack round) or had its lease lapse
in the writer's engine (bounded by `D`, on the straggler's own clock). That replaces the
old view-stability heuristic (`Stability`/`settled()`), which survives only for
`strong-acks` — deprecated, removed next release. `strong-acks` pins the **mechanism**
Expand Down
90 changes: 77 additions & 13 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ blake3 = "1.8.5"
bytes = { version = "1", features = ["serde"] }
futures = "0.3"
getrandom = "0.4"
groupnet = { git = "https://github.com/napbat/groupnet", rev = "fb97452387e396c9b77e91f15c654c69489e2378", features = ["consistency-leases", "consistency-volatile-recovery", "consistency-volatile-bootstrap-bulk", "dns", "tcp", "udp"] }
groupnet = { git = "https://github.com/napbat/groupnet", rev = "30407c0bb057a7f12bf1520397f7398b38f97103", default-features = false, features = ["runtime", "consistency-leases", "consistency-volatile-recovery", "consistency-volatile-bootstrap-bulk", "dns", "tcp", "udp"] }
http = "1"
http-body-util = "0.1"
# Already under hyper-util/s3s; direct so the metrics endpoint can build its own responses.
Expand Down Expand Up @@ -106,6 +106,8 @@ debug = "line-tables-only"
opt-level = 2

[dev-dependencies]
# Managed in-process control links are test-only; production binds UDP.
groupnet = { workspace = true, features = ["mem"] }
# Real bincode 1 bytes: the fixtures that prove `codec::legacy` still reads what
# earlier releases stored. Never a runtime dependency.
bincode.workspace = true
Expand Down
39 changes: 28 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,8 +102,13 @@ The fast path stays in local memory (LIST is served from RAM; nothing remote is
read path). Loss is **detected, never silent**: a peer that falls behind the feed's ring,
or a peer restart, surfaces as a *gap* — the node distrusts every local body and resyncs
its index from the origin, which is the authority the index caches, so a copy is served
again only once that index has proved it current. Only seeds need static addressing;
every other peer resolves through gossiped advertisements. The ring is bounded by both
again only once that index has proved it current. `S3CACHE_GOSSIP_SEEDS` explicitly
admits routing neighbors: configure both endpoints of every intended link. DNS and
gossiped advertisements refresh admitted neighbors' addresses; discovering another
member does not grant admission. The Helm chart configures every ordinal, including
later-starting peers. Groupnet starts one managed network asynchronously and closes
its links and resolver after the planned mutation drain and feed seal.
The ring is bounded by both
event count and encoded bytes: sustained long-key writes shorten its replay window and
surface the same explicit gap instead of growing one state entry past the transport
envelope and silently arresting peer application.
Expand Down Expand Up @@ -208,13 +213,16 @@ heuristic earlier releases used:
positive, not an absence oracle, and `false` covers everything: booting, warming up,
lapsed, awaiting a resync, a peer gone silent, or a partition. Every one of them sends
the read to the origin: slower, never stale (`unhealthy_bypasses`).
- **Writes end at an ack *or* at a lapse.** A write returns only after the origin acked
it AND every lease-holder either applied the invalidation (the fast path — one
in-cluster ack round, ~2 gossip hops, behind the origin round-trip already paid) or
had its serve-lease **expire in the writer's own engine** (the slow path — bounded by
`D`, counted as `write_lease_lapses`). So `PUT via A; GET via B` is deterministically
fresh, and a peer that stops answering costs one write up to `D` **once**, not an ack
timeout on every write forever.
- **Writes are fenced first, and end at an ack *or* at a lapse.** A write first
announces itself on the write feed and waits until every lease-holder holds that
announcement (or has lapsed); only then is it sent to the origin. It returns only
after the origin acked it AND every lease-holder either applied the outcome (the fast
path — one in-cluster ack round, ~2 gossip hops) or had its serve-lease **expire in
the writer's own engine** (the slow path — bounded by `D`, counted as
`write_lease_lapses`). A write pays that round twice, once on each side of the origin
round-trip. So `PUT via A; GET via B` is deterministically fresh, and a peer that
stops answering costs one write up to `D` **once**, not an ack timeout on every write
forever.
- **That second path is the whole point.** The old mode ended an unacked write in a
*degradation*: the writer proceeded and correctness then depended on the stale peer
*learning* it should stand down — exactly what an asymmetrically-partitioned peer
Expand All @@ -241,7 +249,7 @@ heuristic earlier releases used:
after the lapse (so every wait held against the old one is resolved — read per granter
rather than off the confirmed watermark, which is a *min* over the roster and advances
when the member pinning it merely departs), settles for the fabric's
entry-propagation bound (`2 × anti_entropy_interval`), checks that no peer which was
entry-propagation bound (`2 × gossip_interval`), checks that no peer which was
live at the lapse has vanished from membership since, and then barriers on every peer's
advertised feed head — re-running the vanished check once more afterwards, because the
barrier waits and a reap can land inside that wait. Past the barrier the apply loop has
Expand Down Expand Up @@ -312,6 +320,15 @@ box is the long form):
rule indexes that event skeletally, which also sends the key's reads to the origin.
Every mutation runs on a task of its own, so a client hanging up mid-write cannot
strand an applied DELETE, copy or completion outside the index.
- **A writer that dies mid-write leaves its keys on the origin.** From a write's fence
until its outcome arrives, every peer answers the key, and any LIST whose prefix
covers it, from the origin. If the writer crashes in between, before its origin call
or after the origin applied it, the peers keep doing so until the writer's next life
reaches them (a feed gap or a renewal); each such key is then unknown, and one origin
HEAD resolves it. A fence that does not reach every lease-holder by the deadline is
answered **503** without calling the origin (`ack_timeouts`). A write the origin
refused abandons its fence. A node holding a peer's open fence offers no index image
to a joining node.
- **A feed gap discards the gapped node's index.** The node missed writes it can no
longer order its rows against, so it keeps nothing for a peer-image install to carry
forward; the install or the origin rebuild repopulates it.
Expand Down Expand Up @@ -507,7 +524,7 @@ Unrelated documentation-only edits do not require a container build.
| `S3CACHE_DISK_CACHE` | (empty) | Directory for the warm disk tier; unset = no disk tier |
| `S3CACHE_DISK_CACHE_BYTES` | `10737418240` (10 GB) | Warm disk tier capacity (bytes) |
| `S3CACHE_GOSSIP_BIND` | (empty) | UDP bind for the gossip write feed (e.g. `0.0.0.0:7946`); unset = single-node, no coherence layer |
| `S3CACHE_GOSSIP_SEEDS` | (empty) | Comma-separated seed peers as `id=host:port`; only seeds need static addressing |
| `S3CACHE_GOSSIP_SEEDS` | (empty) | Explicit UDP routing neighbors as comma-separated `id=host:port` pairs; configure both endpoints (self entries are ignored) |
| `S3CACHE_GOSSIP_ADVERTISE` | bind addr | Address peers should dial back (set under NAT/container networking) |
| `S3CACHE_CONSISTENCY` | `strong` | `strong`: lease-backed — a node serves locally only while it holds a serve-lease, and a write ends at an ack or at a lapse. `strong-acks`: the pre-lease ack-only mechanism, **deprecated**, removed next release. `bounded`: ~one-hop freshness, no per-write ack round (large clusters; set uniformly) |
| `S3CACHE_LEASE_MS` | `2000` | Coherence-lease duration `D` (`strong` only): the writer's worst-case stall on a silent peer, each reader's window between confirmations, and the fleet's membership `dead_timeout`. Renewal traffic is one small entry per node per `D/3` |
Expand Down
Loading
Loading