From b81f00e673d0e0e14c99a49a21b77467a32a3599 Mon Sep 17 00:00:00 2001 From: andreolf Date: Mon, 17 Aug 2026 21:12:17 +0200 Subject: [PATCH 1/3] docs: document the two-tier storage and pinning model --- docs/STORAGE-AND-PINNING.md | 68 +++++++++++++++++++++++++++++++++++++ 1 file changed, 68 insertions(+) create mode 100644 docs/STORAGE-AND-PINNING.md diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md new file mode 100644 index 00000000..235108b9 --- /dev/null +++ b/docs/STORAGE-AND-PINNING.md @@ -0,0 +1,68 @@ +# Storage and pinning + +How a gitlawb node stores git objects and keeps them available. This documents +the behavior implemented in `crates/gitlawb-node/src/ipfs_pin.rs` and +`crates/gitlawb-node/src/pinata.rs`; it does not change any behavior. + +## Two-tier model + +After a push lands, new git objects are pinned to up to two independent sinks. +Both are **opt-in and independent** — a node runs fine with neither, either, or +both configured. + +| Tier | Sink | Module | Enabled by | Purpose | +|------|------|--------|-----------|---------| +| Hot | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is itself an IPFS peer | +| Warm | Pinata (Filecoin-backed) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability | + +If a sink's config value is empty, every call into that sink is a no-op — so +leaving `GITLAWB_PINATA_JWT` unset simply disables the warm tier. + +## Configuration + +| Env var | Default | Meaning | +|---------|---------|---------| +| `GITLAWB_IPFS_API` | `""` (disabled) | Base URL of the local Kubo HTTP API, e.g. `http://127.0.0.1:5001` | +| `GITLAWB_PINATA_JWT` | `""` (disabled) | Pinata bearer JWT enabling the warm tier | +| `GITLAWB_PINATA_UPLOAD_URL` | `https://uploads.pinata.cloud/v3/files` | Pinata v3 upload endpoint | +| `GITLAWB_MAX_CONCURRENT_PIN_TASKS` | `8` | Cap on concurrent post-push pin loops across all repos | + +## How pinning runs + +Pinning happens **after** a push is accepted, not on the push's critical path: + +- The **hot** tier pins inline in the post-push encrypt/pin task. +- The **warm** (Pinata) tier runs in a spawned replication tail, so a slow or + unreachable Pinata never blocks the pusher. + +Both tiers share a single global **pin admission semaphore** +(`max_concurrent_pin_tasks`). The pool **defers rather than sheds**: when it is +saturated, a pin loop waits for a slot instead of dropping the pin. Each batch is +bounded by `PIN_BATCH_BUDGET` (120s) so a single large or slow push cannot hold a +slot indefinitely. The Pinata tail re-derives its object list only *after* +acquiring a slot, which bounds outstanding memory to O(refs) rather than +O(pushes × objects). + +De-duplication is per sink: the `pinned_cids` and `pinata_cids` tables record +what each sink already holds, so re-pushing unchanged objects does no redundant +upload. + +## Durability notes + +- With only the **hot** tier, availability depends on the node (and any IPFS + peers that have fetched the CIDs). If the node is down and no peer holds the + objects, they are unreachable until it returns. +- The **warm** tier adds off-node durability via Pinata's Filecoin-backed + storage and makes objects reachable through the public IPFS gateway. +- Running **both** gives a node-local hot copy plus an off-node warm copy. + +> The two sinks are currently invoked as separate call paths. Unifying them +> behind a single pluggable backend interface (to add providers such as direct +> Filecoin deals or self-hosted clusters without touching push logic) is tracked +> separately. + +## See also + +- [RUN-A-NODE.md](RUN-A-NODE.md) — provisioning and running a node +- `crates/gitlawb-node/src/ipfs_pin.rs` — hot-tier implementation +- `crates/gitlawb-node/src/pinata.rs` — warm-tier implementation From 706db57c6528b76f33572ff0a17d87a8ac0d321c Mon Sep 17 00:00:00 2001 From: andreolf Date: Mon, 17 Aug 2026 22:26:22 +0200 Subject: [PATCH 2/3] docs: clarify pin de-duplication is best-effort, not atomic --- docs/STORAGE-AND-PINNING.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md index 235108b9..31e5ea5d 100644 --- a/docs/STORAGE-AND-PINNING.md +++ b/docs/STORAGE-AND-PINNING.md @@ -43,9 +43,11 @@ slot indefinitely. The Pinata tail re-derives its object list only *after* acquiring a slot, which bounds outstanding memory to O(refs) rather than O(pushes × objects). -De-duplication is per sink: the `pinned_cids` and `pinata_cids` tables record -what each sink already holds, so re-pushing unchanged objects does no redundant -upload. +De-duplication is per sink and best-effort: the `pinned_cids` and `pinata_cids` +tables record what each sink already holds, so later pushes normally skip objects +whose successful pin is already recorded. The check-upload-record sequence is not +atomic, so concurrent post-push tasks for the same object, or a failure to record +after a successful upload, can still cause a repeat upload attempt. ## Durability notes From 8a577d3aa320b840f6952641a165047d08c557a7 Mon Sep 17 00:00:00 2001 From: andreolf Date: Tue, 18 Aug 2026 12:39:11 +0200 Subject: [PATCH 3/3] =?UTF-8?q?docs:=20address=20review=20=E2=80=94=20fix?= =?UTF-8?q?=20table/no-op/peer/budget=20claims,=20link=20from=20README?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 1 + docs/STORAGE-AND-PINNING.md | 32 +++++++++++++++++++------------- 2 files changed, 20 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 643992c2..b8f35509 100644 --- a/README.md +++ b/README.md @@ -392,6 +392,7 @@ Read: - [`docs/RUN-A-NODE.md`](docs/RUN-A-NODE.md) - [`docs/ECONOMICS.md`](docs/ECONOMICS.md) +- [`docs/STORAGE-AND-PINNING.md`](docs/STORAGE-AND-PINNING.md) Use a dedicated low-balance operator wallet. Do not use a treasury wallet as the heartbeat key. diff --git a/docs/STORAGE-AND-PINNING.md b/docs/STORAGE-AND-PINNING.md index 31e5ea5d..8c2321f7 100644 --- a/docs/STORAGE-AND-PINNING.md +++ b/docs/STORAGE-AND-PINNING.md @@ -12,11 +12,14 @@ both configured. | Tier | Sink | Module | Enabled by | Purpose | |------|------|--------|-----------|---------| -| Hot | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is itself an IPFS peer | +| Hot | Local Kubo (IPFS) | `ipfs_pin.rs` | `GITLAWB_IPFS_API` set | Node-local availability; the node is an HTTP client of a co-located Kubo daemon | | Warm | Pinata (Filecoin-backed) | `pinata.rs` | `GITLAWB_PINATA_JWT` set | Off-node durability + public IPFS gateway reachability | -If a sink's config value is empty, every call into that sink is a no-op — so -leaving `GITLAWB_PINATA_JWT` unset simply disables the warm tier. +If a sink's config value is empty, its **pin** paths are no-ops — so leaving +`GITLAWB_PINATA_JWT` unset simply disables warm-tier pinning. Note this applies to +pinning only, not reads: the hot-tier read path (`ipfs_pin::cat`) returns an error +rather than a no-op when `GITLAWB_IPFS_API` is unset, so a node that serves the +encrypted-blob read endpoint needs Kubo configured. ## Configuration @@ -38,16 +41,19 @@ Pinning happens **after** a push is accepted, not on the push's critical path: Both tiers share a single global **pin admission semaphore** (`max_concurrent_pin_tasks`). The pool **defers rather than sheds**: when it is saturated, a pin loop waits for a slot instead of dropping the pin. Each batch is -bounded by `PIN_BATCH_BUDGET` (120s) so a single large or slow push cannot hold a -slot indefinitely. The Pinata tail re-derives its object list only *after* -acquiring a slot, which bounds outstanding memory to O(refs) rather than -O(pushes × objects). - -De-duplication is per sink and best-effort: the `pinned_cids` and `pinata_cids` -tables record what each sink already holds, so later pushes normally skip objects -whose successful pin is already recorded. The check-upload-record sequence is not -atomic, so concurrent post-push tasks for the same object, or a failure to record -after a successful upload, can still cause a repeat upload attempt. +bounded by `PIN_BATCH_BUDGET` (120s) so the pin batch itself cannot hold a slot +indefinitely. The Pinata tail re-derives its object list only *after* acquiring a +slot, which bounds outstanding memory to O(refs) rather than O(pushes × objects). +Note the budget covers the pin batch, not the preceding object-list re-derivation +walk: that walk holds the slot too, and bounds only each child git process +individually (no aggregate deadline). + +De-duplication is per sink and best-effort, backed by the single `pinned_cids` +table: the hot tier keys on its `cid`/`sha256_hex` rows and the warm tier on the +nullable `pinata_cid` column, so later pushes normally skip objects whose +successful pin is already recorded. The check-upload-record sequence is not atomic, +so concurrent post-push tasks for the same object, or a failure to record after a +successful upload, can still cause a repeat upload attempt. ## Durability notes