Grid design: record what building it changed - #60
Merged
Conversation
…hat shipped Building the coordination substrate settled two questions the design left open and found a third the design had answered wrongly. §12 records all three, and each is amended in place where it was originally stated, so a reader who lands mid-document is not told the superseded thing. - **The unit state machine cannot express replication.** §3 gives the work unit AVAILABLE → LEASED → SUBMITTED → CANONICAL and §4.2 flips it to LEASED on claim. Two paragraphs below that diagram the same document sets `required_replicas` to 2 — so a unit routinely needs a second independent result while a first node holds a lease, and under the sketch that second node can never get it. LEASED would have to mean "…and also still claimable", which is not a state. What shipped keeps only the genuinely exclusive milestones and derives claimability from the lease and submission rows, so no replica counter exists to drift. - **`mt_terminal` is out of the agreement test** (§11.3, option 1). The digest demanded an exact match for a value `analyze_biosample` declines to assign because it "is not final on CHM13" — and the Grid realigns to CHM13. It is still published in the records; it just no longer gates canonicalization. This keeps a research question off an infrastructure milestone's critical path. - **Units are per ENA sample**, resolving §9's first open question. It is the grain Navigator analyses at, the grain consensus haplogroups need, and the grain `crawl_project.rs` already groups runs into. D3 is amended too: P1 now carries both data kinds, because the staging existed only to retire an aligner risk that stopped existing when realignment shipped pure-Rust. P2 is kept as a heading so references resolve. §12 also states plainly that the six integration tests compile but have not been run — the dev host has no reachable Postgres — and that the claim SQL is therefore unverified where it matters most. That belongs in the document rather than only in a conversation, because the next person to build on it needs to know. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
…n the credit formula §12.4. Two things worth recording from building `grid-curate`. The design budgeted an ENA integration for §4.5. It did not need one: `crawl_project` had already resolved every run at file level, so curation is a projection of existing tables and makes no network calls. That also delivers §6.2's ENA fair-use control for free — a node gets a finished manifest and never discovers files for itself. And `est_bases` is NULL for essentially every unit, because the crawl leaves `read_length` unset and ENA's `base_count` is not among the fields it requests. The per-Gbp term of the credit formula therefore has nothing to weigh a FASTQ unit by. That is recorded as a gap with its fix rather than papered over with a byte-derived estimate: a fabricated number in a ledger that pays people is worse than a null. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
…amation §12.5, and §4.3 corrected in place so a reader landing there is not told the superseded thing. Two of §4.3's four bullets described behaviour the code does not have. The reaper does not return a freed unit to the pool — expiry alone does, because `claim` ignores any lease past its bound, so the catalogue keeps flowing even while the reaper is down. And the heartbeat does not extend `expires_at`, because a node that can heartbeat but never finish would then hold a unit forever, which is the exact failure a bounded lease exists to prevent. What the reaper is actually for: recording the EXPIRED outcome that trust tiering needs, and letting the same node retry after overrunning. The "have not been run" caveat is gone — all ten pass. §12.5 keeps the local verification recipe, including that the Apple `container` published port is broken and the vmnet address must be re-derived after each recreate, since that cost two failed runs to work out. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
…exist §12.6, plus §5.2 amended in place. The digest's bucketing moves to the server. As written, the node submitted values already bucketed — which makes the bucket function a cross-repo contract, so a Navigator release rounding differently from the AppView would produce unexplained DIVERGENT verdicts against honest nodes. This session has already spent two PRs on contracts drifting between repos; the cheapest such contract is the one that never exists. The spot-check needed no SHADOW state and no schema column either: a lone trusted submission that draws the 5% simply does not canonicalize yet, so the shadow arrives through the ordinary claim path. Also recorded: trust derives from grid work and never from social reputation, since those are different claims; a contested unit blames nobody, because two conflicting clusters are no evidence about which is wrong; and two independent contributors agreeing suffice whatever their tier, or a young fleet with no trusted nodes could never canonicalize anything. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
§12.7, with §4.4 amended in place for the three changes: /grid/node/heartbeat is redundant with an idempotent register, /grid/heartbeat does not extend a lease, and a signed /grid/mine was added because every other endpoint either mutates or is public — leaving messages::poll uncalled and §7.1's Grid panel with no source for the contributor's own standing. The point worth keeping is why submit recomputes the digest hash instead of trusting the one it was handed: a signature over a hash proves only that the signer knew the hash, so without recomputation a node could sign a good result and post a different one and still audit clean. Also records that the est_bases gap is now load-bearing rather than theoretical — grid-validate already pays a per-Gbp term, and with est_bases null a 90 Gbp realignment earns exactly what a passthrough does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
§12.8. ENA had been publishing base_count on read_run the whole time; RUN_FIELDS never asked for it. Migration 0076 gives it a typed column rather than a JSONB provenance slot, because a missing key in a blob reads identically to a zero and this number decides what a contributor is paid. Records why old rows needed a backfill job rather than a re-crawl (ingest_libraries is idempotent at sample granularity, which is worth keeping), and the two ledger safety properties: the backfill only fills a NULL, and an empty figure from ENA stays NULL rather than becoming a zero. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The design-doc half of the Grid work. Code is in
decodingus#32; this is documentation only, so
nothing here alters behaviour.
distributed-compute-grid.mdwas drafted while realignment was still a plan, and it was the firstdesign in this repo written far enough ahead of the code that its mechanisms were guesses. Several
didn't survive contact. New §11 (reconnaissance) and §12 (what building it settled) record
that, and every superseded claim is corrected in place where it was originally stated — a reader
landing mid-document must not be told the old thing. That's the failure mode §4e demonstrated for
three months.
§11 — reconnaissance, before any code
minimap2-rsFFI;navigator-alignshipsminimap2-pure-rs— pure Rust, 99.74 % byte-identical, zero disagreements at MAPQ > 0. §7.3 hadbuilt a two-tier fleet on that FFI assumption, so Windows nodes take FASTQ units on day one and
the P4 Windows milestone doesn't exist.
App::analyze_biosamplefrom PR Make batch analysis tractable, and fix the Y re-placement bugs it uncovered #47.§12 — what building it settled
Seven subsections. The load-bearing ones:
mt_terminalleft the agreement digest —analyze_biosamplewon't assign mtDNA onCHM13 and the Grid realigns to CHM13, so the digest can't demand a value the analysis path
declines to produce;
the code was right and both the design and my test were wrong;
est_bases— ENA had been publishingbase_countall along.Three decisions you made are recorded with their reasoning: units are per ENA sample,
mt_terminalis out of the agreement digest, and P1 carries both data kinds (D3's stagingexisted only to retire an aligner risk that stopped existing).
Also here
BACKLOG.md§2.2 updated, and §11/§12 both close on outcomes rather than trailing off — the lessonfrom
archaic_match.rs, whose last paragraph became a lie the day the work landed.scripts/ste-check.py: 0 violations.🤖 Generated with Claude Code
https://claude.ai/code/session_01SequryftjcFkLnjQRMKCx3