Repository navigation
kernel-wal: fsyncs fail loudly; poison survives restart (round 2) - #730
Merged
Merged
Conversation
…es a restart; a failed roll keeps its batch; a crash mid-commit is a torn tail, never corruption (frame layout 3) (Q128) Audit findings 2-5 and 7-10 on crates/busbar-kernel-wal (1 and 6 are on #588). 2, 8. durable::sync_holding_dir is the one home of directory durability and returns its error; only EINVAL or an Unsupported answer (a filesystem that cannot fsync a directory) is tolerated. durable::write, remove and create_dir_all propagate it. backend.rs's copy is gone: DirectoryFactory::new goes through durable::create_dir_all (every created ancestor is fsynced), segment creation returns the fsync error and removes the file it created (so the next open does not find it existing and skip the fsync), and a quarantine whose directory fsync fails is an error, so recovery leaves the damage in place rather than cutting it. The structure-lint row no longer ledgers backend.rs for fs::create_dir_all. 3. Design: roll eagerly on poison, plus cut the failed batch's bytes. On a failed write or sync the segment shortens itself back to the last good commit and syncs that, so a batch reported lost cannot be read back from the page cache by a restart. The log then opens the next segment at once, before reporting the loss; that segment's directory entry (made durable on creation, and with 2 an error if it is not) is the durable record that the poisoned segment is finished. A restart appends to the HIGHEST segment always, even when it is empty and the tail is read from the one below, so it never resumes in the segment that lost a sync. Chosen over "a boot never appends to a pre-existing segment": that variant puts every boot's records in a segment of its own, and boot recovery (quarantine, identity marks) only covers the tail segment, so damage in a previous boot's records would stop being quarantined. Chosen over a poison marker frame: the next segment's entry already is the record, with no format to add. What no design covers: a medium that refuses the cut as well as the sync (the bytes may read back until the retried batch is durable), or refuses to create the next segment too (a restart resumes in the poisoned one); every call on such a disk reports the loss. 4. The failed-roll arm retains the records it was handed, as every other arm does, so the journal's chain has no hole after a full volume recovers. 5. Frame layout 3: version 2 plus an END MARK (the frame's last byte, never zero; the payload area is 415 bytes) and an OPENS-COMMIT flag on the first frame of each group commit. Recovery classifies by what was acknowledged: damage with a verifying commit-opening frame past it is corruption whatever its shape (that commit was only written after the damaged one synced). In the last commit, damage made only of frames a write left unfinished (zeros, or a prefix of this layout's frame with a zero end mark) is a torn tail and is cut silently, together with any frame of that commit that landed out of order; a whole frame (end mark present) that fails a check, or a frame of another layout, is corruption and is quarantined (THE DESIGN 7). Version 3, not a redefined 2: the payload area and the trailer changed, so a reader must not read 2's frames as 3's; neither was released (1.5.5 had no WAL), and versions 1 and 2 are both unknown layouts, quarantined, never cut. 7. Deleted the never-constructed BufferShipper, Wal::memory_buffered, Journal::memory_buffered and Journal::resuming (THE DESIGN: a memory-store node keeps no history across a restart). Tests use a test-local keeping shipper and memory_buffered_to. 9. The Cargo.toml dependency note names ring (sha2 is dev-only); durable.rs cites the real structure-lint row; write_with carries its own contract, including what an error from the directory fsync means. 10. The durable fault, decoy and fsync recorders live in src/tests/hooks.rs; durable.rs keeps only the hook points. record::reseal moved to the test fixtures. durable_tests mounts from src/tests. RED tests: a_publish_whose_directory_fsync_fails_is_an_error, a_removal_whose_directory_fsync_fails_is_an_error, a_created_directory_whose_parent_fsync_fails_is_an_error, a_segment_whose_directory_fsync_fails_is_not_handed_out, a_quarantine_whose_directory_fsync_fails_leaves_the_segment_uncut, a_deep_data_directory_has_every_directory_it_creates_fsynced, a_batch_reported_lost_does_not_come_back_at_a_restart, a_restart_never_appends_to_the_segment_that_lost_a_sync, a_roll_that_cannot_open_the_next_segment_keeps_the_batch_it_was_handed, a_failed_roll_leaves_no_hole_in_the_chain, a_zero_padded_tear_at_every_byte_offset_is_torn_never_corrupt, a_tear_inside_a_frames_payload_is_a_torn_tail_cut_silently, a_commit_whose_pages_landed_out_of_order_is_a_torn_tail. Rewritten: a_tear_inside_a_whole_header_frame_is_quarantined_not_cut (asserted the old wrong verdict; now a_tear_inside_a_frames_payload_is_a_torn_tail_cut_silently); a_poisoned_segment_never_takes_another_write (lands a batch first, since the failed batch is now cut); chain_verification_catches_a_gap_in_the_numbering (seals the gap by hand instead of through Journal::resuming); a_restart_without_a_data_dir_loses_nothing_that_was_shipped (the resume half went with Journal::resuming; now without_a_data_dir_what_the_store_took_is_the_chain_to_the_nodes_head); the_default_log_is_the_memory_buffered_one, the_bound_is_pinned and the journal/no-disk batteries (memory_buffered_to); the version-1 vector self-check compares the payload up to the end mark; the frame-size pin adds the trailer.
…ble.rs directory fsync goes to #549's new home #549 moves durable.rs into busbar-plugin-loader and the Shipper seam into the contract, and takes busbar-kernel-wal off every loader edge. Resolved per the ARCHITECT ruling (wal-2 lands after #549, its durable.rs directory fsync goes to #549's new home): - crates/plugin-loader/src/durable.rs carries wal-2's fail-loud sync_holding_dir (only EINVAL or an Unsupported answer is tolerated) and write/remove/create_dir_all propagating it. Its test hooks (src/tests/durable_hooks.rs) and tests (src/tests/durable_tests.rs, with the publish, removal, created-directory and unsupported-fsync cases from wal-2's dir_fsync.rs) are mounted from it. - busbar-kernel-wal keeps the same rule for its own segment files and quarantine copies in backend.rs (sync_holding_dir, create_dir_durably), since the log takes no loader edge; its hooks are trimmed to that one directory fsync and dir_fsync.rs keeps the three backend cases. - ship.rs and wal.rs: BufferShipper and Wal::memory_buffered stay deleted (wal-2 finding 7); the shippers are Shipper<Record> (#549).
… published while unconfirmed With kernel-wal finding 4 a failed roll keeps the batch it was handed, so the card is owed (held for the log's retry), not dropped. ARCHITECT 2026-10-07 ruling: keep the batch; the root treats the card as owed and unconfirmed and never publishes it until it is confirmed. refusal_of already answers a retained card as Lost (in doubt): the caller publishes only on Ok, and the in-force card is put back behind the in-doubt one so it never prices. a_rate_card_the_log_drops_outright_is_never_published asserted the old Dropped answer. It is rewritten as a_rate_card_the_log_owes_after_a_failed_roll_is_not_published_while_unconfirmed: the apply is answered Lost, the card in force prices the applied instant and every instant after it, and the log owes the card and its put-back. The CardRefused::Dropped and refusal_of prose now states that no arm of the log refuses before retaining.
…28-kernel-wal-2 # Conflicts: # crates/busbar-kernel-wal/src/wal.rs
…8-kernel-wal-2
…ow; this branch dropped durable_tests' join("c") path literal (read as busbar-secret-c's id), so the cell measures 0 and the ceiling of 1 was slack (law0 18 -> 19)
…his PR's retained batch Journal::resuming stays deleted; Journal::memory_resumed positions the chain itself. Journal::retaining_at_bound and Wal::memory_seeded are kept: a retaining journal never forgets from the batch a failed roll keeps. H3's tests no longer reach for the deleted BufferShipper and Journal::memory_buffered; a kernel test pins a failed roll's batch kept past a retaining journal's bound.
…eding log The predev merge brought H3's Wal::memory_seeded, which names NullShipper; this branch's wal.rs no longer imported it. Supersedes #679.
…-kernel-wal-round2
MattJackson
enabled auto-merge
October 10, 2026 06:55
promote into
|
| gate | row | detail |
|---|---|---|
| construction | one-pick-site |
3 production call site(s) of 'pick_among(' (ceiling 2): crates/busbar-kernel-egress/src/walk.rs:298; crates/busbar-llm/src/engine/exhaustion/fallback.rs:120; crates/busbar-llm/src/engine/pipeline.rs:8 |
| kind-isolation | kind-isolation:deps |
3 finding(s), 93 shipped edge instance(s) over 30 class(es), 93 declaration(s); 56 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge legacy -> plane busbar-llm -> busbar-plane-llm is a shipped edge w |
| kind-isolation | kind-isolation:test-deps |
3 finding(s), 35 test edge instance(s) over 21 class(es), 35 declaration(s); 20 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge cleanliness -> legacy busbar-core-admin -> busbar-a2a is a test edge |
| kind-isolation | kind-isolation:law0 |
11 hit(s) off the [[law0]] ceilings, 5 finding(s) over 14 neutral crate(s): law0-rise busbar × instance:secret 37 hit(s) against a ceiling of 35: this landing grew a neutral crate's instance vocabular |
| kind-isolation-ship | kind-isolation:deps |
10 finding(s) over 93 shipped edge(s): ship-edge kernel -> hooks busbar-kernel -> busbar-hook-ranking is 'not-allowed': the architecture grants no kernel -> hooks edge, and the ship criterion is the a |
| kind-isolation-ship | kind-isolation:test-deps |
10 finding(s) over 35 test edge(s): ship-edge cleanliness -> export busbar-core-admin -> busbar-export-prometheus is 'not-allowed': the architecture grants no cleanliness -> export edge, and the ship |
| kind-isolation-ship | kind-isolation:law0 |
11 hit(s) off the [[law0]] ceilings, 5 finding(s) over 14 neutral crate(s): law0-rise busbar × instance:secret 37 hit(s) against a ceiling of 35: this landing grew a neutral crate's instance vocabular |
| kind-isolation-ship | kind-isolation:faces |
2 finding(s) over 40 crate(s): foreign-entry crates/busbar busbar is kind 'root' and implements 'Transport' 1 time(s) in shipped source — the entry face of kind 'transport'. A trait implementation is |
| kind-isolation-ship | kind-isolation:legacy-drain |
3 finding(s): transitional-live qa/kind-isolation.toml 'busbar-a2a -> busbar-core-admin' (legacy drain: the retiring A2A engine's tests drive the admin surface that drained into the cleanliness tier) |
| ship-ready | ship-ready:ship-twin |
'kind-isolation-ship' is not green: kind-isolation:deps (a dependency the architecture does not grant is still in the graph); kind-isolation:test-deps (a dependency the architecture does not grant is |
| structure-lint | structure-lint:plane-dup:unledgered |
22 finding(s): PLANE-DUPLICATE (module): 'config.rs' — a2a:crates/busbar-a2a/src/a2a/config.rs decisions:crates/busbar-plane-decisions/src/config.rs voice:crates/busbar-voice/src/config.rs (the ledger |
Judged against base 7290204b1: 0 new red, 0 worse, 5 standing (excused).
tests passed: 22992, failed: 0. Run: https://github.com/GetBusbar/busbar/actions/runs/38072813866 . Artifact verdict-d8ec44638ed76efc2974e3ce08355f309e05442c (failures.json, junit.xml, raw.log; 90 days).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kernel-wal: fsyncs fail loudly; poison survives restart (round 2)