Skip to content

kernel-wal: directory fsyncs fail loudly; poison survives a restart; a failed roll keeps its batch; frame layout 3 (Q128, after #549) - #679

Closed
MattJackson wants to merge 12 commits into
predevfrom
q128-kernel-wal-2
Closed

kernel-wal: directory fsyncs fail loudly; poison survives a restart; a failed roll keeps its batch; frame layout 3 (Q128, after #549)#679
MattJackson wants to merge 12 commits into
predevfrom
q128-kernel-wal-2

Conversation

@MattJackson

@MattJackson MattJackson commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator

kernel-wal round 2 (Q128 findings 2, 3, 4, 5, 7, 8, 9 and 10; findings 1 and 6 were #588). Per the ARCHITECT ruling this lands after #549, and #549 (lane-gb-loader-wal-edge, at bb5bc0f) is merged in. #549 has not merged yet, and this PR lands it.

What changed (journal durability, money-adjacent)

  • Findings 2 and 8: directory fsyncs fail loudly. Only EINVAL or an Unsupported answer (a filesystem that cannot fsync a directory) is tolerated.
    • The durable.rs directory fsync goes to plugin-loader: no busbar-kernel-* edge; durable moves into the loader, the shipping seam into the contract #549's new home, crates/plugin-loader/src/durable.rs: write, remove and create_dir_all propagate it. Its hooks and tests are mounted from it (src/tests/durable_hooks.rs, src/tests/durable_tests.rs).
    • busbar-kernel-wal takes no loader edge, so its backend keeps the same rule for the log's own files:
      • DirectoryFactory::new fsyncs every directory it creates.
      • A segment whose directory fsync fails is not handed out, and the file it created is removed.
      • A quarantine copy whose directory fsync fails is an error, so recovery leaves the damage in place instead of cutting it.
  • Finding 3: poison survives a restart. On a failed write or sync, the segment is cut back to its last good commit and synced, and the next segment is opened before the loss is reported. A restart always appends to the highest segment, so it never resumes in the one that lost a sync.
  • Finding 4: a failed roll keeps its batch, as every other arm does, so the chain has no hole once the volume recovers.
    • ARCHITECT 2026-10-07 ruling: the root treats such a rate card as owed and unconfirmed, and never publishes it until confirmed. refusal_of answers a retained card as in doubt (Lost); the caller publishes only on Ok, and the card in force is put back behind the in-doubt one, so the in-doubt card never prices.
    • a_rate_card_the_log_drops_outright_is_never_published is rewritten as a_rate_card_the_log_owes_after_a_failed_roll_is_not_published_while_unconfirmed. It asserts the apply is answered Lost, the card in force prices the applied instant and every instant after it, and the log owes the card and its put-back.
    • The CardRefused::Dropped and refusal_of prose now states that no arm of the log refuses before retaining.
  • Finding 5: frame layout 3. A frame carries an end mark and an opens-commit flag. In the last commit, a crash mid-commit is a torn tail, cut silently. Damage that a later, acknowledged commit stands behind is corruption, and it is quarantined. Neither layout 2 nor 3 has shipped; 1.5.5 had no WAL.
  • Findings 7, 9 and 10. The never-constructed BufferShipper, Wal::memory_buffered, Journal::memory_buffered and Journal::resuming are deleted. Prose and the Cargo note are corrected, and the test hooks live in test files.

How it was proven

Latchkey cli-0a0c7794-d2f3-487b-9f8d-45e011d1a27d ran on 1607b89 (#549 bb5bc0f and predev 1c4e1f0 merged):

  • fmt passed, and clippy -p busbar-kernel-wal -p busbar-plugin-loader -p busbar-kernel -p busbar --all-targets -D warnings passed.
  • Tests, 0 failed in each crate:
    • busbar-kernel-wal: 26 passed
    • busbar-plugin-loader: 685 passed
    • busbar-kernel: 2870 passed
    • busbar (bin and every integration target except ledger_identity, which CI runs with the oracle clone): 1035 passed
  • RED A, with directory fsync failures swallowed in both homes. These FAIL:
    • a_segment_whose_directory_fsync_fails_is_not_handed_out
    • a_quarantine_whose_directory_fsync_fails_leaves_the_segment_uncut
    • a_publish_whose_directory_fsync_fails_is_an_error
    • a_removal_whose_directory_fsync_fails_is_an_error
    • a_created_directory_whose_parent_fsync_fails_is_an_error
  • RED B, with plugin-loader: no busbar-kernel-* edge; durable moves into the loader, the shipping seam into the contract #549's whole busbar-kernel-wal crate in place. These FAIL:
    • a_roll_that_cannot_open_the_next_segment_keeps_the_batch_it_was_handed
    • a_batch_reported_lost_does_not_come_back_at_a_restart
    • a_restart_never_appends_to_the_segment_that_lost_a_sync
    • a_failed_roll_leaves_no_hole_in_the_chain
    • a_zero_padded_tear_at_every_byte_offset_is_torn_never_corrupt
    • a_tear_inside_a_frames_payload_is_a_torn_tail_cut_silently
    • a_commit_whose_pages_landed_out_of_order_is_a_torn_tail
    • the root's a_rate_card_the_log_owes_after_a_failed_roll_is_not_published_while_unconfirmed
    • The guard damage_in_a_commit_a_subsequent_commit_acknowledges_is_corrupt_whatever_its_shape passes on base, as intended.
  • structure-lint is RED only on plane-dup:unledgered, the standing row xtask/src/gates/mod.rs names (24 plane-local reimplementations). This PR does not touch it.

Oracle verdict

Strict replay against golden/1.5.5 over the money, ledger, usage, billing and admin audit/verify cells, base predev 1c4e1f0 against this head:

  • head: 68 PASS, 20 UNBASELINED, 0 FAIL
  • base: 68 PASS, 20 UNBASELINED, 0 FAIL
  • No cell moved, and no unsigned 1.5.5 cell moved.

@MattJackson
plugin-loader: no busbar-kernel-* edge; the durable-write primitive a…
a86084b 
…nd the shipping seam move

The shipping seam the store adapter implements and the log ships through moves into
busbar_contract::ship: Shipper<R> and ShipError, over ShippedRecord (a record's identity
alone). The log's Record and its framing stay in busbar-kernel-wal; Record implements
ShippedRecord, and the log's own shippers and boxes name Shipper<Record>.

durable.rs and its tests move unchanged from busbar-kernel-wal (which never used it) into
busbar-plugin-loader, host-only machinery no plugin image links, so its temp-name counter
stays one per process. busbar-kernel re-exports it from the loader, so every
crate::durable / busbar_kernel::durable call site is unchanged, and the root's keyset
names busbar_kernel::durable (the root names the loader only in root/loader.rs). The structure-lint choke-point owner and class test follow the file.

crates/plugin-loader/Cargo.toml now names no busbar-kernel-* crate (Part 2 #33: plugin
infrastructure is one crate atop the one contract crate). The dead
busbar-plugin-loader -> busbar-kernel-wal [[dep]] row is struck, and both kind-isolation
gates gain a RED plant that grows a loader -> busbar-kernel-ledger edge back: the per-push
gate refuses it as unlisted-dep-edge, the ship twin as a tcb ship-edge.
@MattJackson
Merge remote-tracking branch 'origin/predev' into lane-gb-loader-wal-…
b4b2cd5 
…edge
@MattJackson
Merge remote-tracking branch 'origin/predev' into lane-gb-loader-wal-…
ce60ba3 
…edge

# Conflicts:
#	Cargo.lock
@MattJackson
kernel-wal: directory fsyncs fail loudly from one home; poison surviv…
ee7072e 
…es a restart; a failed roll keeps its batch; a crash mid-commit is a torn tail, never corruption (frame layout 3) (Q128)

Audit findings 2-5 and 7-10 on crates/busbar-kernel-wal (1 and 6 are on #588).

2, 8. durable::sync_holding_dir is the one home of directory durability and
returns its error; only EINVAL or an Unsupported answer (a filesystem that
cannot fsync a directory) is tolerated. durable::write, remove and
create_dir_all propagate it. backend.rs's copy is gone: DirectoryFactory::new
goes through durable::create_dir_all (every created ancestor is fsynced),
segment creation returns the fsync error and removes the file it created (so
the next open does not find it existing and skip the fsync), and a quarantine
whose directory fsync fails is an error, so recovery leaves the damage in
place rather than cutting it. The structure-lint row no longer ledgers
backend.rs for fs::create_dir_all.

3. Design: roll eagerly on poison, plus cut the failed batch's bytes. On a
failed write or sync the segment shortens itself back to the last good commit
and syncs that, so a batch reported lost cannot be read back from the page
cache by a restart. The log then opens the next segment at once, before
reporting the loss; that segment's directory entry (made durable on creation,
and with 2 an error if it is not) is the durable record that the poisoned
segment is finished. A restart appends to the HIGHEST segment always, even
when it is empty and the tail is read from the one below, so it never resumes
in the segment that lost a sync. Chosen over "a boot never appends to a
pre-existing segment": that variant puts every boot's records in a segment of
its own, and boot recovery (quarantine, identity marks) only covers the tail
segment, so damage in a previous boot's records would stop being quarantined.
Chosen over a poison marker frame: the next segment's entry already is the
record, with no format to add. What no design covers: a medium that refuses
the cut as well as the sync (the bytes may read back until the retried batch
is durable), or refuses to create the next segment too (a restart resumes in
the poisoned one); every call on such a disk reports the loss.

4. The failed-roll arm retains the records it was handed, as every other arm
does, so the journal's chain has no hole after a full volume recovers.

5. Frame layout 3: version 2 plus an END MARK (the frame's last byte, never
zero; the payload area is 415 bytes) and an OPENS-COMMIT flag on the first
frame of each group commit. Recovery classifies by what was acknowledged:
damage with a verifying commit-opening frame past it is corruption whatever
its shape (that commit was only written after the damaged one synced). In the
last commit, damage made only of frames a write left unfinished (zeros, or a
prefix of this layout's frame with a zero end mark) is a torn tail and is cut
silently, together with any frame of that commit that landed out of order; a
whole frame (end mark present) that fails a check, or a frame of another
layout, is corruption and is quarantined (THE DESIGN 7). Version 3, not a
redefined 2: the payload area and the trailer changed, so a reader must not
read 2's frames as 3's; neither was released (1.5.5 had no WAL), and versions
1 and 2 are both unknown layouts, quarantined, never cut.

7. Deleted the never-constructed BufferShipper, Wal::memory_buffered,
Journal::memory_buffered and Journal::resuming (THE DESIGN: a memory-store
node keeps no history across a restart). Tests use a test-local keeping
shipper and memory_buffered_to.

9. The Cargo.toml dependency note names ring (sha2 is dev-only); durable.rs
cites the real structure-lint row; write_with carries its own contract,
including what an error from the directory fsync means.

10. The durable fault, decoy and fsync recorders live in src/tests/hooks.rs;
durable.rs keeps only the hook points. record::reseal moved to the test
fixtures. durable_tests mounts from src/tests.

RED tests: a_publish_whose_directory_fsync_fails_is_an_error,
a_removal_whose_directory_fsync_fails_is_an_error,
a_created_directory_whose_parent_fsync_fails_is_an_error,
a_segment_whose_directory_fsync_fails_is_not_handed_out,
a_quarantine_whose_directory_fsync_fails_leaves_the_segment_uncut,
a_deep_data_directory_has_every_directory_it_creates_fsynced,
a_batch_reported_lost_does_not_come_back_at_a_restart,
a_restart_never_appends_to_the_segment_that_lost_a_sync,
a_roll_that_cannot_open_the_next_segment_keeps_the_batch_it_was_handed,
a_failed_roll_leaves_no_hole_in_the_chain,
a_zero_padded_tear_at_every_byte_offset_is_torn_never_corrupt,
a_tear_inside_a_frames_payload_is_a_torn_tail_cut_silently,
a_commit_whose_pages_landed_out_of_order_is_a_torn_tail.

Rewritten: a_tear_inside_a_whole_header_frame_is_quarantined_not_cut (asserted
the old wrong verdict; now a_tear_inside_a_frames_payload_is_a_torn_tail_cut_silently);
a_poisoned_segment_never_takes_another_write (lands a batch first, since the
failed batch is now cut); chain_verification_catches_a_gap_in_the_numbering
(seals the gap by hand instead of through Journal::resuming);
a_restart_without_a_data_dir_loses_nothing_that_was_shipped (the resume half
went with Journal::resuming; now
without_a_data_dir_what_the_store_took_is_the_chain_to_the_nodes_head);
the_default_log_is_the_memory_buffered_one, the_bound_is_pinned and the
journal/no-disk batteries (memory_buffered_to); the version-1 vector
self-check compares the payload up to the end mark; the frame-size pin adds
the trailer.
@MattJackson
Merge origin/predev into lane-gb-loader-wal-edge
bb5bc0f
@MattJackson
Merge remote-tracking branch 'origin/predev' into q128-kernel-wal-2
f53311e
@MattJackson
Merge lane-gb-loader-wal-edge (#549) into q128-kernel-wal-2: the dura…
04917e3 
…ble.rs directory fsync goes to #549's new home

#549 moves durable.rs into busbar-plugin-loader and the Shipper seam into the contract, and takes
busbar-kernel-wal off every loader edge. Resolved per the ARCHITECT ruling (wal-2 lands after #549,
its durable.rs directory fsync goes to #549's new home):

- crates/plugin-loader/src/durable.rs carries wal-2's fail-loud sync_holding_dir (only EINVAL or an
  Unsupported answer is tolerated) and write/remove/create_dir_all propagating it. Its test hooks
  (src/tests/durable_hooks.rs) and tests (src/tests/durable_tests.rs, with the publish, removal,
  created-directory and unsupported-fsync cases from wal-2's dir_fsync.rs) are mounted from it.
- busbar-kernel-wal keeps the same rule for its own segment files and quarantine copies in
  backend.rs (sync_holding_dir, create_dir_durably), since the log takes no loader edge; its
  hooks are trimmed to that one directory fsync and dir_fsync.rs keeps the three backend cases.
- ship.rs and wal.rs: BufferShipper and Wal::memory_buffered stay deleted (wal-2 finding 7); the
  shippers are Shipper<Record> (#549).
@MattJackson
root: a rate card the log owes after a failed roll is in doubt, never…
b6a6697 
… published while unconfirmed

With kernel-wal finding 4 a failed roll keeps the batch it was handed, so the card is owed (held for
the log's retry), not dropped. ARCHITECT 2026-10-07 ruling: keep the batch; the root treats the card
as owed and unconfirmed and never publishes it until it is confirmed. refusal_of already answers a
retained card as Lost (in doubt): the caller publishes only on Ok, and the in-force card is put back
behind the in-doubt one so it never prices.

a_rate_card_the_log_drops_outright_is_never_published asserted the old Dropped answer. It is
rewritten as a_rate_card_the_log_owes_after_a_failed_roll_is_not_published_while_unconfirmed: the
apply is answered Lost, the card in force prices the applied instant and every instant after it,
and the log owes the card and its put-back. The CardRefused::Dropped and refusal_of prose now
states that no arm of the log refuses before retaining.
@MattJackson
Merge remote-tracking branch 'origin/lane-gb-loader-wal-edge' into q1…
689b7f6 
…28-kernel-wal-2

# Conflicts:
#	crates/busbar-kernel-wal/src/wal.rs
@MattJackson
Merge remote-tracking branch 'origin/predev' into q128-kernel-wal-2
1607b89
@MattJackson
MattJackson enabled auto-merge October 8, 2026 02:47
@MattJackson
MattJackson marked this pull request as draft October 8, 2026 03:33
auto-merge was automatically disabled October 8, 2026 03:33

Pull request was converted to draft

@MattJackson

MattJackson commented Oct 8, 2026

Copy link
Copy Markdown
Collaborator Author

Back to draft automatically: this branch contains #549 (lane-gb-loader-wal-edge), which is also ready. Two hops on the same code is duplicate spend (owner rule 2026-10-08). Mark this ready again after #549 merges.

@github-actions

github-actions Bot commented Oct 8, 2026 •
edited
Loading

Copy link
Copy Markdown

promote into predev: BOARD @4012392fb: 0 failing test row(s), 11 DENY row(s)

DENY rows (11)

gate row detail
construction one-pick-site 3 production call site(s) of 'pick_among(' (ceiling 2): crates/busbar-kernel-egress/src/walk.rs:298; crates/busbar-llm/src/engine/exhaustion/fallback.rs:120; crates/busbar-llm/src/engine/pipeline.rs:8
kind-isolation kind-isolation:deps 3 finding(s), 93 shipped edge instance(s) over 30 class(es), 93 declaration(s); 56 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge legacy -> plane busbar-llm -> busbar-plane-llm is a shipped edge w
kind-isolation kind-isolation:test-deps 3 finding(s), 35 test edge instance(s) over 21 class(es), 35 declaration(s); 20 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge cleanliness -> legacy busbar-core-admin -> busbar-a2a is a test edge
kind-isolation kind-isolation:law0 18 finding(s) over 14 neutral crate(s): law0-rise busbar × auth 51 hit(s) against a ceiling of 46: this landing grew a neutral crate's instance vocabulary. Ceilings only fall (measured on predev f830f
kind-isolation-ship kind-isolation:deps 10 finding(s) over 93 shipped edge(s): ship-edge kernel -> hooks busbar-kernel -> busbar-hook-ranking is 'not-allowed': the architecture grants no kernel -> hooks edge, and the ship criterion is the a
kind-isolation-ship kind-isolation:test-deps 10 finding(s) over 35 test edge(s): ship-edge cleanliness -> export busbar-core-admin -> busbar-export-prometheus is 'not-allowed': the architecture grants no cleanliness -> export edge, and the ship
kind-isolation-ship kind-isolation:law0 18 finding(s) over 14 neutral crate(s): law0-rise busbar × auth 51 hit(s) against a ceiling of 46: this landing grew a neutral crate's instance vocabulary. Ceilings only fall (measured on predev f830f
kind-isolation-ship kind-isolation:faces 2 finding(s) over 40 crate(s): foreign-entry crates/busbar busbar is kind 'root' and implements 'Transport' 1 time(s) in shipped source — the entry face of kind 'transport'. A trait implementation is
kind-isolation-ship kind-isolation:legacy-drain 3 finding(s): transitional-live qa/kind-isolation.toml 'busbar-a2a -> busbar-core-admin' (legacy drain: the retiring A2A engine's tests drive the admin surface that drained into the cleanliness tier)
ship-ready ship-ready:ship-twin 'kind-isolation-ship' is not green: kind-isolation:deps (a dependency the architecture does not grant is still in the graph); kind-isolation:test-deps (a dependency the architecture does not grant is
structure-lint structure-lint:plane-dup:unledgered 22 finding(s): PLANE-DUPLICATE (module): 'config.rs' — a2a:crates/busbar-a2a/src/a2a/config.rs decisions:crates/busbar-plane-decisions/src/config.rs voice:crates/busbar-voice/src/config.rs (the ledger

Judged against base abc07fea9: 0 new red, 0 worse, 5 standing (excused).

tests passed: 24181, failed: 0. Run: https://github.com/GetBusbar/busbar/actions/runs/38022766232 . Artifact verdict-4012392fb76b81799b7304e3bc2c72a27de4913f (failures.json, junit.xml, raw.log; 90 days).

@MattJackson
Merge branch 'predev' of https://github.com/GetBusbar/busbar into q12…
72c4449 
…8-kernel-wal-2
@MattJackson
kind-isolation:law0: strike the busbar-kernel-wal x instance:secret r…
4012392 
…ow; this branch dropped durable_tests' join("c") path literal (read as busbar-secret-c's id), so the cell measures 0 and the ceiling of 1 was slack (law0 18 -> 19)
@MattJackson MattJackson mentioned this pull request Oct 10, 2026
Merged
@MattJackson

MattJackson commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator Author

Closed: superseded by #730 (kernel-wal round 2, landed from this draft).

@MattJackson
MattJackson deleted the q128-kernel-wal-2 branch October 10, 2026 22:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@MattJackson