Skip to content

Commit 4e41bf3

Browse files
committed
fix: preserve queued node identity across full-tree crash scenarios
1 parent f6dc2ff commit 4e41bf3

11 files changed

Lines changed: 689 additions & 9 deletions

‎docs/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ The synchronized unit is one recursive included namespace. Git repositories are
6969
| `src/v3/setup.ts` | Authority setup and target-local enrollment |
7070
| `src/v3/distributed.ts` | Resumable multi-machine setup and per-target approval |
7171
| `src/v3/acceptance.ts` | Witnessed two-run real-snapshot pseudo-fleet execution |
72+
| `src/v3/acceptance-faults.ts` | Full-corpus process-crash, move, conflict, outage, and corruption scenarios |
7273
| `src/v3/acceptance-reflink.ts` | Private bounded-volume validation and immutable object seeding |
7374
| `scripts/run_bounded_archived_acceptance.mjs` | Sequential historical Drive restores and bounded full-tree runner lifecycle |
7475
| `src/v3/backup.ts` | Streaming encrypted capture and portable/platform recovery oracles |

‎docs/acceptance-scenarios.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@ The repository retains older sentinel-protected safety and churn harnesses. They
6666
- pseudo-fleet cutover, peer activation, normal sync, and full verification use the pinned packaged peer processes, each with a private run-local HOME and XDG directories and no global/system Git configuration;
6767
- every pseudo-fleet peer then performs a no-change sync that must publish nothing and transfer zero objects; any offline or inconclusive target verification fails the repetition even when its captured digest matches;
6868
- each peer's force verification must bind to the same accepted checkpoint sequence and digest; the coordinator reads only that checkpoint rather than rescanning workspaces with its own Git environment;
69+
- both pseudo-fleet repetitions run the fault stages in [the full-tree fault runner](full-view-storage.md#full-tree-fault-stages), retaining the entire adopted corpus through real process crashes, a subtree move, concurrent writes and private recovery, hub outage, and deliberately corrupted new content;
6970
- the AI workload requires at least 50 changed TypeScript files across eight directories, verifies required create, move, and delete behavior, records a private cassette, and replays it to the same final digest;
7071
- the Linux durability lane injects real `ENOSPC` write and `EIO` fsync results below materialization staging, requires an inconclusive result with retained prior bytes, then retries to clean full verification;
7172
- the Linux watcher lane freezes the daemon, proves `IN_Q_OVERFLOW` with a separate native watcher, writes the included marker after saturation, and requires the scheduled full scan to publish it;

‎docs/conflicts.md‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ V3 never performs line merge or timestamp last-writer-wins.
44

55
## Adoption conflicts
66

7-
During adoption the source checkpoint is always canonical. Target-only, divergent, type-conflicting, and divergent Git paths move outside the synchronized root before source materialization. Each retained record names the original target path, recovery path, peer, reason, and recoverable manifest when one exists.
7+
During adoption the source checkpoint is canonical within the selected eligible scope. Target-only, divergent, and type-conflicting working paths move outside the synchronized root before source materialization. Each retained record names the original target path, recovery path, peer, reason, and recoverable manifest when one exists. Git metadata and suspended repository scopes stay local.
88

99
Adoption does not create conflict siblings inside the authority tree. It can therefore converge every included live target path exactly while preserving all displaced evidence.
1010

@@ -22,13 +22,17 @@ The name uses peer identity and immutable event identity, not a clock. The sibli
2222

2323
The three-way merge compares accepted baseline, local scan, and current hub state. A remote mutation remains canonical. Losing leaf bytes receive a sibling; losing deletes remain explicit conflict evidence. Parent directories required by a preserved sibling are restored from a verified side.
2424

25-
Stable node IDs allow an uncontested subtree move without descendant uploads. A losing directory version receives one visible sibling plus an immutable subtree manifest containing every included descendant and nested Git boundary, so recovery does not depend on the sibling remaining live. An ambiguous concurrent structural case is never allowed to overwrite stable content; it yields explicit evidence or an inconclusive result.
25+
Stable node IDs allow an uncontested subtree move without descendant uploads. A losing directory version receives one visible sibling plus an immutable subtree manifest containing every included descendant, so recovery does not depend on the sibling remaining live. An ambiguous concurrent structural case is never allowed to overwrite stable content; it yields explicit evidence or an inconclusive result.
2626

2727
Normal conflicts are signed inside the publishing snapshot and commit atomically with that checkpoint. A crash cannot accept canonical content while omitting its recovery record.
2828

29-
## Git conflicts
29+
## Local Git state
3030

31-
Git boundaries are complete immutable manifests. A concurrent losing Git state is recorded with its manifest rather than merged file by file. Current V3 exposes recovery evidence and does not provide the removed V2 `resolve-git` command.
31+
Git metadata is not synchronized or conflict-merged. Default-branch working files
32+
pause independently when repository identity, branch, or HEAD differs. Selected
33+
files follow their allowlist across branches without changing checkout, index,
34+
remotes, or hooks. Retained old-context events and interrupted applies are private
35+
recovery evidence, not Git metadata conflict copies.
3236

3337
## Operator workflow
3438

‎docs/durability-and-recovery.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,14 @@ verified local objects
2727

2828
The hub verifies the peer signature over the snapshot, explicit mutations, peer sequence, and normal conflicts, then commits them in one SQLite transaction. If an acknowledgement is lost after commit, retry returns the original event sequence. If a different peer advanced the hub first, the stale serialization is replaced only after the live proposal remains available for three-way merge.
2929

30+
A publisher can die before new leaf identities reach its accepted catalog.
31+
Restart authenticates its queued proposals against the enrolled local public key
32+
and adds their new node observations to the scan references. These references
33+
survive the current Git-context preparation and outbox acknowledgement, but do
34+
not advance the accepted baseline, cursor, or catalog history. A newly created
35+
file therefore retains its queued identity instead of becoming a second create
36+
or a false concurrent conflict after an unknown acknowledgement.
37+
3038
Default-branch queues also retain their repository context. When any scope is suspended, the engine keeps mixed-context queued records private and publishes eligible sibling changes as fresh CAS events. It does not retry the old mixed snapshot in a different checkout. Returning to the matching context permits normal idempotent retry or stale-CAS reconciliation. There is no automatic immutable-object cleanup.
3139

3240
A changed HEAD moves the exact queued event and its object closure from `outbox` to `held_outbox` in one SQLite transaction. The engine never replays a held event. `recover-held <event-id> --path <relative-leaf>` verifies the enrolled signature and immutable manifest, then materializes one retained version under owner-only private state. It does not alter the current checkout. Publication rechecks active and enclosing Git contexts after object upload and before accepting the signed event.

‎docs/full-view-storage.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,39 @@ Each repetition clones the closed, protected `objects.sqlite` with `--reflink=al
5050

5151
The external controller creates and formats only a new exclusive-created image, owns namespace/process lifetime, restores the full historical masters, and builds the template. It must stop all children and unmount before outer-root cleanup. The product does not mount, format, elevate privileges, or install a permanent filesystem. This storage support is not a completed original-archive campaign, native fleet test, or real-folder stress result.
5252

53+
## Full-tree fault stages
54+
55+
`src/v3/acceptance-faults.ts` runs inside each fresh pseudo-fleet repetition after
56+
initial adoption and workload convergence. It keeps every adopted path available;
57+
its generated probes are mutations within that complete corpus, not replacement
58+
input trees. All probe operations contribute to a deterministic scenario digest.
59+
60+
The packaged product processes are killed with `SIGKILL` after durable outbox
61+
creation, hub acceptance before acknowledgement, and move staging. A private
62+
phase marker and the actual child termination signal prove the boundary. Restart
63+
must drain queued work, retry an already accepted event without a new sequence,
64+
and finish the journaled subtree move. The move must upload zero objects.
65+
66+
Concurrent writes must retain the first accepted bytes and the exact losing bytes
67+
in a visible conflict sibling on every peer. The losing peer also recovers the
68+
immutable conflict privately through its packaged agent. The outage stage moves
69+
only the run-owned hub aside and temporarily puts an exact marker at its configured
70+
path. Sync and verification must reject that unavailable hub without losing the
71+
offline edit. The exact marker is removed and the held hub restored before retry.
72+
73+
The corruption stage damages only the newly generated canary's known chunk in
74+
the disposable hub database. A target that lacks that chunk must reject apply,
75+
and its force verifier must reject the missing canary. The runner repairs that
76+
exact probe chunk before requiring complete convergence. Final force verification
77+
and the no-change pass still cover every peer's entire included namespace.
78+
79+
Crash and recovery agent actions require an exact disposable-run sentinel and
80+
physical contained peer roots. Recovery destinations stay below private peer
81+
state with checked ancestors. Results contain only counts, pass flags, crash
82+
phases, and the scenario digest, never probe contents or real archived paths.
83+
These stages do not substitute for the remaining native write/fsync, sustained
84+
churn, Git-state, AI/replay, and machine-platform gates.
85+
5386
## Historical full-archive controller
5487

5588
After building and verifying the candidate, run the external controller inside a fresh private mount namespace:

0 commit comments

Comments
 (0)