Skip to content

Fix concurrent same-content transfers, disk-full handling and small-frame writes - #45

Merged
TeoSlayer merged 6 commits into
mainfrom
hosted-node-fixes
Oct 7, 2026
Merged

TeoSlayer merged 6 commits into
mainfrom
hosted-node-fixes

Conversation

@TeoSlayer

Copy link
Copy Markdown
Contributor

Summary

Four fixes to file transfer and framing, found by running a node in a 1 GB container under load.

Changes

  • Several transfers of the same content at once no longer fail. The .partial is named after the content hash so a retry can resume it. Two peers sending the same file (or one sender on two connections) therefore wrote the same file, and all but the first failed at the final rename. The first transfer keeps the resumable .partial; a concurrent one gets a private file, removed if it does not finish.
  • A file that cannot fit on the receiver's disk is refused at INIT, keeping 16 MiB in reserve. The byte quota is a fixed number and can be larger than the disk: a transfer ran until the disk was full, failed, and left its bytes in .partial, where they kept the disk full and blocked incoming messages. If a write still hits ENOSPC, that .partial is deleted.
  • A refusal at INIT is reported as a refusal. The sender took any non-INIT_ACK reply for a receiver without streamed-transfer support, so the caller fell back to pushing the whole file in one frame at a peer that had just declined it.
  • WriteFrame sends a frame of up to 64 KiB as one write. Header and payload as two writes made the payload wait on Nagle and the peer's delayed ACK.

Wire format is unchanged. The free-space check uses statfs on unix and is skipped elsewhere.

Test Plan

  • go build ./..., go vet ./...
  • go test ./... -count=1 and -race
  • New tests: four concurrent transfers of one payload all succeed and leave no .partial; a transfer is refused when free space is one byte short and accepted when it fits; a disk-full write removes the .partial while other write errors keep it; small frames are one write and round-trip.
  • In a two-container lab: four concurrent transfers of the same 20 MB file went from 3 of 4 failing to 0 of 4.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Oct 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@TeoSlayer

Copy link
Copy Markdown
Contributor Author

Converted to draft after review; two things to fix before this merges (nothing ships until a stable tag, but both are regressions against main).

  1. Resume regression (filestream.go around lines 642-651). A retry that arrives while the stalled connection is still open on the receiver (idle timeout is 2 minutes) gets a private .partial and restarts at offset 0; main resumes mid-file in the same scenario. The stale .partial is then orphaned and counts against the quota, and with a quota set the retry is refused outright ("receiver disk quota exceeded") where main accepts it.
  2. Build break on some unix targets (filestream_disk_unix.go:3). //go:build unix no longer compiles on openbsd, netbsd, solaris and illumos; main does. linux, darwin, windows, freebsd and android are fine.

The rest reviewed well: wire bytes are unchanged, pilotctl already handles res.OK == false, and the new concurrency test fails on main and passes here. The handleChunk ENOSPC path is not covered by the PR's own tests (it does work against /dev/full).

Teo Calin and others added 2 commits October 7, 2026 16:03
…rame writes

Found by running a node in a 1 GB container under load.

- Several transfers of the same content at once failed: the .partial is
  named after the content hash, so they shared one file and all but the
  first failed at the final rename. The first transfer keeps the resumable
  .partial; a concurrent one gets a private file, removed if it does not
  finish.
- A file that cannot fit on the receiver's disk is refused at INIT (16 MiB
  kept in reserve) instead of running until the disk is full. If a write
  still hits ENOSPC the .partial is deleted rather than left holding the
  disk full.
- A refusal at INIT is reported as a refusal. The sender took it for a
  receiver without streamed-transfer support, and the caller fell back to
  pushing the whole file in one frame.
- WriteFrame sends a frame of up to 64 KiB as one write, so the payload is
  not held behind the header by Nagle and the peer's delayed ACK.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eck builds everywhere

Review of this branch found two regressions against main.

A sender that stalls and retries reconnects while the receiver still holds
the stalled connection open (idle timeout, minutes). The retry found the
content-addressed .partial claimed and started over at offset 0 in a
private file, orphaning the stale .partial, which then counted against the
quota and could get the retry refused. A claim whose holder has written
nothing for staleClaimAfter (10s) can now be taken over: the retry resumes
the .partial where the stalled transfer stopped, and the stalled transfer
is refused any further write or rename. Two live transfers of the same
content still get separate files.

//go:build unix included openbsd, netbsd, solaris and illumos, whose
syscall.Statfs_t lacks the fields used; the free-space check is now built
for linux, darwin and freebsd and other platforms rely on the byte quota.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TeoSlayer
TeoSlayer marked this pull request as ready for review October 7, 2026 13:05
@TeoSlayer

Copy link
Copy Markdown
Contributor Author

Both review findings fixed in c6b706d, rebased onto main.

  1. Resume regression. A claim on a .partial whose holder has written nothing for 10 s (staleClaimAfter) can now be taken over: a retry after a stall resumes where the stalled transfer stopped, and the stalled transfer is refused any further write or rename ("a newer transfer of this file has taken it over"). Two live transfers of the same content still get separate files. TestFileStream_RetryTakesOverAStalledTransfer covers it: the retry resumes at the stalled offset, the stalled connection's next chunk is refused, the file completes intact, and no .partial is left.
  2. Build. The free-space check is built for linux, darwin and freebsd; other platforms rely on the byte quota. Cross-compiled for openbsd, netbsd, solaris, illumos, freebsd, windows, darwin and linux.

go test -race ./... passes twice. Marking ready.

Teo Calin and others added 4 commits October 7, 2026 16:44
A receiver that refuses a transfer at INIT answers with a COMPLETE, and
SendFileStream returns that as a StreamResult with OK false and the
reason in Message, not as ErrStreamUnsupported. The doc comment only
described ErrStreamUnsupported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…after two minutes

Review of the takeover in the previous commit found these holes.

- A second INIT on a connection whose transfer a retry had taken over
  reused that transfer's path and dead claim: it opened the retry's file,
  could truncate it or delete it on a disk-check refusal, and left the id
  refusing every chunk. Such a transfer is now dropped, without touching
  the file, and the INIT goes through the claim like a new one.
- DONE checked the claim only on entry, then ran fsync, a hash over the
  whole file and onPrepare before renaming. It checks again right before
  the rename, and refreshes the claim while hashing.
- An INIT refused for quota, disk space or too many transfers took the
  claim over first and then released it, so the quiet holder lost its file
  to nobody. The checks now run before the claim is taken, and a failure
  after it hands the claim back. The quota check also compares the declared
  size unsigned: one of 2^63 bytes or more came out negative and passed.
- A claim went stale after 10s, while a sender waits up to 60s (pilotctl:
  90s) for an ACK, so a live transfer on a slow link could lose its file.
  staleClaimAfter is now twice streamStepTimeout. A retry sooner than that
  gets a private file and starts from 0; the comment says so.
- A DONE refused for a takeover forgot the transfer without closing its
  file. It and a chunk refused for a takeover now discard the transfer.
- A transfer abandoned for a full disk released its claim before deleting
  the file, so it could delete a new holder's file. It deletes first, and
  only while it still holds the claim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
syscall.ENOSPC does not exist on plan9, so the disk-full check broke the
plan9 build. The errors that mean a full disk now live behind a build-tag
split: ENOSPC and EDQUOT (a full disk quota) everywhere but Windows and
Plan 9, ERROR_DISK_FULL and ERROR_HANDLE_DISK_FULL on Windows, none on
Plan 9, whose errors are strings.

FreeBSD's Statfs_t.Bavail is signed and goes negative once root's
reserved blocks are in use; that now counts as no free space rather than
converting to an enormous number.

A test drives a disk-full write through handleChunk, for each error the
platform recognizes, using a writePartialAt seam.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
INIT creates the .partial before it records the transfer, so under load the
test could abandon a transfer that was not recorded yet, which does nothing,
and fail with "partial kept after disk-full". Review saw 4 of 12 stressed
runs fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@TeoSlayer
TeoSlayer merged commit ab7a6dd into main Oct 7, 2026
5 checks passed
@TeoSlayer
TeoSlayer deleted the hosted-node-fixes branch October 7, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant