Skip to content

feat(sftp): add downloadToRandomAccess for offset-based download#173

Merged
vicajilau merged 2 commits into
TerminalStudio:masterfrom
lollipopkit:upstream/sftp-random-access-download
Jun 30, 2026
Merged

feat(sftp): add downloadToRandomAccess for offset-based download#173
vicajilau merged 2 commits into
TerminalStudio:masterfrom
lollipopkit:upstream/sftp-random-access-download

Conversation

@GT-610

@GT-610 GT-610 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds SftpFile.downloadToRandomAccess, which downloads a file into a dart:io RandomAccessFile, writing each SFTP read reply at its own offset rather than requiring replies to arrive in order.

Unlike downloadTo (which streams sequentially into a StreamSink), downloadToRandomAccess uses a completion queue that decouples the order in which pipelined read replies arrive from the order in which they are written to disk. This lets pipelined reads keep making progress when a later offset completes before an earlier one, improving throughput on high-latency or reordering links.

Chain note

This PR is built on top of #172 (feat(sftp): prefer posix-rename@openssh.com). If #172 is merged first, only the downloadToRandomAccess commits will show in this PR's diff. If #172 is still open, both features are included here — please review and merge #172 first, then this PR will auto-rebase to show only the remaining changes.

Changes

  • Add SftpFile.downloadToRandomAccess(RandomAccessFile destination, ...):
    • Adaptive concurrency: starts with one outstanding read and ramps up to maxPendingRequests as replies succeed, backing off the chunk size on short reads to avoid over-requesting past EOF.
    • Validates chunkSize and maxPendingRequests as positive.
    • Throws SftpError on incomplete downloads (byte count mismatch).
  • Add a private _ReadCompletion helper class (kept SDK 2.17-compatible — no Dart 3 record syntax).
  • Add a protocol-level test that issues reads in pipelined fashion and replies out of order, asserting the final file contents are written at the correct offsets.

Test plan

  • test/src/sftp/sftp_client_protocol_test.dart — all 13 tests pass (including the new out-of-order test).
  • Verified the test replies read3 before read2 and the output file is still in offset order.
  • Ran dart analyze — no new warnings introduced.

Backport of ServerBox's forked dartssh2.

GT-610 and others added 2 commits June 30, 2026 19:53
Adds SftpFile.downloadToRandomAccess, which downloads a file into a
dart:io RandomAccessFile, writing each SFTP read reply at its own offset
rather than requiring replies to arrive in order.

Unlike downloadTo (which streams sequentially into a StreamSink),
downloadToRandomAccess uses a completion queue that decouples the order
in which pipelined read replies arrive from the order in which they are
written to disk. This lets pipelined reads keep making progress when a
later offset completes before an earlier one, improving throughput on
high-latency or reordering links.

Implementation notes:
- Adaptive concurrency: starts with one outstanding read and ramps up to
  maxPendingRequests as replies succeed, backing off the chunk size on
  short reads to avoid over-requesting past EOF.
- Validates chunkSize and maxPendingRequests as positive.
- Throws SftpError on incomplete downloads (byte count mismatch).

Add a protocol-level test that issues reads in pipelined fashion and
replies out of order, asserting the final file contents are written at
the correct offsets.
@vicajilau vicajilau merged commit 6930bc5 into TerminalStudio:master Jun 30, 2026
1 check passed
@codecov

codecov Bot commented Jun 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 67.60563% with 23 lines in your changes missing coverage. Please review.
✅ Project coverage is 53.99%. Comparing base (5c77884) to head (41ed8e1).
⚠️ Report is 3 commits behind head on master.

Files with missing lines Patch % Lines
lib/src/sftp/sftp_client.dart 67.60% 23 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #173      +/-   ##
==========================================
+ Coverage   53.84%   53.99%   +0.14%     
==========================================
  Files          64       64              
  Lines        5198     5269      +71     
==========================================
+ Hits         2799     2845      +46     
- Misses       2399     2424      +25     
Flag Coverage Δ
unittests 53.99% <67.60%> (+0.14%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
lib/src/sftp/sftp_client.dart 58.38% <67.60%> (+1.20%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@GT-610 GT-610 deleted the upstream/sftp-random-access-download branch July 1, 2026 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants