Skip to content

ssh-ng client ~66× slower than upstream nix 2.34.7 for closure copies (per-path connection reopen) #533

Closed
#538
Closed
ssh-ng client ~66× slower than upstream nix 2.34.7 for closure copies (per-path connection reopen)#533
#538

Description

@brett
brett
opened on Jul 3, 2026

Describe the bug

Copying a derivation closure to an ssh-ng:// store with Determinate Nix 3.21.2 (nix 2.34.7) is drastically slower than doing the identical copy with upstream nix 2.34.7 — the same nix version. The ssh-ng client reopens the SSH connection per path during the valid-paths / narFromPath handshake, so the cost scales with the number of paths in the closure rather than the bytes transferred, even when nothing needs to be transferred.

Because upstream 2.34.7 is fast and Determinate 2.34.7 is ~66× slower on the same operation, this looks like a fork-specific regression rather than a missing upstream fix — most likely in Determinate's SinkToSource / narFromPath coroutine connection-handling patches (e.g. a51e93ad, 034fe11d), which touch the same area upstream addressed in NixOS/nix#6950 / PR #14998 ("Do not mark connections as bad when RemoteStore::narFromPath is called as a coroutine").

Practical impact: deploy-rs --remote-build (which copies the system derivation closure over ssh-ng) stalls for 15+ minutes in the handshake on a ~5000-path NixOS system closure, appearing hung. Regular deploys use ssh:// and are unaffected.

Steps to reproduce

Pick a large derivation closure the destination already has (so 0 paths transfer), and copy it over both store types:

# closure the remote already holds → nothing should be sent
$ nix path-info -r <system>.drv | grep -c '\.drv$'
5100

# LEGACY ssh:// — one batched valid-paths query
$ time nix copy -s --derivation --to ssh://user@remote '<system>.drv^out'
copying 0 paths...
real  0m6s

# ssh-ng:// — same closure, same 0 paths, Determinate client
$ time nix copy -s --derivation --to ssh-ng://user@remote '<system>.drv^out'
copying 0 paths...
real  >6m40s   # timed out at 400s, still not finished

Measurements

Same 5100-path closure, already present on the destination (0 paths copied), same host/link:

client (all nix 2.34.7) / store time
Determinate 3.21.2, ssh:// 6 s
Determinate 3.21.2, ssh-ng:// >400 s (timed out)
upstream nixpkgs#nix 2.34.7, ssh-ng:// 8 s

So it is specific to the Determinate ssh-ng client — upstream nix of the same version (2.34.7) does the identical ssh-ng handshake in ~8 s. strace shows the client blocking ~1 s per path, and (with an SSH agent) a signing round-trip per path, consistent with a connection reopen per narFromPath coroutine call.

Workaround: run deploys with upstream nix in PATH (nix shell nixpkgs#nix -c deploy --remote-build ...) — 38 s end-to-end vs 25+ min.

Expected behavior

Determinate's ssh-ng client should match upstream nix 2.34.7: do the valid-paths negotiation in one batched round-trip (like ssh://) instead of reopening the connection per path.

Metadata

Determinate Nixd daemon version: 3.21.2
Determinate Nixd client version: 3.21.2
nix (Determinate Nix 3.21.2) 2.34.7

Likely related: #441 (ssh-ng fails with SSH ControlMaster multiplexing) — same connection-handling layer.

Activity

  1. edolstra commented on Jul 7, 2026

    @edolstra
    edolstra
    on Jul 7, 2026
    Collaborator

    This is fixed in #538 (which should be in the next release). However, it will require upgrading both the client and the server.

  2. edolstra
    closed this as completedin #538on Jul 7, 2026
  3. andre4ik3
    added a commit that references this issue on Jul 9, 2026

    libstore: Add an AddTempRoots daemon protocol operation

    7beda55

  4. szlend commented on Jul 10, 2026

    @szlend
    szlend
    on Jul 10, 2026

    However, it will require upgrading both the client and the server.

    Does this mean DetNix is now incompatible with upstream Nix? DetNix v3.21.5 hangs for me when copying a large derivation to a Nix v2.34.7 host.

  5. brett commented on Jul 10, 2026

    @brett
    brett
    on Jul 10, 2026
    Author

    Does this mean DetNix is now incompatible with upstream Nix? DetNix v3.21.5 hangs for me when copying a large derivation to a Nix v2.34.7 host.

    Yes. Unless DetNix adds a fallback mode, the best answer I've found is to just use nixpkgs#nix for those operations.

  6. CertainLach commented on Jul 10, 2026

    @CertainLach
    CertainLach
    on Jul 10, 2026

    Unless DetNix adds a fallback mode

    It uses featureAddTempRoots protocol-level feature flag, so it should work, and it works for me

    The fallback is here:
    https://github.com/DeterminateSystems/nix-src/pull/538/changes#diff-9338f330e24f13e71e33fcd53698bb563f916fb823331f0484c1a1674974e896R703-R715

  7. brett commented on Jul 11, 2026

    @brett
    brett
    on Jul 11, 2026
    Author

    It uses featureAddTempRoots protocol-level feature flag, so it should work, and it works for me

    featureAddTempRoots is already in v3.21.5, but it's protocol-negotiated — against an upstream nix 2.34.7 server it falls back to the per-path path, which on a large closure over high latency is the "hang" I'm seeing.

    Unless I'm doing something wrong, that doesn't help until upstream ships the batched op server-side. If I'm missing something obvious, please let me know!

  8. szlend commented on Jul 11, 2026

    @szlend
    szlend
    on Jul 11, 2026

    Yeah, to me it looks like the fallback is to the regressed slow path. So right now it's not usable for managing upstream NixOS hosts.

    Edit: It's been more than a month that I've been unable to manage NixOS hosts with Determinate Nix. At this point it looks like there's very little interest in maintaining compatibility with upstream Nix, and as such I've migrated all my machines back to upstream Nix. Good bye.

  9. brett commented on Jul 21, 2026

    @brett
    brett
    on Jul 21, 2026
    Author

    @edolstra : can we reopen this since other users will likely keep hitting this issue?

  10. simonzkl commented on Jul 23, 2026

    @simonzkl
    simonzkl
    on Jul 23, 2026

    I don't understand what we're supposed to do now when using Determinate Nix. It's not like Determinate makes it easy to maintain Determinate Nix on NixOS hosts. It's not packaged for nixpkgs nor as a downstream flake package. The official suggestion is to manually install it on every single host, and then manually maintain it, ironically making Nix the only package not managed through Nix.

    I think at this point I'd rather just use upstream Nix which has none of these problems.

  11. kidonng
    added a commit that references this issue on Aug 3, 2026

    Use Lix instead of Determinate Nix

    032f7a6

  12. edolstra commented on Aug 20, 2026

    @edolstra
    edolstra
    on Aug 20, 2026
    Collaborator

    We've dropped the registering of remote roots when talking to a daemon that doesn't support efficient remote root registration (#598). This will be included in the next release.

  13. edolstra
    added a commit that references this issue on Aug 26, 2026

    libstore: Add an AddTempRoots daemon protocol operation

    9b1503f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions