Skip to content

cli/command: use longer init-ping timeout for ssh connections - #7337

Open
adisivaprasad wants to merge 1 commit into
docker:masterfrom
adisivaprasad:ssh-init-ping-timeout
Open

adisivaprasad wants to merge 1 commit into
docker:masterfrom
adisivaprasad:ssh-init-ping-timeout

Conversation

@adisivaprasad

Copy link
Copy Markdown

- What I did

Fixes #6125

The CLI performs an initial ping to the daemon during (lazy) initialization, which is used for API-version negotiation and to collect server info (e.g. whether experimental features are enabled, and the daemon's OS type). This ping has a short (2 second) timeout to prevent the CLI from hanging when connecting to a local daemon that is not running (#3652, #4226).

Establishing a connection through the ssh connection helper requires a TCP connection, host key verification, and authentication before the connection to the daemon is ready to be used, which can take longer than this timeout, in particular when connecting to a host for the first time. In that case the initial ping is aborted, and API-version negotiation is skipped: the client keeps its default (maximum) API version, and requests to a daemon with a lower API version fail with errors such as:

Error response from daemon: client version 1.56 is too new. Maximum supported API version is 1.55

This can be observed with docker --debug --context <ssh-context> version, which dials the daemon twice on a slow (first) connection (the first connection is killed when the ping's 2-second deadline expires), and prints the client's default (maximum) API version instead of the negotiated version:

$ DOCKER_HOST=ssh://remote docker --debug version
time="..." level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- remote docker system dial-stdio]"
time="..." level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- remote docker system dial-stdio]"
Client:
 Version:           29.8.1
 API version:       1.56      <- not negotiated
...

- How I did it

The ssh connection helper sets a default ConnectTimeout=30 for the ssh process (cli/connhelper), so this change allows the initial ping for ssh:// endpoints to complete within that time, plus some additional time for the handshake and the ping itself (sshInitTimeout = 32s). Non-ssh connections (local sockets, tcp) keep the default (short) timeout, as those connections are either established quickly, or fail quickly when the daemon is not running, so their fast-fail behavior is unchanged.

- How I verified it

Reproduced the issue by simulating a slow ssh connection (an ssh wrapper that delays the connection by ~3 seconds, i.e. longer than defaultInitTimeout) against a local daemon with a lower API version than the client:

Before (client built from master):

$ PATH=/tmp/slowssh:$PATH DOCKER_HOST=ssh://localhost ./docker --debug version
time="..." level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- localhost docker system dial-stdio]"
time="..." level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- localhost docker system dial-stdio]"
 API version:       1.56

After:

$ PATH=/tmp/slowssh:$PATH DOCKER_HOST=ssh://localhost ./docker --debug version
time="..." level=debug msg="commandconn: starting ssh with [-o ConnectTimeout=30 -T -- localhost docker system dial-stdio]"
 API version:       1.55 (downgraded from 1.56)

The initial ping now completes within the longer timeout, the connection is reused for the subsequent request (only one ssh connection is started), and the API version is negotiated correctly.

Also added unit tests:

  • TestGetInitTimeout: verifies the timeout selection per endpoint type (default for unix/tcp, longer for ssh, explicit initTimeout override takes precedence for both).
  • TestInitializeFromClientSlowConnection: simulates a daemon that is slow to connect to (dial takes longer than defaultInitTimeout), and verifies that for an ssh:// endpoint the initial ping completes and the API version is negotiated, whereas non-ssh endpoints keep the default fast-fail behavior.

go test ./cli/command/... passes, and golangci-lint (v2.10.1, as used in CI) reports no issues.

The CLI performs an initial ping to the daemon during (lazy)
initialization, which is used for API-version negotiation and to
collect server info (e.g. whether experimental features are enabled,
and the daemon's OS type). This ping has a short (2 second) timeout
to prevent the CLI from hanging when connecting to a local daemon
that is not running.

Establishing a connection through the ssh connection helper requires
a TCP connection, host key verification, and authentication before
the connection to the daemon is ready to be used, which can take
longer than this timeout, in particular when connecting to a host
for the first time. In that case, the initial ping is aborted and
API-version negotiation is skipped: the client uses its default
(maximum) API version, and requests to a daemon with a lower API
version fail with errors such as:

    Error response from daemon: client version 1.56 is too new.
    Maximum supported API version is 1.55

This can be observed with 'docker --debug --context <ssh-context>
version', which dials the daemon twice on a slow (first) connection,
and prints the client's default (maximum) API version instead of the
negotiated version.

The ssh connection helper sets a default ConnectTimeout=30 for the
ssh process, so allow the initial ping for ssh connections to complete
within that time, plus some additional time for the handshake and the
ping itself. Non-ssh connections (local sockets, tcp) keep the default
(short) timeout, as those connections are either established quickly,
or fail quickly when the daemon is not running.

Fixes docker#6125

Signed-off-by: adisivaprasad <adisivaprasad@users.noreply.github.com>
@pengu-fr

pengu-fr commented Oct 5, 2026

Copy link
Copy Markdown

Another case hit by the 2s init-ping deadline, which this PR helps with but does not fully fix: interactive SSH authentication (passphrase-protected key loaded on demand with AddKeysToAgent, prompted through SSH_ASKPASS; same with a FIDO2 -sk key waiting for a touch).

With such a setup, every docker command that needs to load the key prompts twice:

  1. The init ping starts ssh, which opens the askpass dialog.
  2. After 2 seconds the ping is cancelled and commandConn.kill() sends SIGTERM to ssh only. The askpass process is in the same process group (the helper runs with Setsid) and keeps running: the dialog stays on screen, and the passphrase typed into it is discarded.
  3. cmd.Wait() then blocks until that dialog is closed, because the askpass process still holds the inherited stderr pipe.
  4. The actual request starts a second ssh, which prompts again.

Reproduction without a real key (empty agent, fake askpass that waits 5s), docker 25.0.2. The relevant code is unchanged on master:

SSH_AUTH_SOCK=/nonexistent SSH_ASKPASS=./fake-askpass SSH_ASKPASS_REQUIRE=force strace -f -tt -e trace=execve,kill docker -D ps

09:34:41.270 execve("/usr/bin/ssh", [... "localhost", "docker", "system", "dial-stdio"])   # ping
09:34:41.344 execve("./fake-askpass", [... "Enter passphrase for key ..."])
09:34:43.270 kill(91472, SIGTERM)                                                        # 2s deadline
09:34:46.357 askpass exits, ssh 91472 already gone
09:34:46.366 execve("/usr/bin/ssh", [... "dial-stdio"])                                   # actual request
09:34:46.438 execve("./fake-askpass", [...])                                             # second prompt

About the 32s value: ConnectTimeout does not bound authentication. Per ssh_config(5), it "is applied both to establishing the connection and to performing the initial SSH protocol handshake and key exchange". So a user typing a passphrase, or reaching for a security key, can still take longer than 32s, and will get the double prompt again.

I'd suggest not applying an init deadline at all for ssh:// endpoints:

  • an unreachable or slow host is already bounded by ConnectTimeout=30;

  • authentication is bounded by the user, as with any ssh invocation;

  • a daemon that accepts the connection but never answers is not protected by the init deadline anyway. With a unix socket that accepts and never replies, docker ps still hangs after the 2s ping fails (as noted in cli: set timeout connection ping on sockets as well #3722):

    DOCKER_HOST=unix:///tmp/dead.sock timeout 20 docker --debug ps   # exit 124 after 20s
    

Separately, commandConn.kill() could signal the process group (syscall.Kill(-pid, SIGTERM)) rather than the ssh process alone. Any helper ssh spawned (askpass, ProxyCommand) would then be terminated together with it, whatever the reason for the cancellation. I can open a separate issue for that if preferred.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

api-version negotiation sometimes fails over SSH connection

2 participants