fix(agent): install agent using an absolute remote path by aqandrew · Pull Request #10 · coder/mutagen

aqandrew · 2026-09-01T05:04:56Z

fixes DEVEX-522

Summary

Mutagen copies the agent binary to the remote with scp and then executes it over ssh, both using a ~/-prefixed path (~/.mutagen-agent<UUID>). This assumes ~ resolves to the same place for both steps, which only holds when the remote SSH/SFTP working directory is the user's $HOME.

Coder workspaces can configure a different working directory:

coder_agent.dir set to a non-home path (File sync fails when coder_agent.dir is set to a non-home directory coder-desktop-macos#238), or
a devcontainer whose workspaceFolder differs from $HOME (Linear DEVEX-522), e.g. workspaceFolder=/home/ubuntu/figma/figma while $HOME=/home/ubuntu.

In those cases scp writes the binary relative to the working directory while the ssh exec expands ~ to $HOME, so installation fails:

unable to invoke agent installation: remote error:
zsh:1: no such file or directory: /home/ubuntu/.mutagen-agent<UUID>

Change

In pkg/agent/install.go, resolve the absolute remote home directory once (via echo "$HOME" over the transport) and use it to build an absolute path for both the scp copy and the ssh invocation, so they always agree regardless of the remote working directory. If home resolution fails or returns a non-absolute value, it falls back to the previous ~/-relative behavior. Scoped to POSIX remotes; cmd.exe behavior is unchanged (it already uses a relative name).

Why absolute rather than a relative path: absolute is the only form that is correct across all Coder agent versions (before and after the SFTP working-directory change in coder/coder#21194), because it bypasses cwd/tilde conventions entirely.

Testing

go build ./pkg/agent/ and go vet ./pkg/agent/ pass; gofmt clean.
New unit test TestRemoteHomeDirectory covers trimming, empty/non-absolute/unexpanded output, and command failure. Passes.
Path resolution was verified empirically against a live Coder agent (dir != $HOME): scp host:~/x and scp host:x both land in the working dir (scp strips ~/), ssh '~/x' resolves to $HOME, and absolute paths resolve identically for both — confirming this fix aligns the two steps.
Pre-existing TestExecutableForPlatform* failures are unrelated (they require a built agent bundle in ./build).

Rollout (follow-ups, not in this PR)

Cut a coder/mutagen release and publish artifacts, then bump Resources/.mutagenversion in the Coder Desktop clients (macOS and Windows).

Context

Chosen approach per Linear discussion (fix in the Mutagen fork; "most compatible behaviour" and the fork is already maintained). Alternatives considered — changing coder ssh to land in $HOME, or a devcontainer cwd toggle — were rejected to preserve the "SSH lands in the workspace folder" behavior. A coder/coder SFTP ~-expansion fix is not viable because scp strips ~/ before the server sees it.

_{🤖 Opened by Coder Agents on behalf of @aqandrew. Draft for review.}

Implementation plan / decision log

DEVEX-522 — File Sync fails when the remote working directory differs from `$HOME`

Tracking: coder/coder-desktop-macos#238 (the coder_agent.dir variant, closed via
deprecation) and DEVEX-522 (the devcontainer workspaceFolder variant, still open).

1. Problem statement

Coder Desktop File Sync (Mutagen) fails to bootstrap its agent binary whenever the
remote SSH session's working directory is not the user's $HOME. The user sees:

unable to install agent: unable to invoke agent installation: remote error:
zsh:1: no such file or directory: /home/ubuntu/.mutagen-agent<UUID>

Two triggers, same root cause:

coder_agent.dir set to a non-home directory (issue Mutagen compose without mutagen compose mutagen-io/mutagen#238).
A devcontainer whose workspaceFolder differs from $HOME (DEVEX-522), e.g.
workspaceFolder = /home/ubuntu/figma/figma while $HOME = /home/ubuntu.
This reproduces even with coder_agent.dir unset, which is why Mutagen compose without mutagen compose mutagen-io/mutagen#238's
"unset dir" workaround and the provider deprecation do not resolve it.

2. Root cause (confirmed in code)

Mutagen bootstraps its agent in two steps that resolve paths through different
mechanisms:

Upload (scp/SFTP) — coder/mutagen pkg/agent/install.go:
```
fullRemotePath := remotePathFromHome(cmdExe, remoteFileName) // "~/.mutagen-agent<UUID>"
transport.Copy(agentExecutable, fullRemotePath)
```
transport.Copy (pkg/agent/transport/ssh/transport.go) runs scp with
destination host:~/.mutagen-agent<UUID>. The Coder agent's SFTP server
(agent/agentssh/agentssh.go → sftpHandler → sftp.WithServerWorkingDirectory(dir))
sets the SFTP working directory to the resolved session directory
(coder_agent.dir or, for a devcontainer sub-agent, workspaceFolder). The
binary therefore lands under that directory, not $HOME.
Exec (SSH shell) — pkg/agent/install.go installCommand := fullRemotePath + " install"
runs ~/.mutagen-agent<UUID> install over SSH. The remote login shell expands
~ to $HOME, so it looks in $HOME and fails with the absolute-path error
above.

The Coder agent SFTP working-directory change landed in coder/coder#21194
(SFTP/SCP now respect coder_agent.dir); the original scp-vs-ssh CWD mismatch is
coder/coder#16568.

Empirically verified path resolution (live probe against a real agent)

Tested against a Coder workspace whose agent has dir = /home/coder/coder and
$HOME = /home/coder (scp/sftp/ssh via the Coder net):

Client sends	Lands / resolves at	Mechanism
`scp host:X` (relative)	`dir`	SFTP working dir (#21194)
`scp host:~/X`	`dir` (no literal `~` dir created)	scp strips `~/` client-side → relative → SFTP working dir
`scp host:/abs/X`	`/abs`	absolute bypasses working dir
`ssh host '~/X'`	`$HOME`	login shell expands `~`
`ssh host 'X'` (relative)	`dir`	SSH command cwd = dir (#21194)

Decisive consequences:

scp upload dir and ssh command cwd are already identical (dir). The
mismatch survives only because Mutagen uses ~/, which scp strips to dir
but the shell expands to $HOME.
scp removes the leading ~/ before sending, so the agent SFTP server never
receives a tilde. A server-side ~-expansion fix therefore cannot intercept
it (confirmed: the interactive sftp client's cd ~ resolves to
dir/~ and fails — the server treats ~ as a literal component and never maps it
to home). This rules out a coder/coder SFTP fix for the scp/Mutagen path.
Absolute paths work correctly and identically for both scp and ssh, bypassing all
cwd/tilde conventions.

Note: the connect/invocation path (pkg/agent/dial.go agentInvocationPath →
~/.mutagen/agents/<version>/mutagen-agent) is not affected, because the
agent's own Install() relocates the binary using the remote's real home
(filesystem.Mutagen(...)) and the invocation runs over SSH where ~ also
resolves to $HOME. Both sides of that path agree. The only cross-transport
(scp + ssh) inconsistency is the temporary .mutagen-agent<UUID> upload path in
install.go. That is the single bug locus.

Devcontainer routing (confirmed in `coder/coder`)

File Sync lists every online agent (FileSyncSessionModal.swift), including
devcontainer sub-agents, and the user picks one; Mutagen SSHes to that host.
A devcontainer sub-agent is a separate agent process running its own
agentssh.Server inside the container (agent/agentcontainers/api.go
runSubAgentInContainer → cli/agent.go → agent.New/agent.go:415 SSH server).
Its WorkingDirectory comes from its own manifest Directory, set to the
container-internal workspaceFolder from devcontainer read-configuration
(api.go subAgentConfig.Directory = workspaceFolder).
The sub-agent's SFTP working directory is therefore workspaceFolder (via
resolveWorkingDirectory → WithServerWorkingDirectory).
The ExperimentalContainers && container != "" SFTP guard does not fire for a
direct connection to the sub-agent: container is only populated when the SSH
client sends CODER_CONTAINER (that's the coder ssh --container parent path).
The sub-agent's own env sets only CODER_AGENT_URL/TOKEN, so container == ""
and SFTP is allowed. BlockFileTransfer defaults to false.
Parent vs sub-agent are distinct tailnet hosts (per-agent UUID addresses), so
selecting the sub-agent routes straight to it.

Conclusion: in the Figma scenario (dir unset, devcontainer
workspaceFolder = /home/ubuntu/figma/figma), the failing transfer is served by the
devcontainer sub-agent (SFTP allowed, cwd = workspaceFolder). Syncing to the
parent agent (cwd falls back to $HOME) would not hit this bug. remoteUser /
containerUser can further shift $HOME and are worth capturing in repro.

3. Is Blink's analysis good? (answer to the third attachment)

Directionally correct and operationally useful, with two real flaws.

Right:

Same root cause as Mutagen compose without mutagen compose mutagen-io/mutagen#238 — workspaceFolder vs $HOME mismatch. Correct; later
confirmed by Beebs's ls (binaries under /home/ubuntu/figma/figma, error looks
in /home/ubuntu).
Diagnostic steps are genuinely useful: SSH into the container, compare $HOME
vs pwd, inspect remoteUser/containerUser, and test parent-agent vs
sub-agent targets. The user/home angle is a legitimate extra devcontainer variable.
Correctly recommends filing a separate issue for the workspaceFolder variant.

Wrong / weak:

The "suspicious absolute path" tangent is incorrect and contradicts its own
point 1. /home/ubuntu/.mutagen-agent... is exactly the expected output: the
exec command is ~/.mutagen-agent<UUID> install and the remote shell expands ~
to $HOME. It is not evidence of a different filesystem/user namespace or a
resolution anomaly. The two speculative theories overcomplicate a well-understood
mismatch.
The suggested workaround — set the File Sync session's remote path to an absolute
path under $HOME — will not fix it. The failure is in Mutagen's agent-binary
bootstrap path (~/.mutagen-agent<UUID>), which is independent of the sync
session's beta/remote path. Changing the sync target does not relocate the agent
upload/exec.
Stops at diagnosis; does not identify the fix (absolute-path resolution in the
Mutagen fork) or the container-SFTP-blocked constraint. Acceptable for a support
reply, but not a solution.

Blink's earlier (first) message adds nothing to the root cause but reinforces the
pattern:

Cause ssh: explicitly set ~ for home directory unless using cmd.exe #1 (coder_agent.dir non-home + #21194) was identified correctly from the
start, but hedged with "(or vice versa)" — the scp-vs-ssh direction was never
pinned. Evidence pins it: scp lands in dir/workspaceFolder, exec resolves
~→$HOME.
Cause fix: use a more generic powershell error fragment for Windows probing #2 ("zsh startup files changing the working directory") is a misdiagnosis.
The exec command is ~/.mutagen-agent<UUID> install; zsh expands ~ to $HOME
regardless of any cd in ~/.zshrc/~/.zprofile. The zsh:1: prefix only
identifies the login shell. This is another speculative wrong tangent on top of a
correct core diagnosis.
Its workaround ("remove/change dir so it defaults to $HOME") is correct for the
Mutagen compose without mutagen compose mutagen-io/mutagen#238 variant (unlike the message-3 remote-path workaround), but still does not fix
the DEVEX-522 devcontainer variant, where workspaceFolder reintroduces the
mismatch with dir unset.

Verdict: solid triage and a correct core diagnosis; discount the speculative theories
(zsh-startup, "suspicious path", filesystem/namespace) and the message-3 remote-path
workaround.

4. Proposed fix

4.0 Key finding: no server-side fix that preserves the workspace-folder cwd

The live probe (Section 2) plus the Linear discussion settle the fix location:

scp strips the leading ~/ client-side, so the agent never receives a tilde
to expand. A server-side ~-expansion option in pkg/sftp would be dead code for
the scp/Mutagen path (nobody proposed it, correctly).
The server cannot distinguish scp coder:~/foo (user means home) from
scp coder:foo (user means cwd) — both arrive as the same relative path.
The only coder/coder options that would work change or gate the devcontainer
cwd itself (Asher's options 1 and 2 — see 4d), sacrificing or opt-outing the
"SSH lands in the workspace folder" behaviour. The team rejected those in favour of
keeping that behaviour.

Decision (Linear, Atif): fix it in the coder/mutagen fork — "most compatible
behaviour" and "we already maintain a fork." This reverses the earlier
"no fork patch" stance specifically for the devcontainer case, where deprecation
cannot help.

4a. Primary change — `coder/mutagen` fork: absolute agent path

In pkg/agent/install.go (install), resolve the remote home directory once and use
an absolute path for both the scp transport.Copy and the ... install exec, so
they agree regardless of cwd/tilde conventions.

After probe(...), resolve $HOME over the SSH transport (POSIX: printf %s "$HOME"; cmd.exe: echo %USERPROFILE%). Validate non-empty + absolute; on failure
fall back to today's ~-relative behavior.
Build fullRemotePath := join(remoteHome, remoteFileName) (absolute) and use it for
transport.Copy, the chmod +x command, and the install command.
Leave remotePathFromHome / agentInvocationPath (dial.go) unchanged — that path
is already consistent.

Why absolute (not "drop the ~/ and go relative"): absolute is the only variant
correct across all server versions. The probe shows why:

Mutagen path style	pre-#21194 server (scp default = home, ssh cwd = dir)	#21194+ server (scp cwd = ssh cwd = dir)
`~/X` (current)	works	broken (this bug)
`X` (relative)	broken (#16568)	works
`/home/user/X` (absolute)	works	works

This is exactly the approach Atif described considering. His objection
("only fixes Coder Desktop; plain scp/vanilla Mutagen still misbehave when dir is
set") remains factually true — the probe confirms scp coder:~/foo still lands in
dir. But that objection was raised when deprecating dir was the chosen
alternative; for devcontainers there is no such alternative, and this change only
affects Mutagen's internal agent bootstrap (it does not alter user-facing scp
semantics). Requires team sign-off since it reverses the prior decision — see 4d.

4b. Tests — `coder/mutagen`

Unit-test remote-home resolution + absolute-path construction (POSIX and cmd.exe),
including the empty/$HOME-unset fallback.
Extend pkg/agent/install_test.go so a non-home working directory yields matching
upload + exec paths.

4c. Ship to Coder Desktop

Cut a coder/mutagen release (scripts/ci/build.sh builds mutagen-agents.tar.gz
- mutagen-darwin-{arm64,amd64}), publish to gs://coder-desktop/mutagen/<ver>/.
Bump Coder-Desktop/Coder-Desktop/Resources/.mutagenversion (currently v0.18.3)
in coder-desktop-macos; make clean/mutagen && make.
The Windows desktop client bundles the same fork — bump it there too.

4d. Decision (made in Linear) and the alternatives it beat

Atif selected option 3 — modify the Mutagen fork (this plan's 4a). Rationale:
most compatible behaviour; the fork is already maintained. Asher's other two options
were considered and rejected:

Change coder ssh to land in $HOME, not the workspace folder. Would fix the
mismatch (realigns scp-relative and ssh-~) but is enforced for all users
immediately — jarring, and drops the devcontainer "land in the repo" UX.
Add a toggle (e.g. agent flag/env devcontainers-ssh-into-workspace-folder,
default true) to opt out of workspace-folder cwd. Works, but adds surface area and
still doesn't fix the default configuration.
Modify the Mutagen fork (chosen). Keeps the workspace-folder cwd behaviour;
cost is that other SFTP tools (scp coder:~/foo, vanilla Mutagen) remain
inconsistent when dir/workspaceFolder is set — acceptable, and worth a docs
note (4c).

Implementation note: the fork change is the absolute-path bootstrap in 4a. Confirm
with Asher/Atif whether they want absolute-path resolution (robust across server
versions) versus any narrower variant before opening the PR.

4e. Non-goals

coder/coder SFTP ~-expansion: ineffective (scp strips the tilde). Dropped.
Weaken #21194 (make SFTP cwd $HOME while ssh stays dir): reintroduces
#16568.
Deprecate coder_agent.dir (already shipped for Mutagen compose without mutagen compose mutagen-io/mutagen#238): does not cover the
devcontainer workspaceFolder variant — this ticket is the follow-up.

5. Validation

Repro on a workspace with coder_agent.dir = /home/coder/coder (dogfood "Write
Coder on Coder"): confirm current failure, then confirm the patched build installs
the agent and sync reaches "Watching".
Repro the devcontainer variant: workspaceFolder != $HOME, coder_agent.dir
unset. Confirm target selection (sub-agent vs parent), then confirm the fix.
Regression: default template (dir = ~/home) still works.
ls -la ~/.mutagen/agents/ on the remote shows the installed agent; no stray
.mutagen-agent<UUID> temp files left under the working directory.
Cross-platform sanity: Linux amd64/arm64 remotes; a Windows remote if feasible.

6. Open questions / decisions before coding

Decision (made): Atif chose the Mutagen-fork workaround (Linear). Remaining
micro-decision: confirm absolute-path resolution vs. a narrower variant with
Asher/Atif before opening the PR.
Confirmed — devcontainer routing: the failing transfer is served by the
devcontainer sub-agent (its own agentssh.Server, cwd = workspaceFolder,
container == "" so SFTP is allowed). No separate "SFTP blocked" gap for the
sub-agent path. (One caveat noted by the trace: agent.go disables
a.devcontainers after seeing ParentID; it does not change the conclusion since
the guard needs container != "", which is empty here. Worth a quick runtime
confirmation during implementation.)
Confirmed — no server-side fix: scp strips ~/ before the agent sees it, so a
pkg/sftp ~-expansion option cannot help the scp/Mutagen path.
Remote-home resolution details in Mutagen: pick a robust probe (printf %s "$HOME") and define the fallback when it is empty/unresolvable.

7. Deliverables

coder/mutagen PR: absolute-path agent bootstrap in pkg/agent/install.go +
tests + release tag.
Artifacts published to gs://coder-desktop/mutagen/<ver>/.
coder-desktop-macos PR: .mutagenversion bump (+ proto regen only if the proto
surface changed). Same bump for the Windows desktop client.
Docs: note that with dir/workspaceFolder set, scp coder:~/foo lands in the
session working directory (expected #21194 behavior); File Sync now works via the
absolute-path bootstrap.
Explicitly not pursuing a coder/coder SFTP ~-expansion change (ineffective).

linear-code · 2026-09-01T05:05:00Z

DEVEX-522

Mutagen copies the agent binary with scp and then executes it over ssh using the same "~/"-prefixed path. This relies on "~" resolving to the same location for both steps, which only holds when the remote SSH/SFTP working directory is the user's home directory. Coder workspaces can configure a different working directory (via coder_agent.dir, or a devcontainer whose workspaceFolder differs from $HOME). In that case scp resolves the path relative to the working directory while the ssh exec expands "~" to $HOME, so the freshly-copied agent binary can't be found and installation fails with e.g.: no such file or directory: /home/ubuntu/.mutagen-agent<UUID> Resolve the absolute remote home directory once and use it for both the copy and the invocation so they agree regardless of the remote working directory. If resolution fails, fall back to the previous behavior. Fixes the file sync failure tracked in coder/coder-desktop-macos#238 and Linear DEVEX-522. Signed-off-by: Andrew Aquino <dawneraq@gmail.com>

aqandrew changed the title ~~fix(agent): install agent using an absolute remote path~~ fix: install agent using an absolute remote path Sep 1, 2026

aqandrew changed the title ~~fix: install agent using an absolute remote path~~ fix(agent): install agent using an absolute remote path Sep 1, 2026

aqandrew force-pushed the andrew/devex-522-agent-absolute-path branch from cc1eb98 to 20bde82 Compare September 1, 2026 18:36

aqandrew mentioned this pull request Sep 1, 2026

Revisit "dir breaks Coder Desktop file sync" warning after Mutagen fix ships coder/terraform-provider-coder#543

Open

aqandrew marked this pull request as ready for review September 2, 2026 18:15

aqandrew requested review from code-asher and spikecurtis September 2, 2026 18:19

Navigation Menu

Uh oh!

fix(agent): install agent using an absolute remote path - #10

fix(agent): install agent using an absolute remote path#10
aqandrew wants to merge 1 commit into
mainfrom
andrew/devex-522-agent-absolute-path

aqandrew commented Sep 1, 2026 •

edited

Loading

Uh oh!

linear-code Bot commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

aqandrew commented Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Change

Testing

Rollout (follow-ups, not in this PR)

Context

DEVEX-522 — File Sync fails when the remote working directory differs from $HOME

1. Problem statement

2. Root cause (confirmed in code)

Empirically verified path resolution (live probe against a real agent)

Devcontainer routing (confirmed in coder/coder)

3. Is Blink's analysis good? (answer to the third attachment)

4. Proposed fix

4.0 Key finding: no server-side fix that preserves the workspace-folder cwd

4a. Primary change — coder/mutagen fork: absolute agent path

4b. Tests — coder/mutagen

4c. Ship to Coder Desktop

4d. Decision (made in Linear) and the alternatives it beat

4e. Non-goals

5. Validation

6. Open questions / decisions before coding

7. Deliverables

Uh oh!

linear-code Bot commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

aqandrew commented Sep 1, 2026 •

edited

Loading

DEVEX-522 — File Sync fails when the remote working directory differs from `$HOME`

Devcontainer routing (confirmed in `coder/coder`)

4a. Primary change — `coder/mutagen` fork: absolute agent path

4b. Tests — `coder/mutagen`