Skip to content

Commit 03ab6c7

Browse files
sunibclaude
andauthored
fix(e2e): fail fast on a wedged cluster; devcontainer Git identity + SSH signing bootstrap (#309)
* fix(e2e): fail fast when a stamped cluster is wedged `.stamps/cluster/<ctx>/ready` only proves a past bring-up succeeded. A cluster left wedged by a host suspend/resume or OOM keeps its stamp, so every task depending on `_cluster-ready` reads "up to date" and the run walks straight into the suite, which then hangs until the 30m ginkgo timeout. Add a live probe as a `preconditions:` entry rather than a `status:` one. As a status entry a failed probe would only re-run start-cluster.sh, which has to re-derive the same verdict slowly inside its 60-attempt retry loop; as a precondition it aborts immediately with the cleanup instructions. `test -f .../ready || exit 0` keeps the first run (nothing stamped, no cluster yet) out of the way. Also lower the health-path kubectl timeouts in start-cluster.sh to 5s and give the post-bring-up sanity `get ns` a timeout, so no step in the bring-up graph can hang unbounded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(devcontainer): let Git enforce signing; decouple it from container creation Creating a usable development environment, personalizing Git, and having signing credentials available are three separate things. The scripts treated them as one: post-create.sh required an identity, and sync-signing-key.sh required a forwarded SSH agent, both under `set -e`. A container created before any interactive session attaches -- a Dev Containers CLI `devcontainer up`, or a platform that personalizes the workspace afterwards -- could therefore never finish creating. Signing stays mandatory, but plain Git configuration now enforces it rather than shell logic: commit.gpgsign=true gpg.format=ssh gpg.ssh.defaultKeyCommand=ssh-add -L With those three set, a commit with no key reachable fails on its own ("fatal: either user.signingkey or gpg.ssh.defaultKeyCommand needs to be configured"), so no hook has to fail to keep unsigned commits impossible, and nothing else in the environment is held back by a missing key. Key selection keeps its existing meaning -- an agent key whose comment contains the Git email wins over agent order -- but the match is now pinned as a literal `key::` value instead of a public key file this script wrote and configured by path. With no match there is no pin at all, and the key command resolves the key per commit, so the common case can no longer go stale. The previous file-based pin is retired on the next sync, and only when it is still the exact path this script managed. A signing key configured by the developer or by the platform is never overwritten. Fixes the allowed_signers principal, which was "Name <email>". That file takes bare identities, so ssh-keygen rejected the line and a correctly signed commit failed to verify with "allowed_signers:1: invalid key / No principal matched". It is now the Git email, with the key comment kept, and every key in the agent is listed so verification works whichever one signed. Identity resolution keeps its existing Git-config-then-GIT_USER_NAME/ GIT_USER_EMAIL precedence and is now a warning rather than a hard stop. gpg.format, gpg.ssh.defaultKeyCommand and gpg.ssh.allowedSignersFile are only filled in when unset, so a developer or platform that has already chosen a signing mechanism keeps it. Nothing platform-specific is added. test-signing.sh covers all of it against a disposable ssh-agent and generated throwaway keys, including that a commit is refused when no key is available and verifies for the configured email when one is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(devcontainer): keep the gh CLI logged in across rebuilds ~/.config/gh was not among the persisted volumes, so every container rebuild dropped the token and gh went back to "To get started with GitHub CLI, please run: gh auth login". Mount it as a named volume, the same way .claude, .codex and .kube are already handled. The first rebuild after this starts from an empty volume and needs one more login; after that it sticks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
1 parent 487917c commit 03ab6c7

8 files changed

Lines changed: 490 additions & 76 deletions

File tree

‎.devcontainer/README.md‎

Lines changed: 88 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,11 @@ git config --global user.email "you@example.com"
2424

2525
If `ssh-add -L` shows no keys, commit signing inside the devcontainer will fail.
2626

27+
If the environment creating the devcontainer has no Git config to hand over, set `GIT_USER_NAME` and
28+
`GIT_USER_EMAIL` in that environment instead; [`devcontainer.json`](./devcontainer.json) forwards
29+
both through `${localEnv:...}` and `post-create.sh` writes them into the container's global Git
30+
config. Configured Git config always wins over these variables.
31+
2732
### VS Code
2833

2934
1. Install the [Dev Containers extension](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers)
@@ -43,31 +48,66 @@ bash -ic 'complete -p task >/dev/null && echo task completion ok'
4348
docker version
4449
gh --version
4550
git config --get gpg.format
46-
git config --get user.signingkey
4751
ssh-add -L
4852
```
4953

5054
Expected Git signing values:
5155

5256
```bash
53-
git config --get gpg.format # ssh
54-
git config --get commit.gpgsign # true
55-
git config --get user.signingkey # /home/vscode/.ssh/devcontainer_signing_key.pub
57+
git config --get gpg.format # ssh
58+
git config --get commit.gpgsign # true
59+
git config --get gpg.ssh.defaultKeyCommand # ssh-add -L
60+
git config --get gpg.ssh.allowedSignersFile # /home/vscode/.config/git/allowed_signers
5661
```
5762

63+
`user.signingkey` is set only when an agent key comment matches your Git email; otherwise it stays
64+
unset and the key is read from the agent at commit time.
65+
5866
## How SSH Signing Works Here
5967

6068
The devcontainer uses SSH commit signing, not GPG keyring-based signing.
6169

62-
- [`post-create.sh`](./post-create.sh) bootstraps Git identity and calls the signing sync helper
63-
- [`sync-signing-key.sh`](./sync-signing-key.sh) reads the forwarded SSH agent and refreshes:
64-
- `${HOME}/.ssh/devcontainer_signing_key.pub`
65-
- `${HOME}/.config/git/allowed_signers`
66-
- [`devcontainer.json`](./devcontainer.json) runs the helper on:
67-
- `postCreateCommand`
68-
- `postStartCommand`
70+
Most of it is plain Git configuration rather than shell logic:
71+
72+
```bash
73+
commit.gpgsign=true # every commit is signed
74+
gpg.format=ssh # signed with an SSH key
75+
gpg.ssh.defaultKeyCommand="ssh-add -L" # the key comes from the live agent
76+
```
77+
78+
With those three set, Git signs with whatever the agent currently holds, and refuses to commit
79+
when it holds nothing. No key is cached anywhere, so there is nothing to go stale when a
80+
reconnect changes the forwarded socket or the loaded key.
81+
82+
[`sync-signing-key.sh`](./sync-signing-key.sh) applies that baseline and adds the one thing Git
83+
cannot derive on its own: `${HOME}/.config/git/allowed_signers`, which maps the Git email to the
84+
keys currently in the agent so `git log --show-signature` can verify. It runs from
85+
[`post-start.sh`](./post-start.sh) on every start, and can be run by hand at any time.
86+
87+
If one of the agent keys has a comment containing your Git email, that key is pinned as
88+
`user.signingkey` so selection does not depend on agent order. With no such key the pin is left
89+
unset, which is both simpler and immune to going stale. A signing key you configured yourself is
90+
never overwritten.
91+
92+
[`post-create.sh`](./post-create.sh) does not configure signing. Creating the environment,
93+
personalizing Git, and having credentials available are three separate things, and only the first
94+
has to succeed for the container to be usable.
95+
96+
### When No Agent Has Attached Yet
97+
98+
A devcontainer can be created before any interactive session attaches, for example by
99+
`devcontainer up` from the Dev Containers CLI, or by a platform that personalizes the workspace
100+
afterwards. No SSH agent is forwarded at that point, and no Git identity may exist yet. Neither is
101+
treated as a failure to create the environment: the hooks report what is missing and finish.
69102

70-
This keeps the configured public key file aligned with the currently forwarded agent key.
103+
Signing is not relaxed to achieve that. `commit.gpgsign` is on from the start, so a commit without
104+
a usable key fails:
105+
106+
```text
107+
fatal: either user.signingkey or gpg.ssh.defaultKeyCommand needs to be configured
108+
```
109+
110+
Nothing else in the environment needs a key, so builds, tests and the E2E suite run regardless.
71111

72112
## Best Practices
73113

@@ -96,18 +136,24 @@ workflows.
96136
3. Keep signing setup in one small script
97137
This repo uses [`sync-signing-key.sh`](./sync-signing-key.sh) for that reason.
98138

99-
4. Prefer deterministic key selection
100-
The helper first looks for an SSH key whose comment matches `git user.email`.
101-
If none matches, it falls back to the first key from `ssh-add -L`.
139+
4. Prefer standard Git mechanisms over shell logic
140+
`gpg.ssh.defaultKeyCommand` selects the key whenever no comment matches the Git email, so there
141+
is usually no pinned key to become stale.
102142

103-
5. Keep signing and verification data in sync
104-
`user.signingkey` and `gpg.ssh.allowedSignersFile` are regenerated together.
143+
5. Trust every key the agent currently holds
144+
`allowed_signers` lists them all under the Git email, so verification works whichever one the
145+
key command picks.
105146

106147
6. Store only public key material in the container
107148
The private key stays on the host and is accessed through the forwarded SSH agent.
108149

109-
7. Fail early with actionable messages
110-
Missing agent, missing keys, or missing Git identity should stop setup immediately.
150+
7. Report, do not fail
151+
A missing identity or agent is diagnosed and setup continues. Git, not a shell script, is what
152+
refuses an unsigned commit.
153+
154+
8. Use the Git email as the `allowed_signers` principal
155+
That file takes bare identities, so a `Name <email>` principal makes `ssh-keygen -Y verify`
156+
report `invalid key` and `No principal matched` for a correctly signed commit.
111157

112158
### General Devcontainer Maintenance
113159

@@ -123,9 +169,8 @@ workflows.
123169
```bash
124170
echo "$SSH_AUTH_SOCK"
125171
ssh-add -L
126-
cat ~/.ssh/devcontainer_signing_key.pub
127172
cat ~/.config/git/allowed_signers
128-
git config --show-origin --get user.signingkey
173+
git config --show-origin --get-regexp '^(commit\.gpgsign|gpg\.)'
129174
```
130175

131176
## Optional `.env` for `gh`
@@ -151,6 +196,10 @@ Recommended token scopes:
151196

152197
The repo-root `.env` must stay local. It is already gitignored.
153198

199+
Alternatively just run `gh auth login` once. `~/.config/gh` is a named volume (`ghconfig`), so the
200+
login survives container rebuilds; the first rebuild after this mount was added starts from an
201+
empty volume and needs one more login.
202+
154203
## Troubleshooting
155204

156205
### Commit Signing Fails
@@ -159,8 +208,8 @@ Check:
159208

160209
```bash
161210
ssh-add -L
162-
cat ~/.ssh/devcontainer_signing_key.pub
163-
git config --get user.signingkey
211+
git config --get commit.gpgsign
212+
git config --get gpg.ssh.defaultKeyCommand
164213
```
165214

166215
Then refresh the signing setup manually:
@@ -173,15 +222,26 @@ Common causes:
173222

174223
- no host `ssh-agent` running
175224
- no key loaded into the host agent
176-
- the forwarded agent key changed after the container was created
177225
- `user.name` or `user.email` is missing
178226

179-
Typical signing error when the configured public key is stale:
227+
Typical error when no key is reachable:
180228

181229
```text
182-
error: No private key found for public key "/home/vscode/.ssh/devcontainer_signing_key.pub"?
230+
fatal: either user.signingkey or gpg.ssh.defaultKeyCommand needs to be configured
183231
```
184232

233+
### Commits Sign But Do Not Verify
234+
235+
```text
236+
allowed_signers:1: invalid key
237+
No principal matched.
238+
```
239+
240+
The principal in `~/.config/git/allowed_signers` is malformed. Each line is
241+
`<email> <keytype> <key> [comment]`; re-run `bash .devcontainer/sync-signing-key.sh` to regenerate
242+
it. Verify with `git log -1 --show-signature`, and check the helper itself with
243+
`bash .devcontainer/test-signing.sh`.
244+
185245
### Push Fails But Commit Signing Works
186246

187247
Commit signing and Git push authentication are separate concerns.
@@ -213,7 +273,9 @@ Usually normal. Rebuild time mostly depends on tool installation and cache reuse
213273
- [`Dockerfile`](./Dockerfile) - Multi-stage container image
214274
- [`devcontainer.json`](./devcontainer.json) - VS Code devcontainer configuration
215275
- [`post-create.sh`](./post-create.sh) - Initial bootstrap
276+
- [`post-start.sh`](./post-start.sh) - Per-start refresh
216277
- [`sync-signing-key.sh`](./sync-signing-key.sh) - SSH signing refresh helper
278+
- [`test-signing.sh`](./test-signing.sh) - Tests for the signing helper, using a disposable key
217279
- [`README.md`](./README.md) - This document
218280

219281
## References

‎.devcontainer/devcontainer.json‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,9 @@
9797
"source=codexconfig,target=/home/vscode/.codex,type=volume",
9898
"source=claudeconfig,target=/home/vscode/.claude,type=volume",
9999
"source=dotkube,target=/home/vscode/.kube,type=volume",
100+
// Keeps the gh CLI logged in across rebuilds. Without it every rebuild drops
101+
// ~/.config/gh and gh asks for `gh auth login` again.
102+
"source=ghconfig,target=/home/vscode/.config/gh,type=volume",
100103
"source=persisted-home,target=/home/vscode/persisted-home,type=volume"
101104
],
102105
"containerEnv": {

‎.devcontainer/post-create.sh‎

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -6,15 +6,13 @@ log() {
66
echo "[post-create] $*"
77
}
88

9-
fail() {
10-
echo "[post-create] ERROR: $*" >&2
11-
exit 1
12-
}
13-
149
workspace_dir="${1:-${containerWorkspaceFolder:-${WORKSPACE_FOLDER:-$(pwd)}}}"
1510
log "Using workspace directory: ${workspace_dir}"
1611

17-
# Resolve Git identity from effective config first, then fallback env vars.
12+
# Git identity is developer personalization, not part of building a usable
13+
# development environment, so its absence is reported and not fatal. Whatever
14+
# created this container (VS Code copying the host Git config, a platform's own
15+
# personalization step) may supply it before, during or after this hook.
1816
git_name="$(git config --get user.name || true)"
1917
git_email="$(git config --get user.email || true)"
2018

@@ -26,21 +24,23 @@ if [ -z "${git_email}" ] && [ -n "${GIT_USER_EMAIL:-}" ]; then
2624
git_email="${GIT_USER_EMAIL}"
2725
fi
2826

29-
if [ -z "${git_name}" ] || [ -z "${git_email}" ]; then
30-
fail "Missing Git identity. Set user.name and user.email in Git, or provide GIT_USER_NAME and GIT_USER_EMAIL to the devcontainer environment."
31-
fi
32-
33-
# Persist identity globally in the container if it is not already configured there.
34-
if ! git config --global --get user.name >/dev/null 2>&1; then
27+
if [ -n "${git_name}" ] && [ -z "$(git config --global --get user.name || true)" ]; then
3528
git config --global user.name "${git_name}"
3629
fi
3730

38-
if ! git config --global --get user.email >/dev/null 2>&1; then
31+
if [ -n "${git_email}" ] && [ -z "$(git config --global --get user.email || true)" ]; then
3932
git config --global user.email "${git_email}"
4033
fi
4134

42-
log "Refreshing Git SSH signing configuration"
43-
bash "${workspace_dir}/.devcontainer/sync-signing-key.sh"
35+
if [ -z "${git_name}" ] || [ -z "${git_email}" ]; then
36+
log "WARNING: no Git identity yet. Set user.name and user.email, or provide GIT_USER_NAME and"
37+
log "WARNING: GIT_USER_EMAIL to the devcontainer environment. Commits need one; the rest of the"
38+
log "WARNING: environment does not, so setup continues."
39+
fi
40+
41+
# Signing is not configured here. post-start.sh runs on every start, including
42+
# the first one right after this hook, and that is where an SSH agent can
43+
# actually be expected to exist.
4444

4545
log "Ensuring Go cache directories exist"
4646
sudo mkdir -p \

‎.devcontainer/post-start.sh‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,12 @@ set -euo pipefail
44

55
workspace_dir="${1:-${containerWorkspaceFolder:-${WORKSPACE_FOLDER:-$(pwd)}}}"
66

7-
bash "${workspace_dir}/.devcontainer/sync-signing-key.sh"
7+
# Refresh signing on every start: this is the first hook that can expect a
8+
# forwarded SSH agent, and a reconnect can change SSH_AUTH_SOCK or the loaded
9+
# key. The helper reports missing credentials rather than failing, and Git
10+
# itself is what refuses an unsigned commit, so startup never depends on them.
11+
bash "${workspace_dir}/.devcontainer/sync-signing-key.sh" ||
12+
echo "[post-start] WARNING: sync-signing-key.sh failed; run it by hand to see why." >&2
813

914
if [[ ! -f "${workspace_dir}/.env" ]]; then
1015
echo "hint: ${workspace_dir}/.env is absent; add GH_TOKEN there if you want read-only gh CLI access."

0 commit comments

Comments
 (0)