Skip to content

Commit 77827fd

Browse files
committed
fix(ci): fit the install retry inside every caller and read a failed probe's log from its kept pod
1 parent f8c188a commit 77827fd

3 files changed

Lines changed: 17 additions & 15 deletions

File tree

‎.github/actions/setup/action.yml‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -69,16 +69,17 @@ runs:
6969
# Bun retries a failed request itself, but a registry connection that stalls
7070
# without failing leaves `bun install` waiting until the job's own timeout
7171
# cancels it, the whole budget spent on one hung step. Healthy installs finish
72-
# in under three minutes cold, so an attempt past five is stalled: kill it and
73-
# retry. Any other failure is real and fails at once.
72+
# in under two and a half minutes cold, so an attempt past three is stalled:
73+
# kill it and retry once, which leaves room inside the shortest caller's
74+
# 10-minute job timeout. Any other failure is real and fails at once.
7475
- name: Install dependencies
7576
shell: bash
7677
run: |
77-
for attempt in 1 2 3; do
78-
timeout --kill-after=30s 5m bun install --frozen-lockfile --ignore-scripts && exit 0
78+
for attempt in 1 2; do
79+
timeout --kill-after=15s 3m bun install --frozen-lockfile --ignore-scripts && exit 0
7980
status=$?
8081
if [ "$status" -ne 124 ] && [ "$status" -ne 137 ]; then exit "$status"; fi
81-
echo "::warning title=bun install stalled::attempt ${attempt} of 3 ran past 5 minutes"
82+
echo "::warning title=bun install stalled::attempt ${attempt} of 2 ran past 3 minutes"
8283
done
83-
echo "::error title=bun install stalled::all 3 attempts ran past 5 minutes"
84+
echo "::error title=bun install stalled::both attempts ran past 3 minutes"
8485
exit 1

‎.github/workflows/helm.yml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -188,11 +188,12 @@ jobs:
188188
--values helm/sim/ci/kind-overlay.yaml \
189189
--wait --timeout 15m
190190
191-
# `--logs` prints the probe's output, so a failure says which URL refused.
192191
- name: Run helm test
193-
run: helm test sim --namespace sim --timeout 5m --logs
192+
run: helm test sim --namespace sim --timeout 5m
194193

195-
# After the test, so it also explains a failed probe, not only a failed install.
194+
# After the test, so it also explains a failed probe, not only a failed
195+
# install. A failed test pod is kept (the hook deletes it only on success),
196+
# so its log says which URL refused.
196197
- name: Diagnostics on failure
197198
if: failure()
198199
run: |
@@ -202,6 +203,7 @@ jobs:
202203
kubectl -n sim logs deploy/sim-app -c migrations --tail=50 || true
203204
kubectl -n sim logs deploy/sim-app --tail=80 || true
204205
kubectl -n sim logs deploy/sim-realtime --tail=80 || true
206+
kubectl -n sim logs sim-test-connection || true
205207
206208
# Resolved once and shared, so the two publish paths cannot package the same
207209
# immutable chart version with different appVersions -- they run in parallel,

‎helm/sim/templates/tests/test-connection.yaml‎

Lines changed: 5 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -42,16 +42,15 @@ spec:
4242
set -eu
4343
# A pod turns Ready before every node routes its Service to it, so a
4444
# probe sent the moment `helm install --wait` returns can be refused.
45-
# Each URL gets a minute to answer before the test fails.
45+
# Each URL is retried until a minute has passed, then the test fails.
4646
probe() {
47-
attempt=1
47+
deadline=$(($(date +%s) + 60))
4848
while ! wget -q -T {{ .Values.tests.timeoutSeconds }} --spider "$1"; do
49-
if [ "$attempt" -ge 12 ]; then
50-
echo "FAILED: $1 did not answer after $attempt attempts"
49+
if [ "$(date +%s)" -ge "$deadline" ]; then
50+
echo "FAILED: $1 did not answer within 60s"
5151
return 1
5252
fi
53-
echo "Attempt $attempt: $1 did not answer; retrying in 5s"
54-
attempt=$((attempt + 1))
53+
echo "$1 did not answer; retrying in 5s"
5554
sleep 5
5655
done
5756
echo "OK: $1"

0 commit comments

Comments
 (0)