Skip to content

Fix flaky up --wait failure on bucket-creation exit - #269

Merged
thunderkatz merged 2 commits into
devfrom
jkatzman/create-buckets-wait-exit-fix
Sep 8, 2026
Merged

Fix flaky up --wait failure on bucket-creation exit#269
thunderkatz merged 2 commits into
devfrom
jkatzman/create-buckets-wait-exit-fix

Conversation

@thunderkatz

Copy link
Copy Markdown

Summary

docker compose up --wait treats a container exiting — even with code 0 — as a failure unless some other service depends on it via condition: service_completed_successfully (docker/compose#10596, open since 2023, no upstream fix landed). Neither sublime_create_buckets nor sublime_create_azure_blob_containers has such a dependent: sublime_nipper/sublime_screenshot_service list sublime_create_buckets under plain depends_on (which only waits for it to start, not finish), and nothing depends on sublime_create_azure_blob_containers at all. So their normal, successful exit races the rest of the stack's healthchecks and can fail up --wait for no real reason.

This is what tripped a go-mantis CI shard:

container sublime-platform-sublime_create_buckets-1 exited (0)
Process completed with exit code 1.

(https://github.com/sublime-security/go-mantis/actions/runs/34249933616/job/102141472992)

sublime_storage_ready is a long-running no-op (sleep infinity) that depends on both one-shot containers via condition: service_completed_successfully, giving compose the dependent it needs to recognize their exit as expected — the documented workaround for this issue.

Is this related to #268?

No — orthogonal. #268 fixed a real but different bug: sublime_create_buckets racing MinIO startup with a blind sleep 15 and swallowing errors via an unconditional exit 0. That's about whether bucket creation succeeds. This PR is about --wait's handling of a successful exit, which is gated purely by the depends_on graph — something #268 didn't touch.

Concretely: the container still runs to completion and exits either way, before or after #268. In the failing run, sublime_create_buckets didn't exit until ~14s in, coinciding with sublime_hydra's custom healthcheck (which takes a while to pass) — but the pre-#268 entrypoint's fixed sleep 15 would have landed the exit in essentially the same window, so #268 doesn't look like it shifted the exposure meaningfully. It's plausible but not verified that #268 changed timing at the margins; either way it doesn't touch the actual root cause (the missing service_completed_successfully dependent).

Test plan

  • docker compose config --quiet on the resulting file — valid.
  • Reproduced the fix's effect against an isolated project (trimmed to sublimes3, sublime_azurite, sublime_create_buckets, sublime_create_azure_blob_containers, sublime_storage_ready): with the sentinel, docker compose up -d --wait exits 0 after both one-shot containers exit.
  • Watch go-mantis main-* CI shards over the next several days for absence of recurrence (this is an intermittent race — a single green run doesn't prove it).

Note: local testing used a newer docker compose (v5.5.1) than the CI runner, which reported the one-shot containers as Healthy rather than Exited even on the unmodified dev compose file — so the exact upstream trigger condition may be version-dependent, and I couldn't reproduce the CI failure signature locally byte-for-byte. The fix (adding a service_completed_successfully dependent) is the documented, version-agnostic workaround for this class of bug regardless.

🤖 Generated with Claude Code

docker compose up --wait fails whenever a container exits (even with
code 0) unless some other service depends on it via
condition: service_completed_successfully (docker/compose#10596, open
since 2023). Neither sublime_create_buckets nor
sublime_create_azure_blob_containers has such a dependent, so their
normal successful exit races the rest of the stack's health checks
and intermittently fails up --wait for no real reason. This is what
tripped a go-mantis CI shard: container sublime_create_buckets-1
exited (0), then "Process completed with exit code 1."

sublime_storage_ready is a long-running no-op that depends on both
one-shot containers completing, giving compose the dependent it needs
to treat their exit as expected.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@thunderkatz
thunderkatz requested a lite review from Copilot September 8, 2026 16:56
@datadog-sublime-security

datadog-sublime-security Bot commented Sep 8, 2026

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

⚠️ Your PR has warnings. Please review the issues below.

🚦 1 Pipeline job failed

Platform PR CI | Validate Platform Installation

View in Datadog · View in GitHub Actions

Bora container not found

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fe6d3df | Docs | View more details | Give us feedback!

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a known docker compose up --wait flake where one-shot “init” containers exiting successfully can still cause --wait to fail unless another service depends on them via condition: service_completed_successfully.

Changes:

  • Add a long-running sentinel service (sublime_storage_ready) that depends on sublime_create_buckets and sublime_create_azure_blob_containers via condition: service_completed_successfully.
  • Document the rationale and link the upstream Compose issue to explain the workaround.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docker-compose.yml Outdated
sleep infinity works with the pinned alpine:3.20 BusyBox (verified),
but tail -f /dev/null is the more universally portable no-op-forever
idiom, per PR review.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

@thunderkatz
thunderkatz marked this pull request as ready for review September 8, 2026 17:25
@thunderkatz
thunderkatz requested a review from a team September 8, 2026 18:00
@thunderkatz
thunderkatz merged commit 8408272 into dev Sep 8, 2026
3 of 4 checks passed
@thunderkatz
thunderkatz deleted the jkatzman/create-buckets-wait-exit-fix branch September 8, 2026 19:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants