Make MinIO bucket creation wait for readiness and fail loudly

[thunderkatz]

(HUMAN): Gimlet diagnosed a flake that struck a few times today, where we make a bucket, sleep for 15s, and then just hope it worked in time. Use a retry loop to ensure creation has succeeded. Opted to make this change for dev only since I'm unsure how much it matters for prod (if creation is just slow then it doesn't really matter the same way it matters for tests). The rest of the description is all Claude.

Summary

sublime_create_buckets currently races MinIO startup with a blind sleep 15 and swallows all errors via an unconditional exit 0. When MinIO is slow to accept connections, mc alias set / mc mb fail, but the container still exits 0, docker compose up --wait is satisfied, and the stack comes up with no buckets. Consumers then hit a confusing PutObject → 404 (NoSuchBucket) far from the real cause — this is how the issue surfaced as a go-mantis unit-test flake in TestQuantumHuntPhaseOneProcessor_StartQuantumHunt.

This replaces the fixed sleep with a bounded readiness retry loop and makes genuine failures fail the container, while keeping re-runs against an existing s3_data volume idempotent.

• Replace sleep 15 with a bounded retry loop (60 attempts × 2s) polling mc alias set — a real authenticated round-trip against the endpoint.
• Replace the ;-separated mc mb calls with &&-chained, --ignore-existing invocations so pre-existing buckets on a persistent s3_data volume are a no-op rather than a failure.
• Remove the unconditional exit 0 so real failures propagate and --wait actually gates on success.
• Pin minio/minio and minio/mc to explicit tags (previously unpinned latest), matching what was currently resolving.

Scope / who this affects

dev's docker-compose.yml is not the customer-facing installer path — install-and-launch.sh (the public installer at curl -sL https://sublimesecurity.com/install.sh | sh) does a plain git clone with no branch flag, so it checks out the repo's default branch (main). main's compose includes the real sublime_mantis / sublime_bora_lite services at pinned release versions; dev's compose omits both (dashboard points at host.docker.internal:8000), because dev is the dependency stack Sublime engineers and go-mantis CI use to run mantis/bora from source against containerized dependencies (postgres, redis, minio, strelka, tika, azurite, hydra, localstack).

So this change is scoped to that internal dev/CI dependency stack, not the shipped customer product. Installs that previously "succeeded" while silently failing bucket creation will now fail loudly instead for engineers/CI running the dev stack — that's the intended correction, not a regression.

A forward-port of this same fix to main should follow as a separate PR, since that's the branch with actual customer-facing risk (existing installs with a populated s3_data volume must stay idempotent there too).

Context

• go-mantis flake: https://github.com/sublime-security/go-mantis/actions/runs/33653429352 (sibling shard passed)
• Assertion site: https://github.com/sublime-security/go-mantis/blob/main/internal/services/hunts/quantum_hunt_phase_one_processor_test.go#L57
• No existing open PR addresses this (checked minio, create_buckets, healthcheck search terms and all 8 open PRs).

Follow-ups (not in this PR)

• Forward-port to main (see Scope note above).
• A healthcheck on sublimes3 + condition: service_healthy was considered but deferred — no probe binary has been verified against the pinned minio/minio tag (recent images dropped curl), and a wrong probe would mark the whole stack unhealthy. The retry loop is sufficient on its own.
• Recommended on the go-mantis side: a bucket preflight in the hunts TestMain so a missing bucket reports itself directly instead of surfacing as a 404 inside an unrelated test.
• Unrelated pre-existing CI break on dev: the "Validate Platform Installation" check has failed on every push to dev since at least April 2026 (Bora container not found) because install-and-launch.sh's health check looks for sublime_bora_lite, which only exists on main's compose. Not touched by this PR.

Test plan

• docker compose up -d --wait sublimes3 sublime_create_buckets on a running dev stack with a populated s3_data volume: retry loop absorbed one connection-refused attempt, --ignore-existing treated all pre-existing buckets as no-ops, container exited 0, --wait reported healthy.
• Verified pinned tags' digests match what latest currently resolves to.
• Verified --ignore-existing is supported by the pinned mc build.
• Verified the bounded-failure path (bad hostname) exits non-zero after max_attempts rather than hanging.
• Watch go-mantis main-* CI shards over the next several days for absence of recurrence (this was an intermittent race — a single green run doesn't prove the fix).

🤖 Generated with Claude Code

[datadog-sublime-security]

⚠️ Warnings

⚠️ Your PR has warnings. Please review the issues below.

🚦 1 Pipeline job failed

Platform PR CI | Validate Platform Installation

View in Datadog · View in GitHub Actions

Bora container not found

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8a3ab87 | Docs | View more details | Give us feedback!}

[Copilot]

Pull request overview

This PR hardens local/self-hosted MinIO bucket provisioning by replacing a fixed startup delay with a bounded readiness check and ensuring bucket-creation failures correctly fail the sublime_create_buckets container (so docker compose up --wait doesn’t report success when buckets were not created).

Changes:

• Pin minio/minio and minio/mc to explicit release tags instead of relying on latest.
• Replace sleep 15 with a bounded retry loop that waits for an authenticated mc alias set round-trip to succeed.
• Make bucket creation idempotent via mc mb --ignore-existing and remove the unconditional exit 0 so genuine failures propagate.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[cameron-dunn-sublime]

Nice!

This probably isn't important for main, but it's also not a bad idea. It's not important just because if there's a human in the loop it's generally going to be plenty of down time as someone goes through setup.

[thunderkatz]

Nice!

This probably isn't important for main, but it's also not a bad idea. It's not important just because if there's a human in the loop it's generally going to be plenty of down time as someone goes through setup.

Yup, that's what I figured.