benchmark: clear another entry's compose stack before starting one, and say why a stack failed - #1189
Merged
Merged
Conversation
…nd say why a stack failed production-stack failed on #1182 with nothing in the log but an exit code: Container httparena-fulmine-production-stack-authsvc-1 Error dependency authsvc failed to start dependency failed to start: container ...-authsvc-1 exited (101) [FAIL] gateway compose up failed 101 is a Rust panic. Building frameworks/_shared/authsvc and running it twice reproduces it exactly: on a clean host it logs "authsvc listening on 0.0.0.0:9090" and answers /_health 200; with 9090 already held it exits 101 with "bind 0.0.0.0:9090: Address in use (os error 98)". Nothing about that is fulmine's. authsvc is the shared sidecar every production-stack entry builds, and every gateway and production stack runs network_mode: host on the same fixed ports - edge 8443, authsvc 9090, server 8080 - so no two can coexist. gateway_up already runs `down --remove-orphans`, but scoped to this framework's own compose project, so a stack left behind by another entry, or by a run killed between profiles, survives it. The only broader sweep is the one benchmark.sh does once at startup. _gateway_clear_stale() now removes running containers belonging to any other httparena-* compose project just before `up`, naming what it removed. It matches on the compose project label, so the harness's own `docker run` sidecars - httparena-postgres, httparena-redis - carry no such label and are untouched. The listing is split on `|` rather than whitespace: an unlabelled container prints an empty field, and with whitespace splitting its name would shift into the label's position and match the httparena- test by accident. The second half is why this took a rebuild to diagnose at all. compose reports the exit code and nothing else, so the reason never reached the run log. _gateway_dump_logs() prints each container of the failed stack through the existing dump_container_logs(), so the next failure names itself: ─── httparena-...-authsvc-1 — status=exited exit=101 oom=false error= ─── httparena-...-authsvc-1 — last 120 log lines ─── | bind 0.0.0.0:9090: Address in use (os error 98) Verified against real containers: two stale stacks from other projects are found, named and removed while an unlabelled sidecar survives, and a stack whose service exits 101 has its stderr printed. Leaves authsvc alone. Refusing to start when its port is taken is correct; the bug was the port still being taken.
This was referenced Aug 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
production-stackfailed on #1182 with nothing in the log but an exit code:What it was
101 is a Rust panic. I built
frameworks/_shared/authsvcand ran it two ways:Byte-for-byte the same failure. Nothing about it is fulmine's — that PR is a one-line
package.jsonbump, and authsvc is the shared sidecarfulmine,aspnet-minimal_nginxandsark-productionall build.Every gateway and production stack runs
network_mode: hoston the same fixed ports — edge 8443, authsvc 9090, server 8080 — so no two can coexist.gateway_upalready runsdown --remove-orphans, but scoped to this framework's own compose project, so a stack left behind by another entry, or by a run killed between profiles, survives it. The only broader sweep is the onebenchmark.shdoes once at startup.The fix
_gateway_clear_stale()removes running containers belonging to any otherhttparena-*compose project just beforeup, naming what it removed:It matches on the compose project label, so the harness's own
docker runsidecars —httparena-postgres,httparena-redis— carry no such label and are left alone. The listing is split on|rather than whitespace: an unlabelled container prints an empty field, and with whitespace splitting its name would shift into the label's position and match thehttparena-test by accident.And why it took a rebuild to diagnose
compose reports the exit code and nothing else, so the reason never reached the run log.
_gateway_dump_logs()prints each container of the failed stack through the existingdump_container_logs(), so the next one names itself:Verified
Against real containers, not by inspection: two stale stacks from other projects are found, named and removed while an unlabelled sidecar survives; and a stack whose service exits 101 has its stderr printed.
Not changed
authsvcitself. Refusing to start when its port is taken is correct behaviour — the bug was the port still being taken. Making it retry or bind elsewhere would hide exactly the collision this fixes.🤖 Generated with Claude Code