benchmark: keep service ports out of the ephemeral pool, and stop the cache failing silently

[MDA2AV]

production-stack fails at the end of a full run and passes when it is the only profile. Both failures were identical:

cache-1    exited 1    Could not create server TCP listening socket 0.0.0.0:6379: bind: Address in use
authsvc-1  exited 101  bind 0.0.0.0:9090: Address in use (os error 98)

Nothing in the entry causes either, and nothing in the repo binds 9090 except authsvc itself.

9090 — the ports are in the ephemeral pool

system.sh widens the range so connection-churning profiles don't exhaust it:

sudo sysctl -w net.ipv4.ip_local_port_range='1024 65535'

Which also puts every port a server listens on into the pool the kernel draws outbound ports from. Across a full run — 17 profiles, millions of connections at up to 16,384 concurrent — a load-generator socket ends up holding 9090, and the stack starting afterwards can't bind it:

run	production-stack position	result
12:01	first and only profile	✅
11:27	last, after 17 profiles	❌ 9090 in use
12:16	last, after 17 profiles	❌ 9090 in use

tcp_tw_reuse doesn't help — it governs outbound reuse, not a listening bind.

ip_local_reserved_ports takes them out of ephemeral allocation while leaving them bindable, which is exactly what it's for. Reserved: 8080, 8081, 8082, 8443 from common.sh, 9090 for the auth sidecar, 6379 and 5432 for the Redis and Postgres sidecars. 8080/8443 haven't been hit yet only because they're bound early in a profile rather than at the end of a run.

6379 — older, never a failure, and worse for it

redis_start runs once per run whenever the entry subscribes to crud, and the sidecar holds the host's 6379 for the whole run. production-stack ships its own cache on the same port. fulmine is the only entry subscribed to both — which is why aspnet-minimal_nginx and sark-production never saw it.

It collided every time, and nothing noticed: the server depended on the cache with the short depends_on: - cache form, which waits only for the container to start. The cache died, the server carried on against the harness Redis, and the profile published numbers measured against a cache it never configured — cpuset 0,64 with one io-thread, instead of the 16,80 and four io-threads the compose file asks for.

Two changes:

• the three production-stack compose files give cache a healthcheck and depend on it being healthy, turning a silent downgrade into a failed stack
• _gateway_yield_redis() stops the harness sidecar before bringing up a stack whose compose defines a cache service, so the port is actually free. gateway_down starts it again — ahead of its early returns, and in a subshell so redis_start's fail can't exit the cleanup trap

Docs

The behaviour changed, so the pages that specify the profile change with it:

• test-profiles/gateway/production-stack/implementation.md — "Required compose settings" now states both dependencies that must be conditions rather than the short list form, with the cache healthcheck written out and the reason spelled out. A new "Ports the stack owns" section lists the four host ports and the two things the driver does to keep them free.
• the same requirement goes into the type-rules widget for Standard and Tuned, since it is now a rule an entry has to satisfy rather than advice
• hardware/_index.md — ip_local_reserved_ports documented next to the range it compensates for, with the failure that motivated it, and a note that tcp_tw_reuse governs outbound reuse rather than a listening bind
• running-locally/scripts/benchmark.md — picks it up in the host-tuning step and links across

Verified

Against real containers:

--- stack that ships a cache (production-stack) ---
[info] stopping the harness redis sidecar: this stack ships its own cache on the same port
  GATEWAY_STOPPED_REDIS=true      (redis released 6379)
--- gateway_down restores it ---
  running: httparena-redis
--- stack with NO cache (gateway-64) leaves redis alone ---
  GATEWAY_STOPPED_REDIS=false     running: httparena-redis

Compose files parse, both scripts pass bash -n, and the doc pages build with every link resolving — the two new cross-references land on /docs/hardware/#kernel-tuning-applied-per-run and /docs/test-profiles/h1/isolated/crud/implementation/, and the type-rules widget still renders its three tabs.

Not exercised locally: the sysctl value — this box has no passwordless sudo. net.ipv4.ip_local_reserved_ports exists on the kernel here and reads empty; worth a glance at the first run's log to confirm it isn't warning.

Follows #1189, which made these failures legible in the first place — without its log dump this was just exited (101).

🤖 Generated with Claude Code