fix(runners): make dedicated-host (mac/windows) ASG deploys resilient to metal capacity - #1113
Open
coderbirju wants to merge 1 commit into
Open
fix(runners): make dedicated-host (mac/windows) ASG deploys resilient to metal capacity#1113coderbirju wants to merge 1 commit into
coderbirju wants to merge 1 commit into
Conversation
… to metal capacity
Rolling updates of the macOS/Windows runner ASGs terminate an instance
before its replacement is InService (minInstancesInService: 0), and hosts
are auto-released between deploys (auto-release-host: true). On scarce
mac2.metal capacity this causes a wedge: the old instance is gone, the
replacement can't launch ("Insufficient capacity"), the batch never
stabilizes, and the CloudFormation update fails and can get stuck in
UPDATE_ROLLBACK_FAILED - blocking Finch releases.
Changes (dedicated-host runners only; non-metal Linux ASGs unchanged):
- maxCapacity = desiredInstances + 1 (N+1 headroom) so a replacement can
launch before the old instance is terminated, and so a bad instance can
always be replaced without deadlocking.
- rollingUpdate.minInstancesInService = desiredInstances
(launch-before-terminate) to avoid the capacity gap.
- host resource group auto-release-host = false to retain the reserved
host slot between deploys instead of re-competing for metal capacity
(macOS hosts have a ~24h min allocation anyway).
Requires N+1 dedicated-host quota headroom (mac2 quota L-5D8DADF5 in
us-west-2 is 10, 7 in use - sufficient). Adds unit tests asserting the
N+1 capacity, launch-before-terminate policy, and host retention for
dedicated-host runners, and that non-metal ASGs keep maxCapacity=desired.
Signed-off-by: Arjun Yogidas <arjunry@amazon.com>
coderbirju
marked this pull request as ready for review
September 4, 2026 07:47
coderbirju
added a commit
that referenced
this pull request
Sep 4, 2026
…lock Prod wave (#1115) mac2.metal ARM capacity in us-west-2 is exhausted: only 1 healthy instance exists and AWS cannot allocate a 2nd (all available mac2 hosts at 0 free capacity). The Production-ASG-mac-finch-15-armStack ASG wants DesiredCapacity=2, so CloudFormation cannot stabilize and the stack wedges (UPDATE_ROLLBACK_FAILED), blocking the Prod wave and the downstream Release wave (which carries the Windows update). Reduce runnerProd mac/15.7/arm/finch desiredInstances 2 -> 1 to match currently available capacity so the deploy stabilizes on the single existing instance. TEMPORARY: restore to 2 (and/or rely on the N+1 durability fix in #1113) once mac2.metal capacity is healthy. Complementary to #1113, which ADDS N+1 headroom (needs more capacity); this REDUCES demand to match current capacity. Signed-off-by: Arjun Yogidas <arjunry@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
FinchPipeline deploys of the macOS/Windows runner ASGs intermittently wedge and block Finch releases. The rolling update terminates an instance before its replacement is InService (
minInstancesInService: 0), and dedicated hosts are auto-released between deploys (auto-release-host: true). On scarce mac2.metal capacity this creates a gap: the old instance is gone, the replacement can't launch (Insufficient capacity. Launching EC2 instance failed.), the batch never stabilizes, and the CloudFormation update fails — sometimes wedging the stack inUPDATE_ROLLBACK_FAILED(observed onProduction-ASG-mac-finch-15-armStack, which blocked the Prod wave and prevented the Release wave from running).The prior
maxCapacity = desiredInstancesalso means a bad instance cannot be replaced without deadlocking (no room to launch a replacement).Fix (dedicated-host runners only; non-metal Linux ASGs unchanged)
maxCapacity = desiredInstances + 1so a replacement can launch before the old instance is terminated, and a bad instance can always be replaced.rollingUpdate.minInstancesInService = desiredInstances— never drop below desired healthy during a rollout.auto-release-host = falseso the reserved host slot persists between deploys instead of re-competing for metal capacity (macOS hosts have a ~24h min allocation regardless, so retaining adds little effective cost).Together these guarantee a reserved slot for the replacement, eliminating the capacity gap and the resulting wedge, while still allowing instance replacement.
Capacity / quota
Requires N+1 dedicated-host quota headroom. In Prod us-west-2 the mac2 dedicated-host quota (
L-5D8DADF5) is 10 with 7 in use → sufficient for the +1 per mac arm stack. Recommend bumping to 12 for comfort during host recycle churn (separate quota request, not in this PR).Testing
npm run buildclean.MaxSize = desired+1,MinInstancesInService = desired,MaxBatchSize = 1, and host resource groupauto-release-host: false; and that non-metal (Linux) ASGs keepMaxSize = desired.Rollout note
This changes the ASG capacity + host retention; first deploy will allocate the extra retained host. Should be rolled out when mac2.metal capacity is available. Does not fix an already-wedged stack (recover that separately via
continue-update-rollback).