fix(runners): pace Parameter Store writes against the account-wide limit

README.md

@@ -228,6 +228,8 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh)
 | <a name="input_scale_down_schedule_expression"></a> [scale\_down\_schedule\_expression](#input\_scale\_down\_schedule\_expression) | Scheduler expression to check every x for scale down. | `string` | `"cron(*/5 * * * ? *)"` | no |
 | <a name="input_scale_errors"></a> [scale\_errors](#input\_scale\_errors) | List of AWS error codes that should trigger retry during scale up. This list replaces the module default scale-up retry errors | `list(string)` | <pre>[<br/>  "UnfulfillableCapacity",<br/>  "MaxSpotInstanceCountExceeded",<br/>  "TargetCapacityLimitExceededException",<br/>  "RequestLimitExceeded",<br/>  "ResourceLimitExceeded",<br/>  "MaxSpotInstanceCountExceeded",<br/>  "MaxSpotFleetRequestCountExceeded",<br/>  "InsufficientInstanceCapacity",<br/>  "InsufficientCapacityOnHost"<br/>]</pre> | no |
 | <a name="input_scale_up_reserved_concurrent_executions"></a> [scale\_up\_reserved\_concurrent\_executions](#input\_scale\_up\_reserved\_concurrent\_executions) | Amount of reserved concurrent executions for the scale-up lambda function. A value of 0 disables lambda from being triggered and -1 removes any concurrency limitations. | `number` | `1` | no |
+| <a name="input_ssm_parameter_store_max_concurrent_invocations"></a> [ssm\_parameter\_store\_max\_concurrent\_invocations](#input\_ssm\_parameter\_store\_max\_concurrent\_invocations) | Expected number of concurrent scale-up/pool lambda invocations writing to Parameter Store, used to pace each invocation's writes to a share of the account-wide write-rate limit. | `number` | `1` | no |
+| <a name="input_ssm_parameter_store_max_writes_per_second"></a> [ssm\_parameter\_store\_max\_writes\_per\_second](#input\_ssm\_parameter\_store\_max\_writes\_per\_second) | Parameter Store write-rate limit to pace against, in writes/second. Defaults to the standard-tier limit; raise this if SSM's higher-throughput tier is enabled (up to several thousand writes/second). | `number` | `40` | no |
 | <a name="input_ssm_paths"></a> [ssm\_paths](#input\_ssm\_paths) | The root path used in SSM to store configuration and secrets. | <pre>object({<br/>    root       = optional(string, "github-action-runners")<br/>    app        = optional(string, "app")<br/>    runners    = optional(string, "runners")<br/>    webhook    = optional(string, "webhook")<br/>    use_prefix = optional(bool, true)<br/>  })</pre> | `{}` | no |
 | <a name="input_ssm_ttl_seconds"></a> [ssm\_ttl\_seconds](#input\_ssm\_ttl\_seconds) | Set tokens to the optional TTL in seconds for the SSM parameters holding the runner registration token / JIT config. When set, the parameters are created with an SSM expiration policy so SSM deletes them itself after the TTL passes. Requires the Advanced parameter tier for every token parameter, which incurs additional costs. Expiration is enforced asynchronously by SSM; the SSM housekeeper lambda remains as a backstop. Must be a positive number, and should comfortably exceed the runner boot time so the config does not expire before the instance reads it. | <pre>object({<br/>    tokens = optional(number, null)<br/>  })</pre> | `{}` | no |
 | <a name="input_state_event_rule_binaries_syncer"></a> [state\_event\_rule\_binaries\_syncer](#input\_state\_event\_rule\_binaries\_syncer) | Option to disable EventBridge Lambda trigger for the binary syncer, useful to stop automatic updates of binary distribution | `string` | `"ENABLED"` | no |

docs/rate-limits-and-tuning.md

@@ -82,7 +82,15 @@ Rate limits are **disabled by default** on GitHub Enterprise Server and must be
 |---|---|
 | Combined throughput (Get + Put) | **40 TPS** (shared per-account per-region) |
 
-Each runner instance requires one `PutParameter` call for its JIT config. At 40 TPS shared across all operations in the account, a burst of 40+ concurrent writes will throttle.
+Each runner instance requires one `PutParameter` call for its JIT config. This is account-wide, not per-invocation or per-pool: several pools' scale-up/pool lambdas can each individually look fine while their combined writes exceed the account limit. The module paces its own writes to stay under a configured ceiling — `ssm_parameter_store_max_writes_per_second` (default `40`, matching the standard tier) — dividing the per-write delay across `ssm_parameter_store_max_concurrent_invocations` (default `1`), which should reflect how many scale-up/pool invocations across all pools can realistically run at the same time.
+
+Where to set the two values:
+
+| Module | Setting |
+| ------ | ------- |
+| Root module | `ssm_parameter_store_max_writes_per_second` and `ssm_parameter_store_max_concurrent_invocations` |
+| Multi-runner (v1 inputs) | The same two variables, applied to all runner configurations |
+| Multi-runner (v2 inputs) | `global_config_storage_provider.aws.ssm.parameters.max_writes_per_second` and `max_concurrent_invocations` for all runner configurations. A runner configuration can override either value under its own `storage_provider.aws.ssm.parameters`. |
 
 **Higher throughput mode** raises the ceiling:
 
@@ -94,6 +102,8 @@ aws ssm update-service-setting \
 
 Cost: $0.05 per 10,000 API interactions beyond the standard tier.
 
+Enabling this alone does nothing for this module's own pacing — raise `ssm_parameter_store_max_writes_per_second` to match the new ceiling too, or the module keeps pacing writes at the old 40 TPS default regardless of what AWS now allows.
+
 ### EC2 CreateFleet
 
 The exact TPS limit for `CreateFleet` is not publicly documented. It uses a token-bucket algorithm per-account per-region. Empirically throttles at low single-digit TPS.
@@ -111,7 +121,7 @@ For deployments running more than a handful of concurrent runners:
 | Running On-Demand Standard (A,C,D,H,I,M,R,T,Z) instances | **5 vCPUs** | Service Quotas console |
 | All Standard Spot Instance Requests | **5 vCPUs** | Service Quotas console |
 | EC2 CreateFleet API rate | Undocumented | AWS Support ticket |
-| SSM Parameter Store throughput | 40 TPS | `update-service-setting` (see above) |
+| SSM Parameter Store throughput | 40 TPS | `update-service-setting` (see above), then raise `ssm_parameter_store_max_writes_per_second` to match |
 
 **vCPU quotas are measured in vCPUs, not instance count.** Running 50× `c5.large` (2 vCPU each) requires a quota of at least 100 vCPUs.
 
@@ -131,7 +141,7 @@ For deployments running more than a handful of concurrent runners:
 | `isJobQueued` calls | 100 | 100 (same total) |
 | JIT config generation calls | 100 | 100 (same total) |
 
-Larger `batch_size` reduces CreateFleet calls (the most constrained AWS API) and Lambda invocations. Per-runner work (SSM writes, JIT config, isJobQueued) stays the same total. SSM peak TPS is lower with larger batches because writes are serialized within each Lambda rather than concurrent across many.
+Larger `batch_size` reduces CreateFleet calls (the most constrained AWS API) and Lambda invocations. Per-runner work (SSM writes, JIT config, isJobQueued) stays the same total. SSM writes are paced within each Lambda regardless of batch size (see `ssm_parameter_store_max_writes_per_second`/`ssm_parameter_store_max_concurrent_invocations` above) — batch size does not change the SSM pacing behavior, only how many Lambda invocations are running concurrently in the first place.
 
 ### Tradeoffs
 
@@ -157,7 +167,7 @@ Higher windows improve batching efficiency but add latency to job pickup.
 |---|---|---|---|
 | Small (<50 concurrent jobs) | 1–5 | 90s | Defaults work |
 | Medium (50–200) | 5–10 | 180s | Monitor SSM throttling |
-| Large (200+) | 10 | 300s | Enable SSM higher throughput, raise vCPU quotas, request CreateFleet rate increase |
+| Large (200+) | 10 | 300s | Enable SSM higher throughput and raise `ssm_parameter_store_max_writes_per_second`/`ssm_parameter_store_max_concurrent_invocations` to match, raise vCPU quotas, request CreateFleet rate increase |
 
 ## Monitoring
 

lambdas/functions/control-plane/src/scale-runners/github-runner.test.ts

@@ -1,4 +1,4 @@
-import { isJobQueued, createStartRunnerConfig } from './github-runner';
+import { addDelay, isJobQueued, createStartRunnerConfig } from './github-runner';
 import { metricGitHubAppRateLimit } from '../github/rate-limit';
 import type { ActionRequestMessage, CreateGitHubRunnerConfig } from './types';
 import type { RunnerConfigStore } from '@aws-github-runner/storage-providers';
@@ -11,8 +11,91 @@ vi.mock('../github/rate-limit', () => ({
 
 const mockedMetricGitHubAppRateLimit = vi.mocked(metricGitHubAppRateLimit);
 
+const cleanEnv = process.env;
+
 beforeEach(() => {
   vi.clearAllMocks();
+  process.env = { ...cleanEnv };
+  delete process.env.SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS;
+});
+
+describe('Test addDelay', () => {
+  it('does not delay when the store has no write limit', () => {
+    const store = { maxWritesPerSecond: undefined } as RunnerConfigStore;
+    const { isDelay, delayMilliseconds } = addDelay(['1', '2'], store);
+    expect(isDelay).toBe(false);
+    expect(delayMilliseconds).toBe(0);
+  });
+
+  it('does not delay for an empty batch', () => {
+    const store = { maxWritesPerSecond: 40 } as RunnerConfigStore;
+    const { isDelay } = addDelay([], store);
+    expect(isDelay).toBe(false);
+  });
+
+  it('paces a batch below the write limit, unlike the previous per-invocation-only guard', () => {
+    const store = { maxWritesPerSecond: 40 } as RunnerConfigStore;
+    // A batch of 2 is far below 40, but the account-wide limit does not care about batch size.
+    const { isDelay, delayMilliseconds } = addDelay(['1', '2'], store);
+    expect(isDelay).toBe(true);
+    expect(delayMilliseconds).toBe(25); // 1000 / 40
+  });
+
+  it('divides the per-write delay across the configured number of concurrent invocations', () => {
+    process.env.SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS = '5';
+    const store = { maxWritesPerSecond: 40 } as RunnerConfigStore;
+    const { delayMilliseconds } = addDelay(['1'], store);
+    expect(delayMilliseconds).toBe(125); // (1000 / 40) * 5
+  });
+
+  it.each(['0', '-1', 'not-a-number', ''])(
+    'treats an invalid concurrency value (%s) as a single invocation',
+    (value) => {
+      process.env.SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS = value;
+      const store = { maxWritesPerSecond: 40 } as RunnerConfigStore;
+      const { delayMilliseconds } = addDelay(['1'], store);
+      expect(delayMilliseconds).toBe(25);
+    },
+  );
+});
+
+describe('Test createStartRunnerConfig registration-token pacing', () => {
+  const mockOctokit = {
+    actions: {
+      createRegistrationTokenForOrg: vi.fn().mockResolvedValue({ data: { token: 'reg-token' } }),
+    },
+  } as unknown as Octokit;
+
+  const githubRunnerConfig: CreateGitHubRunnerConfig = {
+    ephemeral: false,
+    enableJitConfig: false,
+    runnerLabels: 'self-hosted',
+    runnerGroup: 'Default',
+    runnerNamePrefix: 'test-',
+    runnerOwner: 'my-org',
+    runnerType: 'Org',
+    disableAutoUpdate: false,
+  };
+
+  it('paces every write between concurrent pools, not just large single-invocation batches', async () => {
+    vi.useFakeTimers();
+    const create = vi.fn().mockResolvedValue(undefined);
+    const runnerConfigStore = { maxWritesPerSecond: 40, create } as unknown as RunnerConfigStore;
+
+    process.env.SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS = '4';
+    const runPromise = createStartRunnerConfig(githubRunnerConfig, ['a', 'b', 'c'], mockOctokit, {
+      runnerConfigStore,
+    });
+
+    // Each write is followed by a (1000 / 40) * 4 = 100ms pacing delay.
+    await vi.advanceTimersByTimeAsync(100);
+    await vi.advanceTimersByTimeAsync(100);
+    await vi.advanceTimersByTimeAsync(100);
+    await runPromise;
+
+    expect(create).toHaveBeenCalledTimes(3);
+    vi.useRealTimers();
+  });
 });
 
 describe('Test isJobQueued rate-limit metric on error', () => {

lambdas/functions/control-plane/src/scale-runners/github-runner.ts

@@ -227,11 +227,16 @@ export async function createStartRunnerConfig(
   }
 }
 
-function addDelay(runnerIds: string[], runnerConfigStore: RunnerConfigStore) {
+// maxWritesPerSecond is account-wide, so the per-write delay is spread across the configured
+// number of concurrent invocations rather than sized off this invocation's batch alone.
+export function addDelay(runnerIds: string[], runnerConfigStore: RunnerConfigStore) {
   const delay = async (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
   const maxWritesPerSecond = runnerConfigStore.maxWritesPerSecond;
-  const isDelay = maxWritesPerSecond !== undefined && runnerIds.length >= maxWritesPerSecond;
-  const delayMilliseconds = maxWritesPerSecond === undefined ? 0 : 1000 / maxWritesPerSecond;
+  const isDelay = maxWritesPerSecond !== undefined && runnerIds.length > 0;
+  const configuredConcurrency = parseInt(process.env.SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS ?? '', 10);
+  const maxConcurrentInvocations = configuredConcurrency > 0 ? configuredConcurrency : 1;
+  const delayMilliseconds =
+    maxWritesPerSecond === undefined ? 0 : (1000 / maxWritesPerSecond) * maxConcurrentInvocations;
   return { isDelay, delay, delayMilliseconds };
 }
 

lambdas/libs/storage-providers/aws/ssm/runner-config-store.test.ts

@@ -1,7 +1,7 @@
 import { putParameter } from '@aws-github-runner/aws-ssm-util';
 import { beforeEach, describe, expect, it, vi } from 'vitest';
 
-import { createAwsSsmRunnerConfigStore } from './runner-config-store';
+import { createAwsSsmRunnerConfigStore, resolveMaxWritesPerSecond } from './runner-config-store';
 
 vi.mock('@aws-github-runner/aws-ssm-util', () => ({
   putParameter: vi.fn(),
@@ -107,6 +107,34 @@ describe('aws_ssm runner config store', () => {
     expect(putParameterMock).toHaveBeenCalledWith('/runner/tokens/runner-1', 'jit-config', true, { tags: [] });
   });
 
+  it('uses SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND when set, e.g. after enabling higher throughput', () => {
+    process.env.SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND = '10000';
+    const store = createAwsSsmRunnerConfigStore();
+    expect(store.maxWritesPerSecond).toBe(10000);
+  });
+
+  it.each([undefined, '', '0', '-5', 'not-a-number'])(
+    'falls back to the standard-tier default of 40 for an invalid value (%j)',
+    (value) => {
+      if (value === undefined) {
+        delete process.env.SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND;
+      } else {
+        process.env.SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND = value;
+      }
+      const store = createAwsSsmRunnerConfigStore();
+      expect(store.maxWritesPerSecond).toBe(40);
+    },
+  );
+
+  it('honors an explicitly configured maxWritesPerSecond over the default', () => {
+    const store = createAwsSsmRunnerConfigStore({
+      tokenPath: '/runner/tokens',
+      parameterStoreTags: [],
+      maxWritesPerSecond: 1000,
+    });
+    expect(store.maxWritesPerSecond).toBe(1000);
+  });
+
   it('logs safe context when a runner configuration write fails', async () => {
     const error = Object.assign(new Error('encoded-jit-secret'), { name: 'ThrottlingException' });
     putParameterMock.mockRejectedValue(error);
@@ -124,6 +152,20 @@ describe('aws_ssm runner config store', () => {
   });
 });
 
+describe('resolveMaxWritesPerSecond', () => {
+  it.each([
+    ['10000', 10000],
+    ['1', 1],
+    [undefined, 40],
+    ['', 40],
+    ['0', 40],
+    ['-1', 40],
+    ['not-a-number', 40],
+  ])('resolves %j to %i', (rawValue, expected) => {
+    expect(resolveMaxWritesPerSecond(rawValue)).toBe(expected);
+  });
+});
+
 function setTokenPath(tokenPath: string | undefined): void {
   if (tokenPath === undefined) {
     delete process.env.SSM_TOKEN_PATH;

lambdas/libs/storage-providers/aws/ssm/runner-config-store.ts

@@ -9,10 +9,20 @@ import { parseSsmTokenTtlSeconds } from './token-ttl';
 
 const logger = createAwsSsmStorageLogger('runner-config-store');
 
+// SSM Parameter Store's standard-tier default; accounts that enabled the higher-throughput tier
+// (up to several thousand TPS) should override this via SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND.
+const DEFAULT_MAX_WRITES_PER_SECOND = 40;
+
+export function resolveMaxWritesPerSecond(rawValue: string | undefined): number {
+  const parsed = parseInt(rawValue ?? '', 10);
+  return parsed > 0 ? parsed : DEFAULT_MAX_WRITES_PER_SECOND;
+}
+
 export interface AwsSsmRunnerConfigStoreConfig {
   tokenPath: string;
   tokenTtlSeconds?: number;
   parameterStoreTags: ReadonlyArray<Readonly<{ Key: string; Value: string }>>;
+  maxWritesPerSecond?: number;
 }
 
 export function createAwsSsmRunnerConfigStore(config?: AwsSsmRunnerConfigStoreConfig): RunnerConfigStore {
@@ -33,13 +43,16 @@ export function createAwsSsmRunnerConfigStore(config?: AwsSsmRunnerConfigStoreCo
     tokenPath,
     tokenTtlSeconds: parseSsmTokenTtlSeconds(process.env.SSM_TOKEN_TTL_SECONDS),
     parameterStoreTags: loadSsmParameterStoreTagsFromEnvironment(),
+    maxWritesPerSecond: resolveMaxWritesPerSecond(process.env.SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND),
   });
 }
 
 class AwsSsmRunnerConfigStore implements RunnerConfigStore {
-  readonly maxWritesPerSecond = 40;
+  readonly maxWritesPerSecond: number;
 
-  constructor(private readonly config: AwsSsmRunnerConfigStoreConfig) {}
+  constructor(private readonly config: AwsSsmRunnerConfigStoreConfig) {
+    this.maxWritesPerSecond = config.maxWritesPerSecond ?? DEFAULT_MAX_WRITES_PER_SECOND;
+  }
 
   async create(record: RunnerConfigRecord, options: { metadata?: RunnerConfigMetadata[] } = {}): Promise<void> {
     const parameterName = `${this.config.tokenPath}/${record.runnerId}`;

lambdas/libs/storage-providers/storage-providers.test.ts

@@ -3,9 +3,11 @@ import { describe, expect, it, vi } from 'vitest';
 import { createAwsSsmRunnerConfigStore } from './aws/ssm/runner-config-store';
 
 import { createStorageProviders } from './storage-providers';
+import { createAwsSsmRunnerConfigStore } from './aws/ssm/runner-config-store';
 
 vi.mock('./aws/ssm/runner-config-store', () => ({
   createAwsSsmRunnerConfigStore: vi.fn(() => ({ create: vi.fn() })),
+  resolveMaxWritesPerSecond: vi.fn((rawValue: string | undefined) => (rawValue ? parseInt(rawValue, 10) : 40)),
 }));
 vi.mock('./aws/ssm/runner-group-cache-store', () => ({
   createAwsSsmRunnerGroupCacheStore: vi.fn(() => ({ get: vi.fn(), create: vi.fn() })),
@@ -44,6 +46,21 @@ describe('createStorageProviders', () => {
     });
   });
 
+  it('forwards SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND to the runner config store', () => {
+    const environment = Object.freeze({
+      RUNNER_CONFIG_STORAGE_PROVIDER: 'AWS_SSM',
+      SSM_TOKEN_PATH: '/runners/tokens',
+      SSM_CONFIG_PATH: '/runners/config',
+      SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND: '10000',
+      PARAMETER_GITHUB_APP_ID_NAME: 'app-id',
+      PARAMETER_GITHUB_APP_KEY_BASE64_NAME: 'app-key',
+    });
+
+    createStorageProviders(environment);
+
+    expect(createAwsSsmRunnerConfigStore).toHaveBeenCalledWith(expect.objectContaining({ maxWritesPerSecond: 10000 }));
+  });
+
   it('rejects an invalid token TTL before creating a storage provider', () => {
     vi.clearAllMocks();
     expect(() =>

lambdas/libs/storage-providers/storage-providers.ts

@@ -1,7 +1,7 @@
 import { createChildLogger } from '@aws-github-runner/aws-powertools-util';
 import { createAwsSsmGitHubAppCredentialsStore } from './aws/ssm/github-app-credentials-store';
 import { createAwsSsmRunnerConfigConsumer } from './aws/ssm/runner-config-consumer';
-import { createAwsSsmRunnerConfigStore } from './aws/ssm/runner-config-store';
+import { createAwsSsmRunnerConfigStore, resolveMaxWritesPerSecond } from './aws/ssm/runner-config-store';
 import { createAwsSsmRunnerGroupCacheStore } from './aws/ssm/runner-group-cache-store';
 import type { CommonStorage, StorageProviders } from './core';
 import { loadRunnerConfigConsumerConfigFromEnvironment } from './runner-config-consumer';
@@ -32,7 +32,12 @@ export function createStorageProviders(environment: Environment = process.env):
   });
 
   return {
-    runnerConfig: createAwsSsmRunnerConfigStore({ tokenPath, tokenTtlSeconds, parameterStoreTags }),
+    runnerConfig: createAwsSsmRunnerConfigStore({
+      tokenPath,
+      tokenTtlSeconds,
+      parameterStoreTags,
+      maxWritesPerSecond: resolveMaxWritesPerSecond(environment.SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND),
+    }),
     runnerGroupCache: createAwsSsmRunnerGroupCacheStore({ configPath, parameterStoreTags }),
     consumer: createAwsSsmRunnerConfigConsumer({ SSM_TOKEN_PATH: tokenPath }, consumerConfig),
     ...createCommonStorage(environment),