diff --git a/README.md b/README.md index 88f7fbefcb..dce6ceae73 100644 --- a/README.md +++ b/README.md @@ -228,6 +228,8 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh) | [scale\_down\_schedule\_expression](#input\_scale\_down\_schedule\_expression) | Scheduler expression to check every x for scale down. | `string` | `"cron(*/5 * * * ? *)"` | no | | [scale\_errors](#input\_scale\_errors) | List of AWS error codes that should trigger retry during scale up. This list replaces the module default scale-up retry errors | `list(string)` |
[| no | | [scale\_up\_reserved\_concurrent\_executions](#input\_scale\_up\_reserved\_concurrent\_executions) | Amount of reserved concurrent executions for the scale-up lambda function. A value of 0 disables lambda from being triggered and -1 removes any concurrency limitations. | `number` | `1` | no | +| [ssm\_parameter\_store\_max\_concurrent\_invocations](#input\_ssm\_parameter\_store\_max\_concurrent\_invocations) | Expected number of concurrent scale-up/pool lambda invocations writing to Parameter Store, used to pace each invocation's writes to a share of the account-wide write-rate limit. | `number` | `1` | no | +| [ssm\_parameter\_store\_max\_writes\_per\_second](#input\_ssm\_parameter\_store\_max\_writes\_per\_second) | Parameter Store write-rate limit to pace against, in writes/second. Defaults to the standard-tier limit; raise this if SSM's higher-throughput tier is enabled (up to several thousand writes/second). | `number` | `40` | no | | [ssm\_paths](#input\_ssm\_paths) | The root path used in SSM to store configuration and secrets. |
"UnfulfillableCapacity",
"MaxSpotInstanceCountExceeded",
"TargetCapacityLimitExceededException",
"RequestLimitExceeded",
"ResourceLimitExceeded",
"MaxSpotInstanceCountExceeded",
"MaxSpotFleetRequestCountExceeded",
"InsufficientInstanceCapacity",
"InsufficientCapacityOnHost"
]
object({
root = optional(string, "github-action-runners")
app = optional(string, "app")
runners = optional(string, "runners")
webhook = optional(string, "webhook")
use_prefix = optional(bool, true)
}) | `{}` | no |
| [ssm\_ttl\_seconds](#input\_ssm\_ttl\_seconds) | Set tokens to the optional TTL in seconds for the SSM parameters holding the runner registration token / JIT config. When set, the parameters are created with an SSM expiration policy so SSM deletes them itself after the TTL passes. Requires the Advanced parameter tier for every token parameter, which incurs additional costs. Expiration is enforced asynchronously by SSM; the SSM housekeeper lambda remains as a backstop. Must be a positive number, and should comfortably exceed the runner boot time so the config does not expire before the instance reads it. | object({
tokens = optional(number, null)
}) | `{}` | no |
| [state\_event\_rule\_binaries\_syncer](#input\_state\_event\_rule\_binaries\_syncer) | Option to disable EventBridge Lambda trigger for the binary syncer, useful to stop automatic updates of binary distribution | `string` | `"ENABLED"` | no |
diff --git a/docs/rate-limits-and-tuning.md b/docs/rate-limits-and-tuning.md
index 88c74bf6d5..09cf68cbee 100644
--- a/docs/rate-limits-and-tuning.md
+++ b/docs/rate-limits-and-tuning.md
@@ -82,7 +82,15 @@ Rate limits are **disabled by default** on GitHub Enterprise Server and must be
|---|---|
| Combined throughput (Get + Put) | **40 TPS** (shared per-account per-region) |
-Each runner instance requires one `PutParameter` call for its JIT config. At 40 TPS shared across all operations in the account, a burst of 40+ concurrent writes will throttle.
+Each runner instance requires one `PutParameter` call for its JIT config. This is account-wide, not per-invocation or per-pool: several pools' scale-up/pool lambdas can each individually look fine while their combined writes exceed the account limit. The module paces its own writes to stay under a configured ceiling — `ssm_parameter_store_max_writes_per_second` (default `40`, matching the standard tier) — dividing the per-write delay across `ssm_parameter_store_max_concurrent_invocations` (default `1`), which should reflect how many scale-up/pool invocations across all pools can realistically run at the same time.
+
+Where to set the two values:
+
+| Module | Setting |
+| ------ | ------- |
+| Root module | `ssm_parameter_store_max_writes_per_second` and `ssm_parameter_store_max_concurrent_invocations` |
+| Multi-runner (v1 inputs) | The same two variables, applied to all runner configurations |
+| Multi-runner (v2 inputs) | `global_config_storage_provider.aws.ssm.parameters.max_writes_per_second` and `max_concurrent_invocations` for all runner configurations. A runner configuration can override either value under its own `storage_provider.aws.ssm.parameters`. |
**Higher throughput mode** raises the ceiling:
@@ -94,6 +102,8 @@ aws ssm update-service-setting \
Cost: $0.05 per 10,000 API interactions beyond the standard tier.
+Enabling this alone does nothing for this module's own pacing — raise `ssm_parameter_store_max_writes_per_second` to match the new ceiling too, or the module keeps pacing writes at the old 40 TPS default regardless of what AWS now allows.
+
### EC2 CreateFleet
The exact TPS limit for `CreateFleet` is not publicly documented. It uses a token-bucket algorithm per-account per-region. Empirically throttles at low single-digit TPS.
@@ -111,7 +121,7 @@ For deployments running more than a handful of concurrent runners:
| Running On-Demand Standard (A,C,D,H,I,M,R,T,Z) instances | **5 vCPUs** | Service Quotas console |
| All Standard Spot Instance Requests | **5 vCPUs** | Service Quotas console |
| EC2 CreateFleet API rate | Undocumented | AWS Support ticket |
-| SSM Parameter Store throughput | 40 TPS | `update-service-setting` (see above) |
+| SSM Parameter Store throughput | 40 TPS | `update-service-setting` (see above), then raise `ssm_parameter_store_max_writes_per_second` to match |
**vCPU quotas are measured in vCPUs, not instance count.** Running 50× `c5.large` (2 vCPU each) requires a quota of at least 100 vCPUs.
@@ -131,7 +141,7 @@ For deployments running more than a handful of concurrent runners:
| `isJobQueued` calls | 100 | 100 (same total) |
| JIT config generation calls | 100 | 100 (same total) |
-Larger `batch_size` reduces CreateFleet calls (the most constrained AWS API) and Lambda invocations. Per-runner work (SSM writes, JIT config, isJobQueued) stays the same total. SSM peak TPS is lower with larger batches because writes are serialized within each Lambda rather than concurrent across many.
+Larger `batch_size` reduces CreateFleet calls (the most constrained AWS API) and Lambda invocations. Per-runner work (SSM writes, JIT config, isJobQueued) stays the same total. SSM writes are paced within each Lambda regardless of batch size (see `ssm_parameter_store_max_writes_per_second`/`ssm_parameter_store_max_concurrent_invocations` above) — batch size does not change the SSM pacing behavior, only how many Lambda invocations are running concurrently in the first place.
### Tradeoffs
@@ -157,7 +167,7 @@ Higher windows improve batching efficiency but add latency to job pickup.
|---|---|---|---|
| Small (<50 concurrent jobs) | 1–5 | 90s | Defaults work |
| Medium (50–200) | 5–10 | 180s | Monitor SSM throttling |
-| Large (200+) | 10 | 300s | Enable SSM higher throughput, raise vCPU quotas, request CreateFleet rate increase |
+| Large (200+) | 10 | 300s | Enable SSM higher throughput and raise `ssm_parameter_store_max_writes_per_second`/`ssm_parameter_store_max_concurrent_invocations` to match, raise vCPU quotas, request CreateFleet rate increase |
## Monitoring
diff --git a/lambdas/functions/control-plane/src/scale-runners/github-runner.test.ts b/lambdas/functions/control-plane/src/scale-runners/github-runner.test.ts
index 9436700ed6..ec097b1b2c 100644
--- a/lambdas/functions/control-plane/src/scale-runners/github-runner.test.ts
+++ b/lambdas/functions/control-plane/src/scale-runners/github-runner.test.ts
@@ -1,4 +1,4 @@
-import { isJobQueued, createStartRunnerConfig } from './github-runner';
+import { addDelay, isJobQueued, createStartRunnerConfig } from './github-runner';
import { metricGitHubAppRateLimit } from '../github/rate-limit';
import type { ActionRequestMessage, CreateGitHubRunnerConfig } from './types';
import type { RunnerConfigStore } from '@aws-github-runner/storage-providers';
@@ -11,8 +11,91 @@ vi.mock('../github/rate-limit', () => ({
const mockedMetricGitHubAppRateLimit = vi.mocked(metricGitHubAppRateLimit);
+const cleanEnv = process.env;
+
beforeEach(() => {
vi.clearAllMocks();
+ process.env = { ...cleanEnv };
+ delete process.env.SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS;
+});
+
+describe('Test addDelay', () => {
+ it('does not delay when the store has no write limit', () => {
+ const store = { maxWritesPerSecond: undefined } as RunnerConfigStore;
+ const { isDelay, delayMilliseconds } = addDelay(['1', '2'], store);
+ expect(isDelay).toBe(false);
+ expect(delayMilliseconds).toBe(0);
+ });
+
+ it('does not delay for an empty batch', () => {
+ const store = { maxWritesPerSecond: 40 } as RunnerConfigStore;
+ const { isDelay } = addDelay([], store);
+ expect(isDelay).toBe(false);
+ });
+
+ it('paces a batch below the write limit, unlike the previous per-invocation-only guard', () => {
+ const store = { maxWritesPerSecond: 40 } as RunnerConfigStore;
+ // A batch of 2 is far below 40, but the account-wide limit does not care about batch size.
+ const { isDelay, delayMilliseconds } = addDelay(['1', '2'], store);
+ expect(isDelay).toBe(true);
+ expect(delayMilliseconds).toBe(25); // 1000 / 40
+ });
+
+ it('divides the per-write delay across the configured number of concurrent invocations', () => {
+ process.env.SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS = '5';
+ const store = { maxWritesPerSecond: 40 } as RunnerConfigStore;
+ const { delayMilliseconds } = addDelay(['1'], store);
+ expect(delayMilliseconds).toBe(125); // (1000 / 40) * 5
+ });
+
+ it.each(['0', '-1', 'not-a-number', ''])(
+ 'treats an invalid concurrency value (%s) as a single invocation',
+ (value) => {
+ process.env.SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS = value;
+ const store = { maxWritesPerSecond: 40 } as RunnerConfigStore;
+ const { delayMilliseconds } = addDelay(['1'], store);
+ expect(delayMilliseconds).toBe(25);
+ },
+ );
+});
+
+describe('Test createStartRunnerConfig registration-token pacing', () => {
+ const mockOctokit = {
+ actions: {
+ createRegistrationTokenForOrg: vi.fn().mockResolvedValue({ data: { token: 'reg-token' } }),
+ },
+ } as unknown as Octokit;
+
+ const githubRunnerConfig: CreateGitHubRunnerConfig = {
+ ephemeral: false,
+ enableJitConfig: false,
+ runnerLabels: 'self-hosted',
+ runnerGroup: 'Default',
+ runnerNamePrefix: 'test-',
+ runnerOwner: 'my-org',
+ runnerType: 'Org',
+ disableAutoUpdate: false,
+ };
+
+ it('paces every write between concurrent pools, not just large single-invocation batches', async () => {
+ vi.useFakeTimers();
+ const create = vi.fn().mockResolvedValue(undefined);
+ const runnerConfigStore = { maxWritesPerSecond: 40, create } as unknown as RunnerConfigStore;
+
+ process.env.SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS = '4';
+ const runPromise = createStartRunnerConfig(githubRunnerConfig, ['a', 'b', 'c'], mockOctokit, {
+ runnerConfigStore,
+ });
+
+ // Each write is followed by a (1000 / 40) * 4 = 100ms pacing delay.
+ await vi.advanceTimersByTimeAsync(100);
+ await vi.advanceTimersByTimeAsync(100);
+ await vi.advanceTimersByTimeAsync(100);
+ await runPromise;
+
+ expect(create).toHaveBeenCalledTimes(3);
+ vi.useRealTimers();
+ });
});
describe('Test isJobQueued rate-limit metric on error', () => {
diff --git a/lambdas/functions/control-plane/src/scale-runners/github-runner.ts b/lambdas/functions/control-plane/src/scale-runners/github-runner.ts
index f968fa3008..c75fe2408f 100644
--- a/lambdas/functions/control-plane/src/scale-runners/github-runner.ts
+++ b/lambdas/functions/control-plane/src/scale-runners/github-runner.ts
@@ -227,11 +227,16 @@ export async function createStartRunnerConfig(
}
}
-function addDelay(runnerIds: string[], runnerConfigStore: RunnerConfigStore) {
+// maxWritesPerSecond is account-wide, so the per-write delay is spread across the configured
+// number of concurrent invocations rather than sized off this invocation's batch alone.
+export function addDelay(runnerIds: string[], runnerConfigStore: RunnerConfigStore) {
const delay = async (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
const maxWritesPerSecond = runnerConfigStore.maxWritesPerSecond;
- const isDelay = maxWritesPerSecond !== undefined && runnerIds.length >= maxWritesPerSecond;
- const delayMilliseconds = maxWritesPerSecond === undefined ? 0 : 1000 / maxWritesPerSecond;
+ const isDelay = maxWritesPerSecond !== undefined && runnerIds.length > 0;
+ const configuredConcurrency = parseInt(process.env.SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS ?? '', 10);
+ const maxConcurrentInvocations = configuredConcurrency > 0 ? configuredConcurrency : 1;
+ const delayMilliseconds =
+ maxWritesPerSecond === undefined ? 0 : (1000 / maxWritesPerSecond) * maxConcurrentInvocations;
return { isDelay, delay, delayMilliseconds };
}
diff --git a/lambdas/libs/storage-providers/aws/ssm/runner-config-store.test.ts b/lambdas/libs/storage-providers/aws/ssm/runner-config-store.test.ts
index a9a2697ae2..e0fb65664a 100644
--- a/lambdas/libs/storage-providers/aws/ssm/runner-config-store.test.ts
+++ b/lambdas/libs/storage-providers/aws/ssm/runner-config-store.test.ts
@@ -1,7 +1,7 @@
import { putParameter } from '@aws-github-runner/aws-ssm-util';
import { beforeEach, describe, expect, it, vi } from 'vitest';
-import { createAwsSsmRunnerConfigStore } from './runner-config-store';
+import { createAwsSsmRunnerConfigStore, resolveMaxWritesPerSecond } from './runner-config-store';
vi.mock('@aws-github-runner/aws-ssm-util', () => ({
putParameter: vi.fn(),
@@ -107,6 +107,34 @@ describe('aws_ssm runner config store', () => {
expect(putParameterMock).toHaveBeenCalledWith('/runner/tokens/runner-1', 'jit-config', true, { tags: [] });
});
+ it('uses SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND when set, e.g. after enabling higher throughput', () => {
+ process.env.SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND = '10000';
+ const store = createAwsSsmRunnerConfigStore();
+ expect(store.maxWritesPerSecond).toBe(10000);
+ });
+
+ it.each([undefined, '', '0', '-5', 'not-a-number'])(
+ 'falls back to the standard-tier default of 40 for an invalid value (%j)',
+ (value) => {
+ if (value === undefined) {
+ delete process.env.SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND;
+ } else {
+ process.env.SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND = value;
+ }
+ const store = createAwsSsmRunnerConfigStore();
+ expect(store.maxWritesPerSecond).toBe(40);
+ },
+ );
+
+ it('honors an explicitly configured maxWritesPerSecond over the default', () => {
+ const store = createAwsSsmRunnerConfigStore({
+ tokenPath: '/runner/tokens',
+ parameterStoreTags: [],
+ maxWritesPerSecond: 1000,
+ });
+ expect(store.maxWritesPerSecond).toBe(1000);
+ });
+
it('logs safe context when a runner configuration write fails', async () => {
const error = Object.assign(new Error('encoded-jit-secret'), { name: 'ThrottlingException' });
putParameterMock.mockRejectedValue(error);
@@ -124,6 +152,20 @@ describe('aws_ssm runner config store', () => {
});
});
+describe('resolveMaxWritesPerSecond', () => {
+ it.each([
+ ['10000', 10000],
+ ['1', 1],
+ [undefined, 40],
+ ['', 40],
+ ['0', 40],
+ ['-1', 40],
+ ['not-a-number', 40],
+ ])('resolves %j to %i', (rawValue, expected) => {
+ expect(resolveMaxWritesPerSecond(rawValue)).toBe(expected);
+ });
+});
+
function setTokenPath(tokenPath: string | undefined): void {
if (tokenPath === undefined) {
delete process.env.SSM_TOKEN_PATH;
diff --git a/lambdas/libs/storage-providers/aws/ssm/runner-config-store.ts b/lambdas/libs/storage-providers/aws/ssm/runner-config-store.ts
index 8c9297be32..ac71e751aa 100644
--- a/lambdas/libs/storage-providers/aws/ssm/runner-config-store.ts
+++ b/lambdas/libs/storage-providers/aws/ssm/runner-config-store.ts
@@ -9,10 +9,20 @@ import { parseSsmTokenTtlSeconds } from './token-ttl';
const logger = createAwsSsmStorageLogger('runner-config-store');
+// SSM Parameter Store's standard-tier default; accounts that enabled the higher-throughput tier
+// (up to several thousand TPS) should override this via SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND.
+const DEFAULT_MAX_WRITES_PER_SECOND = 40;
+
+export function resolveMaxWritesPerSecond(rawValue: string | undefined): number {
+ const parsed = parseInt(rawValue ?? '', 10);
+ return parsed > 0 ? parsed : DEFAULT_MAX_WRITES_PER_SECOND;
+}
+
export interface AwsSsmRunnerConfigStoreConfig {
tokenPath: string;
tokenTtlSeconds?: number;
parameterStoreTags: ReadonlyArrayobject({
artifact = optional(object({
s3 = optional(object({
bucket = optional(string, null)
}), {})
}), {})
runtime = optional(string, "nodejs24.x")
architecture = optional(string, "arm64")
principals = optional(list(object({
type = string
identifiers = list(string)
})), [])
subnet_ids = optional(list(string), [])
security_group_ids = optional(list(string), [])
tags = optional(map(string), {})
role = optional(object({
path = optional(string, null)
permissions_boundary = optional(string, null)
}), {})
}) | `{}` | no |
| [global\_config\_observability](#input\_global\_config\_observability) | Global observability configuration shared by all runner lanes.object({
logs = optional(object({
level = optional(string, "info")
retention_in_days = optional(number, 180)
kms_key_id = optional(string, null)
class = optional(string, "STANDARD")
tags = optional(map(string), {})
}), {})
tracing = optional(object({
mode = optional(string, null)
capture_http_requests = optional(bool, false)
capture_error = optional(bool, false)
}), {})
metrics = optional(object({
enabled = optional(bool, false)
namespace = optional(string, "GitHub Runners")
metric = optional(object({
github_app_rate_limit = optional(object({
enabled = optional(bool, true)
}), {})
job_retry = optional(object({
enabled = optional(bool, true)
}), {})
spot_termination_warning = optional(object({
enabled = optional(bool, true)
}), {})
}), {})
}), {})
}) | `{}` | no |
| [global\_config\_orchestration\_provider](#input\_global\_config\_orchestration\_provider) | Global orchestration-provider configuration shared by all runner lanes.object({
webhook = optional(object({
queue_selection_strategy = optional(string, "first")
eventbridge = optional(object({
enabled = optional(bool, true)
accept_events = optional(list(string), [])
}), {})
matcher_config_parameter_store_tier = optional(string, "Standard")
runner = optional(object({
boot_time_in_minutes = optional(number, 5)
ephemeral = optional(bool, false)
jit_config_enabled = optional(bool, null)
maximum_count = optional(number, null)
}), {})
github = optional(object({
repository_white_list = optional(list(string), [])
}), {})
lambda = optional(object({
artifact = optional(object({
zip = optional(string, null)
s3 = optional(object({
key = string
object_version = optional(string, null)
}), null)
}), {})
scale = optional(object({
up = optional(object({
memory_size = optional(number, 512)
timeout = optional(number, 30)
reserved_concurrent_executions = optional(number, 1)
job_queued_check_enabled = optional(bool, null)
event_source_mapping = optional(object({
batch_size = optional(number, 10)
maximum_batching_window_in_seconds = optional(number, 0)
}), {})
tags = optional(map(string), {})
}), {})
down = optional(object({
memory_size = optional(number, 512)
timeout = optional(number, 60)
schedule_expression = optional(string, "cron(*/5 * * * ? *)")
minimum_running_time_in_minutes = optional(number, null)
idle_confirmation_seconds = optional(number, 0)
idle_config = optional(list(object({
cron = string
timeZone = string
idleCount = number
evictionStrategy = optional(string, "oldest_first")
})), [])
tags = optional(map(string), {})
}), {})
}), {})
webhook = optional(object({
artifact = optional(object({
zip = optional(string, null)
s3 = optional(object({
key = string
object_version = optional(string, null)
}), null)
}), {})
api_gateway_access_log_settings = optional(object({
destination_arn = string
format = string
}), null)
memory_size = optional(number, 256)
timeout = optional(number, 10)
tags = optional(map(string), {})
}), {})
pool = optional(object({
memory_size = optional(number, 512)
timeout = optional(number, 60)
reserved_concurrent_executions = optional(number, 1)
config = optional(list(object({
schedule_expression = string
schedule_expression_timezone = optional(string)
size = number
})), [])
include_busy_runners = optional(bool, false)
runner_owner = optional(string, null)
tags = optional(map(string), {})
}), {})
}), {})
queue = optional(object({
delay_webhook_event = optional(number, 30)
job_queue_retention_in_seconds = optional(number, 86400)
visibility_timeout_seconds = optional(number, 180)
redrive_build_queue = optional(object({
enabled = optional(bool, false)
maxReceiveCount = optional(number, null)
}), {
enabled = false
maxReceiveCount = null
})
tags = optional(map(string), {})
encryption = optional(object({
kms_data_key_reuse_period_seconds = number
kms_master_key_id = string
sqs_managed_sse_enabled = bool
}), {
kms_data_key_reuse_period_seconds = null
kms_master_key_id = null
sqs_managed_sse_enabled = true
})
}), {})
}), {})
scale_set = optional(object({
grouping = optional(object({
strategy = optional(string, "compute_provider")
custom = optional(object({
groups = map(object({
runner_configs = set(string)
}))
}), null)
}), {})
container = optional(object({
image = optional(string, null)
user = optional(string, "10001:10001")
health_port = optional(number, 8080)
health_path = optional(string, "/healthz")
health_check_command = optional(list(string), null)
health_check_interval = optional(number, 30)
health_check_timeout = optional(number, 5)
health_check_retries = optional(number, 3)
health_check_start_period = optional(number, 30)
health_stale_after_seconds = optional(number, 180)
shutdown_timeout_seconds = optional(number, 110)
session_close_timeout_seconds = optional(number, 10)
reconnect_initial_backoff_seconds = optional(number, 1)
reconnect_max_backoff_seconds = optional(number, 30)
stop_timeout_seconds = optional(number, 120)
}), {})
config_store = optional(object({
path_prefix = optional(string, null)
tier = optional(string, "Standard")
tags = optional(map(string), {})
}), {})
ecs = optional(object({
cluster = optional(object({
mode = optional(string, "managed")
arn = optional(string, null)
name = optional(string, null)
container_insights = optional(bool, true)
}), {})
task = optional(object({
cpu = optional(number, 512)
memory = optional(number, 1024)
cpu_architecture = optional(string, "X86_64")
ephemeral_storage = optional(object({
size_in_gib = number
}), null)
}), {})
service = optional(object({
platform_version = optional(string, "LATEST")
}), {})
iam = optional(object({
path = optional(string, "/")
permissions_boundary = optional(string, null)
}), {})
}), {})
network = optional(object({
vpc_id = optional(string, null)
subnet_ids = optional(set(string), null)
https_egress = optional(object({
ipv4_cidrs = optional(set(string), ["0.0.0.0/0"])
ipv6_cidrs = optional(set(string), [])
}), {})
}), {})
logging = optional(object({
retention_in_days = optional(number, 180)
kms_key_id = optional(string, null)
log_group_class = optional(string, "STANDARD")
tags = optional(map(string), {})
}), {})
tags = optional(map(string), {})
}), {})
}) | `{}` | no |
-| [global\_config\_storage\_provider](#input\_global\_config\_storage\_provider) | Global storage-provider configuration shared by all runner lanes.object({
aws = optional(object({
ssm = optional(object({
paths = optional(object({
root = optional(string, null)
app = optional(string, "app")
webhook = optional(string, "webhook")
tokens = optional(string, "runners/tokens")
config = optional(string, "runners/config")
}), {})
kms_key_id = optional(string, null)
tags = optional(map(string), {})
parameters = optional(object({
tags = optional(map(string), {})
}), {})
housekeeper = optional(object({
schedule_expression = optional(string, "rate(1 day)")
state = optional(string, "ENABLED")
tags = optional(map(string), {})
lambda = optional(object({
artifact = optional(object({
zip = optional(string, null)
s3 = optional(object({
key = string
object_version = optional(string, null)
}), null)
}), {})
memory_size = optional(number, 512)
timeout = optional(number, 60)
}), {})
config = optional(object({
tokenPath = optional(string, null)
minimumDaysOld = optional(number, 1)
dryRun = optional(bool, false)
}), {})
}), {})
}), {})
}), {})
}) | `{}` | no |
+| [global\_config\_storage\_provider](#input\_global\_config\_storage\_provider) | Global storage-provider configuration shared by all runner lanes.object({
aws = optional(object({
ssm = optional(object({
paths = optional(object({
root = optional(string, null)
app = optional(string, "app")
webhook = optional(string, "webhook")
tokens = optional(string, "runners/tokens")
config = optional(string, "runners/config")
}), {})
kms_key_id = optional(string, null)
tags = optional(map(string), {})
parameters = optional(object({
tags = optional(map(string), {})
max_concurrent_invocations = optional(number, 1)
max_writes_per_second = optional(number, 40)
}), {})
housekeeper = optional(object({
schedule_expression = optional(string, "rate(1 day)")
state = optional(string, "ENABLED")
tags = optional(map(string), {})
lambda = optional(object({
artifact = optional(object({
zip = optional(string, null)
s3 = optional(object({
key = string
object_version = optional(string, null)
}), null)
}), {})
memory_size = optional(number, 512)
timeout = optional(number, 60)
}), {})
config = optional(object({
tokenPath = optional(string, null)
minimumDaysOld = optional(number, 1)
dryRun = optional(bool, false)
}), {})
}), {})
}), {})
}), {})
}) | `{}` | no |
| [iam\_overrides](#input\_iam\_overrides) | This map provides the possibility to override some IAM defaults. The following attributes are supported: `instance_profile_name` overrides the instance profile name used in the launch template. `runner_role_arn` overrides the IAM role ARN used for the runner instances. | object({
override_instance_profile = optional(bool, null)
instance_profile_name = optional(string, null)
override_runner_role = optional(bool, null)
runner_role_arn = optional(string, null)
}) | {
"instance_profile_name": null,
"override_instance_profile": false,
"override_runner_role": false,
"runner_role_arn": null
} | no |
| [instance\_profile\_path](#input\_instance\_profile\_path) | The path that will be added to the instance\_profile, if not set the environment name will be used. | `string` | `null` | no |
| [instance\_termination\_watcher](#input\_instance\_termination\_watcher) | Configuration for the spot termination watcher lambda function. This feature is Beta, changes will not trigger a major release as long in beta.object({
enable = optional(bool, false)
features = optional(object({
enable_spot_termination_handler = optional(bool, true)
enable_spot_termination_notification_watcher = optional(bool, true)
}), {})
enable_runner_deregistration = optional(bool, true)
environment_variables = optional(map(string), {})
memory_size = optional(number, null)
s3_key = optional(string, null)
s3_object_version = optional(string, null)
timeout = optional(number, null)
zip = optional(string, null)
}) | `{}` | no |
@@ -189,7 +189,7 @@ module "multi-runner" {
| [logging\_retention\_in\_days](#input\_logging\_retention\_in\_days) | Specifies the number of days you want to retain log events for the lambda log group. Possible values are: 0, 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, and 3653. | `number` | `180` | no |
| [matcher\_config\_parameter\_store\_tier](#input\_matcher\_config\_parameter\_store\_tier) | The tier of the parameter store for the matcher configuration. Valid values are `Standard`, and `Advanced`. | `string` | `"Standard"` | no |
| [metrics](#input\_metrics) | Configuration for metrics created by the module, by default metrics are disabled to avoid additional costs. When metrics are enable all metrics are created unless explicit configured otherwise. | object({
enable = optional(bool, false)
namespace = optional(string, "GitHub Runners")
metric = optional(object({
enable_github_app_rate_limit = optional(bool, true)
enable_job_retry = optional(bool, true)
enable_spot_termination_warning = optional(bool, true)
}), {})
}) | `{}` | no |
-| [multi\_runner\_config](#input\_multi\_runner\_config) | Accepts either the stable v1 runner configuration shape or the provider-boundary v2 shape. Entries with `runner_config` use the v1 shape; entries without `runner_config` use the v2 shape. A v2 entry does not need matcher configuration. A v2 entry must be acknowledged with `experimental_features = ["multi-runner-v2"]`; the v2 shape is experimental and may change before graduation.map(object({
# V1 contract
runner_config = optional(object({
runner_os = string
runner_architecture = string
runner_metadata_options = optional(map(any), {
instance_metadata_tags = "enabled"
http_endpoint = "enabled"
http_tokens = "required"
http_put_response_hop_limit = 1
})
ami = optional(object({
filter = optional(map(list(string)), { state = ["available"] })
owners = optional(list(string), ["amazon"])
id_ssm_parameter_arn = optional(string, null)
kms_key_arn = optional(string, null)
}), null)
create_service_linked_role_spot = optional(bool, false)
credit_specification = optional(string, null)
delay_webhook_event = optional(number, 30)
disable_runner_autoupdate = optional(bool, false)
ebs_optimized = optional(bool, false)
enable_ephemeral_runners = optional(bool, false)
enable_job_queued_check = optional(bool, null)
enable_on_demand_failover_for_errors = optional(list(string), [])
scale_errors = optional(list(string), [
"UnfulfillableCapacity",
"MaxSpotInstanceCountExceeded",
"TargetCapacityLimitExceededException",
"RequestLimitExceeded",
"ResourceLimitExceeded",
"MaxSpotInstanceCountExceeded",
"MaxSpotFleetRequestCountExceeded",
"InsufficientInstanceCapacity",
"InsufficientCapacityOnHost",
])
enable_organization_runners = optional(bool, false)
enable_runner_binaries_syncer = optional(bool, true)
enable_ssm_on_runners = optional(bool, false)
enable_userdata = optional(bool, true)
instance_allocation_strategy = optional(string, "lowest-price")
instance_type_priorities = optional(map(number), null)
instance_max_spot_price = optional(string, null)
instance_target_capacity_type = optional(string, "spot")
instance_types = list(string)
job_queue_retention_in_seconds = optional(number, 86400)
minimum_running_time_in_minutes = optional(number, null)
pool_runner_owner = optional(string, null)
runner_as_root = optional(bool, false)
runner_boot_time_in_minutes = optional(number, 5)
scale_down_idle_confirmation_seconds = optional(number, 0)
runner_disable_default_labels = optional(bool, false)
runner_extra_labels = optional(list(string), [])
runner_group_name = optional(string, "Default")
runner_name_prefix = optional(string, "")
runner_run_as = optional(string, "ec2-user")
runners_maximum_count = number
runner_additional_security_group_ids = optional(list(string), [])
scale_down_schedule_expression = optional(string, "cron(*/5 * * * ? *)")
scale_up_reserved_concurrent_executions = optional(number, 1)
lambda_event_source_mapping_batch_size = optional(number, null)
lambda_event_source_mapping_maximum_batching_window_in_seconds = optional(number, null)
userdata_template = optional(string, null)
userdata_content = optional(string, null)
enable_jit_config = optional(bool, null)
enable_runner_detailed_monitoring = optional(bool, false)
enable_cloudwatch_agent = optional(bool, true)
cloudwatch_config = optional(string, null)
userdata_pre_install = optional(string, "")
userdata_post_install = optional(string, "")
runner_hook_job_started = optional(string, "")
runner_hook_job_completed = optional(string, "")
runner_ec2_tags = optional(map(string), {})
runner_iam_role_managed_policy_arns = optional(list(string), [])
vpc_id = optional(string, null)
subnet_ids = optional(list(string), null)
idle_config = optional(list(object({
cron = string
timeZone = string
idleCount = number
evictionStrategy = optional(string, "oldest_first")
})), [])
cpu_options = optional(object({
core_count = optional(number)
threads_per_core = optional(number)
amd_sev_snp = optional(string)
nested_virtualization = optional(string)
}), null)
network_interfaces = optional(list(object({
associate_carrier_ip_address = optional(bool)
associate_public_ip_address = optional(bool)
delete_on_termination = optional(bool)
description = optional(string)
device_index = optional(number)
interface_type = optional(string)
ipv4_address_count = optional(number)
ipv4_addresses = optional(list(string))
ipv4_prefix_count = optional(number)
ipv4_prefixes = optional(list(string))
ipv6_address_count = optional(number)
ipv6_addresses = optional(list(string))
ipv6_prefix_count = optional(number)
ipv6_prefixes = optional(list(string))
network_card_index = optional(number)
network_interface_id = optional(string)
primary_ipv6 = optional(bool)
private_ip_address = optional(string)
security_groups = optional(list(string))
subnet_id = optional(string)
connection_tracking_specification = optional(object({
tcp_established_timeout = optional(number)
udp_stream_timeout = optional(number)
udp_timeout = optional(number)
}))
ena_srd_specification = optional(object({
ena_srd_enabled = optional(bool)
ena_srd_udp_specification = optional(object({
ena_srd_udp_enabled = optional(bool)
}))
}))
})), [])
placement = optional(object({
affinity = optional(string)
availability_zone = optional(string)
group_id = optional(string)
group_name = optional(string)
host_id = optional(string)
host_resource_group_arn = optional(string)
spread_domain = optional(string)
tenancy = optional(string)
partition_number = optional(number)
}), null)
license_specifications = optional(list(object({
license_configuration_arn = string
})), [])
use_dedicated_host = optional(bool, false)
runner_log_files = optional(list(object({
log_group_name = string
prefix_log_group = bool
file_path = string
log_stream_name = string
log_class = optional(string, "STANDARD")
})), null)
block_device_mappings = optional(list(object({
delete_on_termination = optional(bool, true)
device_name = optional(string, "/dev/xvda")
encrypted = optional(bool, true)
iops = optional(number)
kms_key_id = optional(string)
snapshot_id = optional(string)
throughput = optional(number)
volume_initialization_rate = optional(number)
volume_size = number
volume_type = optional(string, "gp3")
})), [{
volume_size = 30
}])
pool_config = optional(list(object({
schedule_expression = string
schedule_expression_timezone = optional(string)
size = number
})), [])
ssm_ttl_seconds = optional(object({
tokens = optional(number, null)
}), {})
job_retry = optional(object({
enable = optional(bool, false)
delay_in_seconds = optional(number, 300)
delay_backoff = optional(number, 2)
lambda_memory_size = optional(number, 256)
lambda_timeout = optional(number, 30)
max_attempts = optional(number, 1)
}), {})
iam_overrides = optional(object({
override_instance_profile = optional(bool, null)
instance_profile_name = optional(string, null)
override_runner_role = optional(bool, null)
runner_role_arn = optional(string, null)
}), {
override_instance_profile = false
instance_profile_name = null
override_runner_role = false
runner_role_arn = null
})
}), null)
matcherConfig = optional(object({
labelMatchers = list(list(string))
exactMatch = optional(bool, false)
bidirectionalLabelMatch = optional(bool, false)
priority = optional(number, 999)
enableDynamicLabels = optional(bool, false)
awsDynamicLabelsPolicy = optional(any, null)
}), null)
redrive_build_queue = optional(object({
enabled = bool
maxReceiveCount = number
}), {
enabled = false
maxReceiveCount = null
})
# V2 Contract
tags = optional(map(string), {})
runner = optional(object({
os = optional(string, null)
architecture = optional(string, null)
disable_default_labels = optional(bool, null)
extra_labels = optional(list(string), null)
group_name = optional(string, null)
name_prefix = optional(string, null)
run_as_root = optional(bool, null)
run_as = optional(string, null)
auto_update_disabled = optional(bool, null)
tags = optional(map(string), {})
hooks = optional(object({
job_started = optional(string, null)
job_completed = optional(string, null)
}), {})
iam = optional(object({
role = optional(object({
arn = string
}), null)
managed_policy_arns = optional(map(string), null)
additional_trust_policy_json = optional(string, null)
path = optional(string, null)
permissions_boundary = optional(string, null)
}), {})
}), {})
lambda = optional(object({
runtime = optional(string, null)
architecture = optional(string, null)
subnet_ids = optional(list(string), null)
security_group_ids = optional(list(string), null)
tags = optional(map(string), {})
role = optional(object({
path = optional(string, null)
permissions_boundary = optional(string, null)
}), {})
}), {})
orchestration_provider = optional(object({
webhook = optional(object({
runner = optional(object({
boot_time_in_minutes = optional(number, null)
ephemeral = optional(bool, null)
jit_config_enabled = optional(bool, null)
maximum_count = optional(number, null)
}), {})
github = optional(object({
organization_runners = optional(bool, false)
}), {})
matcherConfig = optional(object({
labelMatchers = list(list(string))
exactMatch = optional(bool, false)
bidirectionalLabelMatch = optional(bool, false)
priority = optional(number, 999)
dynamic_labels_enabled = optional(bool, false)
awsDynamicLabelsPolicy = optional(object({
allowed_keys = optional(list(string), [])
blocked_keys = optional(list(string), [])
restricted_keys = optional(map(object({
allowed = optional(list(string), [])
denied = optional(list(string), [])
max = optional(string, null)
})), {})
}), null)
}), null)
queue = optional(object({
delay_webhook_event = optional(number, null)
job_queue_retention_in_seconds = optional(number, null)
visibility_timeout_seconds = optional(number, null)
redrive_build_queue = optional(object({
enabled = optional(bool, null)
maxReceiveCount = optional(number, null)
}), null)
tags = optional(map(string), {})
}), {})
lambda = optional(object({
scale = optional(object({
up = optional(object({
memory_size = optional(number, null)
timeout = optional(number, null)
reserved_concurrent_executions = optional(number, null)
job_queued_check_enabled = optional(bool, null)
event_source_mapping = optional(object({
batch_size = optional(number, null)
maximum_batching_window_in_seconds = optional(number, null)
}), {})
tags = optional(map(string), {})
}), {})
down = optional(object({
memory_size = optional(number, null)
timeout = optional(number, null)
schedule_expression = optional(string, null)
minimum_running_time_in_minutes = optional(number, null)
idle_confirmation_seconds = optional(number, null)
idle_config = optional(list(object({
cron = string
timeZone = string
idleCount = number
evictionStrategy = optional(string, "oldest_first")
})), null)
tags = optional(map(string), {})
}), {})
}), {})
pool = optional(object({
memory_size = optional(number, null)
timeout = optional(number, null)
reserved_concurrent_executions = optional(number, null)
config = optional(list(object({
schedule_expression = string
schedule_expression_timezone = optional(string)
size = number
})), null)
include_busy_runners = optional(bool, null)
runner_owner = optional(string, null)
tags = optional(map(string), {})
}), {})
}), {})
job_retry = optional(object({
enabled = optional(bool, false)
delay_in_seconds = optional(number, 300)
delay_backoff = optional(number, 2)
max_attempts = optional(number, 1)
tags = optional(map(string), {})
lambda = optional(object({
memory_size = optional(number, 256)
reserved_concurrent_executions = optional(number, 1)
timeout = optional(number, 30)
}), {})
}), {})
}), null)
scale_set = optional(object({
name = string
runner = optional(object({
min_runners = optional(number, 0)
max_runners = optional(number, 10)
boot_time_in_minutes = optional(number, 10)
}), {})
}), null)
}), {})
storage_provider = optional(object({
aws = optional(object({
ssm = optional(object({
ttl_seconds = optional(object({
tokens = optional(number, null)
}), {})
paths = optional(object({
root = optional(string, null)
tokens = optional(string, null)
config = optional(string, null)
}), {})
tags = optional(map(string), {})
parameters = optional(object({
tags = optional(map(string), {})
}), {})
housekeeper = optional(object({
schedule_expression = optional(string, null)
state = optional(string, null)
tags = optional(map(string), {})
lambda = optional(object({
artifact = optional(object({
zip = optional(string, null)
s3 = optional(object({
key = string
object_version = optional(string, null)
}), null)
}), {})
memory_size = optional(number, null)
timeout = optional(number, null)
}), {})
config = optional(object({
tokenPath = optional(string, null)
minimumDaysOld = optional(number, null)
dryRun = optional(bool, null)
}), {})
}), {})
}), {})
}), {})
}), {})
observability = optional(object({
logs = optional(object({
level = optional(string, null)
retention_in_days = optional(number, null)
kms_key_id = optional(string, null)
class = optional(string, null)
tags = optional(map(string), {})
}), {})
tracing = optional(object({
mode = optional(string, null)
capture_http_requests = optional(bool, null)
capture_error = optional(bool, null)
}), {})
metrics = optional(object({
enabled = optional(bool, null)
namespace = optional(string, null)
metric = optional(object({
github_app_rate_limit = optional(object({
enabled = optional(bool, null)
}), {})
job_retry = optional(object({
enabled = optional(bool, null)
}), {})
spot_termination_warning = optional(object({
enabled = optional(bool, null)
}), {})
}), {})
}), {})
}), {})
compute_provider = optional(object({
aws = optional(object({
ec2 = optional(object({
metadata_options = optional(object({
instance_metadata_tags = optional(string, "enabled")
http_endpoint = optional(string, "enabled")
http_tokens = optional(string, "required")
http_put_response_hop_limit = optional(number, 1)
}), {})
ami = optional(object({
filter = optional(map(list(string)), { state = ["available"] })
owners = optional(list(string), ["amazon"])
id_ssm_parameter = optional(object({
arn = string
}), null)
kms_key = optional(object({
arn = string
}), null)
}), null)
block_device_mappings = optional(list(object({
delete_on_termination = optional(bool, true)
device_name = optional(string, "/dev/xvda")
encrypted = optional(bool, true)
iops = optional(number)
kms_key_id = optional(string)
snapshot_id = optional(string)
throughput = optional(number)
volume_initialization_rate = optional(number)
volume_size = number
volume_type = optional(string, "gp3")
})), [{ volume_size = 30 }])
create_service_linked_role_spot = optional(bool, false)
credit_specification = optional(string, null)
ebs_optimized = optional(bool, false)
cloudwatch_agent = optional(object({
enabled = optional(bool, true)
config = optional(string, null)
}), {})
binaries_syncer = optional(object({
enabled = optional(bool, null)
}), {})
detailed_monitoring_enabled = optional(bool, false)
ssm_enabled = optional(bool, false)
user_data = optional(object({
enabled = optional(bool, true)
template = optional(string, null)
content = optional(string, null)
pre_install = optional(string, "")
post_install = optional(string, "")
debug_logging_enabled = optional(bool, false)
}), {})
instance_allocation_strategy = optional(string, "lowest-price")
instance_max_spot_price = optional(string, null)
instance_target_capacity_type = optional(string, "spot")
instance_type_priorities = optional(map(number), null)
instance_types = optional(list(string), [])
additional_security_group_ids = optional(list(string), null)
managed_security_group_enabled = optional(bool, null)
egress_rules = optional(list(object({
cidr_blocks = list(string)
ipv6_cidr_blocks = list(string)
prefix_list_ids = list(string)
from_port = number
protocol = string
security_groups = list(string)
self = bool
to_port = number
description = string
})), null)
instance_profile_path = optional(string, null)
key_name = optional(string, null)
associate_public_ipv4_address = optional(bool, null)
instance_profile = optional(object({
name = string
}), null)
on_demand_failover_for_errors = optional(list(string), [])
scale_errors = optional(list(string), [
"UnfulfillableCapacity",
"MaxSpotInstanceCountExceeded",
"TargetCapacityLimitExceededException",
"RequestLimitExceeded",
"ResourceLimitExceeded",
"MaxSpotInstanceCountExceeded",
"MaxSpotFleetRequestCountExceeded",
"InsufficientInstanceCapacity",
"InsufficientCapacityOnHost",
])
subnet_ids = optional(list(string), null)
vpc_id = optional(string, null)
cpu_options = optional(object({
core_count = optional(number)
threads_per_core = optional(number)
amd_sev_snp = optional(string)
nested_virtualization = optional(string)
}), null)
network_interfaces = optional(list(object({
associate_carrier_ip_address = optional(bool)
associate_public_ip_address = optional(bool)
delete_on_termination = optional(bool)
description = optional(string)
device_index = optional(number)
interface_type = optional(string)
ipv4_address_count = optional(number)
ipv4_addresses = optional(list(string))
ipv4_prefix_count = optional(number)
ipv4_prefixes = optional(list(string))
ipv6_address_count = optional(number)
ipv6_addresses = optional(list(string))
ipv6_prefix_count = optional(number)
ipv6_prefixes = optional(list(string))
network_card_index = optional(number)
network_interface_id = optional(string)
primary_ipv6 = optional(bool)
private_ip_address = optional(string)
security_groups = optional(list(string))
subnet_id = optional(string)
connection_tracking_specification = optional(object({
tcp_established_timeout = optional(number)
udp_stream_timeout = optional(number)
udp_timeout = optional(number)
}))
ena_srd_specification = optional(object({
ena_srd_enabled = optional(bool)
ena_srd_udp_specification = optional(object({
ena_srd_udp_enabled = optional(bool)
}))
}))
})), [])
placement = optional(object({
affinity = optional(string)
availability_zone = optional(string)
group_id = optional(string)
group_name = optional(string)
host_id = optional(string)
host_resource_group_arn = optional(string)
spread_domain = optional(string)
tenancy = optional(string)
partition_number = optional(number)
}), null)
license_specifications = optional(list(object({
license_configuration_arn = string
})), [])
use_dedicated_host = optional(bool, false)
log_files = optional(list(object({
log_group_name = string
prefix_log_group = bool
file_path = string
log_stream_name = string
log_class = optional(string, "STANDARD")
})), null)
tags = optional(map(string), {})
}), null)
}), {})
}), {})
})) | `{}` | no |
+| [multi\_runner\_config](#input\_multi\_runner\_config) | Accepts either the stable v1 runner configuration shape or the provider-boundary v2 shape. Entries with `runner_config` use the v1 shape; entries without `runner_config` use the v2 shape. A v2 entry does not need matcher configuration. A v2 entry must be acknowledged with `experimental_features = ["multi-runner-v2"]`; the v2 shape is experimental and may change before graduation.map(object({
# V1 contract
runner_config = optional(object({
runner_os = string
runner_architecture = string
runner_metadata_options = optional(map(any), {
instance_metadata_tags = "enabled"
http_endpoint = "enabled"
http_tokens = "required"
http_put_response_hop_limit = 1
})
ami = optional(object({
filter = optional(map(list(string)), { state = ["available"] })
owners = optional(list(string), ["amazon"])
id_ssm_parameter_arn = optional(string, null)
kms_key_arn = optional(string, null)
}), null)
create_service_linked_role_spot = optional(bool, false)
credit_specification = optional(string, null)
delay_webhook_event = optional(number, 30)
disable_runner_autoupdate = optional(bool, false)
ebs_optimized = optional(bool, false)
enable_ephemeral_runners = optional(bool, false)
enable_job_queued_check = optional(bool, null)
enable_on_demand_failover_for_errors = optional(list(string), [])
scale_errors = optional(list(string), [
"UnfulfillableCapacity",
"MaxSpotInstanceCountExceeded",
"TargetCapacityLimitExceededException",
"RequestLimitExceeded",
"ResourceLimitExceeded",
"MaxSpotInstanceCountExceeded",
"MaxSpotFleetRequestCountExceeded",
"InsufficientInstanceCapacity",
"InsufficientCapacityOnHost",
])
enable_organization_runners = optional(bool, false)
enable_runner_binaries_syncer = optional(bool, true)
enable_ssm_on_runners = optional(bool, false)
enable_userdata = optional(bool, true)
instance_allocation_strategy = optional(string, "lowest-price")
instance_type_priorities = optional(map(number), null)
instance_max_spot_price = optional(string, null)
instance_target_capacity_type = optional(string, "spot")
instance_types = list(string)
job_queue_retention_in_seconds = optional(number, 86400)
minimum_running_time_in_minutes = optional(number, null)
pool_runner_owner = optional(string, null)
runner_as_root = optional(bool, false)
runner_boot_time_in_minutes = optional(number, 5)
scale_down_idle_confirmation_seconds = optional(number, 0)
runner_disable_default_labels = optional(bool, false)
runner_extra_labels = optional(list(string), [])
runner_group_name = optional(string, "Default")
runner_name_prefix = optional(string, "")
runner_run_as = optional(string, "ec2-user")
runners_maximum_count = number
runner_additional_security_group_ids = optional(list(string), [])
scale_down_schedule_expression = optional(string, "cron(*/5 * * * ? *)")
scale_up_reserved_concurrent_executions = optional(number, 1)
lambda_event_source_mapping_batch_size = optional(number, null)
lambda_event_source_mapping_maximum_batching_window_in_seconds = optional(number, null)
userdata_template = optional(string, null)
userdata_content = optional(string, null)
enable_jit_config = optional(bool, null)
enable_runner_detailed_monitoring = optional(bool, false)
enable_cloudwatch_agent = optional(bool, true)
cloudwatch_config = optional(string, null)
userdata_pre_install = optional(string, "")
userdata_post_install = optional(string, "")
runner_hook_job_started = optional(string, "")
runner_hook_job_completed = optional(string, "")
runner_ec2_tags = optional(map(string), {})
runner_iam_role_managed_policy_arns = optional(list(string), [])
vpc_id = optional(string, null)
subnet_ids = optional(list(string), null)
idle_config = optional(list(object({
cron = string
timeZone = string
idleCount = number
evictionStrategy = optional(string, "oldest_first")
})), [])
cpu_options = optional(object({
core_count = optional(number)
threads_per_core = optional(number)
amd_sev_snp = optional(string)
nested_virtualization = optional(string)
}), null)
network_interfaces = optional(list(object({
associate_carrier_ip_address = optional(bool)
associate_public_ip_address = optional(bool)
delete_on_termination = optional(bool)
description = optional(string)
device_index = optional(number)
interface_type = optional(string)
ipv4_address_count = optional(number)
ipv4_addresses = optional(list(string))
ipv4_prefix_count = optional(number)
ipv4_prefixes = optional(list(string))
ipv6_address_count = optional(number)
ipv6_addresses = optional(list(string))
ipv6_prefix_count = optional(number)
ipv6_prefixes = optional(list(string))
network_card_index = optional(number)
network_interface_id = optional(string)
primary_ipv6 = optional(bool)
private_ip_address = optional(string)
security_groups = optional(list(string))
subnet_id = optional(string)
connection_tracking_specification = optional(object({
tcp_established_timeout = optional(number)
udp_stream_timeout = optional(number)
udp_timeout = optional(number)
}))
ena_srd_specification = optional(object({
ena_srd_enabled = optional(bool)
ena_srd_udp_specification = optional(object({
ena_srd_udp_enabled = optional(bool)
}))
}))
})), [])
placement = optional(object({
affinity = optional(string)
availability_zone = optional(string)
group_id = optional(string)
group_name = optional(string)
host_id = optional(string)
host_resource_group_arn = optional(string)
spread_domain = optional(string)
tenancy = optional(string)
partition_number = optional(number)
}), null)
license_specifications = optional(list(object({
license_configuration_arn = string
})), [])
use_dedicated_host = optional(bool, false)
runner_log_files = optional(list(object({
log_group_name = string
prefix_log_group = bool
file_path = string
log_stream_name = string
log_class = optional(string, "STANDARD")
})), null)
block_device_mappings = optional(list(object({
delete_on_termination = optional(bool, true)
device_name = optional(string, "/dev/xvda")
encrypted = optional(bool, true)
iops = optional(number)
kms_key_id = optional(string)
snapshot_id = optional(string)
throughput = optional(number)
volume_initialization_rate = optional(number)
volume_size = number
volume_type = optional(string, "gp3")
})), [{
volume_size = 30
}])
pool_config = optional(list(object({
schedule_expression = string
schedule_expression_timezone = optional(string)
size = number
})), [])
ssm_ttl_seconds = optional(object({
tokens = optional(number, null)
}), {})
job_retry = optional(object({
enable = optional(bool, false)
delay_in_seconds = optional(number, 300)
delay_backoff = optional(number, 2)
lambda_memory_size = optional(number, 256)
lambda_timeout = optional(number, 30)
max_attempts = optional(number, 1)
}), {})
iam_overrides = optional(object({
override_instance_profile = optional(bool, null)
instance_profile_name = optional(string, null)
override_runner_role = optional(bool, null)
runner_role_arn = optional(string, null)
}), {
override_instance_profile = false
instance_profile_name = null
override_runner_role = false
runner_role_arn = null
})
}), null)
matcherConfig = optional(object({
labelMatchers = list(list(string))
exactMatch = optional(bool, false)
bidirectionalLabelMatch = optional(bool, false)
priority = optional(number, 999)
enableDynamicLabels = optional(bool, false)
awsDynamicLabelsPolicy = optional(any, null)
}), null)
redrive_build_queue = optional(object({
enabled = bool
maxReceiveCount = number
}), {
enabled = false
maxReceiveCount = null
})
# V2 Contract
tags = optional(map(string), {})
runner = optional(object({
os = optional(string, null)
architecture = optional(string, null)
disable_default_labels = optional(bool, null)
extra_labels = optional(list(string), null)
group_name = optional(string, null)
name_prefix = optional(string, null)
run_as_root = optional(bool, null)
run_as = optional(string, null)
auto_update_disabled = optional(bool, null)
tags = optional(map(string), {})
hooks = optional(object({
job_started = optional(string, null)
job_completed = optional(string, null)
}), {})
iam = optional(object({
role = optional(object({
arn = string
}), null)
managed_policy_arns = optional(map(string), null)
additional_trust_policy_json = optional(string, null)
path = optional(string, null)
permissions_boundary = optional(string, null)
}), {})
}), {})
lambda = optional(object({
runtime = optional(string, null)
architecture = optional(string, null)
subnet_ids = optional(list(string), null)
security_group_ids = optional(list(string), null)
tags = optional(map(string), {})
role = optional(object({
path = optional(string, null)
permissions_boundary = optional(string, null)
}), {})
}), {})
orchestration_provider = optional(object({
webhook = optional(object({
runner = optional(object({
boot_time_in_minutes = optional(number, null)
ephemeral = optional(bool, null)
jit_config_enabled = optional(bool, null)
maximum_count = optional(number, null)
}), {})
github = optional(object({
organization_runners = optional(bool, false)
}), {})
matcherConfig = optional(object({
labelMatchers = list(list(string))
exactMatch = optional(bool, false)
bidirectionalLabelMatch = optional(bool, false)
priority = optional(number, 999)
dynamic_labels_enabled = optional(bool, false)
awsDynamicLabelsPolicy = optional(object({
allowed_keys = optional(list(string), [])
blocked_keys = optional(list(string), [])
restricted_keys = optional(map(object({
allowed = optional(list(string), [])
denied = optional(list(string), [])
max = optional(string, null)
})), {})
}), null)
}), null)
queue = optional(object({
delay_webhook_event = optional(number, null)
job_queue_retention_in_seconds = optional(number, null)
visibility_timeout_seconds = optional(number, null)
redrive_build_queue = optional(object({
enabled = optional(bool, null)
maxReceiveCount = optional(number, null)
}), null)
tags = optional(map(string), {})
}), {})
lambda = optional(object({
scale = optional(object({
up = optional(object({
memory_size = optional(number, null)
timeout = optional(number, null)
reserved_concurrent_executions = optional(number, null)
job_queued_check_enabled = optional(bool, null)
event_source_mapping = optional(object({
batch_size = optional(number, null)
maximum_batching_window_in_seconds = optional(number, null)
}), {})
tags = optional(map(string), {})
}), {})
down = optional(object({
memory_size = optional(number, null)
timeout = optional(number, null)
schedule_expression = optional(string, null)
minimum_running_time_in_minutes = optional(number, null)
idle_confirmation_seconds = optional(number, null)
idle_config = optional(list(object({
cron = string
timeZone = string
idleCount = number
evictionStrategy = optional(string, "oldest_first")
})), null)
tags = optional(map(string), {})
}), {})
}), {})
pool = optional(object({
memory_size = optional(number, null)
timeout = optional(number, null)
reserved_concurrent_executions = optional(number, null)
config = optional(list(object({
schedule_expression = string
schedule_expression_timezone = optional(string)
size = number
})), null)
include_busy_runners = optional(bool, null)
runner_owner = optional(string, null)
tags = optional(map(string), {})
}), {})
}), {})
job_retry = optional(object({
enabled = optional(bool, false)
delay_in_seconds = optional(number, 300)
delay_backoff = optional(number, 2)
max_attempts = optional(number, 1)
tags = optional(map(string), {})
lambda = optional(object({
memory_size = optional(number, 256)
reserved_concurrent_executions = optional(number, 1)
timeout = optional(number, 30)
}), {})
}), {})
}), null)
scale_set = optional(object({
name = string
runner = optional(object({
min_runners = optional(number, 0)
max_runners = optional(number, 10)
boot_time_in_minutes = optional(number, 10)
}), {})
}), null)
}), {})
storage_provider = optional(object({
aws = optional(object({
ssm = optional(object({
ttl_seconds = optional(object({
tokens = optional(number, null)
}), {})
paths = optional(object({
root = optional(string, null)
tokens = optional(string, null)
config = optional(string, null)
}), {})
tags = optional(map(string), {})
parameters = optional(object({
tags = optional(map(string), {})
max_concurrent_invocations = optional(number, null)
max_writes_per_second = optional(number, null)
}), {})
housekeeper = optional(object({
schedule_expression = optional(string, null)
state = optional(string, null)
tags = optional(map(string), {})
lambda = optional(object({
artifact = optional(object({
zip = optional(string, null)
s3 = optional(object({
key = string
object_version = optional(string, null)
}), null)
}), {})
memory_size = optional(number, null)
timeout = optional(number, null)
}), {})
config = optional(object({
tokenPath = optional(string, null)
minimumDaysOld = optional(number, null)
dryRun = optional(bool, null)
}), {})
}), {})
}), {})
}), {})
}), {})
observability = optional(object({
logs = optional(object({
level = optional(string, null)
retention_in_days = optional(number, null)
kms_key_id = optional(string, null)
class = optional(string, null)
tags = optional(map(string), {})
}), {})
tracing = optional(object({
mode = optional(string, null)
capture_http_requests = optional(bool, null)
capture_error = optional(bool, null)
}), {})
metrics = optional(object({
enabled = optional(bool, null)
namespace = optional(string, null)
metric = optional(object({
github_app_rate_limit = optional(object({
enabled = optional(bool, null)
}), {})
job_retry = optional(object({
enabled = optional(bool, null)
}), {})
spot_termination_warning = optional(object({
enabled = optional(bool, null)
}), {})
}), {})
}), {})
}), {})
compute_provider = optional(object({
aws = optional(object({
ec2 = optional(object({
metadata_options = optional(object({
instance_metadata_tags = optional(string, "enabled")
http_endpoint = optional(string, "enabled")
http_tokens = optional(string, "required")
http_put_response_hop_limit = optional(number, 1)
}), {})
ami = optional(object({
filter = optional(map(list(string)), { state = ["available"] })
owners = optional(list(string), ["amazon"])
id_ssm_parameter = optional(object({
arn = string
}), null)
kms_key = optional(object({
arn = string
}), null)
}), null)
block_device_mappings = optional(list(object({
delete_on_termination = optional(bool, true)
device_name = optional(string, "/dev/xvda")
encrypted = optional(bool, true)
iops = optional(number)
kms_key_id = optional(string)
snapshot_id = optional(string)
throughput = optional(number)
volume_initialization_rate = optional(number)
volume_size = number
volume_type = optional(string, "gp3")
})), [{ volume_size = 30 }])
create_service_linked_role_spot = optional(bool, false)
credit_specification = optional(string, null)
ebs_optimized = optional(bool, false)
cloudwatch_agent = optional(object({
enabled = optional(bool, true)
config = optional(string, null)
}), {})
binaries_syncer = optional(object({
enabled = optional(bool, null)
}), {})
detailed_monitoring_enabled = optional(bool, false)
ssm_enabled = optional(bool, false)
user_data = optional(object({
enabled = optional(bool, true)
template = optional(string, null)
content = optional(string, null)
pre_install = optional(string, "")
post_install = optional(string, "")
debug_logging_enabled = optional(bool, false)
}), {})
instance_allocation_strategy = optional(string, "lowest-price")
instance_max_spot_price = optional(string, null)
instance_target_capacity_type = optional(string, "spot")
instance_type_priorities = optional(map(number), null)
instance_types = optional(list(string), [])
additional_security_group_ids = optional(list(string), null)
managed_security_group_enabled = optional(bool, null)
egress_rules = optional(list(object({
cidr_blocks = list(string)
ipv6_cidr_blocks = list(string)
prefix_list_ids = list(string)
from_port = number
protocol = string
security_groups = list(string)
self = bool
to_port = number
description = string
})), null)
instance_profile_path = optional(string, null)
key_name = optional(string, null)
associate_public_ipv4_address = optional(bool, null)
instance_profile = optional(object({
name = string
}), null)
on_demand_failover_for_errors = optional(list(string), [])
scale_errors = optional(list(string), [
"UnfulfillableCapacity",
"MaxSpotInstanceCountExceeded",
"TargetCapacityLimitExceededException",
"RequestLimitExceeded",
"ResourceLimitExceeded",
"MaxSpotInstanceCountExceeded",
"MaxSpotFleetRequestCountExceeded",
"InsufficientInstanceCapacity",
"InsufficientCapacityOnHost",
])
subnet_ids = optional(list(string), null)
vpc_id = optional(string, null)
cpu_options = optional(object({
core_count = optional(number)
threads_per_core = optional(number)
amd_sev_snp = optional(string)
nested_virtualization = optional(string)
}), null)
network_interfaces = optional(list(object({
associate_carrier_ip_address = optional(bool)
associate_public_ip_address = optional(bool)
delete_on_termination = optional(bool)
description = optional(string)
device_index = optional(number)
interface_type = optional(string)
ipv4_address_count = optional(number)
ipv4_addresses = optional(list(string))
ipv4_prefix_count = optional(number)
ipv4_prefixes = optional(list(string))
ipv6_address_count = optional(number)
ipv6_addresses = optional(list(string))
ipv6_prefix_count = optional(number)
ipv6_prefixes = optional(list(string))
network_card_index = optional(number)
network_interface_id = optional(string)
primary_ipv6 = optional(bool)
private_ip_address = optional(string)
security_groups = optional(list(string))
subnet_id = optional(string)
connection_tracking_specification = optional(object({
tcp_established_timeout = optional(number)
udp_stream_timeout = optional(number)
udp_timeout = optional(number)
}))
ena_srd_specification = optional(object({
ena_srd_enabled = optional(bool)
ena_srd_udp_specification = optional(object({
ena_srd_udp_enabled = optional(bool)
}))
}))
})), [])
placement = optional(object({
affinity = optional(string)
availability_zone = optional(string)
group_id = optional(string)
group_name = optional(string)
host_id = optional(string)
host_resource_group_arn = optional(string)
spread_domain = optional(string)
tenancy = optional(string)
partition_number = optional(number)
}), null)
license_specifications = optional(list(object({
license_configuration_arn = string
})), [])
use_dedicated_host = optional(bool, false)
log_files = optional(list(object({
log_group_name = string
prefix_log_group = bool
file_path = string
log_stream_name = string
log_class = optional(string, "STANDARD")
})), null)
tags = optional(map(string), {})
}), null)
}), {})
}), {})
})) | `{}` | no |
| [parameter\_store\_tags](#input\_parameter\_store\_tags) | Map of tags that will be added to all the SSM Parameter Store parameters created by the Lambda function. | `map(string)` | `{}` | no |
| [pool\_lambda\_reserved\_concurrent\_executions](#input\_pool\_lambda\_reserved\_concurrent\_executions) | Amount of reserved concurrent executions for the scale-up lambda function. A value of 0 disables lambda from being triggered and -1 removes any concurrency limitations. | `number` | `1` | no |
| [pool\_lambda\_timeout](#input\_pool\_lambda\_timeout) | Time out for the pool lambda in seconds. | `number` | `60` | no |
@@ -215,6 +215,8 @@ module "multi-runner" {
| [runners\_ssm\_housekeeper](#input\_runners\_ssm\_housekeeper) | Configuration for the SSM housekeeper lambda. This lambda deletes token / JIT config from SSM.object({
schedule_expression = optional(string, "rate(1 day)")
enabled = optional(bool, true)
lambda_memory_size = optional(number, 512)
lambda_timeout = optional(number, 60)
config = object({
tokenPath = optional(string)
minimumDaysOld = optional(number, 1)
dryRun = optional(bool, false)
})
}) | {
"config": {}
} | no |
| [scale\_down\_lambda\_memory\_size](#input\_scale\_down\_lambda\_memory\_size) | Memory size limit in MB for scale down. | `number` | `512` | no |
| [scale\_up\_lambda\_memory\_size](#input\_scale\_up\_lambda\_memory\_size) | Memory size limit in MB for scale\_up lambda. | `number` | `512` | no |
+| [ssm\_parameter\_store\_max\_concurrent\_invocations](#input\_ssm\_parameter\_store\_max\_concurrent\_invocations) | Expected number of concurrent scale-up/pool lambda invocations writing to Parameter Store, used to pace each invocation's writes to a share of the account-wide write-rate limit. Applies to all runner configurations. | `number` | `1` | no |
+| [ssm\_parameter\_store\_max\_writes\_per\_second](#input\_ssm\_parameter\_store\_max\_writes\_per\_second) | Parameter Store write-rate limit to pace against, in writes/second. Defaults to the standard-tier limit; raise this if SSM's higher-throughput tier is enabled (up to several thousand writes/second). Applies to all runner configurations. | `number` | `40` | no |
| [ssm\_paths](#input\_ssm\_paths) | The root path used in SSM to store configuration and secrets. | object({
root = optional(string, "github-action-runners")
app = optional(string, "app")
runners = optional(string, "runners")
webhook = optional(string, "webhook")
}) | `{}` | no |
| [state\_event\_rule\_binaries\_syncer](#input\_state\_event\_rule\_binaries\_syncer) | Option to disable EventBridge Lambda trigger for the binary syncer, useful to stop automatic updates of binary distribution | `string` | `"ENABLED"` | no |
| [subnet\_ids](#input\_subnet\_ids) | List of subnets in which stable v1 action runners will be launched. Omit when using the experimental v2 interface. | `list(string)` | `null` | no |
diff --git a/modules/multi-runner/config.experimental.resolved.tf b/modules/multi-runner/config.experimental.resolved.tf
index b2b7b68309..54b661ac2c 100644
--- a/modules/multi-runner/config.experimental.resolved.tf
+++ b/modules/multi-runner/config.experimental.resolved.tf
@@ -319,6 +319,14 @@ locals {
tags = merge(local.normalized_config.storage_provider.aws.ssm.tags, v.storage_provider.aws.ssm.tags)
parameters = {
tags = merge(local.normalized_config.storage_provider.aws.ssm.parameters.tags, v.storage_provider.aws.ssm.parameters.tags)
+ max_concurrent_invocations = coalesce(
+ v.storage_provider.aws.ssm.parameters.max_concurrent_invocations,
+ local.normalized_config.storage_provider.aws.ssm.parameters.max_concurrent_invocations,
+ )
+ max_writes_per_second = coalesce(
+ v.storage_provider.aws.ssm.parameters.max_writes_per_second,
+ local.normalized_config.storage_provider.aws.ssm.parameters.max_writes_per_second,
+ )
}
housekeeper = {
schedule_expression = coalesce(
diff --git a/modules/multi-runner/config.experimental.translation.tf b/modules/multi-runner/config.experimental.translation.tf
index f8a5e5693e..befc79117f 100644
--- a/modules/multi-runner/config.experimental.translation.tf
+++ b/modules/multi-runner/config.experimental.translation.tf
@@ -74,7 +74,9 @@ locals {
kms_key_id = var.kms_key_arn
tags = {}
parameters = {
- tags = var.parameter_store_tags
+ tags = var.parameter_store_tags
+ max_concurrent_invocations = var.ssm_parameter_store_max_concurrent_invocations
+ max_writes_per_second = var.ssm_parameter_store_max_writes_per_second
}
housekeeper = {
schedule_expression = var.runners_ssm_housekeeper.schedule_expression
@@ -450,7 +452,9 @@ locals {
}
tags = {}
parameters = {
- tags = {}
+ tags = {}
+ max_concurrent_invocations = null
+ max_writes_per_second = null
}
housekeeper = {
schedule_expression = null
diff --git a/modules/multi-runner/runners.tf b/modules/multi-runner/runners.tf
index 1311ee9dc9..de6fead29a 100644
--- a/modules/multi-runner/runners.tf
+++ b/modules/multi-runner/runners.tf
@@ -97,6 +97,8 @@ module "runners" {
runner_group_name = each.value.runner.group_name
runner_name_prefix = each.value.runner.name_prefix
parameter_store_tags = each.value.storage_provider.aws.ssm.parameters.tags
+ ssm_parameter_store_max_concurrent_invocations = each.value.storage_provider.aws.ssm.parameters.max_concurrent_invocations
+ ssm_parameter_store_max_writes_per_second = each.value.storage_provider.aws.ssm.parameters.max_writes_per_second
scale_up_reserved_concurrent_executions = each.value.orchestration_provider.webhook.lambda.scale.up.reserved_concurrent_executions
diff --git a/modules/multi-runner/tests/config-translation.tftest.hcl b/modules/multi-runner/tests/config-translation.tftest.hcl
index f786fa573c..46485f1e80 100644
--- a/modules/multi-runner/tests/config-translation.tftest.hcl
+++ b/modules/multi-runner/tests/config-translation.tftest.hcl
@@ -204,7 +204,9 @@ run "empty_v2_map_translates_stable_inputs" {
runners = "legacy-runners"
webhook = "legacy-webhook"
}
- parameter_store_tags = { owner = "stable-test" }
+ parameter_store_tags = { owner = "stable-test" }
+ ssm_parameter_store_max_concurrent_invocations = 4
+ ssm_parameter_store_max_writes_per_second = 200
runners_ssm_housekeeper = {
schedule_expression = "rate(2 days)"
enabled = false
@@ -431,6 +433,10 @@ run "empty_v2_map_translates_stable_inputs" {
&& local.stable_to_v2.orchestration_provider.webhook.lambda.scale.up.event_source_mapping.batch_size == var.lambda_event_source_mapping_batch_size
&& local.stable_to_v2.orchestration_provider.webhook.lambda.scale.down.idle_config == []
&& local.stable_to_v2.storage_provider.aws.ssm.parameters.tags.owner == var.parameter_store_tags.owner
+ && local.stable_to_v2.storage_provider.aws.ssm.parameters.max_concurrent_invocations == 4
+ && local.stable_to_v2.storage_provider.aws.ssm.parameters.max_writes_per_second == 200
+ && local.effective_config.multi_runner_config["stable"].storage_provider.aws.ssm.parameters.max_concurrent_invocations == 4
+ && local.effective_config.multi_runner_config["stable"].storage_provider.aws.ssm.parameters.max_writes_per_second == 200
&& local.stable_to_v2.storage_provider.aws.ssm.housekeeper.lambda.memory_size == var.runners_ssm_housekeeper.lambda_memory_size
&& local.stable_to_v2.compute_provider.aws.ec2.runner_binaries.s3.encryption.sse_algorithm == "aws:kms"
&& local.stable_to_v2.compute_provider.aws.ec2.runner_binaries.s3.encryption.kms_master_key_id == "arn:aws:kms:eu-west-1:123456789012:key/binaries"
@@ -643,6 +649,10 @@ run "lane_values_override_experimental_globals" {
global_config_storage_provider = {
aws = {
ssm = {
+ parameters = {
+ max_concurrent_invocations = 8
+ max_writes_per_second = 1000
+ }
housekeeper = {
lambda = {
artifact = {
@@ -710,6 +720,9 @@ run "lane_values_override_experimental_globals" {
storage_provider = {
aws = {
ssm = {
+ parameters = {
+ max_writes_per_second = 3000
+ }
housekeeper = {
lambda = {
artifact = {
@@ -751,6 +764,8 @@ run "lane_values_override_experimental_globals" {
&& local.resolved_config.multi_runner_config["lane"].observability.logs.retention_in_days == 30
&& local.resolved_config.multi_runner_config["lane"].storage_provider.aws.ssm.housekeeper.lambda.artifact.zip == null
&& local.resolved_config.multi_runner_config["lane"].storage_provider.aws.ssm.housekeeper.lambda.artifact.s3.key == "lane-housekeeper.zip"
+ && local.resolved_config.multi_runner_config["lane"].storage_provider.aws.ssm.parameters.max_writes_per_second == 3000
+ && local.resolved_config.multi_runner_config["lane"].storage_provider.aws.ssm.parameters.max_concurrent_invocations == 8
)
error_message = "Lane values must override v2 globals while omitted values inherit their global defaults."
}
diff --git a/modules/multi-runner/variables.experimental.storage-provider.tf b/modules/multi-runner/variables.experimental.storage-provider.tf
index a76c07a45f..2691175a46 100644
--- a/modules/multi-runner/variables.experimental.storage-provider.tf
+++ b/modules/multi-runner/variables.experimental.storage-provider.tf
@@ -12,6 +12,8 @@ variable "global_config_storage_provider" {
aws.ssm.kms_key_id: "KMS key ID used to encrypt SSM parameters."
aws.ssm.tags: "Tags applied to SSM resources."
aws.ssm.parameters.tags: "Tags applied to runner configuration parameters."
+ aws.ssm.parameters.max_concurrent_invocations: "Expected number of concurrent scale-up and pool Lambda invocations writing to Parameter Store, used to pace each invocation's writes to a share of the account-wide write-rate limit."
+ aws.ssm.parameters.max_writes_per_second: "Parameter Store write-rate limit to pace against, in writes per second. Raise this above 40 only when SSM's higher-throughput tier is enabled."
aws.ssm.housekeeper.schedule_expression: "Schedule for the SSM parameter housekeeper."
aws.ssm.housekeeper.state: "EventBridge rule state for the SSM housekeeper."
aws.ssm.housekeeper.tags: "Tags applied to the SSM housekeeper resources."
@@ -38,7 +40,9 @@ variable "global_config_storage_provider" {
kms_key_id = optional(string, null)
tags = optional(map(string), {})
parameters = optional(object({
- tags = optional(map(string), {})
+ tags = optional(map(string), {})
+ max_concurrent_invocations = optional(number, 1)
+ max_writes_per_second = optional(number, 40)
}), {})
housekeeper = optional(object({
schedule_expression = optional(string, "rate(1 day)")
diff --git a/modules/multi-runner/variables.tf b/modules/multi-runner/variables.tf
index 59b54a3978..c3e01cb1f0 100644
--- a/modules/multi-runner/variables.tf
+++ b/modules/multi-runner/variables.tf
@@ -451,7 +451,9 @@ variable "multi_runner_config" {
}), {})
tags = optional(map(string), {})
parameters = optional(object({
- tags = optional(map(string), {})
+ tags = optional(map(string), {})
+ max_concurrent_invocations = optional(number, null)
+ max_writes_per_second = optional(number, null)
}), {})
housekeeper = optional(object({
schedule_expression = optional(string, null)
@@ -1278,3 +1280,15 @@ variable "parameter_store_tags" {
type = map(string)
default = {}
}
+
+variable "ssm_parameter_store_max_concurrent_invocations" {
+ description = "Expected number of concurrent scale-up/pool lambda invocations writing to Parameter Store, used to pace each invocation's writes to a share of the account-wide write-rate limit. Applies to all runner configurations."
+ type = number
+ default = 1
+}
+
+variable "ssm_parameter_store_max_writes_per_second" {
+ description = "Parameter Store write-rate limit to pace against, in writes/second. Defaults to the standard-tier limit; raise this if SSM's higher-throughput tier is enabled (up to several thousand writes/second). Applies to all runner configurations."
+ type = number
+ default = 40
+}
diff --git a/modules/orchestration-providers/webhook/README.md b/modules/orchestration-providers/webhook/README.md
index a03e8b4f1d..449508b1b5 100644
--- a/modules/orchestration-providers/webhook/README.md
+++ b/modules/orchestration-providers/webhook/README.md
@@ -46,7 +46,7 @@ The scale-down lifecycle is documented in the [scale-down state diagram](./scale
| [prefix](#input\_prefix) | Prefix used to identify resources created for this webhook orchestration provider. | `string` | n/a | yes |
| [runner](#input\_runner) | Common runner registration values consumed by webhook demand controls. Lifecycle, boot timeout, and capacity remain provider-owned under config.runner. | object({
os = string
auto_update_disabled = bool
labels = list(string)
group_name = string
name_prefix = string
}) | n/a | yes |
| [runner\_provider](#input\_runner\_provider) | Selected compute-provider capabilities consumed by webhook scale-up, scale-down, and pool controls. | object({
type = string
scale_up = object({
environment_variables = map(string)
iam_policy_json = string
additional_iam_policy_json = optional(string, null)
managed_policy = optional(object({
arn = string
}), null)
})
scale_down = object({
environment_variables = map(string)
iam_policy_json = string
})
pool = object({
environment_variables = map(string)
iam_policy_json = string
managed_policy_enabled = bool
managed_policy_arn = optional(string, null)
})
}) | n/a | yes |
-| [storage\_provider](#input\_storage\_provider) | Resolved storage-provider values and optional provider-owned Lambda capabilities.object({
aws = object({
ssm = object({
token_path = string
token_path_arn = string
config_path = string
config_path_arn = string
kms_key_id = optional(string, null)
parameter_store_tags = string
})
})
scale_up = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
scale_down = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
pool = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
job_retry = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
}) | n/a | yes |
+| [storage\_provider](#input\_storage\_provider) | Resolved storage-provider values and optional provider-owned Lambda capabilities.object({
aws = object({
ssm = object({
token_path = string
token_path_arn = string
config_path = string
config_path_arn = string
kms_key_id = optional(string, null)
parameter_store_tags = string
parameter_store_max_concurrent_invocations = optional(number, 1)
parameter_store_max_writes_per_second = optional(number, 40)
})
})
scale_up = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
scale_down = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
pool = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
job_retry = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
}) | n/a | yes |
| [tags](#input\_tags) | Base tags available to webhook-provider resources. Component-specific tags override this map within their documented scopes. | `map(string)` | `{}` | no |
## Outputs
diff --git a/modules/orchestration-providers/webhook/pool/README.md b/modules/orchestration-providers/webhook/pool/README.md
index a6f7a0e086..4539bcd309 100644
--- a/modules/orchestration-providers/webhook/pool/README.md
+++ b/modules/orchestration-providers/webhook/pool/README.md
@@ -56,7 +56,7 @@ No modules.
| [aws\_partition](#input\_aws\_partition) | (optional) partition for the arn if not 'aws' | `string` | `"aws"` | no |
| [config](#input\_config) | Configuration passed from the webhook orchestration provider to the pool Lambda and scheduler.object({
lambda = object({
log_level = string
logging_retention_in_days = number
logging_kms_key_id = string
log_class = string
reserved_concurrent_executions = number
s3_bucket = string
s3_key = string
s3_object_version = string
security_group_ids = list(string)
runtime = string
architecture = string
memory_size = number
timeout = number
zip = string
subnet_ids = list(string)
principals = optional(list(object({
type = string
identifiers = list(string)
})), [])
})
tags = map(string)
ghes = object({
url = string
ssl_verify = string
})
github_app_parameters = object({
key_base64 = map(string)
id = map(string)
additional_apps_manifest = optional(object({
name = string
arn = string
}), null)
additional_app_parameter_arns = optional(list(string), [])
})
runner = object({
disable_runner_autoupdate = bool
ephemeral = bool
enable_jit_config = bool
labels = list(string)
group_name = string
name_prefix = string
pool_owner = string
boot_time_in_minutes = number
})
runners_maximum_count = number
prefix = string
pool = list(object({
schedule_expression = string
schedule_expression_timezone = string
size = number
}))
include_busy_runners = bool
role_permissions_boundary = string
role_path = string
lambda_tags = map(string)
log_group_tags = optional(map(string), {})
user_agent = string
}) | n/a | yes |
| [runner\_provider](#input\_runner\_provider) | Compute provider integration used by the pool Lambda.object({
type = string
environment_variables = map(string)
iam_policy_json = string
managed_policy_enabled = bool
managed_policy_arn = optional(string, null)
}) | n/a | yes |
-| [storage\_provider](#input\_storage\_provider) | Resolved storage-provider configuration and capability used by the pool Lambda. | object({
aws = object({
ssm = object({
token_path = string
token_path_arn = string
config_path = string
config_path_arn = string
kms_key_id = optional(string, null)
parameter_store_tags = string
})
})
environment_variables = optional(map(string), {})
iam_policy_json = optional(string, null)
}) | n/a | yes |
+| [storage\_provider](#input\_storage\_provider) | Resolved storage-provider configuration and capability used by the pool Lambda. | object({
aws = object({
ssm = object({
token_path = string
token_path_arn = string
config_path = string
config_path_arn = string
kms_key_id = optional(string, null)
parameter_store_tags = string
parameter_store_max_concurrent_invocations = optional(number, 1)
parameter_store_max_writes_per_second = optional(number, 40)
})
})
environment_variables = optional(map(string), {})
iam_policy_json = optional(string, null)
}) | n/a | yes |
| [tracing\_config](#input\_tracing\_config) | Tracing configuration for the pool Lambda.object({
mode = optional(string, null)
capture_http_requests = optional(bool, false)
capture_error = optional(bool, false)
}) | `{}` | no |
## Outputs
diff --git a/modules/orchestration-providers/webhook/pool/pool.tf b/modules/orchestration-providers/webhook/pool/pool.tf
index 1ff7249fd7..43e3653e8f 100644
--- a/modules/orchestration-providers/webhook/pool/pool.tf
+++ b/modules/orchestration-providers/webhook/pool/pool.tf
@@ -30,12 +30,14 @@ locals {
}
ssm_environment_variables = {
- PARAMETER_GITHUB_APP_ID_NAME = var.config.github_app_parameters.id.name
- PARAMETER_GITHUB_APP_KEY_BASE64_NAME = var.config.github_app_parameters.key_base64.name
- PARAMETER_GITHUB_APPS_MANIFEST_NAME = var.config.github_app_parameters.additional_apps_manifest != null ? var.config.github_app_parameters.additional_apps_manifest.name : ""
- SSM_TOKEN_PATH = var.storage_provider.aws.ssm.token_path
- SSM_CONFIG_PATH = var.storage_provider.aws.ssm.config_path
- SSM_PARAMETER_STORE_TAGS = var.storage_provider.aws.ssm.parameter_store_tags
+ PARAMETER_GITHUB_APP_ID_NAME = var.config.github_app_parameters.id.name
+ PARAMETER_GITHUB_APP_KEY_BASE64_NAME = var.config.github_app_parameters.key_base64.name
+ PARAMETER_GITHUB_APPS_MANIFEST_NAME = var.config.github_app_parameters.additional_apps_manifest != null ? var.config.github_app_parameters.additional_apps_manifest.name : ""
+ SSM_TOKEN_PATH = var.storage_provider.aws.ssm.token_path
+ SSM_CONFIG_PATH = var.storage_provider.aws.ssm.config_path
+ SSM_PARAMETER_STORE_TAGS = var.storage_provider.aws.ssm.parameter_store_tags
+ SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS = var.storage_provider.aws.ssm.parameter_store_max_concurrent_invocations
+ SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND = var.storage_provider.aws.ssm.parameter_store_max_writes_per_second
}
}
diff --git a/modules/orchestration-providers/webhook/pool/tests/provider.tftest.hcl b/modules/orchestration-providers/webhook/pool/tests/provider.tftest.hcl
index 25fa6c8a14..88e43352b5 100644
--- a/modules/orchestration-providers/webhook/pool/tests/provider.tftest.hcl
+++ b/modules/orchestration-providers/webhook/pool/tests/provider.tftest.hcl
@@ -16,6 +16,9 @@ variables {
config_path_arn = "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/config"
kms_key_id = "arn:aws:kms:eu-west-1:123456789012:key/pool-test"
parameter_store_tags = "{}"
+
+ parameter_store_max_concurrent_invocations = 3
+ parameter_store_max_writes_per_second = 120
}
}
}
@@ -143,7 +146,9 @@ run "provider_supplies_only_compute_specific_pool_configuration" {
assert {
condition = (
- aws_lambda_function.pool.environment[0].variables["PARAMETER_GITHUB_APP_ID_NAME"] == "/github-runner/app-id"
+ aws_lambda_function.pool.environment[0].variables["SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS"] == "3"
+ && aws_lambda_function.pool.environment[0].variables["SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND"] == "120"
+ && aws_lambda_function.pool.environment[0].variables["PARAMETER_GITHUB_APP_ID_NAME"] == "/github-runner/app-id"
&& aws_lambda_function.pool.environment[0].variables["PARAMETER_GITHUB_APP_KEY_BASE64_NAME"] == "/github-runner/key-base64"
&& aws_lambda_function.pool.environment[0].variables["PARAMETER_GITHUB_APPS_MANIFEST_NAME"] == "/github-runner/additional-apps-manifest"
&& contains(data.aws_iam_policy_document.pool_common.statement[2].resources, "arn:aws:ssm:eu-west-1:123456789012:parameter/github-runner/app-id-2")
diff --git a/modules/orchestration-providers/webhook/pool/variables.tf b/modules/orchestration-providers/webhook/pool/variables.tf
index 8ccb8fa16d..eb10c1e586 100644
--- a/modules/orchestration-providers/webhook/pool/variables.tf
+++ b/modules/orchestration-providers/webhook/pool/variables.tf
@@ -142,6 +142,9 @@ variable "storage_provider" {
config_path_arn = string
kms_key_id = optional(string, null)
parameter_store_tags = string
+
+ parameter_store_max_concurrent_invocations = optional(number, 1)
+ parameter_store_max_writes_per_second = optional(number, 40)
})
})
environment_variables = optional(map(string), {})
diff --git a/modules/orchestration-providers/webhook/scale-runners/README.md b/modules/orchestration-providers/webhook/scale-runners/README.md
index 87e3c1cefb..3b621d6a67 100644
--- a/modules/orchestration-providers/webhook/scale-runners/README.md
+++ b/modules/orchestration-providers/webhook/scale-runners/README.md
@@ -67,9 +67,9 @@ No modules.
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| [aws\_partition](#input\_aws\_partition) | AWS partition used to construct IAM policy ARNs. | `string` | `"aws"` | no |
-| [config](#input\_config) | Provider-neutral scale-up and scale-down configuration assembled by runner-config.object({
prefix = string
lambda = object({
artifact = object({
zip = string
s3 = object({
bucket = optional(string, null)
key = optional(string, null)
object_version = optional(string, null)
})
})
runtime = string
architecture = string
vpc = object({
subnet_ids = list(string)
security_group_ids = list(string)
})
role = object({
path = string
permissions_boundary = optional(string, null)
principals = optional(list(object({
type = string
identifiers = list(string)
})), [])
})
})
runner = object({
os = string
auto_update_disabled = bool
ephemeral = bool
jit_config_enabled = optional(bool, null)
labels = list(string)
group_name = string
name_prefix = string
boot_time_in_minutes = number
maximum_count = number
})
github = object({
organization_runners = bool
enterprise_server = object({
url = optional(string, null)
ssl_verify = bool
})
user_agent = optional(string, null)
app_parameters = object({
key_base64 = map(string)
id = map(string)
additional_apps_manifest = optional(object({
name = string
arn = string
}), null)
additional_app_parameter_arns = optional(list(string), [])
})
})
queue = object({
build = object({
arn = string
})
kms_key_id = optional(string, null)
event_source_mapping = object({
batch_size = number
maximum_batching_window_in_seconds = number
})
})
observability = object({
logs = object({
level = string
retention_in_days = number
kms_key_id = optional(string, null)
class = string
})
tracing = object({
mode = optional(string, null)
capture_http_requests = bool
capture_error = bool
})
metrics = object({
enabled = bool
namespace = string
metric = object({
github_app_rate_limit = object({
enabled = bool
})
})
})
})
scale_up = object({
memory_size = number
timeout = number
reserved_concurrent_executions = number
job_queued_check_enabled = bool
tags = object({
resources = map(string)
lambda = map(string)
log_group = map(string)
event_source_mapping = map(string)
})
})
scale_down = object({
memory_size = number
timeout = number
schedule_expression = string
minimum_running_time_in_minutes = optional(number, null)
idle_confirmation_seconds = optional(number, 0)
idle_config = list(object({
cron = string
timeZone = string
idleCount = number
evictionStrategy = string
}))
tags = object({
resources = map(string)
lambda = map(string)
log_group = map(string)
})
})
job_retry = object({
enabled = bool
max_attempts = number
delay_in_seconds = number
delay_backoff = number
queue = optional(object({
arn = string
url = string
}), null)
})
}) | n/a | yes |
+| [config](#input\_config) | Provider-neutral scale-up and scale-down configuration assembled by runner-config.object({
prefix = string
lambda = object({
artifact = object({
zip = string
s3 = object({
bucket = optional(string, null)
key = optional(string, null)
object_version = optional(string, null)
})
})
runtime = string
architecture = string
vpc = object({
subnet_ids = list(string)
security_group_ids = list(string)
})
role = object({
path = string
permissions_boundary = optional(string, null)
principals = optional(list(object({
type = string
identifiers = list(string)
})), [])
})
})
runner = object({
os = string
auto_update_disabled = bool
ephemeral = bool
jit_config_enabled = optional(bool, null)
labels = list(string)
group_name = string
name_prefix = string
boot_time_in_minutes = number
maximum_count = number
})
github = object({
organization_runners = bool
enterprise_server = object({
url = optional(string, null)
ssl_verify = bool
})
user_agent = optional(string, null)
app_parameters = object({
key_base64 = map(string)
id = map(string)
additional_apps_manifest = optional(object({
name = string
arn = string
}), null)
additional_app_parameter_arns = optional(list(string), [])
})
})
queue = object({
build = object({
arn = string
})
kms_key_id = optional(string, null)
event_source_mapping = object({
batch_size = number
maximum_batching_window_in_seconds = number
})
})
observability = object({
logs = object({
level = string
retention_in_days = number
kms_key_id = optional(string, null)
class = string
})
tracing = object({
mode = optional(string, null)
capture_http_requests = bool
capture_error = bool
})
metrics = object({
enabled = bool
namespace = string
metric = object({
github_app_rate_limit = object({
enabled = bool
})
})
})
})
scale_up = object({
memory_size = number
timeout = number
reserved_concurrent_executions = number
job_queued_check_enabled = bool
tags = object({
resources = map(string)
lambda = map(string)
log_group = map(string)
event_source_mapping = map(string)
})
})
scale_down = object({
memory_size = number
timeout = number
schedule_expression = string
minimum_running_time_in_minutes = optional(number, null)
idle_confirmation_seconds = optional(number, 0)
idle_config = list(object({
cron = string
timeZone = string
idleCount = number
evictionStrategy = string
}))
tags = object({
resources = map(string)
lambda = map(string)
log_group = map(string)
})
})
job_retry = object({
enabled = bool
max_attempts = number
delay_in_seconds = number
delay_backoff = number
queue = optional(object({
arn = string
url = string
}), null)
})
}) | n/a | yes |
| [runner\_provider](#input\_runner\_provider) | Selected compute-provider integration for the scaling control plane.object({
type = string
scale_up = object({
environment_variables = map(string)
iam_policy_json = string
additional_iam_policy_json = optional(string, null)
managed_policy = optional(object({
arn = string
}), null)
})
scale_down = object({
environment_variables = map(string)
iam_policy_json = string
})
}) | n/a | yes |
-| [storage\_provider](#input\_storage\_provider) | Resolved storage-provider configuration and capabilities for scale-up and scale-down. | object({
aws = object({
ssm = object({
token_path = string
token_path_arn = string
config_path = string
config_path_arn = string
parameter_store_tags = string
kms_key_id = optional(string, null)
})
})
scale_up = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
scale_down = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
}) | n/a | yes |
+| [storage\_provider](#input\_storage\_provider) | Resolved storage-provider configuration and capabilities for scale-up and scale-down. | object({
aws = object({
ssm = object({
token_path = string
token_path_arn = string
config_path = string
config_path_arn = string
parameter_store_tags = string
kms_key_id = optional(string, null)
parameter_store_max_concurrent_invocations = optional(number, 1)
parameter_store_max_writes_per_second = optional(number, 40)
})
})
scale_up = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
scale_down = optional(object({
environment_variables = map(string)
iam_policy_json = optional(string, null)
}), {
environment_variables = {}
iam_policy_json = null
})
}) | n/a | yes |
## Outputs
diff --git a/modules/orchestration-providers/webhook/scale-runners/scale-up.tf b/modules/orchestration-providers/webhook/scale-runners/scale-up.tf
index 3b66812f83..cc26020663 100644
--- a/modules/orchestration-providers/webhook/scale-runners/scale-up.tf
+++ b/modules/orchestration-providers/webhook/scale-runners/scale-up.tf
@@ -41,12 +41,14 @@ resource "aws_lambda_function" "scale_up" {
POWERTOOLS_SERVICE_NAME = "${var.config.prefix}-scale-up"
JOB_RETRY_CONFIG = jsonencode(local.job_retry_config)
}, {
- PARAMETER_GITHUB_APP_ID_NAME = var.config.github.app_parameters.id.name
- PARAMETER_GITHUB_APP_KEY_BASE64_NAME = var.config.github.app_parameters.key_base64.name
- PARAMETER_GITHUB_APPS_MANIFEST_NAME = var.config.github.app_parameters.additional_apps_manifest != null ? var.config.github.app_parameters.additional_apps_manifest.name : ""
- SSM_TOKEN_PATH = var.storage_provider.aws.ssm.token_path
- SSM_CONFIG_PATH = var.storage_provider.aws.ssm.config_path
- SSM_PARAMETER_STORE_TAGS = var.storage_provider.aws.ssm.parameter_store_tags
+ PARAMETER_GITHUB_APP_ID_NAME = var.config.github.app_parameters.id.name
+ PARAMETER_GITHUB_APP_KEY_BASE64_NAME = var.config.github.app_parameters.key_base64.name
+ PARAMETER_GITHUB_APPS_MANIFEST_NAME = var.config.github.app_parameters.additional_apps_manifest != null ? var.config.github.app_parameters.additional_apps_manifest.name : ""
+ SSM_TOKEN_PATH = var.storage_provider.aws.ssm.token_path
+ SSM_CONFIG_PATH = var.storage_provider.aws.ssm.config_path
+ SSM_PARAMETER_STORE_TAGS = var.storage_provider.aws.ssm.parameter_store_tags
+ SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS = var.storage_provider.aws.ssm.parameter_store_max_concurrent_invocations
+ SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND = var.storage_provider.aws.ssm.parameter_store_max_writes_per_second
}, var.storage_provider.scale_up.environment_variables)
}
diff --git a/modules/orchestration-providers/webhook/scale-runners/tests/scale-runners.tftest.hcl b/modules/orchestration-providers/webhook/scale-runners/tests/scale-runners.tftest.hcl
index 22b695aac2..6d490d6b70 100644
--- a/modules/orchestration-providers/webhook/scale-runners/tests/scale-runners.tftest.hcl
+++ b/modules/orchestration-providers/webhook/scale-runners/tests/scale-runners.tftest.hcl
@@ -27,6 +27,9 @@ variables {
Value = "test"
}])
kms_key_id = "arn:aws-us-gov:kms:us-gov-west-1:123456789012:key/scale-runners-test"
+
+ parameter_store_max_concurrent_invocations = 3
+ parameter_store_max_writes_per_second = 120
}
}
}
@@ -250,6 +253,8 @@ run "assembles_provider_neutral_scaling_control_plane" {
&& aws_lambda_function.scale_down.environment[0].variables["MINIMUM_RUNNING_TIME_IN_MINUTES"] == "15"
&& aws_lambda_function.scale_up.environment[0].variables["NODE_TLS_REJECT_UNAUTHORIZED"] == "0"
&& jsondecode(aws_lambda_function.scale_up.environment[0].variables["SSM_PARAMETER_STORE_TAGS"])[0].Value == "test"
+ && aws_lambda_function.scale_up.environment[0].variables["SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS"] == "3"
+ && aws_lambda_function.scale_up.environment[0].variables["SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND"] == "120"
)
error_message = "Scale runners must assemble shared runner, logging, TLS, lifetime, and Parameter Store environment variables."
}
diff --git a/modules/orchestration-providers/webhook/scale-runners/variables.tf b/modules/orchestration-providers/webhook/scale-runners/variables.tf
index 982fb4e844..0e35c41160 100644
--- a/modules/orchestration-providers/webhook/scale-runners/variables.tf
+++ b/modules/orchestration-providers/webhook/scale-runners/variables.tf
@@ -47,6 +47,8 @@ variable "config" {
- `storage_provider.aws.ssm.config_path_arn`: ARN of the persistent runner configuration path.
- `storage_provider.aws.ssm.kms_key_id`: Optional KMS key ARN used to decrypt shared parameters. Its value may be unknown until apply.
- `storage_provider.aws.ssm.parameter_store_tags`: JSON-encoded tags applied to parameters created at runtime.
+ - `storage_provider.aws.ssm.parameter_store_max_concurrent_invocations`: Expected number of concurrent scale-up and pool Lambda invocations writing to Parameter Store, used to pace writes.
+ - `storage_provider.aws.ssm.parameter_store_max_writes_per_second`: Parameter Store write-rate limit, in writes per second, that runtime writes are paced against.
- `observability.logs`: Shared logging level, retention, encryption, and log-class configuration.
- `observability.tracing`: Lambda X-Ray and tracing-helper configuration.
- `observability.metrics`: Metrics enablement, namespace, and GitHub rate-limit metric configuration.
@@ -242,6 +244,9 @@ variable "storage_provider" {
config_path_arn = string
parameter_store_tags = string
kms_key_id = optional(string, null)
+
+ parameter_store_max_concurrent_invocations = optional(number, 1)
+ parameter_store_max_writes_per_second = optional(number, 40)
})
})
scale_up = optional(object({
diff --git a/modules/orchestration-providers/webhook/variables.tf b/modules/orchestration-providers/webhook/variables.tf
index e74602fd52..57b583730e 100644
--- a/modules/orchestration-providers/webhook/variables.tf
+++ b/modules/orchestration-providers/webhook/variables.tf
@@ -219,6 +219,8 @@ variable "storage_provider" {
- `storage_provider.aws.ssm.config_path_arn`: ARN of the runner-configuration Parameter Store path.
- `storage_provider.aws.ssm.kms_key_id`: Optional KMS key used to decrypt shared parameters.
- `storage_provider.aws.ssm.parameter_store_tags`: JSON-encoded tags applied to runtime parameters.
+ - `storage_provider.aws.ssm.parameter_store_max_concurrent_invocations`: Expected number of concurrent scale-up and pool Lambda invocations writing to Parameter Store, used to pace writes.
+ - `storage_provider.aws.ssm.parameter_store_max_writes_per_second`: Parameter Store write-rate limit, in writes per second, that runtime writes are paced against.
- `scale_up`, `scale_down`, `pool`, and `job_retry`: Provider-owned environment variables and IAM policy fragments.
EOT
type = object({
@@ -230,6 +232,9 @@ variable "storage_provider" {
config_path_arn = string
kms_key_id = optional(string, null)
parameter_store_tags = string
+
+ parameter_store_max_concurrent_invocations = optional(number, 1)
+ parameter_store_max_writes_per_second = optional(number, 40)
})
})
scale_up = optional(object({
diff --git a/modules/runner-config/README.md b/modules/runner-config/README.md
index 423a6c3c3f..428180afa8 100644
--- a/modules/runner-config/README.md
+++ b/modules/runner-config/README.md
@@ -117,7 +117,7 @@ yarn run dist
| [orchestration\_provider](#input\_orchestration\_provider) | Runner demand-orchestration provider configuration. Exactly one provider block must be non-null. Wrapper presence selects the provider and must therefore be known during planning; values inside the selected provider may remain unknown until apply.object({
webhook = optional(object({
runner = optional(object({
boot_time_in_minutes = optional(number, 5)
ephemeral = optional(bool, false)
jit_config_enabled = optional(bool, null)
maximum_count = optional(number, 3)
}), {})
github = object({
organization_runners = bool
})
queue = object({
build = object({
arn = string
url = string
})
kms_key_id = optional(string, null)
tags = optional(map(string), {})
})
lambda = optional(object({
artifact = optional(object({
zip = optional(string, null)
s3 = optional(object({
key = string
object_version = optional(string, null)
}), null)
}), {})
scale = optional(object({
up = optional(object({
memory_size = optional(number, 512)
timeout = optional(number, 60)
reserved_concurrent_executions = optional(number, 1)
job_queued_check_enabled = optional(bool, null)
event_source_mapping = optional(object({
batch_size = optional(number, 10)
maximum_batching_window_in_seconds = optional(number, 0)
}), {})
tags = optional(map(string), {})
}), {})
down = optional(object({
memory_size = optional(number, 512)
timeout = optional(number, 60)
schedule_expression = optional(string, "cron(*/5 * * * ? *)")
minimum_running_time_in_minutes = optional(number, null)
idle_confirmation_seconds = optional(number, 0)
idle_config = optional(list(object({
cron = string
timeZone = string
idleCount = number
evictionStrategy = optional(string, "oldest_first")
})), [])
tags = optional(map(string), {})
}), {})
}), {})
pool = optional(object({
memory_size = optional(number, 512)
timeout = optional(number, 60)
reserved_concurrent_executions = optional(number, 1)
config = optional(list(object({
schedule_expression = string
schedule_expression_timezone = optional(string)
size = number
})), [])
include_busy_runners = optional(bool, false)
runner_owner = optional(string, null)
tags = optional(map(string), {})
}), {})
}), {})
job_retry = optional(object({
enabled = optional(bool, false)
delay_in_seconds = optional(number, 300)
delay_backoff = optional(number, 2)
max_attempts = optional(number, 1)
tags = optional(map(string), {})
lambda = optional(object({
memory_size = optional(number, 256)
reserved_concurrent_executions = optional(number, 1)
timeout = optional(number, 30)
}), {})
}), {})
}), null)
scale_set = optional(object({
name = string
runner = optional(object({
min_runners = optional(number, 0)
max_runners = optional(number, 10)
boot_time_in_minutes = optional(number, 10)
}), {})
}), null)
}) | n/a | yes |
| [prefix](#input\_prefix) | The prefix used for naming resources. | `string` | `"github-actions"` | no |
| [runner](#input\_runner) | Provider-neutral GitHub runner configuration.object({
os = optional(string, "linux")
architecture = optional(string, "x64")
disable_default_labels = optional(bool, false)
labels = list(string)
group_name = optional(string, "Default")
name_prefix = optional(string, "")
run_as_root = optional(bool, false)
run_as = optional(string, "ec2-user")
auto_update_disabled = optional(bool, false)
tags = optional(map(string), {})
hooks = optional(object({
job_started = optional(string, "")
job_completed = optional(string, "")
}), {})
iam = optional(object({
role = optional(object({
arn = string
}), null)
managed_policy_arns = optional(map(string), {})
additional_trust_policy_json = optional(string, null)
path = optional(string, null)
permissions_boundary = optional(string, null)
}), {})
}) | n/a | yes |
-| [storage\_provider](#input\_storage\_provider) | Parameter Store paths, encryption, tag scopes, and housekeeper configuration.object({
aws = object({
ssm = object({
paths = object({
root = string
tokens = string
config = string
})
kms_key_id = optional(string, null)
tags = optional(map(string), {})
parameters = optional(object({
tags = optional(map(string), {})
}), {})
housekeeper = optional(object({
schedule_expression = optional(string, "rate(1 day)")
state = optional(string, "ENABLED")
tags = optional(map(string), {})
lambda = optional(object({
artifact = optional(object({
zip = optional(string, null)
s3 = optional(object({
key = string
object_version = optional(string, null)
}), null)
}), {})
memory_size = optional(number, 512)
timeout = optional(number, 60)
}), {})
config = optional(object({
tokenPath = optional(string)
minimumDaysOld = optional(number, 1)
dryRun = optional(bool, false)
}), {})
}), {})
})
})
}) | n/a | yes |
+| [storage\_provider](#input\_storage\_provider) | Parameter Store paths, encryption, tag scopes, and housekeeper configuration.object({
aws = object({
ssm = object({
paths = object({
root = string
tokens = string
config = string
})
kms_key_id = optional(string, null)
tags = optional(map(string), {})
parameters = optional(object({
tags = optional(map(string), {})
max_concurrent_invocations = optional(number, 1)
max_writes_per_second = optional(number, 40)
}), {})
housekeeper = optional(object({
schedule_expression = optional(string, "rate(1 day)")
state = optional(string, "ENABLED")
tags = optional(map(string), {})
lambda = optional(object({
artifact = optional(object({
zip = optional(string, null)
s3 = optional(object({
key = string
object_version = optional(string, null)
}), null)
}), {})
memory_size = optional(number, 512)
timeout = optional(number, 60)
}), {})
config = optional(object({
tokenPath = optional(string)
minimumDaysOld = optional(number, 1)
dryRun = optional(bool, false)
}), {})
}), {})
})
})
}) | n/a | yes |
| [tags](#input\_tags) | Base tags added to taggable resources created by this runner configuration. Shared, component, and compute-provider tag maps override matching keys within their documented resource scopes. | `map(string)` | `{}` | no |
## Outputs
diff --git a/modules/runner-config/orchestration-provider.tf b/modules/runner-config/orchestration-provider.tf
index 91f7b26e31..b049c70196 100644
--- a/modules/runner-config/orchestration-provider.tf
+++ b/modules/runner-config/orchestration-provider.tf
@@ -47,12 +47,14 @@ module "orchestration_webhook" {
storage_provider = {
aws = {
ssm = {
- token_path = local.token_path
- token_path_arn = local.arn_ssm_parameters_path_tokens
- config_path = "${var.storage_provider.aws.ssm.paths.root}/${var.storage_provider.aws.ssm.paths.config}"
- config_path_arn = local.arn_ssm_parameters_path_config
- kms_key_id = local.kms_key_id
- parameter_store_tags = local.parameter_store_tags
+ token_path = local.token_path
+ token_path_arn = local.arn_ssm_parameters_path_tokens
+ config_path = "${var.storage_provider.aws.ssm.paths.root}/${var.storage_provider.aws.ssm.paths.config}"
+ config_path_arn = local.arn_ssm_parameters_path_config
+ kms_key_id = local.kms_key_id
+ parameter_store_tags = local.parameter_store_tags
+ parameter_store_max_concurrent_invocations = var.storage_provider.aws.ssm.parameters.max_concurrent_invocations
+ parameter_store_max_writes_per_second = var.storage_provider.aws.ssm.parameters.max_writes_per_second
}
}
}
diff --git a/modules/runner-config/tests/tags.tftest.hcl b/modules/runner-config/tests/tags.tftest.hcl
index 8f9cbb2d0d..c60c238cdf 100644
--- a/modules/runner-config/tests/tags.tftest.hcl
+++ b/modules/runner-config/tests/tags.tftest.hcl
@@ -163,6 +163,8 @@ variables {
precedence = "ssm-parameter"
parameter = "yes"
}
+ max_concurrent_invocations = 5
+ max_writes_per_second = 500
}
housekeeper = {
tags = {
@@ -188,6 +190,14 @@ variables {
run "layered_component_tags" {
command = plan
+ assert {
+ condition = (
+ module.orchestration_webhook[0].scale_up.lambda.environment[0].variables["SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS"] == "5"
+ && module.orchestration_webhook[0].scale_up.lambda.environment[0].variables["SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND"] == "500"
+ )
+ error_message = "The Parameter Store pacing settings must reach the scale-up Lambda environment."
+ }
+
assert {
condition = module.orchestration_webhook[0].scale_up.lambda.environment[0].variables["LOG_LEVEL"] == "DEBUG"
error_message = "The nested observability.logs.level value must configure the control-plane functions."
diff --git a/modules/runner-config/variables.tf b/modules/runner-config/variables.tf
index 5b3cdb0102..d16feedefc 100644
--- a/modules/runner-config/variables.tf
+++ b/modules/runner-config/variables.tf
@@ -148,6 +148,8 @@ variable "storage_provider" {
- `storage_provider.aws.ssm.paths.config`: Path segment under `paths.root` used for persistent runner configuration.
- `storage_provider.aws.ssm.kms_key_id`: Optional customer-managed KMS key ARN used by control-plane IAM policies to decrypt shared GitHub App parameters. The ARN may be unknown until apply; null omits the provider-owned KMS statements. It does not select encryption for runtime-created runner parameters.
- `storage_provider.aws.ssm.tags`: Shared tags for SSM-related resources. These override module-level `tags` and are inherited by parameter and housekeeper resources.
+ - `storage_provider.aws.ssm.parameters.max_concurrent_invocations`: Expected number of concurrent scale-up and pool Lambda invocations writing to Parameter Store. Each invocation paces its writes to a share of the account-wide write-rate limit. The default is `1`.
+ - `storage_provider.aws.ssm.parameters.max_writes_per_second`: Parameter Store write-rate limit to pace against, in writes per second. The default is `40`, the standard-tier limit. Raise it only when the higher-throughput tier is enabled.
- `storage_provider.aws.ssm.parameters.tags`: Tags for Terraform-managed runner configuration parameters and temporary parameters created by the scale-up and pool Lambdas. These override module-level and `storage_provider.aws.ssm.tags` values with the same key.
- `storage_provider.aws.ssm.housekeeper.schedule_expression`: EventBridge schedule expression that invokes the SSM housekeeper.
- `storage_provider.aws.ssm.housekeeper.state`: EventBridge rule state, such as `ENABLED` or `DISABLED`.
@@ -174,7 +176,9 @@ variable "storage_provider" {
kms_key_id = optional(string, null)
tags = optional(map(string), {})
parameters = optional(object({
- tags = optional(map(string), {})
+ tags = optional(map(string), {})
+ max_concurrent_invocations = optional(number, 1)
+ max_writes_per_second = optional(number, 40)
}), {})
housekeeper = optional(object({
schedule_expression = optional(string, "rate(1 day)")
diff --git a/modules/runners/README.md b/modules/runners/README.md
index 78d1932e09..bf7c65ec5c 100644
--- a/modules/runners/README.md
+++ b/modules/runners/README.md
@@ -233,6 +233,8 @@ yarn run dist
| [scale\_up\_reserved\_concurrent\_executions](#input\_scale\_up\_reserved\_concurrent\_executions) | Amount of reserved concurrent executions for the scale-up lambda function. A value of 0 disables lambda from being triggered and -1 removes any concurrency limitations. | `number` | `1` | no |
| [sqs\_build\_queue](#input\_sqs\_build\_queue) | SQS queue to consume accepted build events. | object({
arn = string
url = string
}) | n/a | yes |
| [ssm\_housekeeper](#input\_ssm\_housekeeper) | Configuration for the SSM housekeeper lambda. This lambda deletes token / JIT config from SSM.object({
schedule_expression = optional(string, "rate(1 day)")
state = optional(string, "ENABLED")
lambda_memory_size = optional(number, 512)
lambda_timeout = optional(number, 60)
config = object({
tokenPath = optional(string)
minimumDaysOld = optional(number, 1)
dryRun = optional(bool, false)
})
}) | {
"config": {}
} | no |
+| [ssm\_parameter\_store\_max\_concurrent\_invocations](#input\_ssm\_parameter\_store\_max\_concurrent\_invocations) | Expected number of concurrent scale-up/pool lambda invocations writing to Parameter Store, used to pace each invocation's writes to a share of the account-wide write-rate limit. | `number` | `1` | no |
+| [ssm\_parameter\_store\_max\_writes\_per\_second](#input\_ssm\_parameter\_store\_max\_writes\_per\_second) | Parameter Store write-rate limit to pace against, in writes/second. Defaults to the standard-tier limit; raise this if SSM's higher-throughput tier is enabled (up to several thousand writes/second). | `number` | `40` | no |
| [ssm\_paths](#input\_ssm\_paths) | The root path used in SSM to store configuration and secrets. | object({
root = string
tokens = string
config = string
}) | n/a | yes |
| [ssm\_ttl\_seconds](#input\_ssm\_ttl\_seconds) | Set tokens to the optional TTL in seconds for the SSM parameters holding the runner registration token / JIT config. When set, the parameters are created with an SSM expiration policy so SSM deletes them itself after the TTL passes. Requires the Advanced parameter tier for every token parameter, which incurs additional costs. Expiration is enforced asynchronously by SSM; the SSM housekeeper lambda remains as a backstop. Must be a positive number, and should comfortably exceed the runner boot time so the config does not expire before the instance reads it. | object({
tokens = optional(number, null)
}) | `{}` | no |
| [subnet\_ids](#input\_subnet\_ids) | List of subnets in which the action runners will be launched, the subnets needs to be subnets in the `vpc_id`. | `list(string)` | n/a | yes |
diff --git a/modules/runners/pool.tf b/modules/runners/pool.tf
index 1f4b99dfd8..f7054ee87c 100644
--- a/modules/runners/pool.tf
+++ b/modules/runners/pool.tf
@@ -57,15 +57,17 @@ module "pool" {
role = { arn = var.iam_overrides["override_runner_role"] ? var.iam_overrides["runner_role_arn"] : aws_iam_role.runner[0].arn }
use_dedicated_host = var.use_dedicated_host
}
- subnet_ids = var.subnet_ids
- ssm_token_path = "${var.ssm_paths.root}/${var.ssm_paths.tokens}"
- ssm_ttl_seconds = var.ssm_ttl_seconds
- ssm_config_path = "${var.ssm_paths.root}/${var.ssm_paths.config}"
- ami_id_ssm_parameter_name = local.ami_id_ssm_parameter_name
- ami_id_ssm_parameter_read_policy_arn = local.ami_id_ssm_parameter_name != null ? aws_iam_policy.ami_id_ssm_parameter_read[0].arn : null
- tags = local.tags
- lambda_tags = var.lambda_tags
- arn_ssm_parameters_path_config = local.arn_ssm_parameters_path_config
+ subnet_ids = var.subnet_ids
+ ssm_token_path = "${var.ssm_paths.root}/${var.ssm_paths.tokens}"
+ ssm_ttl_seconds = var.ssm_ttl_seconds
+ ssm_config_path = "${var.ssm_paths.root}/${var.ssm_paths.config}"
+ ssm_parameter_store_max_concurrent_invocations = var.ssm_parameter_store_max_concurrent_invocations
+ ssm_parameter_store_max_writes_per_second = var.ssm_parameter_store_max_writes_per_second
+ ami_id_ssm_parameter_name = local.ami_id_ssm_parameter_name
+ ami_id_ssm_parameter_read_policy_arn = local.ami_id_ssm_parameter_name != null ? aws_iam_policy.ami_id_ssm_parameter_read[0].arn : null
+ tags = local.tags
+ lambda_tags = var.lambda_tags
+ arn_ssm_parameters_path_config = local.arn_ssm_parameters_path_config
}
aws_partition = var.aws_partition
diff --git a/modules/runners/pool/README.md b/modules/runners/pool/README.md
index 1b00e0d052..630cc9808e 100644
--- a/modules/runners/pool/README.md
+++ b/modules/runners/pool/README.md
@@ -49,7 +49,7 @@ No modules.
| Name | Description | Type | Default | Required |
|------|-------------|------|---------|:--------:|
| [aws\_partition](#input\_aws\_partition) | (optional) partition for the arn if not 'aws' | `string` | `"aws"` | no |
-| [config](#input\_config) | Lookup details in parent module. | object({
lambda = object({
log_level = string
logging_retention_in_days = number
logging_kms_key_id = string
log_class = string
reserved_concurrent_executions = number
s3_bucket = string
s3_key = string
s3_object_version = string
security_group_ids = list(string)
runtime = string
architecture = string
memory_size = number
timeout = number
zip = string
subnet_ids = list(string)
parameter_store_tags = string
})
tags = map(string)
ghes = object({
url = string
ssl_verify = string
})
github_app_parameters = object({
key_base64 = map(string)
id = map(string)
additional_apps_manifest = optional(object({
name = string
arn = string
}), null)
additional_app_parameter_arns = optional(list(string), [])
})
subnet_ids = list(string)
runner = object({
disable_runner_autoupdate = bool
ephemeral = bool
enable_jit_config = bool
enable_on_demand_failover_for_errors = list(string)
scale_errors = list(string)
boot_time_in_minutes = number
labels = list(string)
launch_template = object({
name = string
})
group_name = string
name_prefix = string
pool_owner = string
role = object({
arn = string
})
use_dedicated_host = bool
})
runners_maximum_count = number
instance_types = list(string)
instance_type_priorities = optional(map(number))
instance_target_capacity_type = string
instance_allocation_strategy = string
instance_max_spot_price = string
prefix = string
pool = list(object({
schedule_expression = string
schedule_expression_timezone = string
size = number
}))
include_busy_runners = bool
role_permissions_boundary = string
kms_key_arn = string
ami_kms_key_arn = string
ami_id_ssm_parameter_arn = string
role_path = string
ssm_token_path = string
ssm_ttl_seconds = optional(object({
tokens = optional(number, null)
}), {})
ssm_config_path = string
ami_id_ssm_parameter_name = string
ami_id_ssm_parameter_read_policy_arn = string
arn_ssm_parameters_path_config = string
lambda_tags = map(string)
user_agent = string
}) | n/a | yes |
+| [config](#input\_config) | Lookup details in parent module. | object({
lambda = object({
log_level = string
logging_retention_in_days = number
logging_kms_key_id = string
log_class = string
reserved_concurrent_executions = number
s3_bucket = string
s3_key = string
s3_object_version = string
security_group_ids = list(string)
runtime = string
architecture = string
memory_size = number
timeout = number
zip = string
subnet_ids = list(string)
parameter_store_tags = string
})
tags = map(string)
ghes = object({
url = string
ssl_verify = string
})
github_app_parameters = object({
key_base64 = map(string)
id = map(string)
additional_apps_manifest = optional(object({
name = string
arn = string
}), null)
additional_app_parameter_arns = optional(list(string), [])
})
subnet_ids = list(string)
runner = object({
disable_runner_autoupdate = bool
ephemeral = bool
enable_jit_config = bool
enable_on_demand_failover_for_errors = list(string)
scale_errors = list(string)
boot_time_in_minutes = number
labels = list(string)
launch_template = object({
name = string
})
group_name = string
name_prefix = string
pool_owner = string
role = object({
arn = string
})
use_dedicated_host = bool
})
runners_maximum_count = number
instance_types = list(string)
instance_type_priorities = optional(map(number))
instance_target_capacity_type = string
instance_allocation_strategy = string
instance_max_spot_price = string
prefix = string
pool = list(object({
schedule_expression = string
schedule_expression_timezone = string
size = number
}))
include_busy_runners = bool
role_permissions_boundary = string
kms_key_arn = string
ami_kms_key_arn = string
ami_id_ssm_parameter_arn = string
role_path = string
ssm_token_path = string
ssm_ttl_seconds = optional(object({
tokens = optional(number, null)
}), {})
ssm_config_path = string
ssm_parameter_store_max_concurrent_invocations = optional(number, 1)
ssm_parameter_store_max_writes_per_second = optional(number, 40)
ami_id_ssm_parameter_name = string
ami_id_ssm_parameter_read_policy_arn = string
arn_ssm_parameters_path_config = string
lambda_tags = map(string)
user_agent = string
}) | n/a | yes |
| [tracing\_config](#input\_tracing\_config) | Configuration for lambda tracing. | object({
mode = optional(string, null)
capture_http_requests = optional(bool, false)
capture_error = optional(bool, false)
}) | `{}` | no |
## Outputs
diff --git a/modules/runners/pool/main.tf b/modules/runners/pool/main.tf
index 166836b08e..93743e37b1 100644
--- a/modules/runners/pool/main.tf
+++ b/modules/runners/pool/main.tf
@@ -27,43 +27,45 @@ resource "aws_lambda_function" "pool" {
environment {
variables = merge({
- AMI_ID_SSM_PARAMETER_NAME = var.config.ami_id_ssm_parameter_name
- DISABLE_RUNNER_AUTOUPDATE = var.config.runner.disable_runner_autoupdate
- ENABLE_EPHEMERAL_RUNNERS = var.config.runner.ephemeral
- ENABLE_JIT_CONFIG = var.config.runner.enable_jit_config
- ENVIRONMENT = var.config.prefix
- GHES_URL = var.config.ghes.url
- USER_AGENT = var.config.user_agent
- INSTANCE_ALLOCATION_STRATEGY = var.config.instance_allocation_strategy
- INSTANCE_MAX_SPOT_PRICE = var.config.instance_max_spot_price
- INSTANCE_TARGET_CAPACITY_TYPE = var.config.instance_target_capacity_type
- INSTANCE_TYPE_PRIORITIES = var.config.instance_type_priorities != null ? jsonencode(var.config.instance_type_priorities) : ""
- INSTANCE_TYPES = join(",", var.config.instance_types)
- LAUNCH_TEMPLATE_NAME = var.config.runner.launch_template.name
- LOG_LEVEL = upper(var.config.lambda.log_level)
- NODE_TLS_REJECT_UNAUTHORIZED = var.config.ghes.url != null && !var.config.ghes.ssl_verify ? 0 : 1
- PARAMETER_GITHUB_APP_ID_NAME = var.config.github_app_parameters.id.name
- PARAMETER_GITHUB_APP_KEY_BASE64_NAME = var.config.github_app_parameters.key_base64.name
- PARAMETER_GITHUB_APPS_MANIFEST_NAME = var.config.github_app_parameters.additional_apps_manifest != null ? var.config.github_app_parameters.additional_apps_manifest.name : ""
- POWERTOOLS_LOGGER_LOG_EVENT = var.config.lambda.log_level == "debug" ? "true" : "false"
- RUNNER_BOOT_TIME_IN_MINUTES = var.config.runner.boot_time_in_minutes
- RUNNER_LABELS = lower(join(",", var.config.runner.labels))
- RUNNER_GROUP_NAME = var.config.runner.group_name
- RUNNER_NAME_PREFIX = var.config.runner.name_prefix
- RUNNER_OWNER = var.config.runner.pool_owner
- RUNNERS_MAXIMUM_COUNT = var.config.runners_maximum_count
- SSM_TOKEN_PATH = var.config.ssm_token_path
- SSM_CONFIG_PATH = var.config.ssm_config_path
- SUBNET_IDS = join(",", var.config.subnet_ids)
- POWERTOOLS_SERVICE_NAME = "${var.config.prefix}-pool"
- POWERTOOLS_TRACE_ENABLED = var.tracing_config.mode != null ? true : false
- POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS = var.tracing_config.capture_http_requests
- POWERTOOLS_TRACER_CAPTURE_ERROR = var.tracing_config.capture_error
- ENABLE_ON_DEMAND_FAILOVER_FOR_ERRORS = jsonencode(var.config.runner.enable_on_demand_failover_for_errors)
- SSM_PARAMETER_STORE_TAGS = var.config.lambda.parameter_store_tags
- SCALE_ERRORS = jsonencode(var.config.runner.scale_errors)
- USE_DEDICATED_HOST = var.config.runner.use_dedicated_host
- INCLUDE_BUSY_RUNNERS = var.config.include_busy_runners
+ AMI_ID_SSM_PARAMETER_NAME = var.config.ami_id_ssm_parameter_name
+ DISABLE_RUNNER_AUTOUPDATE = var.config.runner.disable_runner_autoupdate
+ ENABLE_EPHEMERAL_RUNNERS = var.config.runner.ephemeral
+ ENABLE_JIT_CONFIG = var.config.runner.enable_jit_config
+ ENVIRONMENT = var.config.prefix
+ GHES_URL = var.config.ghes.url
+ USER_AGENT = var.config.user_agent
+ INSTANCE_ALLOCATION_STRATEGY = var.config.instance_allocation_strategy
+ INSTANCE_MAX_SPOT_PRICE = var.config.instance_max_spot_price
+ INSTANCE_TARGET_CAPACITY_TYPE = var.config.instance_target_capacity_type
+ INSTANCE_TYPE_PRIORITIES = var.config.instance_type_priorities != null ? jsonencode(var.config.instance_type_priorities) : ""
+ INSTANCE_TYPES = join(",", var.config.instance_types)
+ LAUNCH_TEMPLATE_NAME = var.config.runner.launch_template.name
+ LOG_LEVEL = upper(var.config.lambda.log_level)
+ NODE_TLS_REJECT_UNAUTHORIZED = var.config.ghes.url != null && !var.config.ghes.ssl_verify ? 0 : 1
+ PARAMETER_GITHUB_APP_ID_NAME = var.config.github_app_parameters.id.name
+ PARAMETER_GITHUB_APP_KEY_BASE64_NAME = var.config.github_app_parameters.key_base64.name
+ PARAMETER_GITHUB_APPS_MANIFEST_NAME = var.config.github_app_parameters.additional_apps_manifest != null ? var.config.github_app_parameters.additional_apps_manifest.name : ""
+ POWERTOOLS_LOGGER_LOG_EVENT = var.config.lambda.log_level == "debug" ? "true" : "false"
+ RUNNER_BOOT_TIME_IN_MINUTES = var.config.runner.boot_time_in_minutes
+ RUNNER_LABELS = lower(join(",", var.config.runner.labels))
+ RUNNER_GROUP_NAME = var.config.runner.group_name
+ RUNNER_NAME_PREFIX = var.config.runner.name_prefix
+ RUNNER_OWNER = var.config.runner.pool_owner
+ RUNNERS_MAXIMUM_COUNT = var.config.runners_maximum_count
+ SSM_TOKEN_PATH = var.config.ssm_token_path
+ SSM_CONFIG_PATH = var.config.ssm_config_path
+ SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS = var.config.ssm_parameter_store_max_concurrent_invocations
+ SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND = var.config.ssm_parameter_store_max_writes_per_second
+ SUBNET_IDS = join(",", var.config.subnet_ids)
+ POWERTOOLS_SERVICE_NAME = "${var.config.prefix}-pool"
+ POWERTOOLS_TRACE_ENABLED = var.tracing_config.mode != null ? true : false
+ POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS = var.tracing_config.capture_http_requests
+ POWERTOOLS_TRACER_CAPTURE_ERROR = var.tracing_config.capture_error
+ ENABLE_ON_DEMAND_FAILOVER_FOR_ERRORS = jsonencode(var.config.runner.enable_on_demand_failover_for_errors)
+ SSM_PARAMETER_STORE_TAGS = var.config.lambda.parameter_store_tags
+ SCALE_ERRORS = jsonencode(var.config.runner.scale_errors)
+ USE_DEDICATED_HOST = var.config.runner.use_dedicated_host
+ INCLUDE_BUSY_RUNNERS = var.config.include_busy_runners
}, var.config.ssm_ttl_seconds.tokens != null ? {
SSM_TOKEN_TTL_SECONDS = tostring(var.config.ssm_ttl_seconds.tokens)
} : {})
diff --git a/modules/runners/pool/variables.tf b/modules/runners/pool/variables.tf
index 20e5f80c60..8915599e86 100644
--- a/modules/runners/pool/variables.tf
+++ b/modules/runners/pool/variables.tf
@@ -75,12 +75,14 @@ variable "config" {
ssm_ttl_seconds = optional(object({
tokens = optional(number, null)
}), {})
- ssm_config_path = string
- ami_id_ssm_parameter_name = string
- ami_id_ssm_parameter_read_policy_arn = string
- arn_ssm_parameters_path_config = string
- lambda_tags = map(string)
- user_agent = string
+ ssm_config_path = string
+ ssm_parameter_store_max_concurrent_invocations = optional(number, 1)
+ ssm_parameter_store_max_writes_per_second = optional(number, 40)
+ ami_id_ssm_parameter_name = string
+ ami_id_ssm_parameter_read_policy_arn = string
+ arn_ssm_parameters_path_config = string
+ lambda_tags = map(string)
+ user_agent = string
})
}
diff --git a/modules/runners/scale-up.tf b/modules/runners/scale-up.tf
index dfe8c0a561..071c3240f6 100644
--- a/modules/runners/scale-up.tf
+++ b/modules/runners/scale-up.tf
@@ -30,47 +30,49 @@ resource "aws_lambda_function" "scale_up" {
depends_on = [aws_cloudwatch_log_group.scale_up]
environment {
variables = merge({
- AMI_ID_SSM_PARAMETER_NAME = local.ami_id_ssm_parameter_name
- DISABLE_RUNNER_AUTOUPDATE = var.disable_runner_autoupdate
- ENABLE_EPHEMERAL_RUNNERS = var.enable_ephemeral_runners
- ENABLE_JIT_CONFIG = var.enable_jit_config
- ENABLE_JOB_QUEUED_CHECK = local.enable_job_queued_check
- ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = var.metrics.enable && var.metrics.metric.enable_github_app_rate_limit
- ENABLE_ORGANIZATION_RUNNERS = var.enable_organization_runners
- ENVIRONMENT = var.prefix
- GHES_URL = var.ghes_url
- USER_AGENT = var.user_agent
- INSTANCE_ALLOCATION_STRATEGY = var.instance_allocation_strategy
- INSTANCE_MAX_SPOT_PRICE = var.instance_max_spot_price
- INSTANCE_TARGET_CAPACITY_TYPE = var.instance_target_capacity_type
- INSTANCE_TYPE_PRIORITIES = var.instance_type_priorities != null ? jsonencode(var.instance_type_priorities) : ""
- INSTANCE_TYPES = join(",", var.instance_types)
- LAUNCH_TEMPLATE_NAME = aws_launch_template.runner.name
- LOG_LEVEL = upper(var.log_level)
- MINIMUM_RUNNING_TIME_IN_MINUTES = coalesce(var.minimum_running_time_in_minutes, local.min_runtime_defaults[var.runner_os])
- NODE_TLS_REJECT_UNAUTHORIZED = var.ghes_url != null && !var.ghes_ssl_verify ? 0 : 1
- PARAMETER_GITHUB_APP_ID_NAME = var.github_app_parameters.id.name
- PARAMETER_GITHUB_APP_KEY_BASE64_NAME = var.github_app_parameters.key_base64.name
- PARAMETER_GITHUB_APPS_MANIFEST_NAME = var.github_app_parameters.additional_apps_manifest != null ? var.github_app_parameters.additional_apps_manifest.name : ""
- POWERTOOLS_LOGGER_LOG_EVENT = var.log_level == "debug" ? "true" : "false"
- POWERTOOLS_METRICS_NAMESPACE = var.metrics.namespace
- POWERTOOLS_TRACE_ENABLED = var.tracing_config.mode != null ? true : false
- POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS = var.tracing_config.capture_http_requests
- POWERTOOLS_TRACER_CAPTURE_ERROR = var.tracing_config.capture_error
- RUNNER_LABELS = lower(join(",", var.runner_labels))
- RUNNER_GROUP_NAME = var.runner_group_name
- RUNNER_NAME_PREFIX = var.runner_name_prefix
- COMPUTE_PROVIDER_TYPE = "ec2"
- RUNNERS_MAXIMUM_COUNT = var.runners_maximum_count
- POWERTOOLS_SERVICE_NAME = "${var.prefix}-scale-up"
- SSM_TOKEN_PATH = local.token_path
- SSM_CONFIG_PATH = "${var.ssm_paths.root}/${var.ssm_paths.config}"
- SSM_PARAMETER_STORE_TAGS = local.parameter_store_tags
- SUBNET_IDS = join(",", var.subnet_ids)
- ENABLE_ON_DEMAND_FAILOVER_FOR_ERRORS = jsonencode(var.enable_on_demand_failover_for_errors)
- SCALE_ERRORS = jsonencode(var.scale_errors)
- JOB_RETRY_CONFIG = jsonencode(local.job_retry_config)
- USE_DEDICATED_HOST = var.use_dedicated_host
+ AMI_ID_SSM_PARAMETER_NAME = local.ami_id_ssm_parameter_name
+ DISABLE_RUNNER_AUTOUPDATE = var.disable_runner_autoupdate
+ ENABLE_EPHEMERAL_RUNNERS = var.enable_ephemeral_runners
+ ENABLE_JIT_CONFIG = var.enable_jit_config
+ ENABLE_JOB_QUEUED_CHECK = local.enable_job_queued_check
+ ENABLE_METRIC_GITHUB_APP_RATE_LIMIT = var.metrics.enable && var.metrics.metric.enable_github_app_rate_limit
+ ENABLE_ORGANIZATION_RUNNERS = var.enable_organization_runners
+ ENVIRONMENT = var.prefix
+ GHES_URL = var.ghes_url
+ USER_AGENT = var.user_agent
+ INSTANCE_ALLOCATION_STRATEGY = var.instance_allocation_strategy
+ INSTANCE_MAX_SPOT_PRICE = var.instance_max_spot_price
+ INSTANCE_TARGET_CAPACITY_TYPE = var.instance_target_capacity_type
+ INSTANCE_TYPE_PRIORITIES = var.instance_type_priorities != null ? jsonencode(var.instance_type_priorities) : ""
+ INSTANCE_TYPES = join(",", var.instance_types)
+ LAUNCH_TEMPLATE_NAME = aws_launch_template.runner.name
+ LOG_LEVEL = upper(var.log_level)
+ MINIMUM_RUNNING_TIME_IN_MINUTES = coalesce(var.minimum_running_time_in_minutes, local.min_runtime_defaults[var.runner_os])
+ NODE_TLS_REJECT_UNAUTHORIZED = var.ghes_url != null && !var.ghes_ssl_verify ? 0 : 1
+ PARAMETER_GITHUB_APP_ID_NAME = var.github_app_parameters.id.name
+ PARAMETER_GITHUB_APP_KEY_BASE64_NAME = var.github_app_parameters.key_base64.name
+ PARAMETER_GITHUB_APPS_MANIFEST_NAME = var.github_app_parameters.additional_apps_manifest != null ? var.github_app_parameters.additional_apps_manifest.name : ""
+ POWERTOOLS_LOGGER_LOG_EVENT = var.log_level == "debug" ? "true" : "false"
+ POWERTOOLS_METRICS_NAMESPACE = var.metrics.namespace
+ POWERTOOLS_TRACE_ENABLED = var.tracing_config.mode != null ? true : false
+ POWERTOOLS_TRACER_CAPTURE_HTTPS_REQUESTS = var.tracing_config.capture_http_requests
+ POWERTOOLS_TRACER_CAPTURE_ERROR = var.tracing_config.capture_error
+ RUNNER_LABELS = lower(join(",", var.runner_labels))
+ RUNNER_GROUP_NAME = var.runner_group_name
+ RUNNER_NAME_PREFIX = var.runner_name_prefix
+ COMPUTE_PROVIDER_TYPE = "ec2"
+ RUNNERS_MAXIMUM_COUNT = var.runners_maximum_count
+ POWERTOOLS_SERVICE_NAME = "${var.prefix}-scale-up"
+ SSM_TOKEN_PATH = local.token_path
+ SSM_CONFIG_PATH = "${var.ssm_paths.root}/${var.ssm_paths.config}"
+ SSM_PARAMETER_STORE_TAGS = local.parameter_store_tags
+ SSM_PARAMETER_STORE_MAX_CONCURRENT_INVOCATIONS = var.ssm_parameter_store_max_concurrent_invocations
+ SSM_PARAMETER_STORE_MAX_WRITES_PER_SECOND = var.ssm_parameter_store_max_writes_per_second
+ SUBNET_IDS = join(",", var.subnet_ids)
+ ENABLE_ON_DEMAND_FAILOVER_FOR_ERRORS = jsonencode(var.enable_on_demand_failover_for_errors)
+ SCALE_ERRORS = jsonencode(var.scale_errors)
+ JOB_RETRY_CONFIG = jsonencode(local.job_retry_config)
+ USE_DEDICATED_HOST = var.use_dedicated_host
}, var.ssm_ttl_seconds.tokens != null ? {
SSM_TOKEN_TTL_SECONDS = tostring(var.ssm_ttl_seconds.tokens)
} : {})
diff --git a/modules/runners/variables.tf b/modules/runners/variables.tf
index b99e5c991e..4037a31ab4 100644
--- a/modules/runners/variables.tf
+++ b/modules/runners/variables.tf
@@ -320,6 +320,18 @@ variable "scale_up_reserved_concurrent_executions" {
default = 1
}
+variable "ssm_parameter_store_max_concurrent_invocations" {
+ description = "Expected number of concurrent scale-up/pool lambda invocations writing to Parameter Store, used to pace each invocation's writes to a share of the account-wide write-rate limit."
+ type = number
+ default = 1
+}
+
+variable "ssm_parameter_store_max_writes_per_second" {
+ description = "Parameter Store write-rate limit to pace against, in writes/second. Defaults to the standard-tier limit; raise this if SSM's higher-throughput tier is enabled (up to several thousand writes/second)."
+ type = number
+ default = 40
+}
+
variable "lambda_scale_up_memory_size" {
description = "Memory size limit in MB for scale-up lambda."
type = number
diff --git a/variables.tf b/variables.tf
index 6bad7a21b0..2a117b4d2d 100644
--- a/variables.tf
+++ b/variables.tf
@@ -168,6 +168,18 @@ variable "scale_up_reserved_concurrent_executions" {
default = 1
}
+variable "ssm_parameter_store_max_concurrent_invocations" {
+ description = "Expected number of concurrent scale-up/pool lambda invocations writing to Parameter Store, used to pace each invocation's writes to a share of the account-wide write-rate limit."
+ type = number
+ default = 1
+}
+
+variable "ssm_parameter_store_max_writes_per_second" {
+ description = "Parameter Store write-rate limit to pace against, in writes/second. Defaults to the standard-tier limit; raise this if SSM's higher-throughput tier is enabled (up to several thousand writes/second)."
+ type = number
+ default = 40
+}
+
variable "webhook_lambda_zip" {
description = "File location of the webhook lambda zip file."
type = string