Repository navigation
feat(aws-credentials-imds): retry IMDS credential requests - #150
rohangavankar wants to merge 2 commits into
Conversation
6881b9a to
38f64cd
Compare
IMDS requests are now retried with capped exponential backoff. The token fetch and the metadata read are retried together, so a retry gets a new token when it needs one. Every failure is retried except a token request rejected with 400 or 403 and a metadata request answered with 403 or 404. A 401 drops the rejected token before the retry, so the next attempt fetches a new one. Any error from sending a request or reading its body counts as a failed request, since urllib3 and aiohttp raise different types for the same dropped or reset connection. IMDSConfig.retry_strategy is replaced by max_attempts. It defaults to 4, or 3 retries, and can be set with AWS_METADATA_SERVICE_NUM_ATTEMPTS or metadata_service_num_attempts, the same settings botocore reads. Backoff waits 0.25 s, 0.5 s and then 1 s. IMDS requests now time out after 1 second. The async client enforces this itself, because aiohttp ignores the per-request read timeout. A caller's sync client keeps its own connect timeout, because a request can't override it. Session tokens now expire after the TTL IMDS returns. Both token caches read the cached token once, so a token invalidated by another caller can't fail a request that already checked it. Response bodies are closed after every read, so a failed attempt doesn't hold a connection from the caller's pool.
38f64cd to
995a10b
Compare
…ing it IMDSConfig.retry_strategy stays as a deprecated argument and attribute, so existing callers keep working. Passing it warns, and only its max_attempts is used: IMDS is link-local, so a service client's retry strategy, with its retry quota and longer throttling backoff, doesn't fit it. An explicit max_attempts takes precedence. The fixed backoff now comes from smithy_core's ExponentialRetryBackoffStrategy rather than a second copy of the calculation.
jonathan343
left a comment
There was a problem hiding this comment.
Thanks @rohangavankar! Let a few pieces of feedback
| attempts = 0 | ||
| last_exc: Exception | None = None | ||
| while attempts < config.max_attempts: |
There was a problem hiding this comment.
This loop repeats the logic smithy_core.retries.SimpleRetryStrategy already implements. That class takes a backoff_strategy and max_attempts, and retries errors where is_retry_safe is true.
Could we use it here, and keep both config options instead of deprecating retry_strategy?
max_attemptsstays the simple option. We buildSimpleRetryStrategy(backoff_strategy=_BACKOFF, max_attempts=max_attempts)from it, so callers don't need to import anything from smithy.retry_strategystays as an optional override and wins if both are set. That's the same precedenceRetryStrategyResolveruses, and it lets callers swap the backoff.- Set
is_retry_safeon_IMDSRequestErrorfrom yourretryablelogic.
| DEFAULT_MAX_ATTEMPTS = 4 | ||
| # IMDS is a link-local endpoint, so a healthy instance answers well within a second. | ||
| TIMEOUT_SECONDS = 1.0 | ||
| _REQUEST_CONFIG = HTTPRequestConfiguration(read_timeout=TIMEOUT_SECONDS) |
There was a problem hiding this comment.
In the default chain, providers.py passes setup.http_client to the resolver. That's the service client's transport, which has a 60 s connect timeout. This config only overrides the read timeout, so if the network silently drops IMDS traffic, each attempt waits 60 s. I reproduced it against an address that drops connections:
develop: 1 attempt, fails in 60 s- this PR: 4 attempts, fails in 242 s
Could providers.py stop passing setup.http_client, so the resolver uses its own 1 s client? That brings the same repro down to 6 s, and it's similar to what botocore does. A per-request connect_timeout in smithy-http is the better long-term fix, but that can be a follow-up.
If we do this, the resolver should create its client once and keep it. When no client is passed in, it currently builds a new one on every get_identity, so the token is never reused. With 10 concurrent callers I saw 10 token PUTs instead of 1.
| _ENDPOINT_MODE_PROFILE = "ec2_metadata_service_endpoint_mode" | ||
| _PROFILE_NAME_ENV = "AWS_EC2_INSTANCE_PROFILE_NAME" | ||
| _PROFILE_NAME_PROFILE = "ec2_instance_profile_name" | ||
| _MAX_ATTEMPTS_ENV = "AWS_METADATA_SERVICE_NUM_ATTEMPTS" |
There was a problem hiding this comment.
Just noticing that we don't read the AWS_METADATA_SERVICE_TIMEOUT option
Summary
IMDS credential requests are now retried with capped exponential backoff. The token fetch and the metadata read are retried as one unit, so a retry gets a new token when it needs one. Every failure is retried except a token request rejected with 400 or 403 and a metadata request answered with 403 or 404. A 401 drops the rejected token first, so the next attempt fetches a new one. Any error from sending a request or reading its body counts as a failed request, because urllib3 and aiohttp raise different exception types for the same dropped or reset connection.
The new
IMDSConfig.max_attemptsdefaults to 4, or 3 retries, and can be set withAWS_METADATA_SERVICE_NUM_ATTEMPTSormetadata_service_num_attempts, the same settings botocore reads. Backoff waits 0.25 s, 0.5 s and then 1 s, computed by smithy_core'sExponentialRetryBackoffStrategy. There's no backoff override, matchingaws-credentials-http. It can be added later as an optional argument without breaking anyone.IMDSConfig.retry_strategywas accepted but never used. It's now deprecated rather than removed, so existing callers keep working: passing it warns, and only itsmax_attemptsis read. IMDS is link-local, so a service client's retry strategy, with its retry quota and longer throttling backoff, doesn't fit it.IMDS requests now time out after 1 second. The async client enforces this itself, because aiohttp ignores the per-request read timeout. A caller's sync client keeps its own connect timeout, because a request can't override it.
Session tokens now expire after the TTL IMDS returns, which can differ from the one requested. Both token caches read the cached token once, so a token invalidated by another caller can't fail a request that already checked it. Response bodies are closed after every read, so a failed attempt doesn't hold a connection from the caller's pool.
Trade-offs worth a look:
IMDSClient. Those now surface as "after 4 attempt(s)", with the real error in the cause chain.resolve_credentialsmakes twogetcalls when no profile name is configured, and each gets its ownmax_attempts.Testing
Unit, functional and static analysis checks pass (
make static-analysis,make test-unit,make test-functional). The new functional tests run the real urllib3 and aiohttp clients against a fake IMDS on 127.0.0.1 that refuses connections, closes them without replying, resets them, or never replies. Each case is retried on both clients. Before the change to retry any send or read error, a closed or reset connection wasn't retried. I haven't run this against IMDS on a real EC2 instance.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.