Skip to content

feat(aws-credentials-imds): retry IMDS credential requests - #150

Open
rohangavankar wants to merge 2 commits into
developfrom
rohangavankar/feat/imds-retries
Open

rohangavankar wants to merge 2 commits into
developfrom
rohangavankar/feat/imds-retries

Conversation

@rohangavankar

@rohangavankar rohangavankar commented Nov 18, 2026 •

Copy link
Copy Markdown

Summary

IMDS credential requests are now retried with capped exponential backoff. The token fetch and the metadata read are retried as one unit, so a retry gets a new token when it needs one. Every failure is retried except a token request rejected with 400 or 403 and a metadata request answered with 403 or 404. A 401 drops the rejected token first, so the next attempt fetches a new one. Any error from sending a request or reading its body counts as a failed request, because urllib3 and aiohttp raise different exception types for the same dropped or reset connection.

The new IMDSConfig.max_attempts defaults to 4, or 3 retries, and can be set with AWS_METADATA_SERVICE_NUM_ATTEMPTS or metadata_service_num_attempts, the same settings botocore reads. Backoff waits 0.25 s, 0.5 s and then 1 s, computed by smithy_core's ExponentialRetryBackoffStrategy. There's no backoff override, matching aws-credentials-http. It can be added later as an optional argument without breaking anyone.

IMDSConfig.retry_strategy was accepted but never used. It's now deprecated rather than removed, so existing callers keep working: passing it warns, and only its max_attempts is read. IMDS is link-local, so a service client's retry strategy, with its retry quota and longer throttling backoff, doesn't fit it.

IMDS requests now time out after 1 second. The async client enforces this itself, because aiohttp ignores the per-request read timeout. A caller's sync client keeps its own connect timeout, because a request can't override it.

Session tokens now expire after the TTL IMDS returns, which can differ from the one requested. Both token caches read the cached token once, so a token invalidated by another caller can't fail a request that already checked it. Response bodies are closed after every read, so a failed attempt doesn't hold a connection from the caller's pool.

Trade-offs worth a look:

  • Retrying every send or read error also retries bugs, such as an async HTTP client given to the sync IMDSClient. Those now surface as "after 4 attempt(s)", with the real error in the cause chain.
  • With the defaults, an unreachable IMDS now takes about 5.75 s to fail instead of about 1 s, the cost of the 3 retries. With the credential chain's shared sync client, whose connect timeout is 60 s, it can take about 4 minutes. Capping that needs a per-request connect timeout in smithy-http.
  • resolve_credentials makes two get calls when no profile name is configured, and each gets its own max_attempts.
  • Backoff has no jitter.

Testing

Unit, functional and static analysis checks pass (make static-analysis, make test-unit, make test-functional). The new functional tests run the real urllib3 and aiohttp clients against a fake IMDS on 127.0.0.1 that refuses connections, closes them without replying, resets them, or never replies. Each case is retried on both clients. Before the change to retry any send or read error, a closed or reset connection wasn't retried. I haven't run this against IMDS on a real EC2 instance.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@rohangavankar
rohangavankar requested a review from a team as a code owner November 18, 2026 17:44
@rohangavankar
rohangavankar force-pushed the rohangavankar/feat/imds-retries branch from 6881b9a to 38f64cd Compare November 18, 2026 17:49
IMDS requests are now retried with capped exponential backoff. The token
fetch and the metadata read are retried together, so a retry gets a new
token when it needs one. Every failure is retried except a token request
rejected with 400 or 403 and a metadata request answered with 403 or 404. A
401 drops the rejected token before the retry, so the next attempt fetches a
new one. Any error from sending a request or reading its body counts as a
failed request, since urllib3 and aiohttp raise different types for the same
dropped or reset connection.

IMDSConfig.retry_strategy is replaced by max_attempts. It defaults to 4, or
3 retries, and can be set with AWS_METADATA_SERVICE_NUM_ATTEMPTS or
metadata_service_num_attempts, the same settings botocore reads. Backoff
waits 0.25 s, 0.5 s and then 1 s.

IMDS requests now time out after 1 second. The async client enforces this
itself, because aiohttp ignores the per-request read timeout. A caller's
sync client keeps its own connect timeout, because a request can't override
it.

Session tokens now expire after the TTL IMDS returns. Both token caches read
the cached token once, so a token invalidated by another caller can't fail a
request that already checked it. Response bodies are closed after every
read, so a failed attempt doesn't hold a connection from the caller's pool.
@rohangavankar
rohangavankar force-pushed the rohangavankar/feat/imds-retries branch from 38f64cd to 995a10b Compare November 18, 2026 17:50
@rohangavankar
rohangavankar deleted the rohangavankar/feat/imds-retries branch November 18, 2026 18:11
…ing it

IMDSConfig.retry_strategy stays as a deprecated argument and attribute, so
existing callers keep working. Passing it warns, and only its max_attempts
is used: IMDS is link-local, so a service client's retry strategy, with its
retry quota and longer throttling backoff, doesn't fit it. An explicit
max_attempts takes precedence. The fixed backoff now comes from
smithy_core's ExponentialRetryBackoffStrategy rather than a second copy of
the calculation.
@rohangavankar
rohangavankar restored the rohangavankar/feat/imds-retries branch November 18, 2026 18:26
@rohangavankar rohangavankar reopened this Nov 18, 2026

@jonathan343 jonathan343 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rohangavankar! Let a few pieces of feedback

Comment on lines +289 to +291
attempts = 0
last_exc: Exception | None = None
while attempts < config.max_attempts:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loop repeats the logic smithy_core.retries.SimpleRetryStrategy already implements. That class takes a backoff_strategy and max_attempts, and retries errors where is_retry_safe is true.

Could we use it here, and keep both config options instead of deprecating retry_strategy?

  • max_attempts stays the simple option. We build SimpleRetryStrategy(backoff_strategy=_BACKOFF, max_attempts=max_attempts) from it, so callers don't need to import anything from smithy.
  • retry_strategy stays as an optional override and wins if both are set. That's the same precedence RetryStrategyResolver uses, and it lets callers swap the backoff.
  • Set is_retry_safe on _IMDSRequestError from your retryable logic.

DEFAULT_MAX_ATTEMPTS = 4
# IMDS is a link-local endpoint, so a healthy instance answers well within a second.
TIMEOUT_SECONDS = 1.0
_REQUEST_CONFIG = HTTPRequestConfiguration(read_timeout=TIMEOUT_SECONDS)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the default chain, providers.py passes setup.http_client to the resolver. That's the service client's transport, which has a 60 s connect timeout. This config only overrides the read timeout, so if the network silently drops IMDS traffic, each attempt waits 60 s. I reproduced it against an address that drops connections:

  • develop: 1 attempt, fails in 60 s
  • this PR: 4 attempts, fails in 242 s

Could providers.py stop passing setup.http_client, so the resolver uses its own 1 s client? That brings the same repro down to 6 s, and it's similar to what botocore does. A per-request connect_timeout in smithy-http is the better long-term fix, but that can be a follow-up.

If we do this, the resolver should create its client once and keep it. When no client is passed in, it currently builds a new one on every get_identity, so the token is never reused. With 10 concurrent callers I saw 10 token PUTs instead of 1.

_ENDPOINT_MODE_PROFILE = "ec2_metadata_service_endpoint_mode"
_PROFILE_NAME_ENV = "AWS_EC2_INSTANCE_PROFILE_NAME"
_PROFILE_NAME_PROFILE = "ec2_instance_profile_name"
_MAX_ATTEMPTS_ENV = "AWS_METADATA_SERVICE_NUM_ATTEMPTS"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just noticing that we don't read the AWS_METADATA_SERVICE_TIMEOUT option

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants