[ParidelPooya]

Fixes #797.

Summary

After a suspend that the service refused for now, the MicroVM worker now waits at least 30 seconds before its next SuspendMicrovm call. The docs now say which errors reach "refused for now" with the default client. The JavaScript worker made the same change in aws/aws-durable-execution-sdk-js#1040.

The warnings for a failed SuspendMicrovm call now match the JavaScript worker after aws/aws-durable-execution-sdk-js#1049. The "suspending the idle MicroVM" line now logs waitedSeconds.

Why

1. feat(microvm-worker): act on lifecycle hooks #782 added the worker's own suspend when idle. A first-attempt error that the service answers before it acts is "refused for now". The worker then accepts jobs again at once and starts the idle time again.
2. Some of these errors repeat on every call. A wrong secret key gives a signature error each time. A static session token gives ExpiredTokenException each time after it expires.
3. The run hook accepts any positive autoSuspendIdleSeconds. So an idle MicroVM made one call per idle time, with no lower bound, and wrote two log lines per call.
4. A probe with an idle time of 0.0001 s counted 15 calls in 3 s of real time. The count depends on the machine.
5. The docs said that a first-attempt throttle is the usual case for "refused for now". The default client uses botocore's standard retry mode with 3 attempts, and that mode retries ThrottlingException. A probe showed that a 429 on every attempt ends with RetryAttempts 2. So the error is classified unknown.
6. botocore retries no signature, clock-skew or expired-token error. So those errors stay "refused for now" with the default client.

Changes

The 30-second minimum

1. _start_idle_time(after_refusal=True) uses max(autoSuspendIdleSeconds, 30 s) for the next idle time.
2. The refused-for-now branch runs while the call is still counted in _suspend_calls. So the request waits in _idle_after_call, and _idle_after_call_floor records the minimum. _suspend_call_ended applies it.
3. A later request during the same call overwrites the recorded minimum. So the latest request decides.
4. A resume hook or a job that ends during the call does not shorten the wait, because the refused outcome arrives after it.
5. A job or a resume hook after the refusal starts the normal idle time.
6. While no job ends and no lifecycle hook arrives, the worker makes at most one call per 30 seconds.

Warnings

A new function _failed_call_warning chooses the warning for a failed call. Its conditions, their order, and its texts match failedCallWarning in the JavaScript worker (#1049), word for word.

1. The locked block that handles the outcome reads four values: the terminate flag, the closed flag, whether a suspend hook arrived since the idle time ended, and whether a refusal of jobs is in effect. It reads them before it changes them.
2. The worker logs the warning after that block. So the text matches the decision that the block made. The rejection warning was logged before the block. It now comes after it too.
3. Each text below is one line in the log. The first matching condition chooses it.

An unknown outcome, such as a timeout, a conflict, a server error, or an error after a retry:

• A terminate hook arrived: "the suspend call for the idle MicroVM had an unknown outcome. The MicroVM is terminating, so the worker refuses jobs and does not try the call again."
• Otherwise: "the suspend call for the idle MicroVM had an unknown outcome. The worker refuses jobs until the MicroVM resumes, or for 30 seconds."

A rejection, such as AccessDeniedException or missing credentials:

• A terminate hook arrived: "could not suspend the idle MicroVM, because the service rejected the call. The MicroVM is terminating, so the worker does not try the call again. A missing permission needs lambda:SuspendMicrovm on the MicroVM image in its execution role."
• A suspend hook arrived during the call, and its refusal is still in effect: "could not suspend the idle MicroVM, because the service rejected the call. A suspend hook arrived during the call, so the MicroVM is suspending anyway. The hook's refusal of jobs stays until the resume hook, or 30 seconds after the last suspend hook. A missing permission needs lambda:SuspendMicrovm on the MicroVM image in its execution role."
• Otherwise, also after close(): "could not suspend the idle MicroVM, because the service rejected the call. It keeps running. A missing permission needs lambda:SuspendMicrovm on the MicroVM image in its execution role."

A refusal for now, such as expired credentials or a signature error:

• A terminate hook arrived: "could not suspend the idle MicroVM, because the service refused the call for now. The MicroVM is terminating, so the worker does not try the call again."
• close() came: "could not suspend the idle MicroVM, because the service refused the call for now. The worker is closed, so it does not try the call again."
• A suspend hook arrived during the call, and its refusal is still in effect: "could not suspend the idle MicroVM, because the service refused the call for now. A suspend hook arrived during the call, so this call does not end the hook's refusal. The idle time starts again after the resume hook, or 30 seconds after the last suspend hook."
• A suspend hook arrived during the call, and its refusal ended before the call ended, on a resume hook or after 30 seconds: "could not suspend the idle MicroVM, because the service refused the call for now. A suspend hook arrived during the call, and the hook's refusal of jobs ended before the call ended. So the worker accepts jobs. It tries the call again one idle time after the call and every job end."
• Otherwise: "could not suspend the idle MicroVM, because the service refused the call for now. It keeps running. The worker tries the call again after the next idle time. That idle time lasts at least 30 seconds, unless a job or a resume hook starts the idle time again."

The earlier suspend-hook text said that the hook's refusal stays. That was false when the refusal had ended during the call: jobs got HTTP 202. The earlier texts also said that the worker goes on as before after a terminate hook or close(). The behavior is unchanged. Only the texts differ.

Log line

The info line "suspending the idle MicroVM" logged only idleSeconds. After a refusal for now, the wait before the next call lasts at least 30 seconds. So the line now also logs waitedSeconds, the delay that the idle timer waited. _start_idle_time passes that delay to the call. The JavaScript worker logs the same value as waitedMs.

Docs

• README item 7 and the docstrings of SuspendOutcome, classify_suspend_error, create_microvms_client and the listener say that the default client retries a throttle. So a throttle usually ends as unknown. "Refused for now" covers expired credentials and a signature error, with or without a clock offset. A throttle counts only from a client that does not retry it.
• The JavaScript default client also retries clock-skew errors. So there, a clock-skew error usually ends as unknown. README item 7 states this difference.
• The create_microvms_client option doc says that a caller client that retries must report its retries in ResponseMetadata.RetryAttempts on each ClientError. Otherwise a retried throttle counts as a first attempt.
• README item 6 and rule 2 of classify_suspend_error give the JavaScript reason why missing credentials count as a rejection: a later call in the same MicroVM most likely finds no credentials either.
• README items 6 and 7 and the listener docstring name the exception from #1049. A suspend hook whose refusal is still in effect when the call ends keeps that refusal. It ends on the resume hook, or 30 seconds after the last suspend hook. README item 6 and the listener docstring also name the resume hook during a rejected call: the worker then retries one idle time after the call ends.

Testing

• Package: 436 tests pass. worker.py and self_suspend.py are at 100 % line and branch coverage. Typecheck, hatch fmt --check and the commit lint pass.
• The package tests pass on each of the five new commits.
• New tests with fake timers:
  • the minimum at idle times of 1 s, 10 s and 45 s;
  • one call per 30 s while the refusal lasts;
  • the normal idle time after a job or a resume hook that follows the refusal;
  • the minimum after a resume hook, or a job that starts and ends, during the call;
  • the normal idle time after a job that runs across the refused outcome;
  • every warning text above, in full, logged once.
• The warning tests also check the HTTP statuses and the call times:
  • a terminate hook during the call, also after a suspend hook: jobs get 503, also after a resume hook, and no call follows;
  • close() during the call: jobs get 202, and no call follows;
  • close() and a terminate hook: the terminating text;
  • two suspend hooks 10 s apart during a rejected or refused call: jobs get 503 until the resume hook, or until 30 s after the second hook, and the next call comes one idle time later;
  • a suspend hook whose refusal ended during the call: jobs get 202, and the next call comes one idle time after the call or after the job.
• A test checks idleSeconds and waitedSeconds for a 1-second and a 45-second idle time after a refusal for now.
• A second test checks waitedSeconds on four other paths, with a 1-second idle time:
  • a job end after a refusal for now: 1 s;
  • a resume hook after a refusal for now: 1 s;
  • the end of the 30-second grace after an unknown outcome: 1 s;
  • a resume hook during a call that ends refused for now: 30 s, because the refusal asks for the idle time last.
• The new tests against the previous worker.py: every status and timing assertion passes. Only the text and waitedSeconds assertions fail.
• A test with the real default client checks the retry count and the outcome for a throttle, a skewed signature error, a signature mismatch and an expired token.
• Six mutations of the minimum each fail at least one test. Examples: no minimum, min instead of max, the minimum on every start, the first request during the call deciding instead of the latest.
• 16 mutations of the warning choice and the log field. 15 fail at least one test. Examples: each branch removed, each suspending check removed, closed checked before terminating, terminating checked after the hook, waitedSeconds taken from the idle time. Reading the refusal flag after the block passes every test. That mutation does not change the result, because the block changes the flag only when no suspend hook arrived.
• With the second waitedSeconds test, the mutation that takes waitedSeconds from the idle time fails 2 test cases: the 1-second refusal and the resume hook during a refused call. A mutation that logs at least 30 s on every call fails all 4 cases of the second test.
• The test helper settle now takes the listener lock after its wait, as finish_job does. A 20 ms sleep inside the listener's call-end block made 14 tests fail without this change, and none with it.
• A loop probe over 3 s of real time made 1 call for every idle time and error after the change. Before the change it made up to 15.
• The package suite passed 2 of 2 runs with 8 parallel workers at 436 tests.
• Three rounds of review. The review docs are not part of the PR.
• Not run against the live service.

[github-actions]

Codex AI review

No actionable findings in the PR changes. Tests were inspected but not executed under the read-only constraints, leaving runtime concurrency and service-integration test risk.

Reviewed commit 691de074e23dda7fdf464e7349baf2af201c02d3. Workflow run