fix(microvm-worker): wait at least 30 s after a suspend refused for now

packages/aws-durable-execution-sdk-python-microvm-worker/README.md

@@ -109,15 +109,15 @@ A session's run hook payload can set `autoSuspendIdleSeconds`. The session sets
 3. When the idle time ends, the worker calls `SuspendMicrovm` with the MicroVM ID from the `run` hook, never one from a job request. The call returned in under 100 milliseconds in testing, before the MicroVM froze. No attempt of the call starts after 10 seconds, because a `before-send` handler on the client stops botocore's retries at that deadline. botocore cannot cancel an attempt in flight, and it opens the connection after `before-send`. So an attempt that started before the deadline can still reach the service after it. The worker waits for that attempt to end, and refuses jobs until then. The client's connect and read timeouts bound the attempt: 5 and 10 seconds for the default client. A client from `create_microvms_client` should use similar timeouts. botocore's own defaults are 60 seconds each, so with them the refusal can last minutes. The durable function resumes the MicroVM before the next job, and the process continues with its memory and files.
 4. From the moment the idle time ends, the worker answers every job with HTTP 503 and does not start it. A job accepted then would freeze with the MicroVM, and nothing would resume it. After a 503, the durable function checks the state, resumes the MicroVM, and delivers the job again.
 5. The refusal ends when the `resume` lifecycle hook arrives, or 30 seconds after the call returned, if no hook arrives. The idle time then starts again. At most one call runs at a time. A `resume` hook that arrives during the call ends the refusal at once, and the next idle time starts when the call ends. So enable the `resume` hook in the image. Without it, the first job after a resume can wait a little over 30 seconds: the 30 seconds, plus up to 4 seconds until the durable function's next delivery attempt. A test measured 32 seconds.
-6. A call that the service rejected on its first attempt, such as `AccessDeniedException` for a missing permission, is logged as a warning, and the MicroVM keeps running. Missing credentials count as a rejection, because the call sends no request. The worker accepts jobs again at once and does not retry the call. The next job that ends starts the idle time again.
-7. A call that the service refused for now on its first attempt is logged as a warning too, and the worker accepts jobs again at once. This covers a throttle, expired credentials, and a clock offset, whatever the HTTP status. The service answers these before it acts. The idle time then starts again, so the worker tries the call again. The JavaScript worker decides these cases by HTTP status.
-8. A call whose outcome is unknown is handled like a successful call, because the service may have suspended the MicroVM anyway. This covers a timeout, a conflict, a server error, and a dropped connection. An error after botocore retried an earlier attempt also counts as unknown, because the earlier attempt may have suspended the MicroVM. botocore retries a throttle, so a throttle usually ends here. The refusal continues until the `resume` hook or the 30-second limit. The idle time then starts again, so the worker tries the call again.
+6. A call that the service rejected on its first attempt, such as `AccessDeniedException` for a missing permission, is logged as a warning, and the MicroVM keeps running. Missing credentials count as a rejection. A later call in the same MicroVM most likely finds no credentials either. The worker accepts jobs again at once and does not retry the call. The next job that ends starts the idle time again. The exception is a `resume` hook that arrived during the call. The idle time then starts when the call ends. So the worker retries the call one idle time after the call ends, if no job runs and no `suspend` hook refusal is in effect when the call ends. A second exception is a `suspend` hook that arrived during the call and whose refusal is still in effect when the call ends. The MicroVM is then suspending anyway. So the rejection does not end the hook's refusal. That refusal ends on the `resume` hook, or 30 seconds after the last `suspend` hook. The idle time then starts again.
+7. A call that the service refused for now on its first attempt is logged as a warning too, and the worker accepts jobs again at once. This covers expired credentials and a signature error, with or without a clock offset, whatever the HTTP status. The service answers these before it acts. botocore retries none of these errors, so they reach the worker on the first attempt. The idle time then starts again, so the worker tries the call again. That idle time lasts at least 30 seconds, also when `autoSuspendIdleSeconds` is shorter. The same error can repeat on every call, for example with a wrong secret key or an expired session token from environment variables. So while no job ends and no lifecycle hook arrives, the worker makes at most one call per 30 seconds, and logs one warning per call. A job that ends after the refusal, or a `resume` hook after it, starts the normal idle time instead. A `resume` hook or a job that ends during the call does not shorten the idle time after the refusal, because the refused outcome arrives after it. A `suspend` hook whose refusal is still in effect when the call ends is the exception, as in item 6. Its refusal stays, and the idle time starts when it ends. The default client retries a throttle, so a throttle from it usually ends as an unknown outcome, as in item 8. A throttle counts as refused for now only from a client that does not retry it, such as a client from `create_microvms_client` with one attempt. The JavaScript worker classifies these errors the same way. Its default client retries a clock-skew error. So there, a clock-skew error usually ends as an unknown outcome.
+8. A call whose outcome is unknown is handled like a successful call, because the service may have suspended the MicroVM anyway. This covers a timeout, a conflict, a server error, and a dropped connection. An error after botocore retried an earlier attempt also counts as unknown, because the earlier attempt may have suspended the MicroVM. The default client uses botocore's standard retry mode with 3 attempts, and that mode retries a throttle. So a throttle usually ends here. The refusal continues until the `resume` hook or the 30-second limit. The idle time then starts again, so the worker tries the call again.
 
 `worker.close()` stops the idle time. A closed worker never suspends its MicroVM again. A refusal that is already running still ends on the `resume` hook or after 30 seconds.
 
 The worker counts only jobs. It cannot see background threads or processes, or inbound traffic that is not a job. A session with such work leaves `auto_suspend_on_idle` off, and the payload then has no `autoSuspendIdleSeconds`.
 
-The worker creates a `lambda-microvms` client for each suspend, and closes it after the call. Pass `create_microvms_client` to provide the client. The worker never closes a client that it returns.
+The worker creates a `lambda-microvms` client for each suspend, and closes it after the call. Pass `create_microvms_client` to provide the client. The worker never closes a client that it returns. A client that retries must report its retries in `ResponseMetadata.RetryAttempts` of the error that it raises, as botocore does. The worker otherwise counts a retried call as a first attempt. An earlier attempt of that call may have suspended the MicroVM. So a retried throttle would count as refused for now, and the worker would accept a job that then freezes with the MicroVM.
 
 ## Suspend and terminate hooks
 

packages/aws-durable-execution-sdk-python-microvm-worker/src/aws_durable_execution_sdk_python_microvm_worker/self_suspend.py

@@ -74,8 +74,15 @@ class SuspendOutcome(Enum):
     """The service did nothing, and a later call would most likely fail the
     same way, for example for a missing permission."""
     REFUSED_FOR_NOW = "refused_for_now"
-    """The service did nothing, and a later call can succeed: a throttle,
-    expired credentials, or a clock offset."""
+    """The service did nothing on the call's first and only attempt, and a
+    later call can succeed.
+
+    Examples are expired credentials and a signature error, with or without
+    a clock offset. botocore retries none of these, so they keep this
+    outcome. The default client retries a throttle. So a throttle usually
+    ends as :attr:`UNKNOWN`. A throttle has this outcome only when it
+    reaches the worker without a retry, for example from a caller's client
+    that does not retry it."""
     UNKNOWN = "unknown"
     """The service may have suspended the MicroVM."""
 
@@ -93,6 +100,12 @@ def create_microvms_client(region: str) -> LambdaMicrovmsClient:
 
     Each call gets its own session, because boto3's default session is not
     safe to use from several threads at once.
+
+    The client uses botocore's standard retry mode with 3 attempts. That
+    mode retries a throttle error code, such as ``ThrottlingException``, a
+    5xx answer, and a connection error. It retries no credential or
+    signature error, also with a clock offset. Each ``ClientError`` reports the
+    retries in ``ResponseMetadata.RetryAttempts``.
     """
     return boto3.session.Session().client(
         "lambda-microvms",
@@ -165,16 +178,21 @@ def classify_suspend_error(error: BaseException) -> SuspendOutcome:
 
     1. A botocore retry inside the call means that an earlier attempt may
        have suspended the MicroVM before its answer was lost. So any error
-       after a retry is :attr:`SuspendOutcome.UNKNOWN`.
-    2. Missing credentials stop the call before any request. They are
-       :attr:`SuspendOutcome.REJECTED`, as in the JavaScript worker.
+       after a retry is :attr:`SuspendOutcome.UNKNOWN`. The default client
+       retries a throttle. So a throttle from it usually ends here.
+    2. Missing credentials are :attr:`SuspendOutcome.REJECTED`, as in the
+       JavaScript worker. A later call in the same MicroVM most likely finds
+       no credentials either.
     3. A known rejection code is :attr:`SuspendOutcome.REJECTED`.
     4. A conflict is :attr:`SuspendOutcome.UNKNOWN`, whatever its status. It
        can mean that the MicroVM is already suspending.
-    5. A throttle, expired credentials, or a clock offset on the first
+    5. Expired credentials, a signature error, or a throttle on the first
        attempt is :attr:`SuspendOutcome.REFUSED_FOR_NOW`, whatever its
        status. The service answers them before it acts, and a later call can
-       succeed. Some services answer a throttle with HTTP 400.
+       succeed. Some services answer a throttle with HTTP 400. botocore
+       retries no credential or signature error, also with a clock offset.
+       So these errors reach this rule. A throttle reaches it only from a
+       client that did not retry it.
     6. Any other 4xx answer except 408, 409, and 429 is
        :attr:`SuspendOutcome.REJECTED`. A 429 is
        :attr:`SuspendOutcome.REFUSED_FOR_NOW`.