Skip to content

MicroVM worker: wait at least 30 seconds after a suspend refused for now #797

Closed

Description

@ParidelPooya

Summary

A persistent "refused for now" error makes the MicroVM worker call SuspendMicrovm once per idle time, with no lower bound.

Details

  1. feat(microvm-worker): act on lifecycle hooks #782 added the worker's own suspend when idle. classify_suspend_error returns REFUSED_FOR_NOW for a first-attempt error that the service answers before it acts. Examples are an expired token and a signature error.
  2. After that outcome, the worker starts the idle time again at once.
  3. The run hook accepts any positive autoSuspendIdleSeconds.
  4. Some of these errors repeat on every call. A wrong secret key gives a signature error each time. A static session token gives ExpiredTokenException each time after it expires.
  5. So an idle MicroVM calls SuspendMicrovm once per idle time for as long as the error lasts. Each call also writes two log lines.
  6. A probe with an idle time of 0.0001 s counted 15 calls in 3 s. The count depends on the machine.

A second gap is in the docs. The default client uses botocore's standard retry mode with 3 attempts. It retries ThrottlingException. So a throttle usually ends as an unknown outcome, not as "refused for now". The README and docstrings say otherwise.

Proposed fix

Match the JavaScript worker (aws/aws-durable-execution-sdk-js#1040):

  1. After a "refused for now" outcome, the next idle time lasts at least 30 seconds. A job or a resume hook after the refusal starts the normal idle time.
  2. Correct the docs on which errors reach "refused for now" with the default client.

The branch fix/microvm-worker-refused-for-now-floor implements both. It has no PR yet.

Raised in the review of aws/aws-durable-execution-sdk-js#1042.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions