Repository navigation
fix(extras): record the session resume mode - #811
Conversation
The resume mode says whether each session job checks the MicroVM state, and resumes a suspended MicroVM, before delivery. The session computed it from the current config on every replay. 1. The worker reads its idle time once, from the run hook payload of the launch. RunMicrovm sets the idle policy once, at the launch. 2. So a deploy that turns auto_suspend_on_idle off, or removes idle_policy, does not stop a running MicroVM from suspending. 3. The next job then skipped the state check. The suspended endpoint answered 502. 4. The request step retries a 502 as a MicroVM that is still starting. So the job failed with MicrovmDeliveryError only after both retry tiers. A session launch now writes the resume mode to the launch checkpoint, as "resumeBeforeDelivery": true or false. Replay follows the recorded mode, not the current config. The JavaScript SDK writes the same field, in aws/aws-durable-execution-sdk-js#1042. LaunchResult reads the field only when it is a bool. In Python True == 1, so a check by equality would accept 1. A checkpoint without the field, or with another value, falls back to the current config. A microvm() launch records no resume mode. So its checkpoint keeps only microvmId, endpoint, and delivery. Fixes #798
With auto_suspend_on_idle=True and no auto_suspend_idle_time, the default idle time is the session timeout if that is shorter than 60 seconds. The idle time must be at least 10 seconds. So a session whose timeout is under 10 seconds does not suspend by default, and it records "resumeBeforeDelivery": false. The README did not state this case. The JavaScript README does.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| result = run(handler, None, [("build", succeed_with(1)), ("test", succeed_with(2))]) | ||
|
|
||
| assert outcome(result)["result"]["test"] == 2 | ||
| assert len(invocations) > 1 |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_ajwnsebjasbacyyspt6wbmvvj4
[P2] Guarantee a replay before delivering the second job
The run fixture completes callbacks as soon as wait_for_callback sees their creation; it does not wait for suspension. If the first callback completes before the request step checkpoints success, that checkpoint supplies the callback result and both jobs can be delivered in the first invocation. A suspension while awaiting the second callback then satisfies len(invocations) > 1 without exercising delivery under the changed config. If both callbacks complete early, the assertion instead fails despite correct behavior. Both new deployment tests share this race. Pause the execution before completing the first callback, then complete it and resume, and assert that the second request executes in a later invocation.
To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.
Codex AI reviewOne P2 test-reliability finding at Reviewed commit |
This branch was successfully deployed
2 participants
Fixes #798.
Summary
A
microvm_sessionnow writes its resume mode to the launch checkpoint as"resumeBeforeDelivery": trueorfalse. Replay follows the recorded mode, not the current config. This matches aws/aws-durable-execution-sdk-js#1042.Why
The resume mode says whether each session job checks the MicroVM state, and resumes a suspended MicroVM, before delivery. The session computed it from the current config on every replay.
auto_suspend_on_idleoff, or removesidle_policy, does not stop a running MicroVM from suspending.MicrovmDeliveryErroronly after both retry tiers.Changes
LaunchResulthas a new field,resume_before_delivery: bool | None. Its wire name isresumeBeforeDelivery.from_dictreads the field only whenisinstance(value, bool). In PythonTrue == 1, so a check by equality would accept1.to_dictomits the key when the value isNone. So themicrovm()checkpoint is unchanged._SessionScope.resume_before_deliveryis renamed toresume_from_config, because it now reflects only the current config.lambda:GetMicrovmandlambda:ResumeMicrovmuntil every session that launched with suspending on has ended. A session whosetimeoutis under 10 seconds does not suspend by default.Checkpoint format
The JS and Python SDKs share the launch checkpoint format. So both SDKs must use the exact field name.
Session launch:
{"microvmId":"mvm-1","endpoint":"mvm-1.lambda-microvm.us-east-1.on.aws","delivery":"http","resumeBeforeDelivery":true}microvm()launch, unchanged:{"microvmId":"mvm-1","endpoint":"mvm-1.lambda-microvm.us-east-1.on.aws","delivery":"http"}Python writes the same keys in the same order. Its JSON has spaces after
:and,, as it did before for the other keys. A JS reader parses it, and a Python reader parses the JS form.The extras package is not published to PyPI. So no released version wrote a launch checkpoint without the field. The fallback covers checkpoints from a build of git main.
Testing
hatch run dev-extras:cov: 279 passed, 100 % coverage. On origin/main: 259 passed, 100 % coverage.dev-extras:typecheck,hatch fmt --check, and the commit lint pass.hatch run test:cov: 5408 passed, 2 skipped.MicrovmDeliveryErrorafter 4 attempts. On this branch it resumes the MicroVM and succeeds.isinstance, always write the key, fall back toTrueorFalse, record nothing, coerce withbool().Known gap, not changed here
A launch step that fails, and is retried after a deploy, computes the resume mode from the new config. If the first attempt created the MicroVM and only its response was lost, the recorded mode can be wrong. The JavaScript operation has the same gap. A fix belongs in both SDKs together.