Skip to content

fix(microvm-worker): a job whose heartbeats cannot start is never reported, and a long Content-Length with Expect gets no answer #770

Open

Description

@Vivek0712

Two small gaps in the MicroVM worker's listener from #767, both reproduced on main at 66aee24.

1. A job whose heartbeat thread cannot start is accepted and never reported

Expected Behavior

If the worker cannot run a job it has already accepted, it fails the job's callback, as it does when the job's own thread cannot start (test_fails_a_run_hook_job_whose_thread_cannot_start).

Actual Behavior

Heartbeats.start (worker.py:627) raises when thread.start() fails, for example under thread exhaustion during a burst of jobs. _run_job logs job failed inside the worker, and its finally records the callback ID as finished. By then the job has had its 202, or the run hook its 200, so no delivery follows. The handler never runs, the callback is never failed, and the durable function waits for its heartbeat timeout.

Steps to Reproduce

  1. Patch Heartbeats._start_thread (or threading.Thread.start for threads named heartbeats-*) to raise RuntimeError("can't start new thread").
  2. Start a worker with a route and the test fakes, and POST a job with heartbeatTimeoutSeconds: 60.
  3. The answer is 202 and idle() returns True, but the Lambda client records no calls.

2. Expect: 100-continue with a very long Content-Length gets no answer

Expected Behavior

HTTP 400 for an invalid Content-Length, as the same request without Expect gets.

Actual Behavior

_CONTENT_LENGTH ([0-9]+) matches a value of any length, and declared_length (worker.py:1013) calls int(raw). Past 4,300 digits Python raises ValueError: Exceeds the limit (4300 digits) for integer string conversion. handle_expect_100 does not catch it, and _Server.handle_error passes it to the base class. So stderr gets a traceback and the connection closes with no answer.

Steps to Reproduce

Send POST /aws/lambda-microvms/runtime/v1/run with Expect: 100-continue and Content-Length: followed by 5,000 nines. The response is empty. Without the Expect header it is a 400.

SDK Version

aws-durable-execution-sdk-python-microvm-worker from main (66aee24), not yet released.

Python Version

3.13.2 (macOS) for the repros. The listener itself also ran in a MicroVM on Python 3.12, where the JS SDK's e2e scenarios passed against it.

Is this a regression?

No. Both are in the new listener.

Additional Context

I have a PR with fixes and a regression test for each.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions