Repository navigation
fix: checkpoint the wrapped error for failed contexts - #807
ParidelPooya wants to merge 4 commits into
Conversation
When an SDK wrapper (StepError, ChildContextError, InvokeError, WaitForConditionError, CallbackExternalError, CallbackSubmitterError or DurableOperationError) escaped a child context, map item, parallel branch or wait_for_callback context, the context checkpointed the wrapper's class as its ErrorType. The type of the error that caused the failure was lost one level up. ErrorObject.from_exception now unwraps the SDK's own wrappers (exact class match, at most 16 steps, stopping at a repeated cause) and records the wrapped error's type, message, data and stack trace. This applies to every context checkpoint and to the execution result. CallbackTimeoutError, CallbackError, SerDesError and other SDK errors are not unwrapped. A callback failed externally now carries the sender's ErrorType on CallbackExternalError.error_type. A missing or blank type keeps the SDK class; the explicit type "Error" is kept as "Error". wait_for_callback tells a submitter failure from a callback failure by reading the checkpoints of the context's two children, so first run and replay raise the same error. Checkpoints that still hold StepError or a callback error type surface as before.
GitHub Actions
Error
The core now records the error a ChildContextError wraps. A MicroVM error that leaves a child context of the session handler therefore reaches the caller with its MicroVM class, and a step failure that leaves the handler carries the step's error type. Update the session docs and tests to match.
GitHub Actions
Error
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| # error is recovered from which child failed: the submitter step or the | ||
| # callback. Both are read from checkpoints, so first run and replay | ||
| # raise the same error. | ||
| context_operation_id: str = self._peek_next_operation_id() |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_h4qgnm736fmhb4frpc4ll4xnns
[P2] Use the allocated context ID for failure lookup
Peeking does not reserve the operation ID. Two concurrent callers sharing a context can both capture the same ID before run_in_child_context allocates distinct IDs. The second caller then classifies its failure using the first operation's children, potentially raising the wrong callback error and message or leaking ChildContextError. Allocate the identifier once and reuse it for both child execution and failure translation. Add a shared-context concurrency test covering different callback outcomes.
To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.
| if isinstance(inner, CallbackError): | ||
| raise inner | ||
| if isinstance(inner, StepError): |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_3uosxfehowzjhkpifltveglt52
[P2] Check child outcomes before applying legacy error classification
The reconstructed cause now reflects the sender's error type, so these checks no longer establish which operation failed. An external callback failure with ErrorType="aws_durable_execution_sdk_python.exceptions.StepError" now raises CallbackSubmitterError despite a successful submitter; a sender type matching CallbackTimeoutError similarly turns an external failure into a timeout. This bypasses the correct exception handlers on both live execution and replay. Consult the child checkpoint statuses first and retain class-based classification only as a fallback when child evidence is unavailable. Add regression cases for sender types matching SDK exception names.
To suppress repeats, reply /ai verdict deferred, /ai verdict false-positive, or /ai verdict out-of-scope on the first nonblank line. Add rationale on later lines. A new /ai verdict accepted allows future repeats again.
Codex AI reviewTwo confirmed findings at Reviewed commit |
|
Closing in favour of a simpler approach. We no longer try to make |
This branch was successfully deployed
2 participants
Problem
When an SDK wrapper error escaped a child context, map item, parallel branch or
wait_for_callbackcontext, the context's FAIL checkpoint recorded the wrapper's class asErrorType. For example, a step that raisedPaymentErrorinsiderun_in_child_contextleftaws_durable_execution_sdk_python.exceptions.StepErroron the context checkpoint and on the execution result. The type of the error that caused the failure was lost one level up. On replay the context's__cause__was rebuilt as aStepErrorwhoseerror_typewasStepError, so the original type was lost there too.Change
ErrorObject.from_exceptionunwraps the SDK's own wrappers before it records an error:DurableOperationError,StepError,InvokeError,ChildContextError,WaitForConditionError,CallbackExternalErrorandCallbackSubmitterError. Classes are matched exactly, so a user subclass of a wrapper is never unwrapped. A wrapper already carries the wrapped error's fields, so the record uses itserror_type,message,dataandstack_trace. Whenerror_typenames another wrapper, the walk continues with__cause__. The walk takes at most 16 steps and stops at a cause it has already visited. A wrapper it stops at is recorded by itserror_type.CallbackTimeoutError, the baseCallbackError,SerDesError,BatchCompletionError,ValidationErrorand other SDK errors are not unwrapped.wait_for_callback, the map/parallel operation itself), step andwait_for_conditionFAIL checkpoints, and the execution result. They all build their error withfrom_exception.ErrorTypeonCallbackExternalError.error_type. A missing or blank type (ASCII whitespace only) keeps the SDK class. The explicit type"Error"stays"Error".wait_for_callbackused to tell a submitter failure from a callback failure by the class rebuilt from the context checkpoint. That checkpoint now holds the user's type, so it reads the checkpoints of the context's two children instead (the callback, then the submitter step). First run and replay read the same checkpoints and raise the same error.Live and replay still go through one path: the live failure is rebuilt from the
ErrorObjectit checkpointed (raise_as_operation_error), the same as replay. The new e2e test records the error shape user code sees on the first run and on replay and asserts they are equal.User-visible changes
ErrorType: a failed child context, map item, parallel branch orwait_for_callbackcontext records the wrapped error's type (for examplemy_module.PaymentError, or a builtin such asValueError) instead of...exceptions.StepError,...ChildContextError,...InvokeErroror...CallbackSubmitterError.Error.ErrorType: an execution that fails because a wrapper propagated now reports the wrapped type. Before:...exceptions.StepError,...ChildContextError,...InvokeErroror...CallbackSubmitterError. After: the user's error type, or the invoked function's error type for an invoke.ErrorMessageis unchanged.ChildContextError.error_type(and theChildContextErrorraised byBatchResult.throw_if_error) is the original type rather than...StepErrorwhen a step failed inside the context.__cause__is now aDurableOperationErrorstand-in carrying that type, instead of aStepErrorstand-in. Code that checkedisinstance(e.__cause__, StepError)on aChildContextErrorno longer matches. Usee.error_typeinstead.CallbackSubmitterErrorfromwait_for_callback:error_typeis the submitter's error type, and__cause__is aDurableOperationErrorstand-in rather than aStepError. The class, message, data and stack trace are unchanged.CallbackExternalError.error_typecarries the type sent withSendDurableExecutionCallbackFailure. It used to always be...CallbackExternalError. Await_for_callbackcontext failed this way records the sender's type.microvm_session: a MicroVM error that leaves a child context the session handler created now reaches the caller with its MicroVM class (for exampleMicrovmJobFailedError), not asChildContextError. The session context now records the MicroVM type instead ofChildContextError. A step failure that leaves the handler still reaches the caller asChildContextError, and itserror_typeis the step's error type. The extras README andmicrovm_sessiondocstring are updated.StepError,ChildContextError,InvokeError,WaitForConditionError,CallbackSubmitterError,CallbackExternalErrorandCallbackTimeoutErrorare raised in the same places as before.Compatibility with old checkpoints
Checkpoints written by an earlier SDK version still replay:
...StepErroror another wrapper type is rebuilt as before.wait_for_callbackstill raisesCallbackSubmitterErrorfor a context holdingStepError, and the callback error for a context holding a callback error type....StepError. The earlier SDK reported...CallbackSubmitterErrorthere. No cause is guessed beyond what the checkpoint holds.ErrorObjectfields only. No new fields are written.Tests
tests/e2e/checkpoint_wrapped_error_int_test.py: 15 scenarios through the local test runner, each run live and replayed (30 tests). The replayed run catches the error, waits so that a new invocation starts, and rethrows. Covered: child context with a failing step, nested child contexts, map item step, two map items with different types, parallel branch step, two parallel branches with different types,wait_for_conditionandinvokeinside a child context,wait_for_callbacksubmitter failure, external failure with typecom.acme.PaymentError,"Error", missing type and blank type, callback timeout, and a plain step. Each test asserts every failed operation's checkpointedErrorType, the executionErrorTypeand message, and that the error user code sees has the same class,error_type, message, data, stack trace and cause on the first run and on replay. Without the source change, 24 of the 30 fail.tests/lambda_service_test.py: unwrap unit tests for every wrapper class, a cause walk through wrapper types, a user cause, a chain that ends at a wrapper, the 16-step bound (16 wrappers resolve, the 17th is reported), a cause cycle, and the errors that are not unwrapped (CallbackTimeoutError,CallbackError, a user subclass ofStepError).e2e/error_hierarchy_int_test.py(the executionErrorTypeisValueError, and an oldStepErrorcontext checkpoint reportsStepError),e2e/invoke_propagation_int_test.py(the executionErrorTypeis the invoked function's type),context_test.py(CallbackExternalError.error_type, plus a mock state for the unrelated-error case), the examples testwait_for_callback/test_wait_for_callback_submitter_failure.py(the executionErrorTypeis the submitter'sException), and two extrasmicrovm/session_test.pytests (item 6).context.py,exceptions.pyandlambda_service.pyat 100%). Testing, examples and conformance packages: 1801 passed, 2 skipped. Extras, otel, microvm-worker and otel conformance packages: 1409 passed. Fullhatch run test:cov: see the CI run.hatch fmt --check,hatch run types:check,hatch buildandlintcommitpass.