Idea: Monorepo for aws-durable-execution-sdk-python-testing and aws-durable-execution-sdk-python

[zhongkechen]

Currently there are two separate Github repos for AWS Durable Functions - aws-durable-execution-sdk-python (SDK repo for short) is for the runtime SDK and aws-durable-execution-sdk-python-testing (testing repo for short) for testing utilities.

Problems with Separate Repos

• Sync drift: SDK changes and their corresponding tests live in different repos, making it easy for tests to fall out of date or reference stale APIs (e.g., the imports from aws_durable_execution_sdk_python.state and aws_durable_execution_sdk_python.threading in your active test file could break silently if the SDK renames modules).
• Cross-repo CI complexity: Every SDK PR needs a way to trigger the test repo's CI against the proposed change.
• Atomic changes are impossible: A single feature (like the empty-checkpoint coalescing in issue #325) requires two separate PRs in two repos, reviewed and merged in the right order. If the test PR merges first, CI breaks; if the SDK PR merges first, there's a window with no test coverage.
• Dependency management overhead: The test repo must declare a dependency on a specific SDK version/branch. This means maintaining version pins, dealing with pre-release builds, or pointing at git refs — all of which add friction.
• Discoverability: New contributors have to find and clone two repos. The relationship between a feature and its tests isn't obvious from either repo alone.
• Inconsistent tooling: Linting, formatting, and CI configs diverge over time since each repo has its own setup.

Recommended solution: monorepo

Hatch supports monorepo:

• https://hatch.pypa.io/1.16/how-to/environment/workspace/

[yaythomas]

Just thinking through how will versioning & publishing separately to aws-durable-execution-sdk-python and aws-durable-execution-sdk-python-testing work?

  monorepo-root/
  ├── aws-durable-execution-sdk-python/
  │   ├── pyproject.toml
  │   └── src/
  │       └── aws_durable_execution_sdk/
  │           ├── __init__.py
  │           └── __about__.py
  ├── aws-durable-execution-sdk-python-testing/
  │   ├── pyproject.toml
  │   └── src/
  │       └── aws_durable_execution_sdk_testing/
  │           ├── __init__.py
  │           └── __about__.py

Tagging a release would have to take a specific format, like

aws-durable-execution-sdk-python/v1.2.3
aws-durable-execution-sdk-python-testing/v3.1.2

Even so, the tag would then apply to the entire repo, i.e you're effectively tagging SDK code when you tag a testing release.

Then there would have to be some parsing in the gh action [published]

1. check tag and parse out package name
2. run hatch build in that subdir
3. selectively publish specific sub dir per https://github.com/pypa/gh-action-pypi-publish#customizing-target-package-dists-directory

[zhongkechen]

Just use one version number for all packages in monorepo.

[bchampp]

It's not as simple as this. We need to maintain independent versions between the language SDK and testing library. We also have much more freedom and autonomy for how we treat version bumps in the testing library as opposed to the SDK since the SDK is bundled in the lambda runtime.

[bchampp]

I'm in favour of a monorepo, but there are some design constraints we faced when setting the project structure up that we need to remain cognizant of. If hatch has the tooling to support this while allowing isolated versioning and publishing across the modules, then great, I'm all for it.

I agree this will simplify how we test code, and allow us to atomically release changes together (which we can already do in the JS SDK).

[zhongkechen]

Is the testing library used in somewhere else, such as sam cli? If so, I'm afraid we'd better keep this separate.

Update: it's used by sam cli via a docker image. So probably we are OK to put the Python package anywhere we want.

https://github.com/aws/aws-sam-cli/blob/5a390570519635d6b80ad8d2272f1ebcfec9462f/samcli/local/docker/durable_functions_emulator_container.py#L33C64-L33C94

[SilanHe]

Yeah, if we move to more of a mono/poly repo, I think we just need to ensure we're still building and publishing the image based on the testing sdk.

[zhongkechen]

1. A quick improvement can be done is to make testing library a dependency of SDK in test env

This extra is only used when we run tests for the SDK. And the examples in testing can be moved into SDK if they are test cases for the SDK library features.

This change will introduce a circular dependency between SDK and testing library. But I think it's fine for now as long as the dependency (SDK -> testing) is optional and only needed in test env.

2. And the next next step would be to remove SDK as a dependency from testing library.

With the examples moved out of the testing repo, the testing library doesn't really depend on the durable execution SDK. This needs to be further investigated though.

[zhongkechen]

First step is done with the following PRs:

• move examples to SDK #218
• move examples from testing library aws-durable-execution-sdk-python#356