From df3a925db341978c13b5f3ccab7d059e0a86132d Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 14:12:25 -0500 Subject: [PATCH 1/2] refactor!: remove the Python extension crate The Python extension (crate morphir-python-binding 0.4.0, bundle morphir-python, short id python) now lives in finos/morphir-python at rust/crates/morphir-python-binding, with its tests, fixtures, examples and Gherkin acceptance. This workspace no longer builds, tests or releases it. The extension/python/v0.1.0 to v0.4.0 releases stay published here. Removed: - the crate, its default-members entry, the Ruff workspace dependencies and the lockfile entries only it used (cargo update --workspace --offline; no other package version changed); - [extensions.python] in .github/extensions.toml, the extension:artifact:python task, the python case of test:cli-release and the CI step for generated Python functions. Host tests: crates/morphir-host-native/tests/python_extension.rs is deleted. Both of its tests were ignored and read fixtures from the deleted crate. The packaged test checked publish to a local repository, version-2 claims with frontend and backend, Unchecked claim checks before and after install, install from LocalIndex, activation from Morphir Home after the repository is removed, MEP negotiation of frontend languages and backend targets, and compile, generate and close in IR v3 and v4. The same host path is covered for remaining guests by rust_extension.rs, gleam_extension.rs and elm_native_extension.rs, which their extension:artifact tasks run in the CI extension-bundle job. gleam_extension.rs also compiles, generates and compiles again in both IR versions. Direct Extism container calls (initialize, capabilities, generate, lifecycle order) stay covered by extism_extension.rs. No host behavior loses its only coverage. Python compile and generate lifecycle checks move to finos/morphir-python, which runs them with the morphir CLI as an offline MEP host. No support code was used only by this file. CI tests: the classifier, graph, output, change-detection, consistency, routing, packaging and Windows identity tests now use morphir-rust-binding, rust or openapi as their sample, so they test the same behavior. The test_python_extension_release.py checks that are not covered elsewhere (selection keeps frontend languages and refuses a descriptor without them; packaging accepts frontend-only entries and refuses invalid languages) move to test_frontend_extension_release.py with a Rust sample. Its tag routing and CI bundle checks were duplicates of test_extension_release_routing.py and test_ci_workflow_definition.py and are removed. README drops the two Python rows and points to finos/morphir-python. BREAKING CHANGE: build morphir-python-binding from finos/morphir-python. Refs: morphir-ukvn.1.6.7 (OpenSpec migrate-python-binding task 7.1) --- .github/extensions.toml | 11 - .github/workflows/ci.yml | 6 - .mise/tasks/extension/artifact/python | 194 ------ .mise/tasks/test/cli-release | 38 +- CHANGELOG.md | 11 + Cargo.lock | 323 +--------- Cargo.toml | 6 - README.md | 7 +- .../tests/python_extension.rs | 294 --------- crates/morphir-python-binding/Cargo.toml | 31 - crates/morphir-python-binding/README.md | 572 ------------------ .../examples/python_adt.rs | 85 --- .../examples/python_modules.rs | 81 --- crates/morphir-python-binding/src/backend.rs | 300 --------- .../src/backend/functions.rs | 160 ----- .../src/backend/imports.rs | 140 ----- crates/morphir-python-binding/src/frontend.rs | 390 ------------ .../src/frontend/functions.rs | 352 ----------- .../src/frontend/imports.rs | 215 ------- .../src/frontend/project.rs | 138 ----- crates/morphir-python-binding/src/ir.rs | 58 -- .../morphir-python-binding/src/ir/classic.rs | 225 ------- .../src/ir/classic/expressions.rs | 136 ----- .../src/ir/classic/read.rs | 102 ---- crates/morphir-python-binding/src/lib.rs | 151 ----- crates/morphir-python-binding/src/modules.rs | 111 ---- crates/morphir-python-binding/src/names.rs | 99 --- crates/morphir-python-binding/src/values.rs | 224 ------- .../src/values/check.rs | 208 ------- .../tests/acceptance.rs | 171 ------ .../tests/conditionals.rs | 275 --------- .../tests/features/adt.feature | 47 -- .../tests/fixtures/conditionals.py | 35 -- .../tests/fixtures/functions.py | 36 -- .../tests/fixtures/models.py | 24 - .../tests/fixtures/modules/rules.py | 13 - .../tests/fixtures/tuples.py | 13 - .../morphir-python-binding/tests/functions.rs | 378 ------------ .../tests/ir_versions.rs | 261 -------- .../morphir-python-binding/tests/modules.rs | 376 ------------ .../morphir-python-binding/tests/pipeline.rs | 341 ----------- .../tests/support/native.rs | 29 - tests/ci/ci_impact_test_support.py | 6 +- tests/ci/test_ci_change_detection.py | 12 +- tests/ci/test_ci_impact_classify.py | 16 +- tests/ci/test_ci_impact_consistency.py | 4 +- tests/ci/test_ci_impact_graph.py | 4 +- tests/ci/test_ci_impact_outputs.py | 8 +- tests/ci/test_extension_release_routing.py | 3 +- tests/ci/test_frontend_extension_release.py | 50 ++ tests/ci/test_package_extension_packaging.py | 14 +- tests/ci/test_python_extension_release.py | 72 --- tests/ci/test_windows_source_identity.py | 2 +- 53 files changed, 104 insertions(+), 6754 deletions(-) delete mode 100755 .mise/tasks/extension/artifact/python delete mode 100644 crates/morphir-host-native/tests/python_extension.rs delete mode 100644 crates/morphir-python-binding/Cargo.toml delete mode 100644 crates/morphir-python-binding/README.md delete mode 100644 crates/morphir-python-binding/examples/python_adt.rs delete mode 100644 crates/morphir-python-binding/examples/python_modules.rs delete mode 100644 crates/morphir-python-binding/src/backend.rs delete mode 100644 crates/morphir-python-binding/src/backend/functions.rs delete mode 100644 crates/morphir-python-binding/src/backend/imports.rs delete mode 100644 crates/morphir-python-binding/src/frontend.rs delete mode 100644 crates/morphir-python-binding/src/frontend/functions.rs delete mode 100644 crates/morphir-python-binding/src/frontend/imports.rs delete mode 100644 crates/morphir-python-binding/src/frontend/project.rs delete mode 100644 crates/morphir-python-binding/src/ir.rs delete mode 100644 crates/morphir-python-binding/src/ir/classic.rs delete mode 100644 crates/morphir-python-binding/src/ir/classic/expressions.rs delete mode 100644 crates/morphir-python-binding/src/ir/classic/read.rs delete mode 100644 crates/morphir-python-binding/src/lib.rs delete mode 100644 crates/morphir-python-binding/src/modules.rs delete mode 100644 crates/morphir-python-binding/src/names.rs delete mode 100644 crates/morphir-python-binding/src/values.rs delete mode 100644 crates/morphir-python-binding/src/values/check.rs delete mode 100644 crates/morphir-python-binding/tests/acceptance.rs delete mode 100644 crates/morphir-python-binding/tests/conditionals.rs delete mode 100644 crates/morphir-python-binding/tests/features/adt.feature delete mode 100644 crates/morphir-python-binding/tests/fixtures/conditionals.py delete mode 100644 crates/morphir-python-binding/tests/fixtures/functions.py delete mode 100644 crates/morphir-python-binding/tests/fixtures/models.py delete mode 100644 crates/morphir-python-binding/tests/fixtures/modules/rules.py delete mode 100644 crates/morphir-python-binding/tests/fixtures/tuples.py delete mode 100644 crates/morphir-python-binding/tests/functions.rs delete mode 100644 crates/morphir-python-binding/tests/ir_versions.rs delete mode 100644 crates/morphir-python-binding/tests/modules.rs delete mode 100644 crates/morphir-python-binding/tests/pipeline.rs delete mode 100644 crates/morphir-python-binding/tests/support/native.rs create mode 100644 tests/ci/test_frontend_extension_release.py delete mode 100644 tests/ci/test_python_extension_release.py diff --git a/.github/extensions.toml b/.github/extensions.toml index 10e9282da..dcfacc146 100644 --- a/.github/extensions.toml +++ b/.github/extensions.toml @@ -32,17 +32,6 @@ workspace_discovery = true ir_versions = ["3", "4"] release_with_workspace = true -[extensions.python] -package = "morphir-python-binding" -artifact = "morphir-python-binding" -extension_id = "morphir-python" -name = "Morphir Python" -mep_versions = ["0.1"] -targets = ["python"] -languages = [{ id = "python", file_extensions = [".py"] }] -ir_versions = ["3", "4"] -release_with_workspace = true - [extensions.gleam] package = "morphir-gleam-binding" artifact = "morphir-gleam-binding" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0956995b6..6ba9bd65c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -474,12 +474,6 @@ jobs: if-no-files-found: error retention-days: 7 - - name: Test generated Python functions and closures - if: matrix.id == 'python' - env: - MORPHIR_TEST_PYTHON: python3 - run: cargo test --locked -p morphir-python-binding --test functions -- --ignored - workspace-wasm: name: Build (workspace discovery WASM) runs-on: ubuntu-latest diff --git a/.mise/tasks/extension/artifact/python b/.mise/tasks/extension/artifact/python deleted file mode 100755 index 87b9cc150..000000000 --- a/.mise/tasks/extension/artifact/python +++ /dev/null @@ -1,194 +0,0 @@ -#!/bin/sh -#MISE description="Build and validate the independently versioned Python extension bundle" - -set -eu -export PYTHONDONTWRITEBYTECODE=1 - -SNAPSHOT_MODE="${MORPHIR_PYTHON_SNAPSHOT_MODE:-0}" -TASK_DIR="$(CDPATH='' cd -- "$(dirname -- "$0")" && pwd -P)" -case "$SNAPSHOT_MODE" in - 0) - REPO_ROOT="$(git -C "$TASK_DIR" rev-parse --show-toplevel)" - ;; - 1) - REPO_ROOT="$(CDPATH='' cd -- "$TASK_DIR/../../../.." && pwd -P)" - ;; - *) - echo "MORPHIR_PYTHON_SNAPSHOT_MODE must be 0 or 1" >&2 - exit 1 - ;; -esac -cd "$REPO_ROOT" - -STAGING_DIR="$REPO_ROOT/.morphir/build/extensions/python" -WASM="target/wasm32-unknown-unknown/release/morphir_python_binding.wasm" - -if ! command -v python3 >/dev/null 2>&1; then - echo "Python 3.11 or newer is required to package extensions" >&2 - exit 1 -fi -if ! python3 -c 'import sys; raise SystemExit(0 if sys.version_info >= (3, 11) else 1)'; then - echo "Python 3.11 or newer is required to package extensions" >&2 - exit 1 -fi - -require_snapshot_file() { - relative="$1" - if [ ! -f "$REPO_ROOT/$relative" ] || [ -L "$REPO_ROOT/$relative" ]; then - echo "required provenance input is missing or unsafe: $relative" >&2 - exit 1 - fi -} - -require_snapshot_executable() { - relative="$1" - require_snapshot_file "$relative" - if [ ! -x "$REPO_ROOT/$relative" ]; then - echo "required provenance input is not executable: $relative" >&2 - exit 1 - fi -} - -require_snapshot_tree() { - relative="$1" - if [ ! -d "$REPO_ROOT/$relative" ] || [ -L "$REPO_ROOT/$relative" ]; then - echo "required provenance input is missing or unsafe: $relative" >&2 - exit 1 - fi - if ! find "$REPO_ROOT/$relative" -type f -print -quit | grep -q .; then - echo "required provenance input is missing or empty: $relative" >&2 - exit 1 - fi -} - -require_snapshot_inputs() { - require_snapshot_executable ".mise/tasks/extension/artifact/python" - require_snapshot_file "Cargo.toml" - require_snapshot_file "Cargo.lock" - require_snapshot_file "mise.toml" - require_snapshot_file ".github/extensions.toml" - require_snapshot_file "scripts/package_extension.py" - require_snapshot_file "scripts/extension_packaging/__init__.py" - require_snapshot_file "scripts/extension_packaging/bundles.py" - require_snapshot_file "scripts/extension_packaging/cli.py" - require_snapshot_file "scripts/extension_packaging/errors.py" - require_snapshot_file "scripts/extension_packaging/model.py" - require_snapshot_file "scripts/extension_packaging/paths.py" - require_snapshot_file "tests/ci/test_package_extension.py" - require_snapshot_file "tests/ci/package_extension_test_support.py" - require_snapshot_file "tests/ci/test_package_extension_packaging.py" - require_snapshot_file "tests/ci/test_package_extension_task.py" - require_snapshot_file "tests/ci/test_python_extension_release.py" - require_snapshot_file "crates/morphir-python-binding/Cargo.toml" - require_snapshot_file "crates/morphir-python-binding/src/lib.rs" - require_snapshot_file "crates/morphir-python-binding/tests/pipeline.rs" - require_snapshot_file "crates/morphir-host-native/tests/python_extension.rs" - require_snapshot_file "crates/morphir-host-native/tests/support/mod.rs" - require_snapshot_file "crates/morphir-host-native/tests/support/installed_wasm.rs" - require_snapshot_file "crates/morphir-host-native/tests/support/mep.rs" - require_snapshot_tree "crates/morphir-python-binding/src" - require_snapshot_tree "crates/morphir-python-binding/tests" - require_snapshot_tree "crates/morphir-distribution/src" - if find "$REPO_ROOT" -type l -print -quit | grep -q .; then - echo "provenance snapshot contains a symbolic link" >&2 - exit 1 - fi -} - -run_pipeline() { - source_root="$1" - output="$2" - git_commit="$3" - ( - cd "$source_root" - cargo test --locked -p morphir-python-binding - cargo build --locked --release -p morphir-python-binding --target wasm32-unknown-unknown - wasm-tools validate target/wasm32-unknown-unknown/release/morphir_python_binding.wasm - set -- scripts/package_extension.py \ - --short-id python \ - --wasm "$WASM" \ - --output "$output" - if [ -n "$git_commit" ]; then - set -- "$@" --git-commit "$git_commit" - fi - python3 "$@" - MORPHIR_PYTHON_BUNDLE="$output" cargo test --locked -p morphir-host-native --test python_extension -- --ignored - ) -} - -if [ "$SNAPSHOT_MODE" = 1 ]; then - if [ -z "${MORPHIR_PYTHON_HEAD_COMMIT:-}" ]; then - echo "MORPHIR_PYTHON_HEAD_COMMIT is required in snapshot mode" >&2 - exit 1 - fi - if git -C "$REPO_ROOT" rev-parse --is-inside-work-tree >/dev/null 2>&1; then - echo "snapshot source must not be inside a Git worktree" >&2 - exit 1 - fi - require_snapshot_inputs - export MISE_CONFIG_FILE="$REPO_ROOT/mise.toml" - python3 scripts/package_extension.py --clean-extension-staging python - run_pipeline "$REPO_ROOT" "$STAGING_DIR" "$MORPHIR_PYTHON_HEAD_COMMIT" - exit 0 -fi - -python3 scripts/package_extension.py --validate-extension-staging python -if ! WORKTREE_STATUS="$(git status --porcelain --untracked-files=normal)"; then - echo "Could not determine Git worktree status" >&2 - exit 1 -fi - -if [ -n "$WORKTREE_STATUS" ]; then - python3 scripts/package_extension.py --clean-extension-staging python - run_pipeline "$REPO_ROOT" "$STAGING_DIR" "" - exit 0 -fi - -HEAD_COMMIT="$(git rev-parse HEAD)" -SNAPSHOT_ROOT="" -SNAPSHOT_ACTIVE=false -cleanup_snapshot() { - if [ "$SNAPSHOT_ACTIVE" = true ]; then - python3 "$REPO_ROOT/scripts/package_extension.py" \ - --clean-head-snapshot "$SNAPSHOT_ROOT" --short-id python - fi -} -trap cleanup_snapshot EXIT -trap 'exit 1' HUP INT TERM - -SNAPSHOT_ROOT="$(mktemp -d "${TMPDIR:-/tmp}/morphir-python-head.XXXXXX")" -SNAPSHOT_ACTIVE=true -SNAPSHOT_ROOT="$(CDPATH='' cd -- "$SNAPSHOT_ROOT" && pwd -P)" -case "$SNAPSHOT_ROOT/" in - "$REPO_ROOT/"*) - echo "HEAD snapshot must be outside repository: $SNAPSHOT_ROOT" >&2 - exit 1 - ;; -esac -mkdir "$SNAPSHOT_ROOT/source" -git archive --format=tar --output="$SNAPSHOT_ROOT/head.tar" "$HEAD_COMMIT" -tar -xf "$SNAPSHOT_ROOT/head.tar" -C "$SNAPSHOT_ROOT/source" - -if find "$SNAPSHOT_ROOT/source" -type l -print -quit | grep -q .; then - echo "provenance snapshot contains a symbolic link" >&2 - exit 1 -fi - -ARCHIVED_TASK="$SNAPSHOT_ROOT/source/.mise/tasks/extension/artifact/python" -if [ ! -f "$ARCHIVED_TASK" ] || [ -L "$ARCHIVED_TASK" ] || [ ! -x "$ARCHIVED_TASK" ]; then - echo "required provenance input is missing or unsafe: .mise/tasks/extension/artifact/python" >&2 - exit 1 -fi -MORPHIR_PYTHON_SNAPSHOT_MODE=1 \ - MORPHIR_PYTHON_HEAD_COMMIT="$HEAD_COMMIT" \ - "$ARCHIVED_TASK" - -SNAPSHOT_BUNDLE="$SNAPSHOT_ROOT/source/.morphir/build/extensions/python" -python3 scripts/package_extension.py --clean-extension-staging python -python3 scripts/package_extension.py \ - --transfer-bundle "$SNAPSHOT_BUNDLE" \ - --output "$STAGING_DIR" - -cleanup_snapshot -SNAPSHOT_ACTIVE=false -trap - EXIT HUP INT TERM diff --git a/.mise/tasks/test/cli-release b/.mise/tasks/test/cli-release index 5827bd771..44602b219 100755 --- a/.mise/tasks/test/cli-release +++ b/.mise/tasks/test/cli-release @@ -1,6 +1,6 @@ #!/bin/sh #MISE description="Publish, install and use an extension bundle through the released morphir CLI" -#USAGE arg "" help="Extension short id: avro, elm-native, gleam, openapi, python or rust" +#USAGE arg "" help="Extension short id: avro, elm-native, gleam, openapi or rust" # The extension bundles built here are hosted by the morphir CLI, which finos/morphir releases. # This task is the compatibility check between the two: it takes the bundle that @@ -19,10 +19,9 @@ case "$ID" in elm-native) EXTENSION_ID=morphir-elm-native ;; gleam) EXTENSION_ID=morphir-gleam ;; openapi) EXTENSION_ID=morphir-openapi ;; - python) EXTENSION_ID=morphir-python ;; rust) EXTENSION_ID=morphir-rust ;; *) - echo "usage: mise run test:cli-release " >&2 + echo "usage: mise run test:cli-release " >&2 exit 2 ;; esac @@ -227,39 +226,6 @@ GLEAM same_json "compiled-v$version/morphir-ir.json" "recompiled-v$version/morphir-ir.json" done ;; - python) - mkdir -p "$PROJECT/src/domain" - cat >"$PROJECT/morphir.toml" <<'TOML' -[project] -name = "acme/example" -version = "0.1.0" -source_directory = "src" -[frontend] -language = "python" -emit_parse_stage = false -TOML - cat >"$PROJECT/src/domain/models.py" <<'PYSRC' -from dataclasses import dataclass -@dataclass(frozen=True) -class Point: - coordinates: tuple[int, int] -PYSRC - cat >"$PROJECT/src/domain/rules.py" <<'PYSRC' -from .models import Point -def choose(flag: bool, first: Point, second: Point) -> Point: - if flag: - return first - else: - return second -PYSRC - morphir compile --language python --input src --output compiled - morphir generate --target python --input compiled/morphir-ir.json --output generated - require_file generated/domain/models.py - require_file generated/domain/rules.py - # Generated source has to compile back to the same IR. - morphir compile --language python --input generated --output recompiled - same_json compiled/morphir-ir.json recompiled/morphir-ir.json - ;; rust) mkdir -p "$PROJECT/src" cat >"$PROJECT/morphir.toml" <<'TOML' diff --git a/CHANGELOG.md b/CHANGELOG.md index 04206eb01..0d5387699 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -288,6 +288,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 (finos/morphir#921). ### Removed +- **Breaking:** The Python extension crate `morphir-python-binding` (bundle + `morphir-python`, short id `python`) moved to + [finos/morphir-python](https://github.com/finos/morphir-python), which now + maintains and releases it. This workspace no longer builds, tests or + releases it: the crate, its `.github/extensions.toml` registration, the + `extension:artifact:python` task, the `python` case of `test:cli-release`, + the CI step for generated Python functions and the Ruff workspace + dependencies are gone, and `docs/tutorials/python-extension.md` is now a + pointer. Anyone who built the crate from morphir-rust must build it from + finos/morphir-python. The `extension/python/v0.1.0` to + `extension/python/v0.4.0` releases stay published here. - From `morphir-daemon`, every duplicate of `morphir-host` and `morphir-host-native`, with the `activation`, `connected`, `container`, `host_functions`, `process`, `protocol`, `registry` and `session` modules diff --git a/Cargo.lock b/Cargo.lock index 3138550c8..3baa967e9 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -510,36 +510,6 @@ version = "1.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" -[[package]] -name = "attribute-derive" -version = "0.10.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05832cdddc8f2650cc2cc187cc2e952b8c133a48eb055f35211f61ee81502d77" -dependencies = [ - "attribute-derive-macro", - "derive-where", - "manyhow", - "proc-macro2", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "attribute-derive-macro" -version = "0.10.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0a7cdbbd4bd005c5d3e2e9c885e6fa575db4f4a3572335b974d8db853b6beb61" -dependencies = [ - "collection_literals", - "interpolator", - "manyhow", - "proc-macro-utils", - "proc-macro2", - "quote", - "quote-use", - "syn 2.0.119", -] - [[package]] name = "autocfg" version = "1.5.1" @@ -1063,18 +1033,6 @@ dependencies = [ "zeroize", ] -[[package]] -name = "char_str" -version = "0.0.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c9cba78a5abb515610e10af297615434b019755b5d7be14e559ff5340dff50c4" -dependencies = [ - "castaway", - "get-size2", - "itoa", - "ryu", -] - [[package]] name = "chrono" version = "0.4.45" @@ -1186,12 +1144,6 @@ dependencies = [ "unicode-width", ] -[[package]] -name = "collection_literals" -version = "1.0.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2550f75b8cfac212855f6b1885455df8eaee8fe8e246b647d69146142e016084" - [[package]] name = "colorchoice" version = "1.0.5" @@ -1222,19 +1174,6 @@ dependencies = [ "static_assertions", ] -[[package]] -name = "compact_str" -version = "0.10.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "79fcda08c33bb58b97008b2cdada6622500e949e060f5913361763121abd2416" -dependencies = [ - "castaway", - "cfg-if", - "itoa", - "static_assertions", - "zmij", -] - [[package]] name = "concurrent-queue" version = "2.5.0" @@ -1854,17 +1793,6 @@ dependencies = [ "syn 2.0.119", ] -[[package]] -name = "derive-where" -version = "1.7.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e2b94854e8576378ccda7c8de8a66ed8b4e8acbd2c50ec3418ea6c8aaf4b567" -dependencies = [ - "proc-macro2", - "quote", - "syn 3.0.6", -] - [[package]] name = "derive_more" version = "2.1.1" @@ -1987,12 +1915,6 @@ version = "2.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "117240f60069e65410b3ae1bb213295bd828f707b5bec6596a1afc8793ce0cbc" -[[package]] -name = "drop_bomb" -version = "0.1.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9bda8e21c04aca2ae33ffc2fd8c23134f3cac46db123ba97bd9d3f3b8a4a85e1" - [[package]] name = "dunce" version = "1.0.5" @@ -2651,40 +2573,6 @@ dependencies = [ "version_check", ] -[[package]] -name = "get-size-derive2" -version = "0.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7efe46d2adb529015748742f1a38594039ce357dbb21e6d178de342cfce468d4" -dependencies = [ - "attribute-derive", - "quote", - "syn 2.0.119", -] - -[[package]] -name = "get-size2" -version = "0.11.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d4d338de392104417fe6fdbe588c52383b9236956d8afe5654f0077176edc632" -dependencies = [ - "compact_str 0.10.0", - "get-size-derive2", - "hashbrown 0.17.1", - "ordermap", - "smallvec", - "thin-vec", -] - -[[package]] -name = "getopts" -version = "0.2.24" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfe4fbac503b8d1f88e6676011885f34b7174f46e59956bba534ba83abded4df" -dependencies = [ - "unicode-width", -] - [[package]] name = "getrandom" version = "0.2.17" @@ -3457,12 +3345,6 @@ dependencies = [ "hybrid-array", ] -[[package]] -name = "interpolator" -version = "0.5.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71dd52191aae121e8611f1e8dc3e324dd0dd1dee1e6dd91d10ee07a3cfb4d9d8" - [[package]] name = "intl-memoizer" version = "0.5.3" @@ -3539,7 +3421,7 @@ dependencies = [ "base64 0.12.3", "bumpalo", "chrono", - "compact_str 0.8.2", + "compact_str", "delegate", "generic-array", "ice_code", @@ -3561,18 +3443,6 @@ version = "2.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a756c3fac73139e83f14c2d742155dd2b78d3ee56597b419a0579b7bdd6dd78" -[[package]] -name = "is-macro" -version = "0.3.8" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8267aa6001e25494f3015f9663bbd88a18240c74483afa5f0934a1b3e4c388e9" -dependencies = [ - "heck 0.5.0", - "proc-macro2", - "quote", - "syn 2.0.119", -] - [[package]] name = "is_terminal_polyfill" version = "1.70.2" @@ -4783,23 +4653,6 @@ dependencies = [ "thiserror 2.0.21", ] -[[package]] -name = "morphir-python-binding" -version = "0.4.0" -dependencies = [ - "cucumber", - "morphir-core", - "morphir-extension-sdk", - "num-bigint 0.5.1", - "ruff_python_ast", - "ruff_python_codegen", - "ruff_python_parser", - "ruff_text_size", - "serde_json", - "tempfile", - "tokio", -] - [[package]] name = "morphir-runtime" version = "0.2.0" @@ -5152,15 +5005,6 @@ dependencies = [ "pin-project-lite", ] -[[package]] -name = "ordermap" -version = "1.2.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "16a756ddcdeb6c0efc8927bdf0ed112885815a35ba350472a467be5104f01ca1" -dependencies = [ - "indexmap", -] - [[package]] name = "ouroboros" version = "0.18.5" @@ -5343,16 +5187,6 @@ dependencies = [ "phf_shared", ] -[[package]] -name = "phf_codegen" -version = "0.11.3" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aef8048c789fa5e851558d709946d6d79a8ff88c0440c587967f8e94bfb1216a" -dependencies = [ - "phf_generator", - "phf_shared", -] - [[package]] name = "phf_generator" version = "0.11.3" @@ -5868,28 +5702,6 @@ dependencies = [ "proc-macro2", ] -[[package]] -name = "quote-use" -version = "0.8.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9619db1197b497a36178cfc736dc96b271fe918875fbf1344c436a7e93d0321e" -dependencies = [ - "quote", - "quote-use-macros", -] - -[[package]] -name = "quote-use-macros" -version = "0.8.4" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "82ebfb7faafadc06a7ab141a6f67bcfb24cb8beb158c6fe933f2f035afa99f35" -dependencies = [ - "proc-macro-utils", - "proc-macro2", - "quote", - "syn 2.0.119", -] - [[package]] name = "r-efi" version = "5.3.0" @@ -6273,111 +6085,6 @@ dependencies = [ "unicode-ident", ] -[[package]] -name = "ruff_python_ast" -version = "0.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7739179c404793f47296e14397478ea03dc07cf3cf6632b3f7afeee80bca10f4" -dependencies = [ - "aho-corasick", - "arrayvec 0.7.8", - "bitflags 2.13.1", - "char_str", - "compact_str 0.10.0", - "get-size2", - "is-macro", - "memchr", - "ruff_python_trivia", - "ruff_source_file", - "ruff_text_size", - "rustc-hash 2.1.3", - "thin-vec", - "thiserror 2.0.21", -] - -[[package]] -name = "ruff_python_codegen" -version = "0.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d17c851523888b26872a2c07d736f50e58d408438afe9a8a64ee8e5677ae5489" -dependencies = [ - "ruff_python_ast", - "ruff_python_literal", - "ruff_python_parser", - "ruff_source_file", - "ruff_text_size", -] - -[[package]] -name = "ruff_python_literal" -version = "0.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6146cbd42f7dd229765b2d97410da14dfb708a7b73a514f6b47db7b4e845a4fc" -dependencies = [ - "bitflags 2.13.1", - "icu_properties", - "itertools 0.15.0", - "ruff_python_ast", -] - -[[package]] -name = "ruff_python_parser" -version = "0.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5e8801707fbf7f737fb8c924cf7c47c32d6bea0f5cb6aabdef70db9cd4ee05cb" -dependencies = [ - "bitflags 2.13.1", - "bstr", - "drop_bomb", - "get-size2", - "hashbrown 0.17.1", - "memchr", - "ruff_python_ast", - "ruff_python_trivia", - "ruff_text_size", - "rustc-hash 2.1.3", - "stacker", - "static_assertions", - "thin-vec", - "unicode-ident", - "unicode-normalization", - "unicode_names2", -] - -[[package]] -name = "ruff_python_trivia" -version = "0.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d6fa97e6048b7e69ca7784e538af46935a9e3dc64d7f30ea412298a385898b26" -dependencies = [ - "itertools 0.15.0", - "memchr", - "ruff_source_file", - "ruff_text_size", - "rustc-hash 2.1.3", - "unicode-ident", - "unicode-normalization", -] - -[[package]] -name = "ruff_source_file" -version = "0.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bff6d1182e0b5d7c276a3fc835c03c3618b8585b1d6ab409d8a7915f44bda465" -dependencies = [ - "memchr", - "ruff_text_size", -] - -[[package]] -name = "ruff_text_size" -version = "0.0.15" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ce8825dec40aec41929c539f419c427885a1005de8bfcf85284220b6fde41766" -dependencies = [ - "get-size2", -] - [[package]] name = "rusqlite" version = "0.40.2" @@ -7395,12 +7102,6 @@ dependencies = [ "unicode-width", ] -[[package]] -name = "thin-vec" -version = "0.2.20" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4568d7e143ec86d2021c338bae2afa88699e84b8e0af523626654fe6f03a1748" - [[package]] name = "thiserror" version = "1.0.69" @@ -7968,28 +7669,6 @@ version = "0.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ebc1c04c71510c7f702b52b7c350734c9ff1295c464a03335b00bb84fc54f853" -[[package]] -name = "unicode_names2" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d1673eca9782c84de5f81b82e4109dcfb3611c8ba0d52930ec4a9478f547b2dd" -dependencies = [ - "phf", - "unicode_names2_generator", -] - -[[package]] -name = "unicode_names2_generator" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b91e5b84611016120197efd7dc93ef76774f4e084cd73c9fb3ea4a86c570c56e" -dependencies = [ - "getopts", - "log", - "phf_codegen", - "rand 0.8.8", -] - [[package]] name = "universal-hash" version = "0.5.1" diff --git a/Cargo.toml b/Cargo.toml index 3ccd5c201..d3e4f086e 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -30,7 +30,6 @@ default-members = [ "crates/morphir-openapi-extension", "crates/morphir-package", "crates/morphir-projection", - "crates/morphir-python-binding", "crates/morphir-runtime", "crates/morphir-rust-binding", "crates/morphir-wasm-binding", @@ -56,11 +55,6 @@ quote = { version = "1", default-features = false } proc-macro2 = { version = "1", default-features = false, features = ["span-locations"] } prettyplease = "0.3.0" num-bigint = "0.5" -# Ruff component crates evolve together; update this group as one unit. -ruff_python_parser = "=0.0.15" -ruff_python_ast = "=0.0.15" -ruff_python_codegen = "=0.0.15" -ruff_text_size = "=0.0.15" # CLI framework starbase = "0.13" clap = { version = "4.5", features = ["derive"] } diff --git a/README.md b/README.md index a64b8680e..0d83fb994 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,6 @@ This table lists the crates in the Cargo workspace. | `morphir-openapi-extension` | OpenAPI and JSON Schema backend that projects Morphir v3/v4 specifications to OpenAPI 3.1, OpenAPI 3.0, and JSON Schema 2020-12 documents. | | `morphir-package` | Experimental model-package normalization, digests, offline schema validation, and closed Library-set verification. | | `morphir-projection` | Shared Morphir IR normalization that backend extensions project from. | -| `morphir-python-binding` | Ruff-based frontend and backend for a Morphir IR v4 subset of Python ADTs, tuples and conditional functions, available natively and as a WebAssembly extension. | | `morphir-runtime` | Reserved runtime crate; currently a minimal scaffold. | | `morphir-rust-binding` | Syn-based Rust frontend and code generator for types, conditional functions, exhaustive pattern matching, calls and typed lambdas in Morphir IR v3 and v4, available natively and as a WebAssembly extension. | | `morphir-tests` | Shared acceptance and Cucumber test harness. | @@ -75,9 +74,13 @@ This table lists independently releasable extensions. Registrations come from | `morphir-elm-native` | `morphir-elm-binding` | `0.2.0` | Compiles Elm type declarations to IR v3 and v4 and generates Elm back; incremental. | | `morphir-gleam` | `morphir-gleam-binding` | `0.3.0` | Native Gleam frontend and backend using the official Gleam parser; IR v3/v4 types and incremental compilation. | | `morphir-openapi` | `morphir-openapi-extension` | `0.2.0` | Generates OpenAPI 3.1, OpenAPI 3.0, and JSON Schema 2020-12 documents from Morphir specifications. | -| `morphir-python` | `morphir-python-binding` | `0.4.0` | Compiles and generates Python ADTs, fixed tuples, conditional functions, typed calls and unary lambdas across multiple modules, using IR v3 or v4. | | `morphir-rust` | `morphir-rust-binding` | `0.3.0` | Compiles and generates Rust types, conditional functions, pattern matches, calls and typed lambdas using IR v3 and v4. | +The Python extension (crate `morphir-python-binding`, bundle `morphir-python`) is now +maintained and released from [finos/morphir-python](https://github.com/finos/morphir-python). +The historical `extension/python/v0.1.0` to `extension/python/v0.4.0` bundles remain +available in the [morphir-rust releases](https://github.com/finos/morphir-rust/releases). + ### Read claims from a built extension Build an extension bundle with `mise run extension:artifact:`, then read the diff --git a/crates/morphir-host-native/tests/python_extension.rs b/crates/morphir-host-native/tests/python_extension.rs deleted file mode 100644 index a63895532..000000000 --- a/crates/morphir-host-native/tests/python_extension.rs +++ /dev/null @@ -1,294 +0,0 @@ -//! Build the Python WASM guest, then run this test with `--ignored`. -//! Set MORPHIR_PYTHON_WASM to override the release artifact path. -//! The packaged lifecycle test also requires MORPHIR_PYTHON_BUNDLE. - -mod support; - -use morphir_extension_sdk::{ - prelude::*, - protocol::{InitializeParams, InitializeResult, PeerInfo, methods}, -}; -use morphir_host::Session; -use morphir_host_native::extism::{ExtensionContainer, MorphirHostFunctions}; -use support::mep::host_config; - -struct PythonExtensionDriver { - container: ExtensionContainer, -} - -#[tokio::test] -#[ignore = "requires MORPHIR_PYTHON_BUNDLE from extension:artifact:python"] -async fn packaged_python_installs_and_roundtrips_offline() { - for version in ["3", "4"] { - packaged_roundtrip(version).await; - } -} - -async fn packaged_roundtrip(version: &str) { - use morphir_common::home::MorphirHome; - use morphir_distribution::{ - Channel, ExtensionId, ExtensionInstaller, LocalExtensionRepository, LocalIndex, Platform, - Selection, activate_installed, - }; - let bundle = - std::env::var_os("MORPHIR_PYTHON_BUNDLE").expect("build the Python release bundle first"); - let root = tempfile::tempdir().unwrap(); - let repository = LocalExtensionRepository::init(root.path().join("repository")).unwrap(); - let publication = repository.publish(bundle).unwrap(); - let artifact = &publication.release().artifacts()[0]; - let claims = artifact.claims().expect("version-2 artifact claims"); - assert!(claims.capabilities.contains_key("frontend")); - assert!(claims.capabilities.contains_key("backend")); - assert_eq!( - artifact.claim_check(), - morphir_distribution::ClaimCheck::Unchecked - ); - let id = ExtensionId::parse("morphir-python").unwrap(); - let home = MorphirHome::resolve_from(Some(root.path().join("home").as_os_str()), None).unwrap(); - let selected = LocalIndex::open(repository.root()) - .unwrap() - .resolve( - &id, - Selection::Channel(Channel::Stable), - &Platform::current(), - &"0.4.0".parse().unwrap(), - ) - .unwrap(); - let installed = ExtensionInstaller::new(&home) - .install(selected, &"0.4.0".parse().unwrap()) - .unwrap(); - assert_eq!( - serde_json::to_value(installed.claims()).unwrap(), - serde_json::to_value(claims).unwrap() - ); - assert_eq!( - installed.claim_check(), - morphir_distribution::ClaimCheck::Unchecked - ); - // Removing this fixture's repository proves activation uses the installed copy. - std::fs::remove_dir_all(repository.root()).unwrap(); - let guest = morphir_host_native::activate(activate_installed(&home, &id).unwrap(), root.path()) - .await - .unwrap(); - let mut session = Session::open( - guest.connection, - &host_config("python-release-test", "1.0.0"), - ) - .await - .unwrap_or_else(|error| panic!("negotiation failed: {error}")); - let capabilities = session.negotiated().capabilities(); - assert_eq!( - capabilities.frontend.as_ref().unwrap().languages[0].id, - "python" - ); - assert_eq!(capabilities.backend.as_ref().unwrap().targets, ["python"]); - - macro_rules! invoke { - ($session:expr, $result:ty, $method:expr, $request:expr) => { - match $session.call::<_, $result>($method, $request).await { - Ok(result) => result, - Err(morphir_host::CallError::Rejected(error)) => { - panic!("request rejected: {error}") - } - Err(error) => panic!("{} (IR {}) failed: {}", $method, version, error), - } - }; - } - let source = concat!( - include_str!("../../morphir-python-binding/tests/fixtures/models.py"), - "\n", - include_str!("../../morphir-python-binding/tests/fixtures/conditionals.py"), - "\n", - include_str!("../../morphir-python-binding/tests/fixtures/tuples.py"), - ); - let request = |documents| CompileRequest { - language_id: "python".into(), - sources: SourceSet { - root: None, - documents, - }, - package: CompilePackage { - name: "acme/example".into(), - exposed_modules: None, - }, - dependencies: vec![], - options: CompileOptions { - ir_version: version.into(), - ..Default::default() - }, - baseline: None, - }; - let compiled = invoke!( - session, - CompileResult, - methods::COMPILE, - request(vec![ - SourceDocument { - uri: "functions.py".into(), - language_id: "python".into(), - version: 1, - text: include_str!("../../morphir-python-binding/tests/fixtures/functions.py") - .into(), - }, - SourceDocument { - uri: "models.py".into(), - language_id: "python".into(), - version: 1, - text: source.into() - }, - SourceDocument { - uri: "rules.py".into(), - language_id: "python".into(), - version: 1, - text: include_str!("../../morphir-python-binding/tests/fixtures/modules/rules.py") - .into() - }, - ]) - ); - assert!(compiled.success, "{:?}", compiled.diagnostics); - let ir = compiled.ir.unwrap(); - let generated = invoke!( - session, - GenerateResult, - methods::GENERATE, - GenerateRequest { - ir: ir.clone(), - target: "python".into(), - options: Default::default(), - } - ); - assert!(generated.success, "{:?}", generated.diagnostics); - assert_eq!(generated.artifacts.len(), 3); - assert!( - generated - .artifacts - .iter() - .any(|artifact| artifact.path == "functions.py" && artifact.content.contains("lambda")) - ); - let again = invoke!( - session, - CompileResult, - methods::COMPILE, - request( - generated - .artifacts - .iter() - .map(|a| SourceDocument { - uri: a.path.clone(), - language_id: "python".into(), - version: 1, - text: a.content.clone(), - }) - .collect() - ) - ); - assert!(again.success, "{:?}", again.diagnostics); - assert_eq!(again.ir, Some(ir)); - session - .close() - .await - .unwrap_or_else(|error| panic!("shutdown failed: {error}")); -} - -impl PythonExtensionDriver { - fn load() -> Self { - let path = std::env::var_os("MORPHIR_PYTHON_WASM") - .map(std::path::PathBuf::from) - .unwrap_or_else(|| { - support::installed_wasm::wasm_guest_path("morphir_python_binding.wasm") - }); - Self { - container: ExtensionContainer::new( - "morphir-python", - &path, - MorphirHostFunctions::default(), - ) - .unwrap(), - } - } - - async fn compile(&self, text: String) -> CompileResult { - self.container - .call( - methods::COMPILE, - CompileRequest { - language_id: "python".into(), - sources: SourceSet { - root: None, - documents: vec![SourceDocument { - uri: "models.py".into(), - language_id: "python".into(), - text, - version: 1, - }], - }, - package: CompilePackage { - name: "acme/example".into(), - exposed_modules: None, - }, - dependencies: vec![], - options: CompileOptions { - ir_version: "4.0.0".into(), - types_only: false, - ..Default::default() - }, - baseline: None, - }, - ) - .await - .unwrap() - } -} - -#[tokio::test] -#[ignore = "requires the independently built morphir-python-binding WASM guest"] -async fn python_adt_and_conditional_roundtrip_through_the_real_wasm_extension() { - let driver = PythonExtensionDriver::load(); - let initialized: InitializeResult = driver - .container - .call( - methods::INITIALIZE, - InitializeParams { - model_transport: None, - protocol_versions: vec!["0.1".into()], - host: PeerInfo { - kind: Default::default(), - name: "python-test".into(), - version: "1.0.0".into(), - }, - }, - ) - .await - .unwrap(); - assert_eq!(initialized.extension.id, "morphir-python"); - assert!(initialized.capabilities.frontend.unwrap().compile); - assert!(initialized.capabilities.backend.unwrap().generate); - let compiled = driver - .compile(format!( - "{}\n{}\n{}", - include_str!("../../morphir-python-binding/tests/fixtures/models.py"), - include_str!("../../morphir-python-binding/tests/fixtures/conditionals.py"), - include_str!("../../morphir-python-binding/tests/fixtures/tuples.py") - )) - .await; - assert!(compiled.success, "{:?}", compiled.diagnostics); - let ir = compiled.ir.unwrap(); - assert!(ir["distribution"]["Library"]["def"]["modules"]["models"]["Public"]["values"]["choose"]["Public"]["ExpressionBody"]["body"].get("IfThenElse").is_some()); - let generated: GenerateResult = driver - .container - .call( - methods::GENERATE, - GenerateRequest { - ir: ir.clone(), - target: "python".into(), - options: Default::default(), - }, - ) - .await - .unwrap(); - assert!(generated.success, "{:?}", generated.diagnostics); - assert_eq!(generated.artifacts[0].path, "models.py"); - let recompiled = driver.compile(generated.artifacts[0].content.clone()).await; - assert!(recompiled.success, "{:?}", recompiled.diagnostics); - assert_eq!(recompiled.ir.unwrap(), ir); -} diff --git a/crates/morphir-python-binding/Cargo.toml b/crates/morphir-python-binding/Cargo.toml deleted file mode 100644 index 8604c2cb4..000000000 --- a/crates/morphir-python-binding/Cargo.toml +++ /dev/null @@ -1,31 +0,0 @@ -[package] -name = "morphir-python-binding" -version = "0.4.0" -edition.workspace = true -rust-version.workspace = true -authors.workspace = true -license.workspace = true -repository.workspace = true -description = "Python ADT, tuple, function and lambda frontend and backend for Morphir" - -[lib] -crate-type = ["rlib", "cdylib"] - -[dependencies] -num-bigint.workspace = true -morphir-core = { path = "../morphir-core", version = "0.2.0" } -morphir-extension-sdk = { path = "../morphir-extension-sdk", version = "0.3.0" } -ruff_python_parser.workspace = true -ruff_python_ast.workspace = true -ruff_python_codegen.workspace = true -ruff_text_size.workspace = true -serde_json = "1.0" - -[dev-dependencies] -tempfile.workspace = true -cucumber = "0.23" -tokio = { version = "1", features = ["macros", "rt-multi-thread"] } - -[[test]] -name = "acceptance" -harness = false diff --git a/crates/morphir-python-binding/README.md b/crates/morphir-python-binding/README.md deleted file mode 100644 index 105cff4ca..000000000 --- a/crates/morphir-python-binding/README.md +++ /dev/null @@ -1,572 +0,0 @@ -# Python models, functions and lambdas - -`morphir-python-binding` implements the `morphir-python` extension in Rust. -It provides MEP `frontend/compile` and `backend/generate` through the Morphir -extension SDK, both natively and as an Extism WebAssembly guest. Python source -is parsed with Ruff and never imported or executed. The backend builds -declarations, parses them into Ruff's AST, and uses Ruff's generator to write -Python. All four direct Ruff component dependencies are pinned together. - -This is an initial implementation of a defined subset of Python 3.12 and later -and Morphir IR v3 and v4. It supports ADT declarations, fixed tuples and annotated pure conditional -functions, typed calls and unary lambdas. It does not implement all Python syntax or all Morphir IR nodes, and -does not claim full Morphir Compatibility Kit conformance for this extension. -It has no dependency on `finos/morphir-python`. - -The examples below describe the published `extension/python/v0.4.0` bundle. The earlier -`extension/python/v0.1.0` bundle does not include multi-module, private-module, IR v3 or higher-order -function support, and Morphir CLI `0.4.0-beta.1` and later cannot compile with it. Use the -[installation guide](../../docs/tutorials/python-extension.md) to install the bundle. - -## A complete model - -Save this as `models.py`. It is valid Python and supported frontend input: - -```python -from __future__ import annotations -from dataclasses import dataclass - -@dataclass(frozen=True) -class Pending: - pass - -@dataclass(frozen=True) -class Approved: - amount: float - -type Decision = Pending | Approved - -@dataclass(frozen=True) -class Application: - decision: Decision - verified: bool - -def choose(flag: bool, first: Decision, second: Decision) -> Decision: - return first if flag else second -``` - -`Decision` becomes a custom type with `Pending` and `Approved` constructors. -`Application` becomes a record type alias. `choose` accepts and returns a -`Decision`; it selects an existing value. Constructing an `Approved` inside a -compiled function is not supported yet. - -After importing the generated module in a Python application, callers can -instantiate its dataclasses and call its functions: - -```python -from models import Approved, Pending, choose - -approved = Approved(amount=125.0) -pending = Pending() -assert choose(True, approved, pending) is approved -assert choose(False, approved, pending) is pending -``` - -This caller is ordinary application code, not input to the Morphir frontend. -The frontend rejects its constructor calls, assignments and assertions. - -## Mapping - -| Python declaration | Morphir definition | -| --- | --- | -| Frozen dataclass outside a sum | Type alias whose body is a record | -| `type Decision = Pending \| Approved` | Custom type with named constructors | -| Dataclass named in that alias | Constructor with its fields as ordered parameters | -| Variant with `pass` | Constructor with no parameters | -| `int`, `float`, `bool`, `str` | SDK `basics#int`, `basics#float`, `basics#bool`, `string#string` | -| `tuple[A, B]` | Tuple type | -| `type Point = tuple[float, float]` | Type alias with a tuple body | -| `(x, y)` or `return x, y` | Tuple value with ordered elements | -| Local or imported type annotation | Fully qualified reference to its defining module in the current package | -| Annotated function | Value definition with typed parameters and an expression body | -| `Callable[[A], B]` | Unary `Function` type from `A` to `B` | -| Named function / function call | Same-package `Reference` / ordered `Apply` nodes | -| `lambda item: body` | `Lambda` with a named wildcard `AsPattern` and a lexically scoped body | -| Returning `if`/`elif`/`else`, or `a if condition else b` | `IfThenElse` with `condition`, `then`, and `else` members | - -The scalar references are exactly `morphir/SDK:basics#int`, -`morphir/SDK:basics#float`, `morphir/SDK:basics#bool`, and -`morphir/SDK:string#string`. With package `acme/example` and file `models.py`, -`Decision` is referenced as `acme/example:models#decision`. - -A variant belongs to exactly one sum. It is not also emitted as a record alias, -and its class name cannot be used as a field type. Refer to its enclosing sum -instead. A one-variant sum is spelled `type Wrapped = Variant`. -Recursive fields work with postponed annotations. These mappings preserve -declarations; Python itself does not enforce annotation types at runtime. - -Each request contains one or more `.py` source documents. A source-relative path -determines its module name: `models.py` becomes `models`, and -`domain/order_items.py` becomes `domain/order-items`. The package name must be a -canonical Morphir package path such as `acme/example`. Omitted `exposedModules` -exposes every module. An explicit empty list makes every module private; a -nonempty list selects the public modules, using canonical paths or dotted names -such as `Domain.OrderItems`. Other modules remain private and usable by siblings. -Unknown exposure names are errors. All emitted types, constructors and functions -within those modules are public. Declaration names use Morphir's -existing name codec, including uppercase initialism segments. Collisions after -normalization are errors. Names must be ASCII identifiers without a leading -underscore; Python keywords and the builtins used by this subset are reserved. -Module filenames also exclude Windows device names such as `con` and `aux`. -Names are checked in their generated spelling too: a field named `Class` would -become the reserved word `class`, so it is rejected. `zip_code` becomes the -Morphir name `zip-code` and generates back as `zip_code`. Type and constructor -names generate in PascalCase; field, function and parameter names use snake_case. - -The backend returns one relative `.py` artifact per module. It rejects names that cannot -roundtrip through this spelling, including collisions across types, constructors -and functions after Morphir name normalization. The common Morphir pattern where -a type and its single constructor share a name therefore needs distinct names -for this first Python subset. - -## Multiple modules and imports - -Module privacy is package metadata, not Python source syntax. Both the frontend -and backend support public and private modules. For a project with -`src/domain/models.py` and `src/domain/rules.py`, this configuration exposes only -the rules module: - -```toml -[project] -name = "acme/example" -version = "0.1.0" -source_directory = "src" -exposed_modules = ["domain.rules"] - -[frontend] -language = "python" -``` - -`rules.py` may import types from the private `models.py`. The backend generates -both files. Keep the exposure configuration when recompiling generated files to -preserve their Morphir visibility; ordinary Python imports do not enforce that -visibility at runtime. Private types, constructors, and functions are still -outside the supported subset. - -Compile all source files together in one request. For example, save the complete -model above as `models.py` and this function as `rules.py`: - -```python -from __future__ import annotations -from models import Decision as Result - -def select_decision(flag: bool, first: Result, second: Result) -> Result: - if flag: - return first - else: - return second -``` - -The function's parameter and result types reference -`acme/example:models#decision`. Import aliases do not create new IR types. -Imported record, sum and tuple types work in fields, constructor parameters and -function signatures. Tuple aliases resolve across modules when checking returns. -The order of source documents does not affect the resulting IR. - -Supported imports include `from models import Decision`, optional `as` aliases, -`import models`, `import domain.models as model`, and relative imports such as -`from .models import Decision` or `from ..models import Decision` inside nested -modules. `from domain import models` also binds a source module. Imports must -resolve to files supplied in the request, and named type or function imports must refer to -declarations in that module. Importing constructors or re-exported -names, wildcard imports, lazy imports and external package imports are rejected. -The standard imports `dataclasses.dataclass`, `__future__.annotations`, and -`Callable` from `typing` or `collections.abc` are supported. A sum's variant -classes must stay in the same module as its alias. - -Relative document paths are relative to the source root. For multiple absolute -paths or file URIs, set `sources.root`; the CLI does this when -compiling a directory. URI queries and fragments do not affect module identity; -percent-encoded path segments are decoded before validating module names. -Dot segments and encoded path separators are rejected. Absolute documents outside that root, module paths that -collide after normalization or case folding, and a file that is also a package directory such as -`models.py` alongside `models/item.py` are errors. A single absolute document -without a root retains the original filename-based behavior. - -Nested modules use Python namespace packages. `__init__.py` modules and package -initialization code are not supported. Generated paths preserve the directory -structure, and generated imports use absolute module paths with collision-free -aliases. Put the generated directory on Python's import path. The backend uses -module imports and postponed annotations, so mutually referring record types -can be generated without eager cross-module type imports. Recursive tuple aliases -remain unsupported. No imports execute during compilation or generation. - -Run `cargo run -p morphir-python-binding --example python_modules -- 3` for a v3 -example, or omit the final argument for v4. Both provide a complete -native MEP example with imported ADTs and tuple aliases. - -## Functions and lambdas - -Both IR v3 and v4 support these declarations: - -```python -from collections.abc import Callable - -def identity(value: int) -> int: - return value - -def apply(transform: Callable[[int], int], value: int) -> int: - return transform(value) - -def retain(value: int) -> Callable[[int], int]: - return lambda ignored: value - -def run(value: int) -> int: - return apply(lambda item: identity(item), value) -``` - -Functions may call other annotated functions in the supplied package, including -forward references, recursion, and functions imported from private modules. -Calls are checked against the declared parameter and return types. Lambdas can -capture enclosing parameters and shadow a parameter with the same source name. -Different spellings that would collapse two lexical bindings to the same Morphir -name are rejected. Generated global function references use module aliases that -avoid parameter and lambda names, including when referencing the current module. -Keep generated files together and import them as modules when executing them. - -Import `Callable` from `collections.abc` or `typing`, optionally with an alias; -qualified forms such as `typing.Callable` also work after importing the module. -`Callable` and `callable` are reserved declaration names. Lambda input types come -from an annotated return, a function argument, or a containing tuple/conditional. -An immediately called lambda can infer its input from its argument. - -Morphir function types are unary. First-class function values therefore use -exactly one parameter: `Callable[[A], B]`, a unary named function, or a unary -lambda. For currying, nest the types and lambdas explicitly: - -```python -def select(flag: bool) -> Callable[[int], Callable[[int], int]]: - return lambda first: lambda second: first if flag else second - -def selected(value: int) -> int: - return select(True)(value)(0) -``` - -Ordinary `def` functions may still have multiple parameters, but their direct -calls must supply all arguments positionally. A zero-parameter `def` call maps -to a value `Reference` and generates as a zero-argument call. Zero-parameter or -multi-parameter definitions cannot themselves be passed as callable values. -Partial applications of multi-parameter definitions, multi-argument `Callable` -annotations or lambdas, keyword arguments, starred arguments, defaults, -variadics, untyped lambda inputs without context, and lambda destructuring -patterns are outside this binding's subset. Nested `def`, assignments and -general Python inference remain unsupported. - -The [function fixture](tests/fixtures/functions.py) is complete frontend input. -The [function tests](tests/functions.rs) cover structural round trips, independently -authored IR and executable Python results. CI also sends these functions through -the packaged WASM extension in both IR versions. - -## Conditional function bodies - -This function can be added to `models.py`: - -```python -def classify(value: int) -> str: - if value < 0: - return "negative" - elif value == 0: - return "zero" - else: - return "positive" -``` - -Functions need parameter and return type annotations. They may return parameters, -`bool`, `int`, `float` or `str` literals, fixed tuples, comparisons, conditional -expressions, typed calls, function references and contextual lambdas. -Parameters can also carry local ADTs or fixed tuples. Conditions must have type -`bool`; both branches must have the same type and match the declared return type. -Scalar comparisons `==`, `!=`, `<`, `<=`, `>` and `>=` map respectively to SDK -Basics `equal`, `not-equal`, `less-than`, `less-than-or-equal`, `greater-than`, and -`greater-than-or-equal`. Each is a `Reference` with two nested `Apply` nodes. -Operands must have matching scalar types; booleans support equality only. - -Nested branches and `elif` chains work. An early-return branch can use a following -return as its fallback, for example `if flag: return first` followed by -`return second`. Every path must return a value. The backend generates explicit -returning `if`/`else` blocks, preserving which branch is evaluated. It may use a -conditional expression inside a condition or comparison operand. - -Function and parameter names generate as snake_case. For IR v4, integer literals preserve -arbitrary precision, including decimal, hexadecimal, octal and binary source -literals. The backend emits decimal integers. Float literals must be finite. Python's implicit -truthiness and numeric coercions are not part of this subset. These restrictions -are checked during compilation and generation. - -The `choose` function in the complete model has this value definition at -`values["choose"]["Public"]`, when compiled with package `acme/example`: - -```json -{ - "ExpressionBody": { - "inputTypes": { - "flag": "morphir/SDK:basics#bool", - "first": "acme/example:models#decision", - "second": "acme/example:models#decision" - }, - "outputType": "acme/example:models#decision", - "body": { - "IfThenElse": { - "condition": {"Variable": "flag"}, - "then": {"Variable": "first"}, - "else": {"Variable": "second"} - } - } - } -} -``` - -This shows the extension's direct output. A host such as the CLI may write the -equivalent expanded type encoding, for example -`{"Reference": {"fqname": "morphir/SDK:basics#bool"}}` in place of the compact -string. Such spelling changes do not change the type. - -For v4, the frontend emits explicit parameter and return types, but does not attach -inferred-type or source-location attributes to each IR value node. Its expression -checks cover the supported subset; they are not a general Morphir type checker. - -## Tuples - -This code can also be added to `models.py`: - -```python -type Point = tuple[float, float] -type LabeledPoint = tuple[Point, str] - -def make_point(x: float, y: float) -> Point: - return (x, y) - -def located(flag: bool, location: Point) -> LabeledPoint: - if flag: - return (location, "known") - else: - return ((0.0, 0.0), "unknown") -``` - -Fixed tuples of two or more elements work in dataclass fields, constructor -parameters, function parameters, return types and function expressions. Elements -can be scalars, local types or nested tuples. Named tuple aliases are expanded -when checking expression types, while their names are retained in IR signatures. -Each tuple element and its position must match the annotation. - -The `Point` alias has `TypeAliasDefinition.typeExp`: - -```json -{"Tuple": ["morphir/SDK:basics#float", "morphir/SDK:basics#float"]} -``` - -The body of `make_point` is a value expression: - -```json -{"Tuple": [{"Variable": "x"}, {"Variable": "y"}]} -``` - -The codec writes the tuple type wrapper shown here; it also accepts the IR v4 -compact bare-array spelling in type positions. A tuple **value** always has a -`Tuple` wrapper. In value positions, a bare array means a list and is outside -this binding's subset. - -Empty tuples, one-element tuples, `tuple[T, ...]`, unparameterized `tuple`, -`typing.Tuple`, `NamedTuple`, starred expansion, tuple indexing, destructuring, -tuple comparison and recursive tuple aliases are not supported. These are limits -of this binding, not claims that Morphir IR cannot represent those concepts. - -## Morphir IR support and conformance - -Both directions support IR v3 and v4 through the shared `morphir-core` models. -The frontend uses one v4 language model and encodes the requested version. The -backend detects the document version, migrates v3 through the shared core API, -and applies the same Python subset checks. IR v1 and v2 are unsupported. -Readers follow the shared compatibility intervals `[3.0.0,3.2.0)` and -`[4.0.0,4.1.0)`, accepting later patches but rejecting later minor versions. -Compilation targets the baseline versions 3.0.0 and 4.0.0. - -| Version | Encoding and attributes | -| --- | --- | -| v3 | `formatVersion: 3`, classic tagged arrays and word-array names; function values carry checked inferred types | -| v4 | `formatVersion: 4`, canonical names and the v4 JSON codec; emitted node attributes remain empty | - -V3 generation accepts typed value annotations, checks them against the supported -expression types, and regenerates them on recompilation. Empty documentation -wrappers normalize to absent documentation. Other documentation and metadata -remain unsupported. Untyped v3 function bodies are not accepted by the current -typed distribution reader. V3 whole-number literals must fit the shared codec's -signed 64-bit range, from -9223372036854775808 to 9223372036854775807. Larger -Python integer literals produce a diagnostic for v3 and remain supported in v4. -These are current binding/codec limits, not limits attributed to Morphir IR. -V3 compilation also checks that every name survives the shared legacy name -conversion. For example, `a_b` would become the initialism `AB`, so it is rejected -with `PY003` for v3; rename it or target v4. Initialisms such as `APIResponse` -remain supported. - -The type and value subset below applies to both versions; the displayed JSON -examples elsewhere in this README use v4. - -| IR area | Frontend emits / backend accepts | -| --- | --- | -| Distribution | `Library`, one or more public or private modules, empty dependency map; definitions within modules are public | -| Type definitions | Non-generic record aliases, fixed tuple aliases, custom types with public constructors | -| Type expressions | The four SDK scalar references, same-package type references across modules, fixed tuples, unary `Function`; a record at a record-alias body | -| Value definitions | V4 `ExpressionBody`, or the classic v3 value-definition object, with annotated inputs and a required output type | -| Value expressions | Parameter `Variable`, scalar `Literal`, fixed `Tuple`, `IfThenElse`, same-package function `Reference`, typed `Apply`, unary `Lambda`, and fully applied two-argument SDK scalar comparisons | -| Lambda patterns | `AsPattern` over `WildcardPattern`, binding one parameter; v3 pattern annotations are checked | -| Literal kinds | `BoolLiteral`, finite `FloatLiteral`, `StringLiteral`; v4 arbitrary-precision `IntegerLiteral` or v3 signed 64-bit `WholeNumberLiteral` | -| Metadata | Empty type attributes; v4 empty value attributes or v3 checked inferred types. Nonempty documentation and other retained metadata are rejected | - -Unsupported IR includes `Specs` and `Application` distributions, dependencies, -private types, constructors and values, generic types, opaque types, empty custom types, extensible -records, unit types, free type variables, and other SDK types such -as List, Maybe and Decimal. Arbitrary aliases such as an alias directly to `int` -are not supported. Type references must resolve within the supplied package. - -Unsupported value forms include record construction or field access, -constructor application, external calls or references, lists, arithmetic, -let bindings, general pattern matching, record updates, holes, and -`NativeBody`, `ExternalBody` or `IncompleteBody` definitions. A valid Morphir IR -document can therefore still be outside this extension's supported subset. - -For supported models, the round-trip contract is preservation of the model -through Python → IR → Python → IR. Tests compare compiled IR structurally and -also exercise independently authored IR inputs. This is not source-text or -byte-for-byte JSON preservation: formatting, comments, declaration spelling, -declaration order and accepted alternate JSON encodings may normalize. Numeric -literals may be respelled. The extension does not preserve arbitrary metadata -that the shared IR reader ignores. - -Current evidence is scoped to the checked-in cases: - -| Evidence | What it establishes | -| --- | --- | -| [ADT integration tests](tests/pipeline.rs) | Expected IR mappings, independent custom-type input, recursive references, naming and rejection boundaries | -| [Function and tuple integration tests](tests/conditionals.rs) | Exact `IfThenElse`/tuple encodings, alias expansion, branch/type validation and round-trips | -| [Functions and lambdas](tests/functions.rs) | V3/v4 calls, closures, independent IR, annotation and scope checks; opt-in Python execution runs in CI | -| [Version integration tests](tests/ir_versions.rs) | V3/v4 roundtrips, private imports, independently authored v3 input, integer bounds and annotation validation | -| [Module integration tests](tests/modules.rs) | Absolute and relative imports, nested modules, cyclic record references, import collisions and cross-module tuple checking | -| [Acceptance scenarios](tests/features/adt.feature) | Supported models pass through the public extension API; unsupported input returns diagnostics | -| [WASM host test](../morphir-host-native/tests/python_extension.rs) | Capability negotiation and compile/generate/recompile through an actual Extism guest; run by CI | -| Example verification | Generated example IR checked against the v4 JSON Schema, and selected generated Python branch/tuple results checked in Python 3.14.7 | - -The repository's [Kit conformance job](../../.github/workflows/ci.yml) runs the -shared MCK against the Rust adapter. It does not certify the Python extension's -language mapping. There is currently no complete Python-extension MCK report or -full Python/Morphir semantic equivalence claim. Generated Python type annotations -also do not enforce argument types at runtime. - -## Current boundary - -Supported fields are scalars, same-package references, unary `Callable` types and -fixed tuples of at least two elements. The imports described above, frozen dataclasses, non-generic -`type` aliases of dataclass variants or fixed tuples, and annotated pure functions -are the accepted module statements. -Comments and whitespace are not preserved. Methods, field defaults, -inheritance, arbitrary decorators or imports, docstrings, generic parameters, -containers, optional fields, quoted annotations and cross-package dependencies -are rejected. Constructor calls, assignments, loops, bare returns, -decorated or async functions, parameter defaults, variadic parameters, -positional-only or keyword-only parameters, chained comparisons, and boolean -operators are not supported yet. The backend also rejects private types, constructors and values, documentation, -unsupported metadata, dependencies and IR type forms outside this subset. Failures return -diagnostics with no partial IR or artifacts. - -For example, this is valid Python but rejected with `PY004`, because an integer -condition would use Python's implicit truthiness: - -```python -def invalid(value: int) -> int: - return 1 if value else 0 -``` - -Other rejected examples are `return (1, 2)` for `-> tuple[int, str]`, an `if` -whose false path reaches the end of the function without a return, and -`type Point = tuple[float, ...]`. No incomplete model is emitted for these cases. - -Compile functions with `typesOnly=false`. A request containing functions with -`typesOnly=true` returns a diagnostic; bodies are never silently dropped. ADT-only -sources accept either value. `irVersion` accepts `3`, `3.0.0`, `4` or `4.0.0`. The CLI's -`outputDir` string option is accepted as context; `sources.root` determines -module paths for absolute document URIs. Neither grants filesystem access. -`emitParseStage=true` produces warning `PY006`; combining it -with `emitParseStageFatal=true` fails. Other options are rejected. - -| Code | Meaning | -| --- | --- | -| `PY001` | Unsupported request or option | -| `PY002` | Python syntax error, with a source range | -| `PY003` | Invalid name or naming collision | -| `PY004` | Unsupported declaration, annotation or IR feature | -| `PY005` | IR decoding or serialization, or generated syntax failure | -| `PY006` | Parse-stage output was requested but is unavailable | - -Semantic lowering diagnostics identify the source module; request-option -diagnostics can have no source location. Syntax diagnostics -identify the parser's failing range. Source positions use the SDK's UTF-16 -convention. - -## Run the pipeline - -From the `morphir-rust` checkout: - -```console -cargo test --locked -p morphir-python-binding -cargo run --locked -p morphir-python-binding --example python_adt -rustup target add wasm32-unknown-unknown -cargo build --locked --release -p morphir-python-binding --target wasm32-unknown-unknown -cargo test --locked -p morphir-host-native --test python_extension -- --ignored --exact python_adt_and_conditional_roundtrip_through_the_real_wasm_extension -``` - -The example prints JSON containing compiled IR and generated source for the -[ADT](tests/fixtures/models.py), [conditional](tests/fixtures/conditionals.py), and -[tuple](tests/fixtures/tuples.py) fixtures. Its values and names differ from the -shorter README model. To compile the README model directly, save its complete -model block as `models.py` and run: - -```console -cargo run --locked -p morphir-python-binding --example python_adt -- models.py -``` - -The optional path selects one source file; the example uses package `acme/example` -and returns IR plus generated Python through native MEP. File reading belongs to -this example program, not the extension. The integration test invokes both -capabilities through the actual WASM guest. -`MORPHIR_PYTHON_WASM` can select a different guest file for that test. -The native `NativeExtension::frontend_backend(PythonExtension)` adapter exposes -the same MEP methods without requiring a WASM runtime. - -Build the installable release bundle with `mise run extension:artifact:python`. -This runs native tests, builds and validates the WASM guest, packages its checksum -and descriptor, and tests publication, installation and both capabilities with a -fresh Morphir Home after removing the source repository. The three bundle files -are written to `.morphir/build/extensions/python/`. - -CI uploads these files as `morphir-python-extension-bundle`. Tags such as -`extension/python/v0.4.0` publish the same tested bundle as GitHub release assets. -The descriptor declares language `python` (`.py`), target `python`, MEP `0.1` -and IR `3` and `4`; these declarations cover only the subset documented above. The -release is a Morphir WASM extension, not a PyPI package or a CPython import module. - -For installation with the parent Morphir CLI, see the -[Python extension guide](../../docs/tutorials/python-extension.md). - -## Specification sources - -The implementation uses `morphir-core::ir::v4` rather than a separate IR model. -The parent repository's [IR knowledgebase](https://github.com/finos/morphir/tree/main/kb/bundles/morphir/morphir-ir) -records the naming, record-field, constructor-parameter and SDK package decisions. -The [IR specification](https://morphir.finos.org/docs/spec/ir/) and the -[extension tutorial](../../docs/tutorials/extension-tutorial.md) describe the -contracts this crate uses. Ruff's [parser](https://github.com/astral-sh/ruff/tree/main/crates/ruff_python_parser) -and [generator](https://github.com/astral-sh/ruff/tree/main/crates/ruff_python_codegen) -remain internal component crates; upgrades require running both native and WASM tests. - -### Bundle descriptor compatibility - -Newly built bundles include a version-2 `release.json` with capability claims -read from the exact WASM being packaged and checked against -`.github/extensions.toml`. Publishing requires Morphir CLI `0.4.0-beta.7` or -later; older CLIs cannot read this descriptor. The versioned WASM and -`.wasm.sha256` names are unchanged. The downloaded descriptor is still named -`-.release.json`; rename it to `release.json` before -publishing. WASM installs show `Claims: unchecked`. - -See the [packaging contract](../../docs/extensions/capability-claims.md#packaging-wasm-extensions) -for the descriptor shape and provenance rules. diff --git a/crates/morphir-python-binding/examples/python_adt.rs b/crates/morphir-python-binding/examples/python_adt.rs deleted file mode 100644 index f16482f6d..000000000 --- a/crates/morphir-python-binding/examples/python_adt.rs +++ /dev/null @@ -1,85 +0,0 @@ -//! Print a fixture's IR and generated Python using the same MEP endpoint as a host. -use morphir_extension_sdk::{prelude::*, protocol::methods}; -use morphir_python_binding::PythonExtension; - -fn main() -> Result<()> { - let extension = NativeExtension::frontend_backend(PythonExtension)?; - let (uri, text) = match std::env::args().nth(1) { - Some(path) => { - let text = std::fs::read_to_string(&path) - .map_err(|error| ExtensionError::ExecutionFailed(format!("{path}: {error}")))?; - (path, text) - } - None => ( - "models.py".into(), - format!( - "{}\n{}\n{}", - include_str!("../tests/fixtures/models.py"), - include_str!("../tests/fixtures/conditionals.py"), - include_str!("../tests/fixtures/tuples.py") - ), - ), - }; - let request = CompileRequest { - language_id: "python".into(), - sources: SourceSet { - root: None, - documents: vec![SourceDocument { - uri, - language_id: "python".into(), - version: 1, - text, - }], - }, - package: CompilePackage { - name: "acme/example".into(), - exposed_modules: None, - }, - dependencies: vec![], - options: CompileOptions { - types_only: false, - ir_version: "4".into(), - ..Default::default() - }, - baseline: None, - }; - let compiled = - extension - .protocol() - .handle(ExtensionRequest::new(methods::COMPILE, request, 1)?); - let compiled: CompileResult = serde_json::from_value( - compiled - .result - .ok_or_else(|| ExtensionError::ExecutionFailed(format!("{:?}", compiled.error)))?, - )?; - let ir = compiled - .ir - .ok_or_else(|| ExtensionError::ExecutionFailed(format!("{:?}", compiled.diagnostics)))?; - let generated = extension.protocol().handle(ExtensionRequest::new( - methods::GENERATE, - GenerateRequest { - ir: ir.clone(), - target: "python".into(), - options: Default::default(), - }, - 2, - )?); - let generated: GenerateResult = serde_json::from_value( - generated - .result - .ok_or_else(|| ExtensionError::ExecutionFailed(format!("{:?}", generated.error)))?, - )?; - if !generated.success { - return Err(ExtensionError::ExecutionFailed(format!( - "{:?}", - generated.diagnostics - ))); - } - println!( - "{}", - serde_json::to_string_pretty( - &serde_json::json!({ "ir": ir, "artifacts": generated.artifacts }) - )? - ); - Ok(()) -} diff --git a/crates/morphir-python-binding/examples/python_modules.rs b/crates/morphir-python-binding/examples/python_modules.rs deleted file mode 100644 index 8a39e3c14..000000000 --- a/crates/morphir-python-binding/examples/python_modules.rs +++ /dev/null @@ -1,81 +0,0 @@ -//! Compile and generate multiple modules with imported ADTs and tuple aliases through MEP. -use morphir_extension_sdk::{prelude::*, protocol::methods}; -use morphir_python_binding::PythonExtension; - -fn main() -> Result<()> { - let extension = NativeExtension::frontend_backend(PythonExtension)?; - let request = CompileRequest { - language_id: "python".into(), - sources: SourceSet { - root: None, - documents: vec![ - SourceDocument { - uri: "models.py".into(), - language_id: "python".into(), - version: 1, - text: format!( - "{}\n{}", - include_str!("../tests/fixtures/models.py"), - include_str!("../tests/fixtures/tuples.py") - ), - }, - SourceDocument { - uri: "rules.py".into(), - language_id: "python".into(), - version: 1, - text: include_str!("../tests/fixtures/modules/rules.py").into(), - }, - ], - }, - package: CompilePackage { - name: "acme/example".into(), - exposed_modules: None, - }, - dependencies: vec![], - options: CompileOptions { - types_only: false, - ir_version: std::env::args().nth(1).unwrap_or_else(|| "4".into()), - ..Default::default() - }, - baseline: None, - }; - let compiled = - extension - .protocol() - .handle(ExtensionRequest::new(methods::COMPILE, request, 1)?); - let compiled: CompileResult = serde_json::from_value( - compiled - .result - .ok_or_else(|| ExtensionError::ExecutionFailed(format!("{:?}", compiled.error)))?, - )?; - let ir = compiled - .ir - .ok_or_else(|| ExtensionError::ExecutionFailed(format!("{:?}", compiled.diagnostics)))?; - let generated = extension.protocol().handle(ExtensionRequest::new( - methods::GENERATE, - GenerateRequest { - ir: ir.clone(), - target: "python".into(), - options: Default::default(), - }, - 2, - )?); - let generated: GenerateResult = serde_json::from_value( - generated - .result - .ok_or_else(|| ExtensionError::ExecutionFailed(format!("{:?}", generated.error)))?, - )?; - if !generated.success { - return Err(ExtensionError::ExecutionFailed(format!( - "{:?}", - generated.diagnostics - ))); - } - println!( - "{}", - serde_json::to_string_pretty( - &serde_json::json!({ "ir": ir, "artifacts": generated.artifacts }) - )? - ); - Ok(()) -} diff --git a/crates/morphir-python-binding/src/backend.rs b/crates/morphir-python-binding/src/backend.rs deleted file mode 100644 index 94f75e233..000000000 --- a/crates/morphir-python-binding/src/backend.rs +++ /dev/null @@ -1,300 +0,0 @@ -use crate::{Outcome, error, names}; -use morphir_core::{ir::v4::*, naming::FQName}; -use morphir_extension_sdk::{Artifact, GenerateRequest}; -use std::collections::{BTreeMap, BTreeSet}; - -mod functions; -mod imports; - -pub(crate) fn generate(request: &GenerateRequest) -> Outcome> { - if request.target != "python" || !request.options.is_empty() { - return Err(error( - "PY001", - "Expected target python without backend options", - )); - } - let ir = crate::ir::decode(request.ir.clone())?; - let Distribution::Library(library) = ir.distribution else { - return Err(error("PY004", "Only Library distributions are supported")); - }; - if !library.dependencies.is_empty() || library.def.modules.is_empty() { - return Err(error( - "PY004", - "Expected at least one module with no dependencies", - )); - } - crate::modules::validate_paths(library.def.modules.keys().map(String::as_str))?; - let identities = library - .def - .modules - .keys() - .map(|name| { - Ok(( - name.clone(), - crate::modules::ModuleIdentity::from_canonical(name)?, - )) - }) - .collect::>>()?; - let aliases = crate::modules::tuple_aliases(&library.package_name, library.def.modules.iter())?; - let signatures = crate::values::signatures(&library)?; - library - .def - .modules - .iter() - .map(|(name, module)| { - render_module(&library, name, module, &identities, &aliases, &signatures) - }) - .collect() -} - -fn render_module( - library: &LibraryContent, - module_name: &str, - module: &AccessControlled, - identities: &BTreeMap, - tuple_aliases: &crate::values::TupleAliases, - signatures: &crate::values::Signatures, -) -> Outcome { - if module.value.doc.is_some() { - return Err(error( - "PY004", - "Module documentation is not supported by the Python backend", - )); - } - let mut symbols = BTreeSet::new(); - let mut type_names = BTreeMap::new(); - for name in module.value.types.keys() { - let python = - names::type_name(&Name::from_canonical_string(name).map_err(|e| error("PY003", e))?)?; - reserve(&mut symbols, &python)?; - type_names.insert(name.clone(), python); - } - let mut source = String::from( - "from __future__ import annotations\nfrom dataclasses import dataclass\nfrom collections.abc import Callable\n\n", - ); - let mut value_names = BTreeMap::new(); - let imported = imports::prepare( - library, - module_name, - identities, - &mut type_names, - &mut value_names, - )?; - let functions = functions::Context { - names: &value_names, - signatures, - }; - source.push_str(&imported); - let mut aliases = vec![]; - - for (name, definition) in &module.value.types { - require_public(definition.access)?; - if definition.value.doc.is_some() { - return Err(error("PY004", "Type documentation is not supported yet")); - } - let python = &type_names[name]; - match &definition.value.value { - TypeDefinition::TypeAliasDefinition { - type_params, - type_expr: tpe @ Type::Tuple(..), - } if type_params.is_empty() => { - aliases.push(format!( - "type {python} = {}\n", - annotation(tpe, &library.package_name, module_name, &type_names)? - )); - } - TypeDefinition::TypeAliasDefinition { - type_params, - type_expr: Type::Record(attrs, fields), - } if type_params.is_empty() => { - require_empty_attributes(attrs)?; - render_class( - &mut source, - python, - fields.iter().map(|f| (&f.name, &f.tpe)), - &library.package_name, - module_name, - &type_names, - )?; - } - TypeDefinition::CustomTypeDefinition { - type_params, - constructors, - } if type_params.is_empty() => { - require_public(constructors.access)?; - if constructors.value.is_empty() { - return Err(error( - "PY004", - "Empty custom types cannot be represented by a Python union", - )); - } - let mut variants = vec![]; - for constructor in &constructors.value { - let variant = names::type_name(&constructor.name)?; - reserve(&mut symbols, &variant)?; - render_class( - &mut source, - &variant, - constructor.args.iter().map(|a| (&a.name, &a.arg_type)), - &library.package_name, - module_name, - &type_names, - )?; - variants.push(variant); - } - aliases.push(format!("type {python} = {}\n", variants.join(" | "))); - } - _ => { - return Err(error( - "PY004", - "Only non-generic record or tuple aliases and custom types with public constructors are supported", - )); - } - } - } - source.push_str(&aliases.join("\n")); - for (name, definition) in &module.value.values { - require_public(definition.access)?; - if definition.value.doc.is_some() { - return Err(error( - "PY004", - "Function documentation is not supported yet", - )); - } - let name = - names::field_name(&Name::from_canonical_string(name).map_err(|e| error("PY003", e))?)?; - reserve(&mut symbols, &name)?; - source.push_str(&functions::render( - &name, - &definition.value.value, - &library.package_name, - module_name, - &type_names, - tuple_aliases, - &functions, - )?); - } - // Ruff validates the constructed declarations and owns AST-to-source rendering. - // No Python interpreter or user imports run during generation. - let content = ruff_python_codegen::round_trip(&source) - .map_err(|e| error("PY005", format!("Generated Python is invalid: {e}")))?; - Ok(Artifact { - path: identities[module_name].filename(), - content: format!("{}\n", content.trim_end()), - binary: false, - }) -} - -fn render_class<'a>( - source: &mut String, - name: &str, - fields: impl Iterator, - package: &PackageName, - module: &str, - types: &BTreeMap, -) -> Outcome<()> { - source.push_str(&format!("@dataclass(frozen=True)\nclass {name}:\n")); - let mut seen = BTreeSet::new(); - for (name, tpe) in fields { - let field = names::field_name(name)?; - reserve(&mut seen, &field)?; - source.push_str(&format!( - " {field}: {}\n", - annotation(tpe, package, module, types)? - )); - } - if seen.is_empty() { - source.push_str(" pass\n"); - } - source.push('\n'); - Ok(()) -} - -fn annotation( - tpe: &Type, - package: &PackageName, - module: &str, - types: &BTreeMap, -) -> Outcome { - require_empty_attributes(tpe.attributes())?; - match tpe { - Type::Function(_, input, output) => Ok(format!( - "Callable[[{}], {}]", - annotation(input, package, module, types)?, - annotation(output, package, module, types)? - )), - Type::Reference(_, fq, args) if args.is_empty() => reference(fq, package, module, types), - Type::Tuple(_, items) if items.len() >= 2 => Ok(format!( - "tuple[{}]", - items - .iter() - .map(|t| annotation(t, package, module, types)) - .collect::>>()? - .join(", ") - )), - _ => Err(error( - "PY004", - "Unsupported Python field type; expected scalar, local reference, or fixed tuple", - )), - } -} - -fn reference( - fq: &FQName, - package: &PackageName, - module: &str, - types: &BTreeMap, -) -> Outcome { - let canonical = fq.to_canonical_string(); - let scalar = match canonical.as_str() { - "morphir/SDK:basics#int" => Some("int"), - "morphir/SDK:basics#float" => Some("float"), - "morphir/SDK:basics#bool" => Some("bool"), - "morphir/SDK:string#string" => Some("str"), - _ => None, - }; - if let Some(scalar) = scalar { - return Ok(scalar.into()); - } - if let Some(imported) = types.get(&canonical) { - return Ok(imported.clone()); - } - let prefix = format!("{}:{module}#", package.to_canonical_string()); - canonical - .strip_prefix(&prefix) - .and_then(|local| types.get(local)) - .cloned() - .ok_or_else(|| { - error( - "PY004", - format!("Unresolved or external type reference: {canonical}"), - ) - }) -} - -fn reserve(seen: &mut BTreeSet, name: &str) -> Outcome<()> { - if !seen.insert(names::identifier(name)?.to_canonical_string()) { - return Err(error("PY003", format!("Python name collision: {name}"))); - } - Ok(()) -} - -fn require_public(access: Access) -> Outcome<()> { - if access != Access::Public { - return Err(error( - "PY004", - "Private definitions are not supported by this Python subset", - )); - } - Ok(()) -} - -fn require_empty_attributes(attrs: &TypeAttributes) -> Outcome<()> { - if *attrs != TypeAttributes::default() { - return Err(error( - "PY004", - "Type attributes cannot yet be preserved in Python", - )); - } - Ok(()) -} diff --git a/crates/morphir-python-binding/src/backend/functions.rs b/crates/morphir-python-binding/src/backend/functions.rs deleted file mode 100644 index 67c959028..000000000 --- a/crates/morphir-python-binding/src/backend/functions.rs +++ /dev/null @@ -1,160 +0,0 @@ -use super::{annotation, reserve}; -use crate::{ - Outcome, error, names, - values::{self, unsupported}, -}; -use morphir_core::ir::v4::*; -use std::collections::{BTreeMap, BTreeSet}; - -pub(super) fn render( - name: &str, - definition: &ValueDefinition, - package: &PackageName, - module: &str, - types: &BTreeMap, - aliases: &values::TupleAliases, - context: &Context<'_>, -) -> Outcome { - let ValueBody::Expression(body) = &definition.body else { - return Err(unsupported( - "Only expression function bodies can be generated as Python", - )); - }; - let output_type = definition - .output_type - .as_ref() - .ok_or_else(|| unsupported("Function return type is required"))?; - let output = annotation(output_type, package, module, types)?; - let mut seen = BTreeSet::new(); - let inputs = definition - .input_types - .iter() - .map(|(name, entry)| { - let parameter_name = - Name::from_canonical_string(name).map_err(|e| error("PY003", e))?; - let python = names::field_name(¶meter_name)?; - reserve(&mut seen, &python)?; - Ok(format!( - "{python}: {}", - annotation(entry, package, module, types)? - )) - }) - .collect::>>()?; - values::validate_function(definition, aliases, context.signatures)?; - let mut source = format!("\ndef {name}({}) -> {output}:\n", inputs.join(", ")); - context.render_body(&mut source, body, 1)?; - Ok(source) -} - -pub(super) struct Context<'a> { - pub(super) names: &'a BTreeMap, - pub(super) signatures: &'a values::Signatures, -} - -impl Context<'_> { - fn render_body(&self, source: &mut String, body: &Value, depth: usize) -> Outcome<()> { - let indent = " ".repeat(depth); - match body { - Value::IfThenElse(_, condition, then_branch, else_branch) => { - source.push_str(&format!("{indent}if {}:\n", self.expression(condition)?)); - self.render_body(source, then_branch, depth + 1)?; - source.push_str(&format!("{indent}else:\n")); - self.render_body(source, else_branch, depth + 1)?; - } - _ => source.push_str(&format!("{indent}return {}\n", self.expression(body)?)), - } - Ok(()) - } - - fn expression(&self, value: &Value) -> Outcome { - match value { - Value::Lambda(_, Pattern::AsPattern(_, _, name), body) => Ok(format!( - "(lambda {}: {})", - names::field_name(name)?, - self.expression(body)? - )), - Value::Reference(_, name) => { - let signature = &self.signatures[&name.to_canonical_string()]; - let python = self.reference(name)?; - match signature.inputs.len() { - 0 => Ok(format!("{python}()")), - 1 => Ok(python), - _ => Err(unsupported( - "Only unary named functions can be used as callable values", - )), - } - } - Value::Variable(_, name) => names::field_name(name), - Value::Tuple(_, elements) => Ok(format!( - "({})", - elements - .iter() - .map(|value| self.expression(value)) - .collect::>>()? - .join(", ") - )), - Value::Literal(_, Literal::Bool(value)) => { - Ok(if *value { "True" } else { "False" }.into()) - } - Value::Literal(_, Literal::Integer(value)) => Ok(value.to_string()), - Value::Literal(_, Literal::Float(value)) => Ok(format!("{:?}", value.value())), - // JSON escaping is also valid in a Python string literal, including control characters. - Value::Literal(_, Literal::String(value)) => { - serde_json::to_string(value).map_err(|e| error("PY005", e.to_string())) - } - Value::IfThenElse(_, condition, then_branch, else_branch) => Ok(format!( - "({} if {} else {})", - self.expression(then_branch)?, - self.expression(condition)?, - self.expression(else_branch)? - )), - Value::Apply(..) => { - if let Ok((operator, left, right)) = values::comparison(value) { - return Ok(format!( - "({} {} {})", - self.expression(left)?, - operator.python(), - self.expression(right)? - )); - } - let mut head = value; - let mut arguments = vec![]; - while let Value::Apply(_, function, argument) = head { - arguments.push(argument.as_ref()); - head = function; - } - arguments.reverse(); - let (mut source, consumed) = if let Value::Reference(_, name) = head { - let count = self.signatures[&name.to_canonical_string()].inputs.len(); - if arguments.len() < count { - return Err(unsupported( - "Partial applications of multi-parameter functions are not supported", - )); - } - let args = arguments[..count] - .iter() - .map(|arg| self.expression(arg)) - .collect::>>()?; - ( - format!("{}({})", self.reference(name)?, args.join(", ")), - count, - ) - } else { - (self.expression(head)?, 0) - }; - for argument in &arguments[consumed..] { - source = format!("{source}({})", self.expression(argument)?); - } - Ok(source) - } - _ => Err(unsupported("Unsupported Python function expression")), - } - } - - fn reference(&self, name: &morphir_core::naming::FQName) -> Outcome { - self.names - .get(&name.to_canonical_string()) - .cloned() - .ok_or_else(|| unsupported("Unknown function reference")) - } -} diff --git a/crates/morphir-python-binding/src/backend/imports.rs b/crates/morphir-python-binding/src/backend/imports.rs deleted file mode 100644 index a1830556c..000000000 --- a/crates/morphir-python-binding/src/backend/imports.rs +++ /dev/null @@ -1,140 +0,0 @@ -//! Generate module-qualified type annotations to avoid imported name collisions. - -use crate::{Outcome, error, modules::ModuleIdentity, names}; -use morphir_core::ir::v4::*; -use std::collections::{BTreeMap, BTreeSet}; - -pub(super) fn prepare( - library: &LibraryContent, - current: &str, - identities: &BTreeMap, - scope: &mut BTreeMap, - values: &mut BTreeMap, -) -> Outcome { - let definition = &library.def.modules[current].value; - let mut used: BTreeSet<_> = definition - .types - .keys() - .chain(definition.values.keys()) - .cloned() - .collect(); - let mut references = BTreeSet::new(); - let mut value_references = BTreeSet::new(); - for entry in definition.types.values() { - match &entry.value.value { - TypeDefinition::TypeAliasDefinition { type_expr, .. } => { - collect(type_expr, &mut references) - } - TypeDefinition::CustomTypeDefinition { constructors, .. } => { - for constructor in &constructors.value { - used.insert(constructor.name.to_canonical_string()); - for arg in &constructor.args { - collect(&arg.arg_type, &mut references); - } - } - } - _ => {} - } - } - for entry in definition.values.values() { - used.extend(entry.value.value.input_types.keys().cloned()); - if let ValueBody::Expression(body) = &entry.value.value.body { - collect_value(body, &mut value_references, &mut used); - } - for input in entry.value.value.input_types.values() { - collect(input, &mut references); - } - if let Some(output) = &entry.value.value.output_type { - collect(output, &mut references); - } - } - let mut source = String::new(); - let mut next_alias = 1; - for (module, identity) in identities { - let prefix = format!("{}:{module}#", library.package_name.to_canonical_string()); - if !(module != current && references.iter().any(|fq| fq.starts_with(&prefix))) - && !value_references.iter().any(|fq| fq.starts_with(&prefix)) - { - continue; - } - let alias = loop { - let candidate = format!("morphir_module_{next_alias}"); - next_alias += 1; - if used.insert(Name::from(candidate.as_str()).to_canonical_string()) { - break candidate; - } - }; - source.push_str(&format!("import {} as {alias}\n", identity.python)); - for name in library.def.modules[module].value.types.keys() { - let python = names::type_name( - &Name::from_canonical_string(name).map_err(|e| error("PY003", e))?, - )?; - scope.insert(format!("{prefix}{name}"), format!("{alias}.{python}")); - } - for name in library.def.modules[module].value.values.keys() { - let python = names::field_name( - &Name::from_canonical_string(name).map_err(|e| error("PY003", e))?, - )?; - values.insert(format!("{prefix}{name}"), format!("{alias}.{python}")); - } - } - if !source.is_empty() { - source.push('\n'); - } - Ok(source) -} - -fn collect(tpe: &Type, references: &mut BTreeSet) { - match tpe { - Type::Function(_, input, output) => { - collect(input, references); - collect(output, references); - } - Type::Reference(_, fq, args) => { - references.insert(fq.to_canonical_string()); - for arg in args { - collect(arg, references); - } - } - Type::Tuple(_, items) => { - for item in items { - collect(item, references); - } - } - Type::Record(_, fields) => { - for field in fields { - collect(&field.tpe, references); - } - } - _ => {} - } -} - -fn collect_value(value: &Value, references: &mut BTreeSet, used: &mut BTreeSet) { - match value { - Value::Reference(_, name) => { - references.insert(name.to_canonical_string()); - } - Value::Lambda(_, pattern, body) => { - if let Pattern::AsPattern(_, _, name) = pattern { - used.insert(name.to_canonical_string()); - } - collect_value(body, references, used); - } - Value::Apply(_, function, argument) => { - collect_value(function, references, used); - collect_value(argument, references, used); - } - Value::Tuple(_, elements) => { - for value in elements { - collect_value(value, references, used); - } - } - Value::IfThenElse(_, condition, yes, no) => { - for value in [condition, yes, no] { - collect_value(value, references, used); - } - } - _ => {} - } -} diff --git a/crates/morphir-python-binding/src/frontend.rs b/crates/morphir-python-binding/src/frontend.rs deleted file mode 100644 index 74a6f6319..000000000 --- a/crates/morphir-python-binding/src/frontend.rs +++ /dev/null @@ -1,390 +0,0 @@ -use crate::{Outcome, error, names}; -use morphir_core::{ir::v4::*, naming::FQName}; -use morphir_extension_sdk::{ - CompileRequest, Diagnostic, SourceLocation, SourcePosition, SourceRange, -}; -use ruff_python_ast::{Expr, Operator, Stmt, StmtClassDef}; -use ruff_text_size::TextRange; -use std::collections::{BTreeMap, BTreeSet}; - -mod functions; -mod imports; -mod project; - -#[derive(Clone)] -enum Symbol { - Type(FQName), - Function(FQName), - Callable, -} -type TypeScope = BTreeMap; - -pub(crate) fn compile(request: &CompileRequest) -> Outcome<(IRFile, Vec)> { - crate::ir::Version::parse(&request.options.ir_version)?; - if request.language_id != "python" || !request.dependencies.is_empty() { - return Err(error("PY001", "Expected Python and no dependencies")); - } - for (key, value) in &request.options.extra { - let valid = match key.as_str() { - "outputDir" => value.is_string(), - "emitParseStage" | "emitParseStageFatal" => value.is_boolean(), - _ => false, - }; - if !valid { - return Err(error( - "PY001", - format!("Unknown or invalid compile option: {key}"), - )); - } - } - if parse_stage_requested(request) - && request - .options - .extra - .get("emitParseStageFatal") - .and_then(serde_json::Value::as_bool) - == Some(true) - { - return Err(error( - "PY004", - "Python parse-stage output is not implemented", - )); - } - project::compile(request) -} - -pub(crate) fn parse_stage_requested(request: &CompileRequest) -> bool { - request - .options - .extra - .get("emitParseStage") - .and_then(serde_json::Value::as_bool) - == Some(true) -} - -fn lower( - statements: &[Stmt], - types: &TypeScope, - aliases: Option<&crate::values::TupleAliases>, - signatures: &crate::values::Signatures, -) -> Outcome { - let mut classes = BTreeMap::new(); - let mut sums = BTreeMap::new(); - let mut tuple_aliases = BTreeMap::new(); - let mut declared = BTreeSet::new(); - let mut functions = BTreeMap::new(); - let mut dataclass_imported = false; - for statement in statements { - match statement { - Stmt::ImportFrom(import) => { - // The package resolver validates all imports and their binding names. - if import.level == 0 - && import.module.as_ref().map(|m| m.as_str()) == Some("dataclasses") - { - dataclass_imported = true; - } - } - Stmt::Import(_) => {} - Stmt::ClassDef(class) => { - if !dataclass_imported { - return Err(error( - "PY004", - "Import dataclass before declaring a product or variant", - )); - } - declare(&mut declared, class.name.as_str())?; - names::type_name(&names::identifier(class.name.as_str())?)?; - validate_class(class)?; - classes.insert(class.name.to_string(), class); - } - Stmt::TypeAlias(alias) if alias.type_params.is_none() => { - let name = expr_name(&alias.name)?; - declare(&mut declared, name)?; - names::type_name(&names::identifier(name)?)?; - if matches!(alias.value.as_ref(), Expr::Subscript(subscript) if matches!(subscript.value.as_ref(), Expr::Name(name) if name.id.as_str() == "tuple")) - { - tuple_aliases.insert(name.to_owned(), alias.value.as_ref()); - continue; - } - let mut variants = vec![]; - union_members(&alias.value, &mut variants)?; - sums.insert(name.to_owned(), variants); - } - Stmt::FunctionDef(function) => { - declare(&mut declared, function.name.as_str())?; - names::field_name(&names::identifier(function.name.as_str())?)?; - functions.insert(function.name.to_string(), function); - } - _ => { - return Err(error( - "PY004", - "Only frozen dataclasses, non-generic sum or tuple aliases and annotated pure functions are supported", - )); - } - } - } - let mut owned_variants = BTreeSet::new(); - for variants in sums.values() { - for variant in variants { - if !classes.contains_key(*variant) || !owned_variants.insert(*variant) { - return Err(error( - "PY004", - format!( - "Variant {variant} must name a dataclass and belong to exactly one sum" - ), - )); - } - } - } - let mut definitions = BTreeMap::new(); - - for (name, expression) in &tuple_aliases { - let canonical = names::identifier(name)?.to_canonical_string(); - let type_expr = annotation(expression, types)?; - definitions.insert( - canonical, - TypeDefinition::TypeAliasDefinition { - type_params: vec![], - type_expr, - }, - ); - } - for (name, class) in &classes { - if !owned_variants.contains(name.as_str()) { - definitions.insert( - names::identifier(name)?.to_canonical_string(), - TypeDefinition::TypeAliasDefinition { - type_params: vec![], - type_expr: Type::Record(Default::default(), fields(class, types)?), - }, - ); - } - } - for (name, variants) in &sums { - let constructors = variants - .iter() - .map(|variant| { - Ok(ConstructorDefinition { - name: names::identifier(variant)?, - args: fields(classes[*variant], types)? - .into_iter() - .map(|field| ConstructorArg { - name: field.name, - arg_type: field.tpe, - }) - .collect(), - }) - }) - .collect::>>()?; - definitions.insert( - names::identifier(name)?.to_canonical_string(), - TypeDefinition::CustomTypeDefinition { - type_params: vec![], - constructors: public(constructors), - }, - ); - } - Ok(ModuleDefinition { - types: definitions - .into_iter() - .map(|(name, definition)| (name, public(Documented::new(None, definition)))) - .collect(), - values: functions - .into_iter() - .filter_map(|(name, function)| aliases.map(|aliases| (name, function, aliases))) - .map(|(name, function, aliases)| { - Ok(( - names::identifier(&name)?.to_canonical_string(), - public(Documented::new( - None, - functions::lower(function, types, aliases, signatures)?, - )), - )) - }) - .collect::>()?, - doc: None, - }) -} - -fn declare(declared: &mut BTreeSet, name: &str) -> Outcome<()> { - if !declared.insert(names::identifier(name)?.to_canonical_string()) { - return Err(error( - "PY003", - format!("Duplicate declaration after Morphir name normalization: {name}"), - )); - } - Ok(()) -} - -fn validate_class(class: &StmtClassDef) -> Outcome<()> { - if class.type_params.is_some() || class.arguments.is_some() || class.decorator_list.len() != 1 { - return Err(error( - "PY004", - "Dataclasses cannot have type parameters, bases, metaclasses or additional decorators", - )); - } - let Expr::Call(call) = &class.decorator_list[0].expression else { - return Err(error("PY004", "Expected @dataclass(frozen=True)")); - }; - if expr_name(&call.func)? != "dataclass" - || !call.arguments.args.is_empty() - || call.arguments.keywords.len() != 1 - || call.arguments.keywords[0].arg.as_ref().map(|s| s.as_str()) != Some("frozen") - || !matches!(&call.arguments.keywords[0].value, Expr::BooleanLiteral(v) if v.value) - { - return Err(error("PY004", "Only @dataclass(frozen=True) is supported")); - } - Ok(()) -} - -fn fields(class: &StmtClassDef, types: &TypeScope) -> Outcome> { - let mut seen = BTreeSet::new(); - class - .body - .iter() - .filter(|s| !matches!(s, Stmt::Pass(_))) - .map(|statement| { - let Stmt::AnnAssign(field) = statement else { - return Err(error( - "PY004", - "Dataclass bodies may contain only annotated fields without defaults, or pass", - )); - }; - if field.value.is_some() { - return Err(error("PY004", "Field defaults are not supported")); - } - let name = expr_name(&field.target)?; - declare(&mut seen, name)?; - let name = names::identifier(name)?; - names::field_name(&name)?; - Ok(Field::new(name, annotation(&field.annotation, types)?)) - }) - .collect() -} - -fn annotation(expr: &Expr, types: &TypeScope) -> Outcome { - match expr { - Expr::Name(name) => { - if let Some(Symbol::Type(fq)) = types.get(name.id.as_str()) { - return Ok(Type::Reference(Default::default(), fq.clone(), vec![])); - } - let fq = match name.id.as_str() { - "int" => "morphir/SDK:basics#int", - "float" => "morphir/SDK:basics#float", - "bool" => "morphir/SDK:basics#bool", - "str" => "morphir/SDK:string#string", - other => { - return Err(error( - "PY004", - format!("Unknown type or constructor used as a type: {other}"), - )); - } - }; - Ok(Type::Reference( - Default::default(), - FQName::from_canonical_string(fq).map_err(|e| error("PY003", e))?, - vec![], - )) - } - Expr::Attribute(_) => { - let name = imports::qualified_name(expr)?; - let Some(Symbol::Type(fq)) = types.get(&name) else { - return Err(error("PY004", format!("Unknown imported type: {name}"))); - }; - Ok(Type::Reference(Default::default(), fq.clone(), vec![])) - } - Expr::Subscript(subscript) - if matches!( - types.get(&imports::qualified_name(&subscript.value)?), - Some(Symbol::Callable) - ) => - { - let Expr::Tuple(parts) = subscript.slice.as_ref() else { - return Err(error("PY004", "Expected Callable[[input], output]")); - }; - let [Expr::List(inputs), output] = parts.elts.as_slice() else { - return Err(error("PY004", "Expected Callable[[input], output]")); - }; - let [input] = inputs.elts.as_slice() else { - return Err(error( - "PY004", - "Callable requires exactly one input; nest Callable for curried functions", - )); - }; - Ok(Type::Function( - Default::default(), - Box::new(annotation(input, types)?), - Box::new(annotation(output, types)?), - )) - } - Expr::Subscript(subscript) if expr_name(&subscript.value)? == "tuple" => { - let elements: Vec<&Expr> = match subscript.slice.as_ref() { - Expr::Tuple(tuple) => tuple.elts.iter().collect(), - element => vec![element], - }; - if elements.len() < 2 { - return Err(error("PY004", "Tuples require at least two fixed elements")); - } - Ok(Type::Tuple( - Default::default(), - elements - .into_iter() - .map(|e| annotation(e, types)) - .collect::>()?, - )) - } - _ => Err(error( - "PY004", - "Expected int, float, bool, str, a local type name, or a fixed tuple annotation", - )), - } -} - -fn union_members<'a>(expr: &'a Expr, members: &mut Vec<&'a str>) -> Outcome<()> { - match expr { - Expr::BinOp(binary) if binary.op == Operator::BitOr => { - union_members(&binary.left, members)?; - union_members(&binary.right, members) - } - Expr::Name(name) => { - members.push(name.id.as_str()); - Ok(()) - } - _ => Err(error( - "PY004", - "A sum alias must list dataclass variants separated by |", - )), - } -} - -fn expr_name(expr: &Expr) -> Outcome<&str> { - match expr { - Expr::Name(name) => Ok(name.id.as_str()), - _ => Err(error("PY004", "Expected a simple Python name")), - } -} - -fn public(value: T) -> AccessControlled { - AccessControlled { - access: Access::Public, - value, - } -} - -fn at(mut diagnostic: Diagnostic, uri: &str, source: &str, range: TextRange) -> Diagnostic { - let position = |offset: usize| { - let prefix = &source[..offset.min(source.len())]; - SourcePosition::from_line_prefix( - prefix.bytes().filter(|b| *b == b'\n').count() as u32, - prefix.rsplit('\n').next().unwrap_or(""), - ) - }; - diagnostic.location = Some(SourceLocation { - uri: uri.into(), - range: SourceRange { - start: position(range.start().to_usize()), - end: position(range.end().to_usize()), - }, - }); - diagnostic -} diff --git a/crates/morphir-python-binding/src/frontend/functions.rs b/crates/morphir-python-binding/src/frontend/functions.rs deleted file mode 100644 index 81d227b4a..000000000 --- a/crates/morphir-python-binding/src/frontend/functions.rs +++ /dev/null @@ -1,352 +0,0 @@ -use super::{annotation, declare}; -use crate::{ - Outcome, names, - values::{self, Comparison, unsupported}, -}; -use morphir_core::ir::v4::*; -use num_bigint::BigInt; -use ruff_python_ast::{CmpOp, Expr, Number, Stmt, StmtFunctionDef, UnaryOp}; -use std::collections::BTreeSet; - -pub(super) fn signature( - function: &StmtFunctionDef, - types: &super::TypeScope, -) -> Outcome { - let parameters = &function.parameters; - if function.is_async - || function.type_params.is_some() - || !function.decorator_list.is_empty() - || !parameters.posonlyargs.is_empty() - || !parameters.kwonlyargs.is_empty() - || parameters.vararg.is_some() - || parameters.kwarg.is_some() - { - return Err(unsupported( - "Expected a synchronous, non-generic function with ordinary annotated parameters and no decorators", - )); - } - let mut seen = BTreeSet::new(); - let mut source_names = BTreeSet::new(); - let input_types = parameters - .args - .iter() - .map(|parameter| { - if parameter.default.is_some() { - return Err(unsupported("Parameter defaults are not supported")); - } - let source_name = parameter.parameter.name.as_str(); - declare(&mut seen, source_name)?; - let name = names::identifier(source_name)?; - names::field_name(&name)?; - source_names.insert(source_name); - let tpe = annotation( - parameter.annotation().ok_or_else(|| { - unsupported("Every function parameter needs a type annotation") - })?, - types, - )?; - Ok((name.to_canonical_string(), tpe)) - }) - .collect::>()?; - let output_type = annotation( - function - .returns - .as_deref() - .ok_or_else(|| unsupported("Functions need a return type annotation"))?, - types, - )?; - Ok(ValueSpecification { - annotations: Vec::new().into(), - inputs: input_types, - output: output_type, - }) -} - -pub(super) fn lower( - function: &StmtFunctionDef, - types: &super::TypeScope, - aliases: &values::TupleAliases, - signatures: &values::Signatures, -) -> Outcome { - let signature = signature(function, types)?; - let source_names = function - .parameters - .args - .iter() - .map(|p| p.parameter.name.as_str()) - .collect(); - let context = Context { types, signatures }; - let body = context.block(&function.body, None, &source_names)?; - let definition = ValueDefinition { - input_types: signature.inputs, - output_type: Some(signature.output), - body: ValueBody::Expression(body), - }; - values::validate_function(&definition, aliases, signatures)?; - Ok(definition) -} - -struct Context<'a> { - types: &'a super::TypeScope, - signatures: &'a values::Signatures, -} - -impl Context<'_> { - // The continuation is used only by paths that reach the end of a branch. - fn block( - &self, - statements: &[Stmt], - continuation: Option<&Value>, - parameters: &BTreeSet<&str>, - ) -> Outcome { - let Some((first, rest)) = statements.split_first() else { - return continuation - .cloned() - .ok_or_else(|| unsupported("Every function path must return a value")); - }; - match first { - Stmt::Return(statement) if rest.is_empty() => self.expression( - statement - .value - .as_deref() - .ok_or_else(|| unsupported("Bare return is not supported"))?, - parameters, - ), - Stmt::If(statement) => { - let fallback = if rest.is_empty() { - continuation.cloned() - } else { - Some(self.block(rest, continuation, parameters)?) - }; - let mut otherwise = fallback.clone(); - for clause in statement.elif_else_clauses.iter().rev() { - let branch = self.block(&clause.body, fallback.as_ref(), parameters)?; - otherwise = Some(match &clause.test { - Some(test) => conditional( - self.expression(test, parameters)?, - branch, - otherwise.ok_or_else(|| { - unsupported("Every function path must return a value") - })?, - ), - None => branch, - }); - } - Ok(conditional( - self.expression(&statement.test, parameters)?, - self.block(&statement.body, fallback.as_ref(), parameters)?, - otherwise - .ok_or_else(|| unsupported("Every function path must return a value"))?, - )) - } - _ => Err(unsupported( - "Function bodies may contain only returns and if/elif/else statements; unreachable statements are not supported", - )), - } - } - - fn expression(&self, expr: &Expr, parameters: &BTreeSet<&str>) -> Outcome { - let literal = match expr { - Expr::Lambda(lambda) => { - let p = lambda - .parameters - .as_ref() - .ok_or_else(|| unsupported("A lambda requires one parameter"))?; - if p.args.len() != 1 - || !p.posonlyargs.is_empty() - || !p.kwonlyargs.is_empty() - || p.vararg.is_some() - || p.kwarg.is_some() - || p.args[0].default.is_some() - { - return Err(unsupported( - "Lambdas require exactly one ordinary parameter without a default", - )); - } - let source = p.args[0].parameter.name.as_str(); - let name = names::identifier(source)?; - names::field_name(&name)?; - if parameters.iter().any(|outer| { - *outer != source && names::identifier(outer).is_ok_and(|outer| outer == name) - }) { - return Err(unsupported( - "Lambda parameter spelling collides with an enclosing parameter after name normalization", - )); - } - let mut scope = parameters.clone(); - scope.insert(source); - return Ok(Value::Lambda( - Default::default(), - Pattern::AsPattern( - Default::default(), - Box::new(Pattern::WildcardPattern(Default::default())), - name, - ), - Box::new(self.expression(&lambda.body, &scope)?), - )); - } - Expr::Call(call) => { - if !call.arguments.keywords.is_empty() - || call - .arguments - .args - .iter() - .any(|arg| matches!(arg, Expr::Starred(_))) - { - return Err(unsupported( - "Only positional calls without starred arguments are supported", - )); - } - let function = - if let Some((name, signature)) = self.reference(&call.func, parameters) { - if call.arguments.args.len() != signature.inputs.len() { - return Err(unsupported( - "Named function calls must supply every declared parameter", - )); - } - Value::Reference(Default::default(), name.clone()) - } else { - if call.arguments.args.len() != 1 { - return Err(unsupported("Callable values take exactly one argument")); - } - self.expression(&call.func, parameters)? - }; - return call - .arguments - .args - .iter() - .try_fold(function, |function, arg| { - Ok(Value::Apply( - Default::default(), - Box::new(function), - Box::new(self.expression(arg, parameters)?), - )) - }); - } - Expr::Name(_) | Expr::Attribute(_) if self.reference(expr, parameters).is_some() => { - let (name, signature) = self.reference(expr, parameters).unwrap(); - if signature.inputs.len() != 1 { - return Err(unsupported( - "Only unary named functions can be used as callable values", - )); - } - return Ok(Value::Reference(Default::default(), name.clone())); - } - Expr::Tuple(tuple) if tuple.elts.len() >= 2 => { - return Ok(Value::Tuple( - Default::default(), - tuple - .elts - .iter() - .map(|element| self.expression(element, parameters)) - .collect::>()?, - )); - } - Expr::Name(name) if parameters.contains(name.id.as_str()) => { - return Ok(Value::Variable( - Default::default(), - names::identifier(name.id.as_str())?, - )); - } - Expr::If(expression_if) => { - return Ok(conditional( - self.expression(&expression_if.test, parameters)?, - self.expression(&expression_if.body, parameters)?, - self.expression(&expression_if.orelse, parameters)?, - )); - } - Expr::Compare(compare) if compare.ops.len() == 1 && compare.operands.len() == 2 => { - let operator = match compare.ops[0] { - CmpOp::Eq => Comparison::Equal, - CmpOp::NotEq => Comparison::NotEqual, - CmpOp::Lt => Comparison::Less, - CmpOp::LtE => Comparison::LessEqual, - CmpOp::Gt => Comparison::Greater, - CmpOp::GtE => Comparison::GreaterEqual, - _ => { - return Err(unsupported( - "Identity and membership comparisons are not supported", - )); - } - }; - return Ok(operator.apply( - self.expression(&compare.operands[0], parameters)?, - self.expression(&compare.operands[1], parameters)?, - )); - } - Expr::BooleanLiteral(value) => Literal::Bool(value.value), - Expr::StringLiteral(value) => Literal::String(value.value.to_string()), - Expr::NumberLiteral(value) => number(&value.value, false)?, - Expr::UnaryOp(unary) if unary.op == UnaryOp::USub => { - let Expr::NumberLiteral(value) = unary.operand.as_ref() else { - return Err(unsupported( - "Unary minus is supported only for numeric literals", - )); - }; - number(&value.value, true)? - } - _ => { - return Err(unsupported( - "Expected a parameter, scalar literal, fixed tuple, comparison or conditional expression", - )); - } - }; - Ok(Value::Literal(Default::default(), literal)) - } - - fn reference( - &self, - expr: &Expr, - parameters: &BTreeSet<&str>, - ) -> Option<(&morphir_core::naming::FQName, &ValueSpecification)> { - let name = super::imports::qualified_name(expr).ok()?; - if parameters.contains(name.split('.').next()?) { - return None; - } - let super::Symbol::Function(fq) = self.types.get(&name)? else { - return None; - }; - Some((fq, self.signatures.get(&fq.to_canonical_string())?)) - } -} - -fn conditional(condition: Value, then_branch: Value, else_branch: Value) -> Value { - Value::IfThenElse( - Default::default(), - Box::new(condition), - Box::new(then_branch), - Box::new(else_branch), - ) -} - -fn number(value: &Number, negative: bool) -> Outcome { - match value { - Number::Int(value) => { - // Ruff retains the original token for integers larger than u64. - let token = value.to_string().replace('_', "").to_ascii_lowercase(); - let (digits, radix) = match token.get(..2) { - Some("0x") => (&token[2..], 16), - Some("0o") => (&token[2..], 8), - Some("0b") => (&token[2..], 2), - _ => (token.as_str(), 10), - }; - let magnitude = BigInt::parse_bytes(digits.as_bytes(), radix) - .ok_or_else(|| unsupported("Invalid integer literal"))?; - Ok(Literal::Integer(if negative { - -magnitude - } else { - magnitude - })) - } - Number::Float(value) if value.is_finite() => { - Ok(Literal::Float(FloatLiteral::from_f64(if negative { - -*value - } else { - *value - }))) - } - _ => Err(unsupported( - "Only finite real numeric literals are supported", - )), - } -} diff --git a/crates/morphir-python-binding/src/frontend/imports.rs b/crates/morphir-python-binding/src/frontend/imports.rs deleted file mode 100644 index 30ba1a714..000000000 --- a/crates/morphir-python-binding/src/frontend/imports.rs +++ /dev/null @@ -1,215 +0,0 @@ -//! Static import resolution. Imports never load or execute Python code. - -use super::{Symbol, TypeScope, declare, union_members}; -use crate::{Outcome, error, modules::ModuleIdentity, names}; -use morphir_core::{ir::v4::PackageName, naming::FQName}; -use ruff_python_ast::{Expr, Stmt}; -use std::collections::{BTreeMap, BTreeSet}; - -pub(super) type Exports = BTreeMap; - -pub(super) fn exports( - statements: &[Stmt], - package: &PackageName, - module: &str, -) -> Outcome { - let mut types = BTreeSet::new(); - let mut variants = vec![]; - let mut functions = BTreeSet::new(); - for statement in statements { - match statement { - Stmt::FunctionDef(function) => { - types.insert(function.name.to_string()); - functions.insert(function.name.to_string()); - } - Stmt::ClassDef(class) => { - types.insert(class.name.to_string()); - } - Stmt::TypeAlias(alias) => { - types.insert(super::expr_name(&alias.name)?.to_owned()); - if !matches!(alias.value.as_ref(), Expr::Subscript(_)) { - union_members(&alias.value, &mut variants)?; - } - } - _ => {} - } - } - types - .into_iter() - .filter(|name| !variants.contains(&name.as_str())) - .map(|name| { - let fq = format!( - "{}:{module}#{}", - package.to_canonical_string(), - names::identifier(&name)?.to_canonical_string() - ); - let fq = FQName::from_canonical_string(&fq).map_err(|e| error("PY003", e))?; - let symbol = if functions.contains(&name) { - Symbol::Function(fq) - } else { - Symbol::Type(fq) - }; - Ok((name, symbol)) - }) - .collect() -} - -pub(super) fn scope( - statements: &[Stmt], - identity: &ModuleIdentity, - exports: &Exports, -) -> Outcome { - let mut types = exports[&identity.python].clone(); - let mut bindings = BTreeSet::new(); - let mut module_bindings = BTreeMap::new(); - for statement in statements { - match statement { - Stmt::ClassDef(class) => declare(&mut bindings, class.name.as_str())?, - Stmt::TypeAlias(alias) => declare(&mut bindings, super::expr_name(&alias.name)?)?, - Stmt::FunctionDef(function) => declare(&mut bindings, function.name.as_str())?, - _ => {} - } - } - for (index, statement) in statements.iter().enumerate() { - match statement { - Stmt::ImportFrom(import) => { - let module = import.module.as_ref().map(|m| m.as_str()).unwrap_or(""); - if import.is_lazy { - return Err(error("PY004", "Lazy imports are not supported")); - } - if import.level == 0 && ["typing", "collections.abc"].contains(&module) { - for alias in &import.names { - if alias.name.as_str() != "Callable" { - return Err(error( - "PY004", - "Only Callable is supported from typing or collections.abc", - )); - } - let bound = alias - .asname - .as_ref() - .map(|n| n.as_str()) - .unwrap_or("Callable"); - if bound == "Callable" { - if !bindings.insert("callable".into()) { - return Err(error("PY003", "Duplicate Callable binding")); - } - } else { - declare(&mut bindings, bound)?; - } - types.insert(bound.into(), Symbol::Callable); - } - continue; - } - if import.level == 0 && ["dataclasses", "__future__"].contains(&module) { - let expected = if module == "dataclasses" { - "dataclass" - } else { - "annotations" - }; - if import.names.len() != 1 - || import.names[0].name.as_str() != expected - || import.names[0].asname.is_some() - { - return Err(error( - "PY004", - "Only dataclasses.dataclass and __future__.annotations standard imports are supported", - )); - } - if module == "__future__" && !statements[..index].iter().all(|s| { - matches!(s, Stmt::ImportFrom(i) if i.level == 0 && i.module.as_ref().map(|m| m.as_str()) == Some("__future__")) - }) { - return Err(error("PY004", "Future imports must precede other statements")); - } - continue; - } - let target = absolute_module(&identity.python, module, import.level)?; - for alias in &import.names { - let name = alias.name.as_str(); - let bound = alias.asname.as_ref().map(|a| a.as_str()).unwrap_or(name); - declare(&mut bindings, bound)?; - if let Some(fq) = exports.get(&target).and_then(|types| types.get(name)) { - types.insert(bound.into(), fq.clone()); - } else { - let child = if target.is_empty() { - name.into() - } else { - format!("{target}.{name}") - }; - import_module(&mut types, exports, &child, bound)?; - } - } - } - Stmt::Import(import) => { - if import.is_lazy { - return Err(error("PY004", "Lazy imports are not supported")); - } - for alias in &import.names { - let target = alias.name.as_str(); - let bound = alias.asname.as_ref().map(|a| a.as_str()).unwrap_or(target); - let root = bound.split('.').next().unwrap_or(bound); - let binding = if alias.asname.is_some() { target } else { root }; - match module_bindings.get(root) { - Some(existing) if existing == &binding => {} - _ => { - declare(&mut bindings, root)?; - module_bindings.insert(root, binding); - } - } - if ["typing", "collections.abc"].contains(&target) { - types.insert(format!("{bound}.Callable"), Symbol::Callable); - } else { - import_module(&mut types, exports, target, bound)?; - } - } - } - _ => {} - } - } - Ok(types) -} - -fn absolute_module(current: &str, module: &str, level: u32) -> Outcome { - if level == 0 { - return Ok(module.into()); - } - let mut segments: Vec<_> = current.split('.').collect(); - if level as usize >= segments.len() { - return Err(error("PY004", "Relative import escapes the source package")); - } - segments.truncate(segments.len() - level as usize); - if !module.is_empty() { - segments.push(module); - } - Ok(segments.join(".")) -} - -fn import_module( - scope: &mut TypeScope, - exports: &Exports, - target: &str, - bound: &str, -) -> Outcome<()> { - let types = exports.get(target).ok_or_else(|| { - error( - "PY004", - format!("Unresolved local module or type import: {target}"), - ) - })?; - for (name, fq) in types { - scope.insert(format!("{bound}.{name}"), fq.clone()); - } - Ok(()) -} - -pub(super) fn qualified_name(expr: &Expr) -> Outcome { - match expr { - Expr::Name(name) => Ok(name.id.to_string()), - Expr::Attribute(attribute) => Ok(format!( - "{}.{}", - qualified_name(&attribute.value)?, - attribute.attr - )), - _ => Err(error("PY004", "Expected an imported module and type name")), - } -} diff --git a/crates/morphir-python-binding/src/frontend/project.rs b/crates/morphir-python-binding/src/frontend/project.rs deleted file mode 100644 index b8828aed6..000000000 --- a/crates/morphir-python-binding/src/frontend/project.rs +++ /dev/null @@ -1,138 +0,0 @@ -//! Compile a closed set of source modules in declaration and value passes. - -use super::{at, imports, lower}; -use crate::{ - Outcome, error, - modules::{self, ModuleIdentity}, -}; -use morphir_core::ir::v4::*; -use morphir_extension_sdk::CompileRequest; -use ruff_text_size::Ranged; -use std::collections::{BTreeMap, BTreeSet}; - -pub(super) fn compile(request: &CompileRequest) -> Outcome<(IRFile, Vec)> { - if request.sources.documents.is_empty() { - return Err(error("PY001", "Expected at least one Python source module")); - } - let package = - PackageName::from_canonical_string(&request.package.name).map_err(|e| error("PY001", e))?; - if package.is_empty() { - return Err(error("PY001", "Package name must not be empty")); - } - let paths = request - .source_paths() - .map_err(|e| error("PY001", e.to_string()))?; - let parsed = request - .sources - .documents - .iter() - .zip(&paths) - .map(|(document, path)| { - if document.language_id != "python" { - return Err(error("PY001", "Source document language must be python")); - } - let identity = ModuleIdentity::from_relative_path(path.as_str())?; - let syntax = ruff_python_parser::parse_module(&document.text).map_err(|e| { - at( - error("PY002", e.to_string()), - &document.uri, - &document.text, - e.range(), - ) - })?; - Ok((document, identity, syntax)) - }) - .collect::>>()?; - modules::validate_paths(parsed.iter().map(|(_, id, _)| id.canonical.as_str()))?; - let names: BTreeSet<_> = parsed - .iter() - .map(|(_, id, _)| id.canonical.clone()) - .collect(); - let exposed = request - .package - .exposed_modules - .iter() - .flatten() - .map(|name| ModuleName::parse(&name.replace('.', "/")).to_canonical_string()) - .collect::>(); - if !exposed.is_subset(&names) { - return Err(error( - "PY001", - "exposedModules names an unknown source module", - )); - } - let exports = parsed - .iter() - .map(|(document, id, syntax)| { - imports::exports(syntax.suite(), &package, &id.canonical) - .map(|types| (id.python.clone(), types)) - .map_err(|e| at(e, &document.uri, &document.text, syntax.syntax().range())) - }) - .collect::>()?; - let scopes = parsed - .iter() - .map(|(document, id, syntax)| { - imports::scope(syntax.suite(), id, &exports) - .map_err(|e| at(e, &document.uri, &document.text, syntax.syntax().range())) - }) - .collect::>>()?; - let mut signatures = crate::values::Signatures::new(); - for ((document, _, syntax), scope) in parsed.iter().zip(&scopes) { - for statement in syntax.suite() { - if let ruff_python_ast::Stmt::FunctionDef(function) = statement { - let super::Symbol::Function(name) = &scope[function.name.as_str()] else { - unreachable!() - }; - let signature = super::functions::signature(function, scope) - .map_err(|e| at(e, &document.uri, &document.text, function.range()))?; - signatures.insert(name.to_canonical_string(), signature); - } - } - } - let lower_modules = |aliases| -> Outcome> { - parsed - .iter() - .zip(&scopes) - .map(|((document, id, syntax), scope)| { - lower(syntax.suite(), scope, aliases, &signatures) - .map(|module| { - ( - id.canonical.clone(), - AccessControlled { - access: if request.package.exposed_modules.is_none() - || exposed.contains(&id.canonical) - { - Access::Public - } else { - Access::Private - }, - value: module, - }, - ) - }) - .map_err(|e| at(e, &document.uri, &document.text, syntax.syntax().range())) - }) - .collect() - }; - let definitions = lower_modules(None)?; - let aliases = modules::tuple_aliases(&package, definitions.iter())?; - let definitions = lower_modules(Some(&aliases))?; - if request.options.types_only && definitions.values().any(|m| !m.value.values.is_empty()) { - return Err(error( - "PY004", - "typesOnly is not supported for Python functions; compile with typesOnly false", - )); - } - let ir = IRFile { - format_version: FormatVersion::Integer(4), - metadata: None, - distribution: Distribution::Library(LibraryContent { - package_name: package, - dependencies: Default::default(), - def: PackageDefinition { - modules: definitions.into_iter().collect(), - }, - }), - }; - Ok((ir, names.into_iter().collect())) -} diff --git a/crates/morphir-python-binding/src/ir.rs b/crates/morphir-python-binding/src/ir.rs deleted file mode 100644 index e5efea2c7..000000000 --- a/crates/morphir-python-binding/src/ir.rs +++ /dev/null @@ -1,58 +0,0 @@ -//! Version boundaries around the shared Python language mapping. - -mod classic; - -use crate::{Outcome, error}; -use morphir_core::ir::v4::{self, IRFile}; - -#[derive(Clone, Copy)] -pub(crate) enum Version { - V3, - V4, -} - -impl Version { - pub(crate) fn parse(text: &str) -> Outcome { - match text { - "3" | "3.0.0" => Ok(Self::V3), - "4" | "4.0.0" => Ok(Self::V4), - _ => Err(error("PY001", "Expected IR version 3, 3.0.0, 4 or 4.0.0")), - } - } - - pub(crate) fn major(self) -> &'static str { - match self { - Self::V3 => "3", - Self::V4 => "4", - } - } - - pub(crate) fn encode(self, model: IRFile) -> Outcome { - match self { - Self::V3 => serde_json::to_value(classic::encode(&model)?), - Self::V4 => { - v4::with_type_encoding(v4::TypeEncoding::Compact, || serde_json::to_value(model)) - } - } - .map_err(|e| error("PY005", e.to_string())) - } -} - -pub(crate) fn decode(value: serde_json::Value) -> Outcome { - use morphir_core::format_version::{NormalizedFormatVersion, ScalarValue, SupportTable}; - let header = value - .get("formatVersion") - .ok_or_else(|| error("PY005", "Missing formatVersion"))?; - let scalar = ScalarValue::from_json(header).map_err(|e| error("PY005", e.to_string()))?; - let normalized = NormalizedFormatVersion::from_scalar(&scalar, &SupportTable::reference()) - .map_err(|e| error("PY005", e.to_string()))?; - if !normalized.is_supported() { - return Err(error("PY005", "Unsupported Morphir IR version")); - } - if normalized.release.major() == 3 { - classic::decode(value) - } else { - // Let the shared v4 codec enforce its complete version/header contract. - serde_json::from_value(value).map_err(|e| error("PY005", e.to_string())) - } -} diff --git a/crates/morphir-python-binding/src/ir/classic.rs b/crates/morphir-python-binding/src/ir/classic.rs deleted file mode 100644 index cb98e4e32..000000000 --- a/crates/morphir-python-binding/src/ir/classic.rs +++ /dev/null @@ -1,225 +0,0 @@ -//! Classic v3 encoding for the Python subset, using the shared concrete IR types. - -mod expressions; -mod read; -pub(super) use read::decode; - -use crate::{Outcome, error, values}; -use morphir_core::{ - ir::{classic as c, v4 as v}, - naming, -}; - -type Type = c::Type; -type Definition = c::ValueDefinition; - -fn name(value: &naming::Name) -> Outcome { - let encoded = c::Name::from_str(&value.to_canonical_string()); - let decoded = morphir_core::migration::migrate_name(&encoded, &Default::default()) - .map_err(|e| error("PY003", e.message))?; - if decoded != *value { - return Err(error( - "PY003", - format!( - "Name '{}' cannot roundtrip through classic IR v3 word arrays; rename it or use v4", - value.to_canonical_string() - ), - )); - } - Ok(encoded) -} - -fn canonical_name(value: &str) -> Outcome { - name(&naming::Name::from_canonical_string(value).map_err(|e| error("PY003", e))?) -} - -fn path(value: &naming::Path) -> Outcome { - Ok(c::Path { - segments: value.segments.iter().map(name).collect::>()?, - }) -} - -fn fqname(value: &naming::FQName) -> Outcome { - Ok(c::FQName { - package_path: path(&value.package_path)?, - module_path: path(&value.module_path)?, - local_name: name(&value.local_name)?, - }) -} - -fn access(value: v::Access) -> c::Access { - match value { - v::Access::Public => c::Access::Public, - v::Access::Private => c::Access::Private, - } -} - -fn tpe(value: &v::Type) -> Outcome { - let a = c::Attrs::None; - Ok(match value { - v::Type::Reference(_, reference, args) => c::Type::Reference( - a, - fqname(reference)?, - args.iter().map(tpe).collect::>()?, - ), - v::Type::Tuple(_, elements) => { - c::Type::Tuple(a, elements.iter().map(tpe).collect::>()?) - } - v::Type::Record(_, fields) => c::Type::Record( - a, - fields - .iter() - .map(|f| { - Ok(c::Field { - name: name(&f.name)?, - ty: tpe(&f.tpe)?, - }) - }) - .collect::>()?, - ), - v::Type::Function(_, input, output) => { - c::Type::Function(a, Box::new(tpe(input)?), Box::new(tpe(output)?)) - } - _ => { - return Err(values::unsupported( - "Type cannot be encoded by the Python v3 subset", - )); - } - }) -} - -fn definition( - value: &v::ValueDefinition, - aliases: &values::TupleAliases, - signatures: &values::Signatures, -) -> Outcome { - let v::ValueBody::Expression(body) = &value.body else { - return Err(values::unsupported("Expected an expression body")); - }; - let mut clean = value.clone(); - clean.body = v::ValueBody::Expression(expressions::erase(body)?); - let typed = values::annotate_function(&clean, aliases, signatures)?; - Ok(Definition { - input_types: value - .input_types - .iter() - .map(|(key, value)| { - Ok(c::value::ValueArgument { - name: canonical_name(key)?, - annotation: tpe(value)?, - ty: tpe(value)?, - }) - }) - .collect::>()?, - output_type: tpe(value - .output_type - .as_ref() - .ok_or_else(|| values::unsupported("Expected a return type"))?)?, - body: expressions::encode(body, &typed, aliases)?, - }) -} - -pub(super) fn encode(ir: &v::IRFile) -> Outcome { - let v::Distribution::Library(library) = &ir.distribution else { - return Err(values::unsupported("Expected a Library")); - }; - let signatures = values::signatures(library)?; - let aliases = crate::modules::tuple_aliases(&library.package_name, library.def.modules.iter())?; - let modules = library - .def - .modules - .iter() - .map(|(key, module)| { - let types = module - .value - .types - .iter() - .map(|(key, item)| { - let value = match &item.value.value { - v::TypeDefinition::TypeAliasDefinition { - type_params, - type_expr, - } if type_params.is_empty() => { - c::TypeDefinition::Alias(vec![], tpe(type_expr)?) - } - v::TypeDefinition::CustomTypeDefinition { - type_params, - constructors, - } if type_params.is_empty() => c::TypeDefinition::Custom( - vec![], - c::AccessControlled { - access: access(constructors.access), - value: constructors - .value - .iter() - .map(|ctor| { - Ok(c::Constructor { - name: name(&ctor.name)?, - args: ctor - .args - .iter() - .map(|arg| { - Ok((name(&arg.name)?, tpe(&arg.arg_type)?)) - }) - .collect::>()?, - }) - }) - .collect::>()?, - }, - ), - _ => { - return Err(values::unsupported( - "Type definition cannot be encoded by the Python v3 subset", - )); - } - }; - Ok(( - canonical_name(key)?, - c::AccessControlled { - access: access(item.access), - value: c::Documented::new("", value), - }, - )) - }) - .collect::>()?; - let values = module - .value - .values - .iter() - .map(|(key, item)| { - Ok(( - canonical_name(key)?, - c::AccessControlled { - access: access(item.access), - value: c::Documented::new( - "", - definition(&item.value.value, &aliases, &signatures)?, - ), - }, - )) - }) - .collect::>()?; - Ok(c::module::ModuleEntry { - path: path( - &naming::Path::from_canonical_string(key).map_err(|e| error("PY003", e))?, - )?, - definition: c::AccessControlled { - access: access(module.access), - value: c::ModuleDefinition { - types, - values, - doc: None, - }, - }, - }) - }) - .collect::>()?; - Ok(c::Distribution { - format_version: 3, - distribution: c::DistributionBody::Library( - path(library.package_name.as_path())?, - vec![], - c::PackageDefinition { modules }, - ), - }) -} diff --git a/crates/morphir-python-binding/src/ir/classic/expressions.rs b/crates/morphir-python-binding/src/ir/classic/expressions.rs deleted file mode 100644 index 7668ee62c..000000000 --- a/crates/morphir-python-binding/src/ir/classic/expressions.rs +++ /dev/null @@ -1,136 +0,0 @@ -//! Infer v3 value annotations and verify annotations supplied by a v3 producer. - -use super::{Type, fqname, name, tpe}; -use crate::{Outcome, values}; -use morphir_core::ir::{classic as c, v4 as v}; - -type Value = c::Value; - -fn annotation( - value: &v::ValueAttributes, - expected: &v::Type, - aliases: &values::TupleAliases, -) -> Outcome { - let attrs = value; - if attrs.source.is_some() || !attrs.extensions.is_empty() { - return Err(values::unsupported( - "Value metadata cannot be preserved in Python", - )); - } - if let Some(actual) = &attrs.inferred_type - && values::resolve_aliases(actual, aliases)? != values::resolve_aliases(expected, aliases)? - { - return Err(values::unsupported( - "IR v3 value annotation does not match its expression type", - )); - } - tpe(expected) -} - -pub(super) fn encode( - value: &v::Value, - typed: &v::Value, - aliases: &values::TupleAliases, -) -> Outcome { - let inferred = typed - .attributes() - .inferred_type - .as_ref() - .expect("checked expression"); - let attrs = annotation(value.attributes(), inferred, aliases)?; - Ok(match value { - v::Value::Variable(_, key) => c::Value::Variable(attrs, name(key)?), - v::Value::Literal(_, literal) => c::Value::Literal(attrs, match literal { - v::Literal::Bool(value) => c::Literal::Bool(*value), - v::Literal::Integer(value) => c::Literal::WholeNumber(i64::try_from(value).map_err(|_| values::unsupported("IR v3 whole-number literals must fit the shared codec's signed 64-bit range; use IR v4 for larger integers"))?), - v::Literal::Float(value) => c::Literal::Float(value.value()), - v::Literal::String(value) => c::Literal::String(value.clone()), - _ => return Err(values::unsupported("Unsupported Python literal")), - }), - v::Value::Reference(_, key) => c::Value::Reference(attrs, fqname(key)?), - v::Value::Tuple(_, elements) => { - let v::Value::Tuple(_, checked) = typed else { unreachable!() }; - c::Value::Tuple(attrs, elements.iter().zip(checked).map(|(value, typed)| encode(value, typed, aliases)).collect::>()?) - } - v::Value::IfThenElse(_, condition, yes, no) => { - let v::Value::IfThenElse(_, tc, ty, tn) = typed else { unreachable!() }; - c::Value::IfThenElse(attrs, Box::new(encode(condition, tc, aliases)?), Box::new(encode(yes, ty, aliases)?), Box::new(encode(no, tn, aliases)?)) - } - v::Value::Apply(_, function, argument) => { - let v::Value::Apply(_, tf, ta) = typed else { unreachable!() }; - c::Value::Apply(attrs, Box::new(encode(function, tf, aliases)?), Box::new(encode(argument, ta, aliases)?)) - } - v::Value::Lambda(_, pattern, body) => { - let v::Value::Lambda(_, tp, tb) = typed else { unreachable!() }; - c::Value::Lambda(attrs, encode_pattern(pattern, tp, aliases)?, Box::new(encode(body, tb, aliases)?)) - } _ => return Err(values::unsupported("Unsupported Python expression")), - }) -} - -/// Remove only inferred types after they have been checked against the shared subset inference. -pub(super) fn erase(value: &v::Value) -> Outcome { - let a = Default::default(); - Ok(match value { - v::Value::Variable(_, name) => v::Value::Variable(a, name.clone()), - v::Value::Reference(_, name) => v::Value::Reference(a, name.clone()), - v::Value::Literal(_, literal) => v::Value::Literal(a, literal.clone()), - v::Value::Tuple(_, elements) => { - v::Value::Tuple(a, elements.iter().map(erase).collect::>()?) - } - v::Value::Lambda(_, pattern, body) => { - v::Value::Lambda(a, erase_pattern(pattern)?, Box::new(erase(body)?)) - } - v::Value::Apply(_, function, argument) => { - v::Value::Apply(a, Box::new(erase(function)?), Box::new(erase(argument)?)) - } - v::Value::IfThenElse(_, condition, yes, no) => v::Value::IfThenElse( - a, - Box::new(erase(condition)?), - Box::new(erase(yes)?), - Box::new(erase(no)?), - ), - _ => { - return Err(values::unsupported( - "IR v3 expression is outside the Python subset", - )); - } - }) -} - -fn encode_pattern( - pattern: &v::Pattern, - typed: &v::Pattern, - aliases: &values::TupleAliases, -) -> Outcome> { - let attrs = annotation( - pattern.attributes(), - typed - .attributes() - .inferred_type - .as_ref() - .expect("checked pattern"), - aliases, - )?; - Ok(match (pattern, typed) { - (v::Pattern::WildcardPattern(_), _) => c::Pattern::Wildcard(attrs), - (v::Pattern::AsPattern(_, inner, key), v::Pattern::AsPattern(_, checked, _)) => { - c::Pattern::As( - attrs, - Box::new(encode_pattern(inner, checked, aliases)?), - name(key)?, - ) - } - _ => return Err(values::unsupported("Unsupported lambda pattern")), - }) -} -fn erase_pattern(pattern: &v::Pattern) -> Outcome { - Ok(match pattern { - v::Pattern::WildcardPattern(_) => v::Pattern::WildcardPattern(Default::default()), - v::Pattern::AsPattern(_, inner, name) => v::Pattern::AsPattern( - Default::default(), - Box::new(erase_pattern(inner)?), - name.clone(), - ), - _ => return Err(values::unsupported("Unsupported lambda pattern")), - }) -} diff --git a/crates/morphir-python-binding/src/ir/classic/read.rs b/crates/morphir-python-binding/src/ir/classic/read.rs deleted file mode 100644 index 7b2e62c8d..000000000 --- a/crates/morphir-python-binding/src/ir/classic/read.rs +++ /dev/null @@ -1,102 +0,0 @@ -//! Read classic IR through the core migration, refusing collisions and annotation loss. - -use crate::{Outcome, error, values}; -use morphir_core::{ - ir::{classic as c, v4 as v}, - migration, -}; -use std::collections::BTreeSet; - -fn distinct<'a>(names: impl Iterator) -> Outcome<()> { - let mut seen = BTreeSet::new(); - for name in names { - let key = migration::migrate_name(name, &Default::default()) - .map_err(|e| error("PY003", e.message))? - .to_canonical_string(); - if !seen.insert(key) { - return Err(error("PY003", "Duplicate name in classic IR")); - } - } - Ok(()) -} - -pub(in crate::ir) fn decode(value: serde_json::Value) -> Outcome { - let classic: c::Distribution = - serde_json::from_value(value).map_err(|e| error("PY005", e.to_string()))?; - let c::DistributionBody::Library(_, dependencies, package) = &classic.distribution else { - return Err(values::unsupported( - "a v3 Specs distribution has no definitions to decode", - )); - }; - if !dependencies.is_empty() { - return Err(values::unsupported( - "External dependencies are not supported", - )); - } - let mut module_names = BTreeSet::new(); - for module in &package.modules { - let key = migration::migrate_path(&module.path, &Default::default()) - .map_err(|e| error("PY003", e.message))? - .to_canonical_string(); - if !module_names.insert(key) { - return Err(error("PY003", "Duplicate module in classic IR")); - } - distinct(module.definition.value.types.iter().map(|(key, _)| key))?; - distinct(module.definition.value.values.iter().map(|(key, _)| key))?; - for (_, function) in &module.definition.value.values { - let function = &function.value.value; - distinct(function.input_types.iter().map(|arg| &arg.name))?; - } - } - let migrated = migration::migrate_distribution(&classic, Default::default()) - .map_err(|e| error("PY004", format!("{}: {}", e.path, e.message)))?; - if !migrated.report.can_publish() || !migrated.report.diagnostics().is_empty() { - return Err(values::unsupported( - "IR v3 migration would lose information", - )); - } - let mut ir = migrated.value; - let v::Distribution::Library(library) = &mut ir.distribution else { - unreachable!() - }; - let signatures = values::signatures(library)?; - let aliases = crate::modules::tuple_aliases(&library.package_name, library.def.modules.iter())?; - for module in &package.modules { - for (_, function) in &module.definition.value.values { - for arg in &function.value.value.input_types { - let convert = |ty| { - migration::migrate_type(ty, &mut Default::default()) - .map_err(|e| error("PY004", e.message)) - }; - if values::resolve_aliases(&convert(&arg.annotation)?, &aliases)? - != values::resolve_aliases(&convert(&arg.ty)?, &aliases)? - { - return Err(values::unsupported( - "IR v3 parameter annotation does not match its declared type", - )); - } - } - } - } - for module in library.def.modules.values_mut() { - clear_empty_doc(&mut module.value.doc); - for definition in module.value.types.values_mut() { - clear_empty_doc(&mut definition.value.doc); - } - for definition in module.value.values.values_mut() { - clear_empty_doc(&mut definition.value.doc); - // Encoding checks every supplied inferred type, including intermediate comparison applications. - super::definition(&definition.value.value, &aliases, &signatures)?; - if let v::ValueBody::Expression(body) = &mut definition.value.value.body { - *body = super::expressions::erase(body)?; - } - } - } - Ok(ir) -} - -fn clear_empty_doc(doc: &mut Option) { - if doc.as_ref().is_some_and(|doc| doc.text().is_empty()) { - *doc = None; - } -} diff --git a/crates/morphir-python-binding/src/lib.rs b/crates/morphir-python-binding/src/lib.rs deleted file mode 100644 index 3c8da5618..000000000 --- a/crates/morphir-python-binding/src/lib.rs +++ /dev/null @@ -1,151 +0,0 @@ -//! Python ADT, tuple, function and lambda extension for Morphir. -//! -//! Source is parsed statically, never imported or executed. -//! See the crate README for the supported Python subset. -//! -//! ``` -//! use morphir_extension_sdk::{Extension, NativeExtension}; -//! use morphir_python_binding::PythonExtension; -//! let extension = NativeExtension::frontend_backend(PythonExtension).unwrap(); -//! assert_eq!(PythonExtension::info().id, "morphir-python"); -//! ``` -//! -//! Annotated function bodies can return through conditional branches: -//! -//! ```python -//! def choose(flag: bool, first: int, second: int) -> int: -//! if flag: -//! return first -//! else: -//! return second -//! ``` -//! -//! Both branches become a Morphir `IfThenElse` expression. Compile with -//! `types_only: false`; every path must return the declared result type. - -mod backend; -mod frontend; -mod ir; -mod modules; -mod names; -mod values; - -use morphir_extension_sdk::prelude::*; - -/// Stateless Python frontend and backend, usable natively or as a WASM guest. -#[derive(Default)] -pub struct PythonExtension; - -impl Extension for PythonExtension { - fn info() -> ExtensionInfo { - ExtensionInfo { - id: "morphir-python".into(), - name: "Morphir Python".into(), - version: env!("CARGO_PKG_VERSION").into(), - types: vec![ExtensionType::Frontend, ExtensionType::Backend], - description: Some("Python algebraic data types, tuples, functions and lambdas".into()), - license: Some("Apache-2.0".into()), - ..Default::default() - } - } - - fn capabilities() -> ExtensionCapabilities { - ExtensionCapabilities { - frontend: Some(FrontendCapability { - languages: vec![LanguageCapability { - id: "python".into(), - file_extensions: vec![".py".into()], - }], - ir_versions: vec!["3".into(), "4".into()], - compile: true, - ..Default::default() - }), - backend: Some(BackendCapability { - targets: vec!["python".into()], - ir_versions: vec!["3".into(), "4".into()], - generate: true, - }), - ..Default::default() - } - } -} - -impl Frontend for PythonExtension { - fn compile(&self, request: CompileRequest) -> Result { - let diagnostics = if frontend::parse_stage_requested(&request) { - vec![Diagnostic { - severity: DiagnosticSeverity::Warning, - ..error( - "PY006", - "Python parse-stage output is not implemented; returning compiled IR only", - ) - }] - } else { - vec![] - }; - let compiled = frontend::compile(&request).and_then(|(model, modules)| { - let version = ir::Version::parse(&request.options.ir_version)?; - Ok((version.encode(model)?, modules, version)) - }); - Ok(match compiled { - Ok((ir, modules, version)) => CompileResult { - success: true, - ir_version: Some(version.major().into()), - ir: Some(ir), - modules, - diagnostics, - module_results: vec![], - context_digest: None, - }, - Err(diagnostic) => CompileResult { - success: false, - ir_version: None, - ir: None, - modules: vec![], - diagnostics: vec![diagnostic], - module_results: vec![], - context_digest: None, - }, - }) - } - fn supported_languages() -> Vec { - vec!["python".into()] - } - fn file_extensions() -> Vec { - vec![".py".into()] - } -} - -impl Backend for PythonExtension { - fn generate(&self, request: GenerateRequest) -> Result { - Ok(match backend::generate(&request) { - Ok(artifacts) => GenerateResult { - success: true, - artifacts, - diagnostics: vec![], - }, - Err(diagnostic) => GenerateResult { - success: false, - artifacts: vec![], - diagnostics: vec![diagnostic], - }, - }) - } - fn target_languages() -> Vec { - vec!["python".into()] - } -} - -morphir_extension_sdk::export_extension!(PythonExtension, frontend, backend); - -type Outcome = std::result::Result; - -fn error(code: &str, message: impl Into) -> Diagnostic { - Diagnostic { - severity: DiagnosticSeverity::Error, - code: Some(code.into()), - message: message.into(), - location: None, - related: vec![], - } -} diff --git a/crates/morphir-python-binding/src/modules.rs b/crates/morphir-python-binding/src/modules.rs deleted file mode 100644 index 23391f5b0..000000000 --- a/crates/morphir-python-binding/src/modules.rs +++ /dev/null @@ -1,111 +0,0 @@ -//! Module identity and package-wide type information shared by both directions. - -use crate::{Outcome, error, names, values::TupleAliases}; -use morphir_core::ir::v4::*; - -/// A Python dotted import path and its validated Morphir module name. -pub(crate) struct ModuleIdentity { - pub python: String, - pub canonical: String, -} - -impl ModuleIdentity { - pub fn from_relative_path(relative: &str) -> Outcome { - let stem = relative - .strip_suffix(".py") - .ok_or_else(|| error("PY001", "Source URI must end in .py"))?; - let segments = stem - .split('/') - .map(|part| { - let name = names::identifier(part)?; - names::module_file_stem(&name)?; - Ok(name.to_canonical_string()) - }) - .collect::>>()?; - if segments.first().is_some_and(|s| s == "dataclasses") { - return Err(error("PY003", "A source module cannot shadow dataclasses")); - } - Ok(Self { - python: stem.replace('/', "."), - canonical: segments.join("/"), - }) - } - - pub fn from_canonical(canonical: &str) -> Outcome { - let path = ModuleName::from_canonical_string(canonical).map_err(|e| error("PY003", e))?; - if path.is_empty() { - return Err(error("PY003", "Module name must not be empty")); - } - let parts = path - .as_path() - .segments - .iter() - .map(names::module_file_stem) - .collect::>>()?; - let identity = Self::from_relative_path(&format!("{}.py", parts.join("/")))?; - if identity.canonical != canonical { - return Err(error( - "PY003", - "Module name cannot be represented losslessly", - )); - } - Ok(identity) - } - - pub fn filename(&self) -> String { - format!("{}.py", self.python.replace('.', "/")) - } -} - -/// Reject output collisions and module/package conflicts on every supported OS. -pub(crate) fn validate_paths<'a>(modules: impl Iterator) -> Outcome<()> { - let mut seen = std::collections::BTreeSet::new(); - for module in modules { - if !seen.insert(module.to_ascii_lowercase()) { - return Err(error( - "PY003", - format!("Module paths collide after normalization or case folding: {module}"), - )); - } - } - for module in &seen { - let mut parent = module.as_str(); - while let Some((prefix, _)) = parent.rsplit_once('/') { - if seen.contains(prefix) { - return Err(error( - "PY003", - format!("Module is also a package directory: {prefix}"), - )); - } - parent = prefix; - } - } - Ok(()) -} - -pub(crate) fn tuple_aliases<'a>( - package: &PackageName, - modules: impl Iterator)>, -) -> Outcome { - let aliases: TupleAliases = - modules - .flat_map(|(module, definition)| { - definition.value.types.iter().filter_map( - move |(name, definition)| match &definition.value.value { - TypeDefinition::TypeAliasDefinition { - type_params, - type_expr: tpe @ Type::Tuple(..), - } if type_params.is_empty() => Some(( - format!("{}:{module}#{name}", package.to_canonical_string()), - tpe.clone(), - )), - _ => None, - }, - ) - }) - .collect(); - for alias in aliases.values() { - crate::values::resolve_aliases(alias, &aliases)?; - } - Ok(aliases) -} diff --git a/crates/morphir-python-binding/src/names.rs b/crates/morphir-python-binding/src/names.rs deleted file mode 100644 index 98bbbdcb1..000000000 --- a/crates/morphir-python-binding/src/names.rs +++ /dev/null @@ -1,99 +0,0 @@ -use crate::{Outcome, error}; -use morphir_core::naming::Name; - -const RESERVED: &[&str] = &[ - "False", - "None", - "True", - "and", - "as", - "assert", - "async", - "await", - "break", - "class", - "continue", - "def", - "del", - "elif", - "else", - "except", - "finally", - "for", - "from", - "global", - "if", - "import", - "in", - "is", - "lambda", - "nonlocal", - "not", - "or", - "pass", - "raise", - "return", - "try", - "while", - "with", - "yield", - "dataclass", - "int", - "str", - "float", - "bool", - "tuple", - "Callable", - "callable", -]; - -pub(crate) fn identifier(text: &str) -> Outcome { - if text.is_empty() - || !text.as_bytes()[0].is_ascii_alphabetic() - || !text.bytes().all(|c| c.is_ascii_alphanumeric() || c == b'_') - || RESERVED.contains(&text) - { - return Err(error( - "PY003", - format!("Unsupported or reserved Python identifier: {text}"), - )); - } - Ok(Name::from(text)) -} - -pub(crate) fn type_name(name: &Name) -> Outcome { - let text = name.to_pascal_case(); - faithful(name, text) -} - -pub(crate) fn field_name(name: &Name) -> Outcome { - faithful(name, name.to_canonical_string().replace('-', "_")) -} - -pub(crate) fn module_file_stem(name: &Name) -> Outcome { - let stem = field_name(name)?; - let lower = stem.to_ascii_lowercase(); - if ["con", "prn", "aux", "nul"].contains(&lower.as_str()) - || ["com", "lpt"].iter().any(|prefix| { - lower.strip_prefix(prefix).is_some_and(|suffix| { - suffix.len() == 1 && matches!(suffix.as_bytes()[0], b'1'..=b'9') - }) - }) - { - return Err(error( - "PY003", - format!("Module name is reserved on Windows: {stem}"), - )); - } - Ok(stem) -} - -fn faithful(name: &Name, text: String) -> Outcome { - if identifier(&text)? != *name { - return Err(error( - "PY003", - format!("Name cannot be represented losslessly in Python: {name}"), - )); - } - Ok(text) -} diff --git a/crates/morphir-python-binding/src/values.rs b/crates/morphir-python-binding/src/values.rs deleted file mode 100644 index 4710c29c7..000000000 --- a/crates/morphir-python-binding/src/values.rs +++ /dev/null @@ -1,224 +0,0 @@ -//! Validation shared by Python function compilation and generation. - -use crate::{Outcome, error}; -use morphir_core::{ir::v4::*, naming::FQName}; -use std::collections::{BTreeMap, BTreeSet}; - -mod check; -pub(crate) use check::annotate_function; -pub(crate) type Signatures = BTreeMap; - -pub(crate) fn signatures(library: &LibraryContent) -> Outcome { - library - .def - .modules - .iter() - .flat_map(|(module, definition)| { - definition.value.values.iter().map(move |(name, entry)| { - let definition = &entry.value.value; - Ok(( - format!( - "{}:{module}#{name}", - library.package_name.to_canonical_string() - ), - specification(definition)?, - )) - }) - }) - .collect() -} - -pub(crate) fn specification(definition: &ValueDefinition) -> Outcome { - Ok(ValueSpecification { - annotations: Vec::new().into(), - inputs: definition.input_types.clone(), - output: definition - .output_type - .clone() - .ok_or_else(|| unsupported("Function return type is required"))?, - }) -} - -pub(crate) fn function_type(input: Type, output: Type) -> Type { - Type::Function(Default::default(), Box::new(input), Box::new(output)) -} - -pub(crate) fn signature_type(signature: &ValueSpecification) -> Type { - signature - .inputs - .values() - .rev() - .fold(signature.output.clone(), |output, input| { - function_type(input.clone(), output) - }) -} - -/// Tuple aliases indexed by their canonical fully qualified names. -pub(crate) type TupleAliases = BTreeMap; - -pub(crate) fn resolve_aliases(tpe: &Type, aliases: &TupleAliases) -> Outcome { - fn resolve( - tpe: &Type, - aliases: &TupleAliases, - visiting: &mut BTreeSet, - ) -> Outcome { - match tpe { - Type::Reference(_, name, args) if args.is_empty() => { - let key = name.to_canonical_string(); - if let Some(alias) = aliases.get(&key) { - if !visiting.insert(key.clone()) { - return Err(unsupported("Recursive tuple aliases are not supported")); - } - let result = resolve(alias, aliases, visiting); - visiting.remove(&key); - result - } else { - Ok(tpe.clone()) - } - } - Type::Tuple(attrs, elements) => Ok(Type::Tuple( - attrs.clone(), - elements - .iter() - .map(|element| resolve(element, aliases, visiting)) - .collect::>()?, - )), - Type::Function(attrs, input, output) => Ok(Type::Function( - attrs.clone(), - Box::new(resolve(input, aliases, visiting)?), - Box::new(resolve(output, aliases, visiting)?), - )), - _ => Ok(tpe.clone()), - } - } - resolve(tpe, aliases, &mut BTreeSet::new()) -} - -pub(crate) fn validate_function( - definition: &ValueDefinition, - aliases: &TupleAliases, - signatures: &Signatures, -) -> Outcome<()> { - annotate_function(definition, aliases, signatures).map(|_| ()) -} - -#[derive(Clone, Copy)] -pub(crate) enum Comparison { - Equal, - NotEqual, - Less, - LessEqual, - Greater, - GreaterEqual, -} - -impl Comparison { - pub(crate) fn python(self) -> &'static str { - match self { - Self::Equal => "==", - Self::NotEqual => "!=", - Self::Less => "<", - Self::LessEqual => "<=", - Self::Greater => ">", - Self::GreaterEqual => ">=", - } - } - - pub(crate) fn reference(self) -> FQName { - let name = match self { - Self::Equal => "equal", - Self::NotEqual => "not-equal", - Self::Less => "less-than", - Self::LessEqual => "less-than-or-equal", - Self::Greater => "greater-than", - Self::GreaterEqual => "greater-than-or-equal", - }; - FQName::from_canonical_string(&format!("morphir/SDK:basics#{name}")).expect("SDK name") - } - - fn from_reference(reference: &FQName) -> Outcome { - [ - Self::Equal, - Self::NotEqual, - Self::Less, - Self::LessEqual, - Self::Greater, - Self::GreaterEqual, - ] - .into_iter() - .find(|op| op.reference() == *reference) - .ok_or_else(|| unsupported("Only scalar comparison applications are supported")) - } - - pub(crate) fn apply(self, left: Value, right: Value) -> Value { - Value::Apply( - Default::default(), - Box::new(Value::Apply( - Default::default(), - Box::new(Value::Reference(Default::default(), self.reference())), - Box::new(left), - )), - Box::new(right), - ) - } -} - -pub(crate) fn scalar(name: &str) -> Type { - let path = if name == "string" { "string" } else { "basics" }; - Type::Reference( - Default::default(), - FQName::from_canonical_string(&format!("morphir/SDK:{path}#{name}")).expect("SDK type"), - vec![], - ) -} - -pub(crate) fn unsupported(message: &str) -> morphir_extension_sdk::Diagnostic { - error("PY004", message) -} - -pub(crate) fn require_type(actual: &Type, expected: &Type) -> Outcome<()> { - if actual != expected { - return Err(unsupported( - "Function expression does not match the required type", - )); - } - Ok(()) -} - -pub(crate) fn literal_type(literal: &Literal) -> Outcome { - Ok(scalar(match literal { - Literal::Bool(_) => "bool", - Literal::Integer(_) => "int", - Literal::Float(_) => "float", - Literal::String(_) => "string", - _ => { - return Err(unsupported( - "Only bool, int, float and str literals are supported", - )); - } - })) -} - -pub(crate) fn require_empty_attributes(value: &Value) -> Outcome<()> { - if *value.attributes() != ValueAttributes::default() { - return Err(unsupported( - "Value attributes cannot yet be preserved in Python", - )); - } - Ok(()) -} - -pub(crate) fn comparison(value: &Value) -> Outcome<(Comparison, &Value, &Value)> { - let Value::Apply(_, function, right) = value else { - return Err(unsupported("Expected a comparison")); - }; - require_empty_attributes(function)?; - let Value::Apply(_, reference, left) = function.as_ref() else { - return Err(unsupported("Expected two comparison arguments")); - }; - require_empty_attributes(reference)?; - let Value::Reference(_, reference) = reference.as_ref() else { - return Err(unsupported("Expected an SDK comparison reference")); - }; - Ok((Comparison::from_reference(reference)?, left, right)) -} diff --git a/crates/morphir-python-binding/src/values/check.rs b/crates/morphir-python-binding/src/values/check.rs deleted file mode 100644 index 7f4b11b96..000000000 --- a/crates/morphir-python-binding/src/values/check.rs +++ /dev/null @@ -1,208 +0,0 @@ -//! Bidirectional checking supplies lambda input types from the enclosing annotation. -use super::*; - -pub(crate) fn annotate_function( - definition: &ValueDefinition, - aliases: &TupleAliases, - signatures: &Signatures, -) -> Outcome { - let ValueBody::Expression(body) = &definition.body else { - return Err(unsupported("Only expression function bodies are supported")); - }; - let output = resolve_aliases(&specification(definition)?.output, aliases)?; - let parameters = definition - .input_types - .iter() - .map(|(name, tpe)| Ok((name.clone(), resolve_aliases(tpe, aliases)?))) - .collect::>()?; - Context { - signatures, - aliases, - } - .check(body, Some(&output), ¶meters) -} - -fn attrs(tpe: Type) -> ValueAttributes { - ValueAttributes { - inferred_type: Some(Box::new(tpe)), - ..Default::default() - } -} -fn tpe(value: &Value) -> Type { - *value - .attributes() - .inferred_type - .clone() - .expect("checked expression") -} - -pub(crate) fn binder(pattern: &Pattern) -> Outcome<&Name> { - let Pattern::AsPattern(a, inner, name) = pattern else { - return Err(unsupported("Lambda patterns must bind one named variable")); - }; - if *a != ValueAttributes::default() - || !matches!(inner.as_ref(), Pattern::WildcardPattern(a) if *a == ValueAttributes::default()) - { - return Err(unsupported( - "Lambda patterns must be an unannotated wildcard with a name", - )); - } - crate::names::field_name(name)?; - Ok(name) -} - -struct Context<'a> { - signatures: &'a Signatures, - aliases: &'a TupleAliases, -} - -impl Context<'_> { - fn check( - &self, - value: &Value, - expected: Option<&Type>, - parameters: &BTreeMap, - ) -> Outcome { - require_empty_attributes(value)?; - let result = match value { - Value::Literal(_, literal) => { - Value::Literal(attrs(literal_type(literal)?), literal.clone()) - } - Value::Variable(_, name) => Value::Variable( - attrs( - parameters - .get(&name.to_canonical_string()) - .cloned() - .ok_or_else(|| unsupported("Unknown function parameter"))?, - ), - name.clone(), - ), - Value::Reference(_, name) => { - // Resolve only referenced signatures: rebuilding the whole package for - // every body makes WASM validation quadratic in the number of functions. - let signature = self - .signatures - .get(&name.to_canonical_string()) - .ok_or_else(|| unsupported("Unknown function reference"))?; - Value::Reference( - attrs(resolve_aliases(&signature_type(signature), self.aliases)?), - name.clone(), - ) - } - Value::Lambda(_, pattern, body) => { - let Some(Type::Function(_, input, output)) = expected else { - return Err(unsupported( - "A lambda needs a contextual unary Callable type", - )); - }; - let name = binder(pattern)?; - let mut scope = parameters.clone(); - scope.insert(name.to_canonical_string(), *input.clone()); - let body = self.check(body, Some(output), &scope)?; - let pattern = Pattern::AsPattern( - attrs(*input.clone()), - Box::new(Pattern::WildcardPattern(attrs(*input.clone()))), - name.clone(), - ); - Value::Lambda(attrs(expected.unwrap().clone()), pattern, Box::new(body)) - } - Value::Tuple(_, elements) if elements.len() >= 2 => { - let types = match expected { - Some(Type::Tuple(_, types)) if types.len() == elements.len() => Some(types), - _ => None, - }; - let elements = elements - .iter() - .enumerate() - .map(|(i, element)| self.check(element, types.map(|ts| &ts[i]), parameters)) - .collect::>>()?; - Value::Tuple( - attrs(Type::Tuple( - Default::default(), - elements.iter().map(tpe).collect(), - )), - elements, - ) - } - Value::IfThenElse(_, condition, yes, no) => { - let condition = self.check(condition, Some(&scalar("bool")), parameters)?; - let yes = self.check(yes, expected, parameters)?; - let no = self.check(no, Some(&tpe(&yes)), parameters)?; - Value::IfThenElse( - attrs(tpe(&yes)), - Box::new(condition), - Box::new(yes), - Box::new(no), - ) - } - Value::Apply(_, function, argument) => { - if let Ok((op, left, right)) = comparison(value) { - let left = self.check(left, None, parameters)?; - let operand = tpe(&left); - let right = self.check(right, Some(&operand), parameters)?; - let allowed = ["int", "float", "string"] - .iter() - .any(|name| operand == scalar(name)) - || (matches!(op, Comparison::Equal | Comparison::NotEqual) - && operand == scalar("bool")); - if !allowed { - return Err(unsupported( - "Comparison operands must be matching scalar types; bool supports only equality", - )); - } - let tail = function_type(operand.clone(), scalar("bool")); - let reference = Value::Reference( - attrs(function_type(operand, tail.clone())), - op.reference(), - ); - Value::Apply( - attrs(scalar("bool")), - Box::new(Value::Apply( - attrs(tail), - Box::new(reference), - Box::new(left), - )), - Box::new(right), - ) - } else { - if let Value::Lambda(_, pattern, body) = function.as_ref() { - require_empty_attributes(function)?; - let argument = self.check(argument, None, parameters)?; - let input = tpe(&argument); - let name = binder(pattern)?; - let mut scope = parameters.clone(); - scope.insert(name.to_canonical_string(), input.clone()); - let body = self.check(body, expected, &scope)?; - let output = tpe(&body); - let pattern = Pattern::AsPattern( - attrs(input.clone()), - Box::new(Pattern::WildcardPattern(attrs(input.clone()))), - name.clone(), - ); - let function = Value::Lambda( - attrs(function_type(input, output.clone())), - pattern, - Box::new(body), - ); - return Ok(Value::Apply( - attrs(output), - Box::new(function), - Box::new(argument), - )); - } - let function = self.check(function, None, parameters)?; - let Type::Function(_, input, output) = tpe(&function) else { - return Err(unsupported("Call target must have a function type")); - }; - let argument = self.check(argument, Some(&input), parameters)?; - Value::Apply(attrs(*output), Box::new(function), Box::new(argument)) - } - } - _ => return Err(unsupported("Unsupported Python function expression")), - }; - if let Some(expected) = expected { - require_type(&tpe(&result), expected)?; - } - Ok(result) - } -} diff --git a/crates/morphir-python-binding/tests/acceptance.rs b/crates/morphir-python-binding/tests/acceptance.rs deleted file mode 100644 index 9a47e63e6..000000000 --- a/crates/morphir-python-binding/tests/acceptance.rs +++ /dev/null @@ -1,171 +0,0 @@ -use cucumber::{World, given, then, when}; -use morphir_extension_sdk::prelude::*; -use morphir_python_binding::PythonExtension; - -#[derive(Debug, Default)] -struct TestDriver { - source: String, - ir_version: Option, - additional: Vec, - compiled: Option, - generated: Option, -} - -impl TestDriver { - fn compile(&self, text: String) -> CompileResult { - let mut documents = vec![SourceDocument { - uri: "models.py".into(), - language_id: "python".into(), - version: 1, - text, - }]; - documents.extend(self.additional.clone()); - self.compile_documents(documents) - } - - fn compile_documents(&self, documents: Vec) -> CompileResult { - PythonExtension - .compile(CompileRequest { - language_id: "python".into(), - sources: SourceSet { - root: None, - documents, - }, - package: CompilePackage { - name: "acme/example".into(), - exposed_modules: None, - }, - dependencies: vec![], - options: CompileOptions { - ir_version: self.ir_version.clone().unwrap_or_else(|| "4".into()), - types_only: false, - ..Default::default() - }, - baseline: None, - }) - .unwrap() - } - - fn compile_and_generate(&mut self) { - let compiled = self.compile(self.source.clone()); - assert!(compiled.success, "{:?}", compiled.diagnostics); - let generated = PythonExtension - .generate(GenerateRequest { - ir: compiled.ir.clone().unwrap(), - target: "python".into(), - options: Default::default(), - }) - .unwrap(); - assert!(generated.success, "{:?}", generated.diagnostics); - self.compiled = Some(compiled); - self.generated = Some(generated); - } - - fn assert_roundtrip(&self) { - let recompiled = self.compile_documents( - self.generated - .as_ref() - .unwrap() - .artifacts - .iter() - .map(|a| SourceDocument { - uri: a.path.clone(), - language_id: "python".into(), - version: 1, - text: a.content.clone(), - }) - .collect(), - ); - assert!(recompiled.success, "{:?}", recompiled.diagnostics); - assert_eq!(recompiled.ir, self.compiled.as_ref().unwrap().ir); - } - - fn assert_rejected(&self) { - let result = self.compiled.as_ref().unwrap(); - assert!(!result.success); - assert!(result.ir.is_none()); - assert!( - result - .diagnostics - .iter() - .any(|d| d.severity == DiagnosticSeverity::Error) - ); - } -} - -#[derive(Debug, Default, World)] -struct PythonWorld { - driver: TestDriver, -} - -#[given(expr = "IR version {word}")] -fn ir_version(world: &mut PythonWorld, version: String) { - world.driver.ir_version = Some(version); -} - -#[given("a Python model with a product and a sum with payloads")] -fn model(world: &mut PythonWorld) { - world.driver.source = include_str!("fixtures/models.py").into(); -} - -#[given("a Python model containing an unannotated function")] -fn function(world: &mut PythonWorld) { - world.driver.source = "def value():\n return 1\n".into(); -} - -#[given("annotated Python functions with conditional bodies")] -fn conditionals(world: &mut PythonWorld) { - world.driver.source = include_str!("fixtures/conditionals.py").into(); -} - -#[given("Python tuple aliases and tuple-valued functions")] -fn tuples(world: &mut PythonWorld) { - world.driver.source = include_str!("fixtures/tuples.py").into(); -} - -#[given("Python higher-order functions and captured lambdas")] -fn higher_order(world: &mut PythonWorld) { - world.driver.source = include_str!("fixtures/functions.py").into(); -} - -#[given("Python modules with imported ADTs and tuple aliases")] -fn modules(world: &mut PythonWorld) { - world.driver.source = format!( - "{}\n{}", - include_str!("fixtures/models.py"), - include_str!("fixtures/tuples.py") - ); - world.driver.additional.push(SourceDocument { - uri: "rules.py".into(), - language_id: "python".into(), - version: 1, - text: include_str!("fixtures/modules/rules.py").into(), - }); -} -#[when("I compile the model and generate Python")] -fn roundtrip(world: &mut PythonWorld) { - world.driver.compile_and_generate(); -} - -#[when("I compile the model")] -fn compile(world: &mut PythonWorld) { - world.driver.compiled = Some(world.driver.compile(world.driver.source.clone())); -} - -#[then("compiling the generated Python preserves the model")] -fn preserved(world: &mut PythonWorld) { - world.driver.assert_roundtrip(); -} - -#[then("compilation fails without partial IR")] -fn rejected(world: &mut PythonWorld) { - world.driver.assert_rejected(); -} - -#[tokio::main] -async fn main() { - PythonWorld::cucumber() - .fail_on_skipped() - .run_and_exit(concat!(env!("CARGO_MANIFEST_DIR"), "/tests/features")) - .await; -} diff --git a/crates/morphir-python-binding/tests/conditionals.rs b/crates/morphir-python-binding/tests/conditionals.rs deleted file mode 100644 index 85db7ec30..000000000 --- a/crates/morphir-python-binding/tests/conditionals.rs +++ /dev/null @@ -1,275 +0,0 @@ -#[path = "support/native.rs"] -mod native; - -use morphir_extension_sdk::{ - prelude::*, - protocol::{ExtensionRequest, methods}, -}; -use morphir_python_binding::PythonExtension; -use serde_json::{Value, json}; - -fn request(source: &str) -> CompileRequest { - CompileRequest { - language_id: "python".into(), - sources: SourceSet { - root: None, - documents: vec![SourceDocument { - uri: "models.py".into(), - language_id: "python".into(), - version: 1, - text: source.into(), - }], - }, - package: CompilePackage { - name: "acme/example".into(), - exposed_modules: None, - }, - dependencies: vec![], - options: CompileOptions { - ir_version: "4".into(), - ..Default::default() - }, - baseline: None, - } -} - -fn compile(source: &str) -> CompileResult { - let extension = native::an_initialized_extension(); - let response = extension - .protocol() - .handle(ExtensionRequest::new(methods::COMPILE, request(source), 1).unwrap()); - serde_json::from_value(response.result.unwrap()).unwrap() -} - -fn generate(ir: Value) -> GenerateResult { - PythonExtension - .generate(GenerateRequest { - ir, - target: "python".into(), - options: Default::default(), - }) - .unwrap() -} - -fn roundtrip(source: &str) -> Value { - let compiled = compile(source); - assert!(compiled.success, "{:?}", compiled.diagnostics); - let ir = compiled.ir.unwrap(); - let generated = generate(ir.clone()); - assert!(generated.success, "{:?}", generated.diagnostics); - let again = compile(&generated.artifacts[0].content); - assert!(again.success, "{:?}", again.diagnostics); - assert_eq!(again.ir.as_ref(), Some(&ir)); - ir -} - -#[test] -fn conditionals_lower_to_canonical_if_then_else_and_roundtrip() { - let ir = roundtrip(include_str!("fixtures/conditionals.py")); - let choose = &ir["distribution"]["Library"]["def"]["modules"]["models"]["Public"]["values"]["choose"] - ["Public"]["ExpressionBody"]; - assert_eq!( - choose["inputTypes"], - json!({"flag": "morphir/SDK:basics#bool", "first": "morphir/SDK:basics#int", "second": "morphir/SDK:basics#int"}) - ); - assert_eq!(choose["outputType"], json!("morphir/SDK:basics#int")); - assert_eq!( - choose["body"], - json!({"IfThenElse": {"condition": {"Variable": "flag"}, "then": {"Variable": "first"}, "else": {"Variable": "second"}}}) - ); -} - -#[test] -fn branches_can_select_adt_and_tuple_parameters() { - roundtrip(&format!( - "{}\ndef select(flag: bool, first: Decision, second: Decision) -> Decision:\n return first if flag else second\n\ndef pair(flag: bool, first: tuple[int, str], second: tuple[int, str]) -> tuple[int, str]:\n return first if flag else second\n", - include_str!("fixtures/models.py") - )); -} - -#[test] -fn tuple_types_and_values_have_distinct_ir_encodings() { - let ir = roundtrip( - "def pair(amount: int, label: str) -> tuple[int, str]:\n return amount, label\n", - ); - let definition = &ir["distribution"]["Library"]["def"]["modules"]["models"]["Public"]["values"] - ["pair"]["Public"]["ExpressionBody"]; - assert_eq!( - definition["outputType"], - json!({"Tuple": ["morphir/SDK:basics#int", "morphir/SDK:string#string"]}) - ); - assert_eq!( - definition["body"], - json!({"Tuple": [{"Variable": "amount"}, {"Variable": "label"}]}) - ); -} - -#[test] -fn named_and_nested_tuple_aliases_roundtrip() { - roundtrip(include_str!("fixtures/tuples.py")); - roundtrip( - "type Pair = tuple[int, str]\ndef select(flag: bool, pair: Pair) -> tuple[int, str]:\n return pair if flag else (0, 'none')\n", - ); - roundtrip( - "type Pair = tuple[int, str]\ntype Nested = tuple[Pair, bool]\ndef make_nested(value: int) -> Nested:\n return ((value, 'value'), True)\n", - ); -} - -#[test] -fn invalid_tuple_types_and_values_are_rejected() { - for source in [ - "def f() -> tuple[int, str]:\n return (1, 2)\n", - "def f() -> tuple[int, str]:\n return (1, 'a', True)\n", - "def f(flag: bool) -> tuple[int, str]:\n return (1, 'a') if flag else (2, False)\n", - "def f() -> tuple[int]:\n return (1,)\n", - "def f() -> tuple[()]:\n return ()\n", - "type Pair = tuple[int, ...]\n", - "type Pair = tuple[int, Pair]\n", - "type A = tuple[int, B]\ntype B = tuple[str, A]\n", - "type Pair = tuple[int, str]\ndef f() -> Pair:\n return [1, 'a']\n", - "def f(pair: tuple[int, str]) -> tuple[int, str]:\n return (*pair,)\n", - ] { - let result = compile(source); - assert!(!result.success, "accepted {source}"); - assert!(result.ir.is_none()); - } -} - -#[test] -fn backend_accepts_independent_tuple_value_and_rejects_wrong_elements() { - let mut ir = independent_ir( - json!({"Tuple": [{"Literal": {"IntegerLiteral": 1}}, {"Literal": {"StringLiteral": "one"}}]}), - ); - ir["distribution"]["Library"]["def"]["modules"]["models"]["Public"]["values"]["select"]["Public"] - ["ExpressionBody"]["outputType"] = - json!({"Tuple": ["morphir/SDK:basics#int", "morphir/SDK:string#string"]}); - let generated = generate(ir.clone()); - assert!(generated.success, "{:?}", generated.diagnostics); - assert_eq!( - compile(&generated.artifacts[0].content).ir, - Some(ir.clone()) - ); - ir["distribution"]["Library"]["def"]["modules"]["models"]["Public"]["values"]["select"]["Public"] - ["ExpressionBody"]["body"]["Tuple"][1] = json!({"Literal": {"BoolLiteral": true}}); - let invalid = generate(ir); - assert!(!invalid.success); - assert!(invalid.artifacts.is_empty()); -} - -#[test] -fn comparisons_and_literal_boundaries_roundtrip() { - for operator in ["==", "!=", "<", "<=", ">", ">="] { - roundtrip(&format!( - "def compare(left: int, right: int) -> str:\n return 'yes' if left {operator} right else 'no'\n" - )); - } - roundtrip( - "def limits(flag: bool) -> int:\n return -9223372036854775808 if flag else 9223372036854775807\n", - ); - roundtrip( - "def text(flag: bool) -> str:\n return 'line\\n\\x00\\U0001f600' if flag else '\"quoted\"'\n", - ); -} - -#[test] -fn arbitrary_precision_integers_preserve_exact_values() { - for (literal, decimal) in [ - ("9223372036854775808", "9223372036854775808"), - ("18_446_744_073_709_551_616", "18446744073709551616"), - ("0x1_0000_0000_0000_0000", "18446744073709551616"), - ("0O2000000000000000000000", "18446744073709551616"), - ( - "0B10000000000000000000000000000000000000000000000000000000000000000", - "18446744073709551616", - ), - ( - "1234567890123456789012345678901234567890", - "1234567890123456789012345678901234567890", - ), - ] { - for sign in ["", "-"] { - let ir = roundtrip(&format!( - "def number() -> int:\n return {sign}{literal}\n" - )); - let value = &ir["distribution"]["Library"]["def"]["modules"]["models"]["Public"]["values"] - ["number"]["Public"]["ExpressionBody"]["body"]["Literal"]["IntegerLiteral"]; - assert_eq!(value.to_string(), format!("{sign}{decimal}")); - } - } -} - -#[test] -fn invalid_functions_fail_without_partial_ir() { - for source in [ - "def f(flag: bool) -> int:\n if flag:\n return 1\n", - "def f(flag: bool) -> int:\n return 1 if flag else 'no'\n", - "def f(flag: int) -> int:\n return 1 if flag else 2\n", - "def f(flag: bool) -> int:\n return missing if flag else 2\n", - "def f(flag: bool) -> int:\n if flag:\n print('side effect')\n return 1\n return 2\n", - "def f(flag: bool) -> int:\n result = 1\n return result\n", - "def f(flag: bool = True) -> int:\n return 1\n", - "def f(flag) -> int:\n return 1\n", - "def f(flag: bool):\n return 1\n", - "async def f(flag: bool) -> int:\n return 1\n", - "def f(*, flag: bool) -> int:\n return 1\n", - "def f(flag: bool, /) -> int:\n return 1\n", - "def f(a_b: bool, aB: bool) -> int:\n return 1\n", - "def f(Class: bool) -> int:\n return 1\n", - "def f(flag: bool) -> int:\n return\n", - "def f(flag: bool) -> int:\n return 1\n return 2\n", - "def f(flag: bool) -> float:\n return 1e400\n", - "def f(flag: bool) -> int:\n return 1 if flag < True else 2\n", - "def f(a: int, b: str) -> int:\n return 1 if a == b else 2\n", - "def f(flag: bool) -> int:\n return 1 if 0 < 1 < 2 else 2\n", - ] { - let result = compile(source); - assert!(!result.success, "accepted {source}"); - assert!(result.ir.is_none()); - assert!(!result.diagnostics.is_empty()); - } -} - -#[test] -fn types_only_does_not_silently_emit_function_bodies() { - let mut request = request(include_str!("fixtures/conditionals.py")); - request.options.types_only = true; - let result = PythonExtension.compile(request).unwrap(); - assert!(!result.success); - assert!(result.ir.is_none()); -} - -fn independent_ir(body: Value) -> Value { - let definition = json!({"ExpressionBody": { - "inputTypes": {"flag": "morphir/SDK:basics#bool"}, - "outputType": "morphir/SDK:basics#int", "body": body - }}); - json!({"formatVersion": 4, "distribution": {"Library": { - "packageName": "acme/example", "dependencies": {}, "def": {"modules": { - "models": {"Public": {"types": {}, "values": {"select": {"Public": definition}}}} - }} - }}}) -} - -#[test] -fn backend_accepts_independent_if_then_else_ir() { - let ir = independent_ir( - json!({"IfThenElse": {"condition": {"Variable": "flag"}, "then": {"Literal": {"IntegerLiteral": 1}}, "else": {"Literal": {"IntegerLiteral": 2}}}}), - ); - let result = generate(ir.clone()); - assert!(result.success, "{:?}", result.diagnostics); - assert_eq!(compile(&result.artifacts[0].content).ir, Some(ir)); -} - -#[test] -fn backend_rejects_invalid_bodies_without_artifacts() { - for body in [ - json!({"IfThenElse": {"condition": {"Literal": {"IntegerLiteral": 1}}, "then": {"Literal": {"IntegerLiteral": 1}}, "else": {"Literal": {"IntegerLiteral": 2}}}}), - json!({"IfThenElse": {"condition": {"Variable": "flag"}, "then": {"Literal": {"IntegerLiteral": 1}}, "else": {"Literal": {"StringLiteral": "no"}}}}), - json!({"Variable": "unknown"}), - json!({"Reference": "acme/example:models#select"}), - ] { - let result = generate(independent_ir(body)); - assert!(!result.success); - assert!(result.artifacts.is_empty()); - } -} diff --git a/crates/morphir-python-binding/tests/features/adt.feature b/crates/morphir-python-binding/tests/features/adt.feature deleted file mode 100644 index 2c615f1b7..000000000 --- a/crates/morphir-python-binding/tests/features/adt.feature +++ /dev/null @@ -1,47 +0,0 @@ -Feature: Python algebraic data types - Scenario Outline: Higher-order functions and closures roundtrip in both IR versions - Given IR version - And Python higher-order functions and captured lambdas - When I compile the model and generate Python - Then compiling the generated Python preserves the model - - Examples: - | version | - | 3 | - | 4 | - - Scenario: Model declarations survive a Python and Morphir roundtrip - Given a Python model with a product and a sum with payloads - When I compile the model and generate Python - Then compiling the generated Python preserves the model - - Scenario: Unsupported behavior is diagnosed - Given a Python model containing an unannotated function - When I compile the model - Then compilation fails without partial IR - - Scenario: Conditional function bodies survive a Python and Morphir roundtrip - Given annotated Python functions with conditional bodies - When I compile the model and generate Python - Then compiling the generated Python preserves the model - - Scenario: Tuple types and values survive a Python and Morphir roundtrip - Given Python tuple aliases and tuple-valued functions - When I compile the model and generate Python - Then compiling the generated Python preserves the model - - Scenario: Imports preserve types and conditional bodies across modules - Given Python modules with imported ADTs and tuple aliases - When I compile the model and generate Python - Then compiling the generated Python preserves the model - - Scenario Outline: Versioned multi-module models roundtrip - Given IR version - And Python modules with imported ADTs and tuple aliases - When I compile the model and generate Python - Then compiling the generated Python preserves the model - - Examples: - | version | - | 3 | - | 4 | diff --git a/crates/morphir-python-binding/tests/fixtures/conditionals.py b/crates/morphir-python-binding/tests/fixtures/conditionals.py deleted file mode 100644 index 49a8b7356..000000000 --- a/crates/morphir-python-binding/tests/fixtures/conditionals.py +++ /dev/null @@ -1,35 +0,0 @@ -def choose(flag: bool, first: int, second: int) -> int: - if flag: - return first - else: - return second - - -def classify(value: int) -> str: - if value < 0: - return "negative" - elif value == 0: - return "zero" - else: - return "positive" - - -def nested(first: bool, second: bool) -> int: - if first: - if second: - return 1 - else: - return 2 - else: - return 3 - - -def guarded(first: bool, second: bool) -> int: - if first: - if second: - return 1 - return 2 - - -def expression(flag: bool) -> float: - return -1.5 if flag else 2.5 diff --git a/crates/morphir-python-binding/tests/fixtures/functions.py b/crates/morphir-python-binding/tests/fixtures/functions.py deleted file mode 100644 index 7e4060fa3..000000000 --- a/crates/morphir-python-binding/tests/fixtures/functions.py +++ /dev/null @@ -1,36 +0,0 @@ -from collections.abc import Callable -def identity(value: int) -> int: - return value -def apply(transform: Callable[[int], int], value: int) -> int: - return transform(value) -def capture(value: int) -> Callable[[int], int]: - return lambda ignored: value -def shadow(value: int) -> Callable[[int], int]: - return lambda value: value -def curry(flag: bool) -> Callable[[int], Callable[[int], int]]: - return lambda first: lambda second: first if flag else second -def choose(flag: bool, first: int, second: int) -> int: - return first if flag else second -def run(value: int) -> int: - return choose(True, apply(lambda item: identity(item), value), curry(False)(1)(2)) -def recurse(flag: bool, value: int) -> int: - return recurse(False, value) if flag else value -def constant() -> int: - return 42 -def read_constant(value: int) -> int: - return constant() - -def immediate(value: int) -> int: - return (lambda item: item)(value) - -def tupled(value: int) -> tuple[Callable[[int], int], int]: - return (lambda ignored: value, value) - -def conditional(flag: bool) -> Callable[[int], int]: - return identity if flag else lambda item: item - -def forward(value: int) -> int: - return later(value) - -def later(value: int) -> int: - return value diff --git a/crates/morphir-python-binding/tests/fixtures/models.py b/crates/morphir-python-binding/tests/fixtures/models.py deleted file mode 100644 index 8fb4b26fd..000000000 --- a/crates/morphir-python-binding/tests/fixtures/models.py +++ /dev/null @@ -1,24 +0,0 @@ -from __future__ import annotations -from dataclasses import dataclass - -@dataclass(frozen=True) -class Address: - street: str - zip_code: int - -@dataclass(frozen=True) -class Pending: - pass - -@dataclass(frozen=True) -class Approved: - amount: float - address: Address - -type Decision = Pending | Approved - -@dataclass(frozen=True) -class Application: - decision: Decision - verified: bool - coordinates: tuple[float, float] diff --git a/crates/morphir-python-binding/tests/fixtures/modules/rules.py b/crates/morphir-python-binding/tests/fixtures/modules/rules.py deleted file mode 100644 index d00893ec5..000000000 --- a/crates/morphir-python-binding/tests/fixtures/modules/rules.py +++ /dev/null @@ -1,13 +0,0 @@ -from __future__ import annotations -from models import Decision, Point - - -def select_decision(flag: bool, first: Decision, second: Decision) -> Decision: - if flag: - return first - else: - return second - - -def origin() -> Point: - return (0.0, 0.0) diff --git a/crates/morphir-python-binding/tests/fixtures/tuples.py b/crates/morphir-python-binding/tests/fixtures/tuples.py deleted file mode 100644 index f76562b5f..000000000 --- a/crates/morphir-python-binding/tests/fixtures/tuples.py +++ /dev/null @@ -1,13 +0,0 @@ -type Point = tuple[float, float] -type LabeledPoint = tuple[Point, str] - - -def make_point(x: float, y: float) -> Point: - return (x, y) - - -def located(flag: bool, location: Point) -> LabeledPoint: - if flag: - return (location, "known") - else: - return ((0.0, 0.0), "unknown") diff --git a/crates/morphir-python-binding/tests/functions.rs b/crates/morphir-python-binding/tests/functions.rs deleted file mode 100644 index 9d6288519..000000000 --- a/crates/morphir-python-binding/tests/functions.rs +++ /dev/null @@ -1,378 +0,0 @@ -use morphir_extension_sdk::prelude::*; -use morphir_python_binding::PythonExtension; - -fn request(version: &str, source: &str) -> CompileRequest { - CompileRequest { - language_id: "python".into(), - sources: SourceSet { - root: None, - documents: vec![SourceDocument { - uri: "functions.py".into(), - language_id: "python".into(), - text: source.into(), - ..Default::default() - }], - }, - package: CompilePackage { - name: "acme/functions".into(), - ..Default::default() - }, - options: CompileOptions { - ir_version: version.into(), - types_only: false, - ..Default::default() - }, - ..Default::default() - } -} - -const FUNCTIONS: &str = include_str!("fixtures/functions.py"); - -#[test] -fn functions_calls_and_captured_lambdas_roundtrip_in_both_ir_versions() { - for version in ["3", "4"] { - let mut input = request(version, FUNCTIONS); - input.sources.documents.push(SourceDocument { - uri: "consumer.py".into(), language_id: "python".into(), - text: "from typing import Callable as Fn\nfrom functions import identity as ident, apply\ndef use(value: int) -> int:\n return apply(ident, value)\ndef make(value: int) -> Fn[[int], int]:\n return lambda ignored: ident(value)\n".into(), - ..Default::default() - }); - input.package.exposed_modules = Some(vec!["consumer".into()]); - let compiled = PythonExtension.compile(input.clone()).unwrap(); - assert!(compiled.success, "{version}: {:?}", compiled.diagnostics); - let ir = compiled.ir.unwrap(); - let generated = PythonExtension - .generate(GenerateRequest { - target: "python".into(), - ir: ir.clone(), - options: Default::default(), - }) - .unwrap(); - assert!(generated.success, "{version}: {:?}", generated.diagnostics); - assert!( - generated - .artifacts - .iter() - .any(|a| a.content.contains("lambda")) - ); - input.sources.documents = generated - .artifacts - .into_iter() - .map(|artifact| SourceDocument { - uri: artifact.path, - text: artifact.content, - language_id: "python".into(), - ..Default::default() - }) - .collect(); - let recompiled = PythonExtension.compile(input).unwrap(); - assert!( - recompiled.success, - "{version}: {:?}", - recompiled.diagnostics - ); - assert_eq!(recompiled.ir, Some(ir)); - } -} - -#[test] -fn invalid_calls_and_lambda_types_return_diagnostics_without_ir() { - for version in ["3", "4"] { - for source in [ - "def bad(value: int) -> int:\n return missing(value)\n", - "from typing import Callable\ndef many(first: int, second: int) -> int:\n return first\ndef bad(value: int) -> Callable[[int], Callable[[int], int]]:\n return many\n", - "from typing import Callable\ndef constant() -> int:\n return 1\ndef bad(value: int) -> Callable[[int], int]:\n return constant\n", - "from typing import Callable\ndef bad(value: Callable[[int], int]) -> int:\n return value(True)\n", - "from typing import Callable\ndef bad(value: int) -> Callable[[int], int]:\n return lambda item: absent\n", - "def identity(value: int) -> int:\n return value\ndef bad(value: int) -> int:\n return identity(True)\n", - "def choose(a: int, b: int) -> int:\n return a\ndef bad(value: int) -> int:\n return choose(value)\n", - "from collections.abc import Callable\ndef bad(value: int) -> Callable[[int], int]:\n return lambda item: True\n", - "from collections.abc import Callable\ndef bad(value: int) -> Callable[[int], int]:\n return lambda first, second: first\n", - "from collections.abc import Callable\ndef bad(value: int) -> Callable[[int], int]:\n return lambda item=1: item\n", - "from collections.abc import Callable\ndef bad(fooBar: int) -> Callable[[int], int]:\n return lambda foo_bar: fooBar\n", - "from collections.abc import Callable\ndef bad(value: Callable[[int, int], int]) -> int:\n return value(1, 2)\n", - "def identity(value: int) -> int:\n return value\ndef bad(value: int) -> int:\n return identity(value=value)\n", - "def identity(value: int) -> int:\n return value\ndef bad(value: int) -> int:\n return identity(*value)\n", - ] { - let result = PythonExtension.compile(request(version, source)).unwrap(); - assert!(!result.success, "{version}: {source}"); - assert!(result.ir.is_none()); - assert!(!result.diagnostics.is_empty()); - } - } -} - -fn generate(ir: serde_json::Value) -> GenerateResult { - PythonExtension - .generate(GenerateRequest { - target: "python".into(), - ir, - options: Default::default(), - }) - .unwrap() -} - -/// Authored directly against the classic IR contract, independently of the frontend. -fn classic_lambda() -> serde_json::Value { - use serde_json::json; - let int = json!([ - "Reference", - {}, - [[["morphir"], ["s", "d", "k"]], [["basics"]], ["int"]], - [] - ]); - let function = json!(["Function", {}, int, int]); - let identity = json!([ - "Lambda", - function, - ["AsPattern", int, ["WildcardPattern", int], ["item"]], - ["Variable", int, ["item"]] - ]); - json!({"formatVersion":3,"distribution":["Library",[["acme"],["functions"]],[],{"modules":[ - [[["functions"]],{"access":"Public","value":{"doc":null,"types":[],"values":[ - [["run"],{"access":"Public","value":{"doc":"","value":{ - "inputTypes":[[["value"],int,int]],"outputType":int, - "body":["Apply",int,identity,["Variable",int,["value"]]] - }}}] - ]}}] - ]}]}) -} - -fn independent_closure() -> serde_json::Value { - use morphir_core::ir::v4::*; - use morphir_core::naming::FQName; - let int = Type::Reference( - Default::default(), - FQName::from_canonical_string("morphir/SDK:basics#int").unwrap(), - vec![], - ); - let variable = |name: &str| { - Value::Variable( - Default::default(), - Name::from_canonical_string(name).unwrap(), - ) - }; - let identity = ValueDefinition { - input_types: [("value".into(), int.clone())].into_iter().collect(), - output_type: Some(int.clone()), - body: ValueBody::Expression(variable("value")), - }; - let closure = ValueDefinition { - input_types: [ - ("identity".into(), int.clone()), - ("morphir-module-1".into(), int.clone()), - ] - .into_iter() - .collect(), - output_type: Some(Type::Function( - Default::default(), - Box::new(int.clone()), - Box::new(int), - )), - body: ValueBody::Expression(Value::Lambda( - Default::default(), - Pattern::AsPattern( - Default::default(), - Box::new(Pattern::WildcardPattern(Default::default())), - Name::from_canonical_string("morphir-module-2").unwrap(), - ), - Box::new(Value::Apply( - Default::default(), - Box::new(Value::Reference( - Default::default(), - FQName::from_canonical_string("acme/functions:functions#identity").unwrap(), - )), - Box::new(variable("identity")), - )), - )), - }; - let value = |definition| AccessControlled { - access: Access::Public, - value: Documented { - doc: None, - value: definition, - }, - }; - serde_json::to_value(IRFile { - format_version: FormatVersion::Integer(4), - metadata: None, - distribution: Distribution::Library(LibraryContent { - package_name: PackageName::from_canonical_string("acme/functions").unwrap(), - dependencies: Default::default(), - def: PackageDefinition { - modules: [( - "functions".into(), - AccessControlled { - access: Access::Public, - value: ModuleDefinition { - types: Default::default(), - values: [ - ("identity".into(), value(identity)), - ("make".into(), value(closure)), - ] - .into_iter() - .collect(), - doc: None, - }, - }, - )] - .into_iter() - .collect(), - }, - }), - }) - .unwrap() -} - -#[test] -fn independent_ir_keeps_global_references_distinct_from_captured_parameters() { - let ir = independent_closure(); - let generated = generate(ir.clone()); - assert!(generated.success, "{:?}", generated.diagnostics); - assert!( - generated.artifacts[0] - .content - .contains("import functions as morphir_module_3") - ); - let compiled = PythonExtension - .compile(request("4", &generated.artifacts[0].content)) - .unwrap(); - assert!(compiled.success, "{:?}", compiled.diagnostics); - assert_eq!( - serde_json::from_value::(compiled.ir.unwrap()).unwrap(), - serde_json::from_value::(ir).unwrap() - ); -} - -#[test] -fn backend_rejects_unsupported_lambda_patterns_and_unknown_references() { - use serde_json::json; - for (path, replacement) in [ - ("/pattern", json!({"WildcardPattern": {}})), - ( - "/body/Apply/function", - json!({"Reference": "acme/functions:missing#identity"}), - ), - ("/body/Apply/argument", json!({"Variable": "missing"})), - ] { - let mut ir = independent_closure(); - let body = &mut ir["distribution"]["Library"]["def"]["modules"]["functions"]["Public"]["values"] - ["make"]["Public"]["ExpressionBody"]["body"]["Lambda"]; - *body.pointer_mut(path).unwrap() = replacement; - let result = generate(ir); - assert!(!result.success, "accepted {path}"); - assert!(result.artifacts.is_empty()); - } -} - -#[test] -fn function_exports_cannot_be_used_as_type_references() { - let encoded = independent_closure().to_string().replace( - "morphir/SDK:basics#int", - "acme/functions:functions#identity", - ); - let result = generate(serde_json::from_str(&encoded).unwrap()); - assert!( - !result.success, - "a value reference was accepted in a type position" - ); - assert!(result.artifacts.is_empty()); -} - -#[test] -fn callable_qualified_annotations_and_aliases_resolve_statically() { - for version in ["3", "4"] { - for import in ["import typing as types", "import collections.abc as types"] { - let source = format!( - "{import}\ndef apply(transform: types.Callable[[int], int], value: int) -> int:\n return transform(value)\n" - ); - let result = PythonExtension.compile(request(version, &source)).unwrap(); - assert!(result.success, "{:?}", result.diagnostics); - } - } -} - -#[test] -fn independently_authored_classic_lambda_and_application_generate_and_roundtrip() { - let ir = classic_lambda(); - let result = generate(ir.clone()); - assert!(result.success, "{:?}", result.diagnostics); - let compiled = PythonExtension - .compile(request("3", &result.artifacts[0].content)) - .unwrap(); - assert!(compiled.success, "{:?}", compiled.diagnostics); - assert_eq!(compiled.ir, Some(ir)); -} - -#[test] -fn classic_lambda_and_pattern_annotations_are_checked() { - use serde_json::json; - let boolean = json!([ - "Reference", - {}, - [[["morphir"], ["s", "d", "k"]], [["basics"]], ["bool"]], - [] - ]); - for path in ["/1", "/2/1", "/2/2/1", "/3/1"] { - let mut ir = classic_lambda(); - let lambda = &mut ir["distribution"][3]["modules"][0][1]["value"]["values"][0][1]["value"] - ["value"]["body"][2]; - *lambda.pointer_mut(path).unwrap() = boolean.clone(); - let result = generate(ir); - assert!(!result.success, "accepted malformed annotation {path}"); - assert!(result.artifacts.is_empty()); - } -} - -#[test] -#[ignore = "requires MORPHIR_TEST_PYTHON; exercised in the Python extension CI job"] -fn generated_python_executes_functions_and_closures() { - use std::{fs, process::Command}; - let python = std::env::var_os("MORPHIR_TEST_PYTHON") - .expect("set MORPHIR_TEST_PYTHON to a Python 3.12+ executable"); - for version in ["3", "4"] { - let mut input = request(version, FUNCTIONS); - input.sources.documents.push(SourceDocument { uri: "consumer.py".into(), language_id: "python".into(), text: "from functions import apply, identity\ndef use(value: int) -> int:\n return apply(identity, value)\n".into(), ..Default::default() }); - let compiled = PythonExtension.compile(input).unwrap(); - assert!(compiled.success, "{:?}", compiled.diagnostics); - let generated = generate(compiled.ir.unwrap()); - assert!(generated.success, "{:?}", generated.diagnostics); - let directory = tempfile::tempdir().unwrap(); - for artifact in generated.artifacts { - fs::write(directory.path().join(artifact.path), artifact.content).unwrap(); - } - let output = Command::new(&python).current_dir(directory.path()).arg("-c").arg(concat!( - "import functions as f, consumer\n", - "assert f.run(7) == 7\nassert consumer.use(9) == 9\n", - "assert f.capture(7)(2) == 7\nassert f.shadow(7)(2) == 2\n", - "assert f.curry(True)(1)(2) == 1\nassert f.curry(False)(1)(2) == 2\n", - "assert f.recurse(True, 8) == 8\nassert f.read_constant(0) == 42\nassert f.immediate(5) == 5\n", - "assert f.tupled(7)[0](2) == 7\nassert f.tupled(7)[1] == 7\n", - "assert f.conditional(True)(4) == 4\nassert f.conditional(False)(4) == 4\nassert f.forward(6) == 6\n", - )).output().unwrap(); - assert!( - output.status.success(), - "{}", - String::from_utf8_lossy(&output.stderr) - ); - } - let generated = generate(independent_closure()); - assert!(generated.success, "{:?}", generated.diagnostics); - let directory = tempfile::tempdir().unwrap(); - for artifact in generated.artifacts { - fs::write(directory.path().join(artifact.path), artifact.content).unwrap(); - } - let output = Command::new(&python) - .current_dir(directory.path()) - .args([ - "-c", - "import functions; assert functions.make(7, 99)(3) == 7", - ]) - .output() - .unwrap(); - assert!( - output.status.success(), - "{}", - String::from_utf8_lossy(&output.stderr) - ); -} diff --git a/crates/morphir-python-binding/tests/ir_versions.rs b/crates/morphir-python-binding/tests/ir_versions.rs deleted file mode 100644 index 9da15e60e..000000000 --- a/crates/morphir-python-binding/tests/ir_versions.rs +++ /dev/null @@ -1,261 +0,0 @@ -use morphir_core::ir::classic; -use morphir_extension_sdk::prelude::*; -use morphir_python_binding::PythonExtension; -use serde_json::{Value, json}; - -fn a_request(version: &str) -> CompileRequest { - CompileRequest { - language_id: "python".into(), - sources: SourceSet { - root: None, - documents: vec![ - SourceDocument { - uri: "domain/models.py".into(), language_id: "python".into(), version: 1, - text: concat!( - "from __future__ import annotations\nfrom dataclasses import dataclass\n", - "@dataclass(frozen=True)\nclass Pending:\n pass\n", - "@dataclass(frozen=True)\nclass Approved:\n amount: float\n", - "type Decision = Pending | Approved\n", - "type Point = tuple[float, float]\n", - "@dataclass(frozen=True)\nclass APIResponse:\n decision: Decision\n point: Point\n", - ).into(), - }, - SourceDocument { - uri: "domain/rules.py".into(), language_id: "python".into(), version: 1, - text: concat!( - "from .models import Decision, Point\n", - "def choose(flag: bool, first: Decision, second: Decision) -> Decision:\n", - " return first if flag else second\n", - "def locate(value: int, point: Point) -> tuple[Point, str]:\n", - " if value < 0:\n return (point, 'negative')\n", - " elif value == 0:\n return ((0.0, 0.0), 'zero')\n", - " else:\n return (point, 'positive')\n", - ).into(), - }, - ], - }, - package: CompilePackage { name: "acme/example".into(), exposed_modules: Some(vec!["domain.rules".into()]) }, - dependencies: vec![], - options: CompileOptions { ir_version: version.into(), types_only: false, ..Default::default() }, - baseline: None, - } -} - -fn compile(request: CompileRequest) -> Value { - let expected = if request.options.ir_version.starts_with('3') { - "3" - } else { - "4" - }; - let result = PythonExtension.compile(request).unwrap(); - assert!(result.success, "{:?}", result.diagnostics); - assert_eq!(result.ir_version.as_deref(), Some(expected)); - result.ir.unwrap() -} - -fn generate(ir: Value) -> GenerateResult { - PythonExtension - .generate(GenerateRequest { - ir, - target: "python".into(), - options: Default::default(), - }) - .unwrap() -} - -#[test] -fn both_versions_preserve_imports_privacy_tuples_and_conditionals() { - for version in ["3", "3.0.0", "4", "4.0.0"] { - let mut request = a_request(version); - let ir = compile(request.clone()); - if version.starts_with('3') { - let decoded: classic::Distribution = serde_json::from_value(ir.clone()).unwrap(); - assert_eq!(decoded.format_version, 3); - let classic::DistributionBody::Library(package, _, definition) = decoded.distribution - else { - panic!("a Library") - }; - assert_eq!( - serde_json::to_value(package).unwrap(), - json!([["acme"], ["example"]]) - ); - assert_eq!( - definition.modules[0].definition.access, - classic::Access::Private - ); - assert_eq!( - definition.modules[1].definition.access, - classic::Access::Public - ); - assert_eq!(ir["distribution"][0], "Library"); - } - let generated = generate(ir.clone()); - assert!(generated.success, "{:?}", generated.diagnostics); - assert_eq!(generated.artifacts.len(), 2); - request.sources.documents = generated - .artifacts - .into_iter() - .map(|artifact| SourceDocument { - uri: artifact.path, - text: artifact.content, - language_id: "python".into(), - version: 1, - }) - .collect(); - assert_eq!(compile(request), ir); - } -} - -#[test] -fn independently_authored_v3_record_generates_python() { - let scalar = json!([ - "Reference", - {}, - [[["morphir"], ["s", "d", "k"]], [["basics"]], ["int"]], - [] - ]); - let ir = json!({"formatVersion": 3, "distribution": ["Library", [["acme"], ["example"]], [], {"modules": [ - [[["models"]], {"access":"Public", "value":{"types":[ - [["record"], {"access":"Public", "value":{"doc":"", "value":["TypeAliasDefinition", [], ["Record", {}, [ - {"name":["count"], "tpe":scalar} - ]]]}}] - ], "values":[]}}] - ]}]}); - let result = generate(ir); - assert!(result.success, "{:?}", result.diagnostics); - assert!(result.artifacts[0].content.contains("count: int")); -} - -#[test] -fn independently_authored_typed_v3_conditional_generates_python() { - let scalar = |name| { - json!([ - "Reference", - {}, - [[["morphir"], ["s", "d", "k"]], [["basics"]], [name]], - [] - ]) - }; - let integer = scalar("int"); - let boolean = scalar("bool"); - let ir = json!({"formatVersion":3,"distribution":["Library",[["acme"],["example"]],[],{"modules":[ - [[["models"]],{"access":"Public","value":{"types":[],"values":[ - [["choose"],{"access":"Public","value":{"doc":"","value":{ - "inputTypes":[[["flag"],boolean,boolean]], "outputType":integer, - "body":["IfThenElse",integer,["Variable",boolean,["flag"]], - ["Literal",integer,["WholeNumberLiteral",1]],["Literal",integer,["WholeNumberLiteral",2]]] - }}}] - ]}}] - ]}]}); - let result = generate(ir); - assert!(result.success, "{:?}", result.diagnostics); - assert!( - result.artifacts[0] - .content - .contains("def choose(flag: bool) -> int:") - ); - assert!(result.artifacts[0].content.contains("if flag:")); -} - -#[test] -fn v3_parameter_annotations_accept_an_expanded_tuple_alias() { - let mut ir = compile(a_request("3")); - let float = json!([ - "Reference", - {}, - [[["morphir"], ["s", "d", "k"]], [["basics"]], ["float"]], - [] - ]); - ir["distribution"][3]["modules"][1][1]["value"]["values"][1][1]["value"]["value"]["inputTypes"] - [1][1] = json!(["Tuple", {}, [float.clone(), float]]); - let result = generate(ir); - assert!(result.success, "{:?}", result.diagnostics); -} - -#[test] -fn v3_integer_range_is_checked_without_affecting_v4() { - for number in ["-9223372036854775808", "9223372036854775807"] { - let mut request = a_request("3"); - request.sources.documents[1].text = format!("def number() -> int:\n return {number}\n"); - let result = generate(compile(request)); - assert!(result.success, "{:?}", result.diagnostics); - } - for number in ["9223372036854775808", "-9223372036854775809"] { - let mut request = a_request("3"); - request.sources.documents[1].text = format!("def number() -> int:\n return {number}\n"); - let result = PythonExtension.compile(request.clone()).unwrap(); - assert!(!result.success); - assert!(result.ir.is_none()); - assert!(result.diagnostics[0].message.contains("64-bit")); - request.options.ir_version = "4".into(); - compile(request); - } -} - -#[test] -fn rejects_unsupported_versions_and_inconsistent_v3_value_annotations() { - for version in ["1", "2", "3.1.0", "3.0.1", "4.1.0", "3.0", "03", "latest"] { - let result = PythonExtension.compile(a_request(version)).unwrap(); - assert!(!result.success, "accepted {version}"); - assert!(result.ir.is_none()); - } - let mut ir = compile(a_request("3")); - ir["distribution"][3]["modules"][1][1]["value"]["values"][0][1]["value"]["value"]["body"][1] = - json!(["Unit", {}]); - let result = generate(ir); - assert!(!result.success); - assert!(result.artifacts.is_empty()); -} - -#[test] -fn backend_reads_compatible_patches_but_rejects_other_minor_versions() { - for (baseline, rejected_minor) in [("3", "3.2.0"), ("4", "4.1.0")] { - let mut ir = compile(a_request(baseline)); - ir["formatVersion"] = json!(format!("{baseline}.0.7")); - let result = generate(ir.clone()); - assert!(result.success, "{:?}", result.diagnostics); - ir["formatVersion"] = json!(rejected_minor); - assert!(!generate(ir).success); - } -} - -#[test] -fn v3_rejects_duplicate_names_documentation_and_untyped_function_bodies() { - let source = compile(a_request("3")); - let mut duplicate = source.clone(); - let module = duplicate["distribution"][3]["modules"][0].clone(); - duplicate["distribution"][3]["modules"] - .as_array_mut() - .unwrap() - .push(module); - assert!(!generate(duplicate).success); - let mut docs = source.clone(); - docs["distribution"][3]["modules"][0][1]["value"]["types"][0][1]["value"]["doc"] = - json!("Cannot preserve this"); - assert!(!generate(docs).success); - let mut untyped = source; - untyped["distribution"][3]["modules"][1][1]["value"]["values"][0][1]["value"]["value"]["body"] - [1] = json!({}); - assert!(!generate(untyped).success); -} - -#[test] -fn classic_names_cannot_silently_change_the_python_model() { - let mut request = a_request("4"); - request.sources.documents.truncate(1); - request.package.exposed_modules = None; - request.sources.documents[0].text = - "from dataclasses import dataclass\n@dataclass(frozen=True)\nclass Record:\n a_b: int\n" - .into(); - let expected = generate(compile(request.clone())); - assert!(expected.success); - request.options.ir_version = "3".into(); - let compiled = PythonExtension.compile(request).unwrap(); - if compiled.success { - let actual = generate(compiled.ir.unwrap()); - assert!(actual.success, "{:?}", actual.diagnostics); - assert_eq!(actual.artifacts[0].content, expected.artifacts[0].content); - } else { - assert_eq!(compiled.diagnostics[0].code.as_deref(), Some("PY003")); - } -} diff --git a/crates/morphir-python-binding/tests/modules.rs b/crates/morphir-python-binding/tests/modules.rs deleted file mode 100644 index b52b756bb..000000000 --- a/crates/morphir-python-binding/tests/modules.rs +++ /dev/null @@ -1,376 +0,0 @@ -use morphir_extension_sdk::prelude::*; -use morphir_python_binding::PythonExtension; -use serde_json::{Value, json}; - -fn request(sources: &[(&str, &str)]) -> CompileRequest { - CompileRequest { - language_id: "python".into(), - sources: SourceSet { - root: None, - documents: sources - .iter() - .map(|(uri, text)| SourceDocument { - uri: (*uri).into(), - language_id: "python".into(), - version: 1, - text: (*text).into(), - }) - .collect(), - }, - package: CompilePackage { - name: "acme/example".into(), - exposed_modules: None, - }, - dependencies: vec![], - options: CompileOptions { - ir_version: "4".into(), - types_only: false, - ..Default::default() - }, - baseline: None, - } -} - -fn compile(request: CompileRequest) -> Value { - let result = PythonExtension.compile(request).unwrap(); - assert!(result.success, "{:?}", result.diagnostics); - result.ir.unwrap() -} - -fn roundtrip(request: CompileRequest) -> (Value, Vec) { - let ir = compile(request.clone()); - let mut reversed = request; - reversed.sources.documents.reverse(); - assert_eq!( - ir, - compile(reversed), - "document order must not affect resolution" - ); - let generated = PythonExtension - .generate(GenerateRequest { - ir: ir.clone(), - target: "python".into(), - options: Default::default(), - }) - .unwrap(); - assert!(generated.success, "{:?}", generated.diagnostics); - let sources: Vec<_> = generated - .artifacts - .iter() - .map(|a| (a.path.as_str(), a.content.as_str())) - .collect(); - assert_eq!(ir, compile(self::request(&sources))); - (ir, generated.artifacts) -} - -const TYPES: &str = "from dataclasses import dataclass\n@dataclass(frozen=True)\nclass Address:\n zip_code: int\ntype Pair = tuple[int, str]\n"; - -#[test] -fn private_modules_are_available_inside_the_package_and_survive_generation() { - let mut input = request(&[ - ("internal/models.py", TYPES), - ( - "api.py", - "from internal.models import Pair\ndef origin() -> Pair:\n return (1, 'one')\n", - ), - ]); - input.package.exposed_modules = Some(vec!["api".into()]); - let ir = compile(input.clone()); - assert!( - ir["distribution"]["Library"]["def"]["modules"]["internal/models"]["Private"].is_object() - ); - assert!(ir["distribution"]["Library"]["def"]["modules"]["api"]["Public"].is_object()); - let generated = PythonExtension - .generate(GenerateRequest { - ir: ir.clone(), - target: "python".into(), - options: Default::default(), - }) - .unwrap(); - assert!(generated.success, "{:?}", generated.diagnostics); - input.sources.documents = generated - .artifacts - .iter() - .map(|artifact| SourceDocument { - uri: artifact.path.clone(), - text: artifact.content.clone(), - language_id: "python".into(), - version: 1, - }) - .collect(); - assert_eq!(ir, compile(input)); -} - -#[test] -fn explicitly_empty_exposure_makes_every_module_private() { - let mut input = request(&[("models.py", TYPES)]); - input.package.exposed_modules = Some(vec![]); - let ir = compile(input); - assert!(ir["distribution"]["Library"]["def"]["modules"]["models"]["Private"].is_object()); -} - -#[test] -fn imported_types_and_tuple_aliases_roundtrip_across_modules() { - let (ir, artifacts) = roundtrip(request(&[ - ("models.py", TYPES), - ( - "rules.py", - "from models import Address as Home, Pair\ndef choose(flag: bool, first: Home, second: Home) -> Home:\n return first if flag else second\ndef make_pair() -> Pair:\n return (1, 'one')\n", - ), - ])); - assert_eq!( - artifacts - .iter() - .map(|a| a.path.as_str()) - .collect::>(), - ["models.py", "rules.py"] - ); - assert_eq!( - ir["distribution"]["Library"]["def"]["modules"]["rules"]["Public"]["values"]["choose"]["Public"] - ["ExpressionBody"]["outputType"], - json!("acme/example:models#address") - ); -} - -#[test] -fn nested_modules_resolve_absolute_relative_and_qualified_imports() { - let mut input = request(&[ - ("file:///project/src/domain/models.py", TYPES), - ( - "file:///project/src/domain/rules.py", - "from .models import Pair\nimport domain.models as model\nfrom dataclasses import dataclass\n@dataclass(frozen=True)\nclass Holder:\n home: model.Address\ndef make_pair() -> Pair:\n return (1, 'one')\n", - ), - ( - "file:///project/src/app.py", - "import domain.models\nfrom domain import rules as rule\nfrom dataclasses import dataclass\n@dataclass(frozen=True)\nclass App:\n home: domain.models.Address\n holder: rule.Holder\n", - ), - ]); - input.sources.root = Some("file:///project/src".into()); - input.package.exposed_modules = Some(vec![ - "App".into(), - "Domain.Models".into(), - "Domain.Rules".into(), - ]); - let (_, artifacts) = roundtrip(input); - assert_eq!( - artifacts - .iter() - .map(|a| a.path.as_str()) - .collect::>(), - ["app.py", "domain/models.py", "domain/rules.py"] - ); -} - -#[test] -fn mutually_referencing_records_and_same_named_types_roundtrip() { - roundtrip(request(&[ - ( - "first.py", - "from dataclasses import dataclass\nimport second\n@dataclass(frozen=True)\nclass Item:\n other: second.Item\n", - ), - ( - "second.py", - "from dataclasses import dataclass\nimport first\n@dataclass(frozen=True)\nclass Item:\n other: first.Item\n", - ), - ])); -} - -#[test] -fn rejects_invalid_module_graphs_without_partial_ir() { - for sources in [ - vec![], - vec![("models.py", TYPES), ("models.py", TYPES)], - vec![("foo_bar.py", ""), ("fooBar.py", "")], - vec![ - ("models.py", TYPES), - ("rules.py", "from missing import Address\n"), - ], - vec![ - ("models.py", TYPES), - ("rules.py", "from models import Missing\n"), - ], - vec![("models.py", TYPES), ("rules.py", "from models import *\n")], - vec![ - ("models.py", TYPES), - ("rules.py", "from models import Address as int\n"), - ], - vec![ - ("models.py", TYPES), - ( - "rules.py", - "from models import Address\nfrom dataclasses import dataclass\n@dataclass(frozen=True)\nclass Address:\n x: int\n", - ), - ], - vec![ - ("models.py", TYPES), - ("rules.py", "from .models import Address\n"), - ], - vec![ - ( - "a.py", - "from b import Pair as Other\ntype Pair = tuple[Other, int]\n", - ), - ( - "b.py", - "from a import Pair as Other\ntype Pair = tuple[Other, int]\n", - ), - ], - vec![("../models.py", TYPES)], - vec![("models/__init__.py", TYPES)], - vec![("models.py", TYPES), ("models/child.py", TYPES)], - ] { - let result = PythonExtension.compile(request(&sources)).unwrap(); - assert!(!result.success, "accepted {sources:?}"); - assert!(result.ir.is_none()); - assert!(result.modules.is_empty()); - } -} - -#[test] -fn rejects_sources_outside_root_and_unknown_exposed_modules() { - let mut input = request(&[("file:///other/models.py", TYPES)]); - input.sources.root = Some("file:///project".into()); - assert!(!PythonExtension.compile(input).unwrap().success); - let mut input = request(&[("models.py", TYPES), ("rules.py", "")]); - input.package.exposed_modules = Some(vec!["Missing".into()]); - assert!(!PythonExtension.compile(input).unwrap().success); -} - -#[test] -fn imports_sharing_a_package_prefix_and_parent_relative_imports_roundtrip() { - roundtrip(request(&[ - ("domain/models.py", TYPES), - ("domain/other.py", TYPES), - ( - "domain/rules/select.py", - "from ..models import Address\nfrom .. import other\nimport domain.models\nimport domain.other\nfrom dataclasses import dataclass\n@dataclass(frozen=True)\nclass Choice:\n first: domain.models.Address\n second: domain.other.Address\n third: other.Address\n fourth: Address\n", - ), - ])); -} - -#[test] -fn generated_module_imports_avoid_declarations_and_constructor_names() { - let (_, artifacts) = roundtrip(request(&[ - ("models.py", TYPES), - ( - "rules.py", - "from models import Address\nfrom dataclasses import dataclass\n@dataclass(frozen=True)\nclass MorphirModule1:\n home: Address\n@dataclass(frozen=True)\nclass MorphirModule2:\n pass\ntype Choice = MorphirModule2\ndef morphir_module_1(value: Address) -> Address:\n return value\ndef morphir_module_2(value: Address) -> Address:\n return value\n", - ), - ])); - assert!( - artifacts[1] - .content - .contains("import models as morphir_module_3") - ); -} - -#[test] -fn reports_every_module_and_locates_bad_imports() { - let result = PythonExtension - .compile(request(&[("b.py", ""), ("a.py", "")])) - .unwrap(); - assert_eq!(result.modules, ["a", "b"]); - let result = PythonExtension - .compile(request(&[ - ("models.py", TYPES), - ("bad.py", "from models import Missing\n"), - ])) - .unwrap(); - assert!(!result.success); - assert_eq!( - result.diagnostics[0].location.as_ref().unwrap().uri, - "bad.py" - ); -} - -#[test] -fn backend_rejects_unresolved_cross_module_references_atomically() { - let mut ir = compile(request(&[ - ("models.py", TYPES), - ( - "rules.py", - "from models import Address\nfrom dataclasses import dataclass\n@dataclass(frozen=True)\nclass Holder:\n home: Address\n", - ), - ])); - ir["distribution"]["Library"]["def"]["modules"]["rules"]["Public"]["types"]["holder"]["Public"] - ["TypeAliasDefinition"]["typeExp"]["Record"]["fields"]["home"] = - json!("acme/example:models#missing"); - let result = PythonExtension - .generate(GenerateRequest { - ir, - target: "python".into(), - options: Default::default(), - }) - .unwrap(); - assert!(!result.success); - assert!(result.artifacts.is_empty()); -} - -#[test] -fn types_only_applies_to_every_module() { - let mut input = request(&[ - ("models.py", TYPES), - ("rules.py", "def count() -> int:\n return 1\n"), - ]); - input.options.types_only = true; - assert!(!PythonExtension.compile(input).unwrap().success); -} - -#[test] -fn rejects_module_paths_that_collide_on_case_insensitive_filesystems() { - for sources in [ - vec![("API.py", ""), ("api.py", "")], - vec![("API.py", ""), ("api/models.py", "")], - vec![("domain/API.py", ""), ("domain/api.py", "")], - ] { - assert!( - !PythonExtension.compile(request(&sources)).unwrap().success, - "accepted {sources:?}" - ); - } - let ir = json!({"formatVersion": 4, "distribution": {"Library": { - "packageName": "acme/example", "dependencies": {}, "def": {"modules": { - "API": {"Public": {"types": {}, "values": {}}}, - "api": {"Public": {"types": {}, "values": {}}} - }} - }}}); - let result = PythonExtension - .generate(GenerateRequest { - ir, - target: "python".into(), - options: Default::default(), - }) - .unwrap(); - assert!(!result.success); - assert!(result.artifacts.is_empty()); -} - -#[test] -fn document_uri_metadata_and_percent_encoding_do_not_change_module_identity() { - let mut input = request(&[ - ( - "file:///my%20project/src/domain/%6dodels.py?rev=1#selection", - TYPES, - ), - ( - "file:///my%20project/src/domain/rules.py?rev=2", - "from .models import Pair\ndef origin() -> Pair:\n return (0, '')\n", - ), - ]); - input.sources.root = Some("file:///my%20project/src/".into()); - roundtrip(input); - for uri in [ - "file://other/project/src/models.py", - "file:///project/src/../models.py", - "file:///project/src/%2e%2e/models.py", - "file:///project/src/domain%2Fmodels.py", - "file:///project/src/domain%5Cmodels.py", - ] { - let mut input = request(&[(uri, TYPES)]); - input.sources.root = Some("file:///project/src".into()); - assert!( - !PythonExtension.compile(input).unwrap().success, - "accepted {uri}" - ); - } -} diff --git a/crates/morphir-python-binding/tests/pipeline.rs b/crates/morphir-python-binding/tests/pipeline.rs deleted file mode 100644 index c464cb1db..000000000 --- a/crates/morphir-python-binding/tests/pipeline.rs +++ /dev/null @@ -1,341 +0,0 @@ -#[path = "support/native.rs"] -mod native; - -use morphir_extension_sdk::{ - prelude::*, - protocol::{ExtensionRequest, methods}, -}; -use morphir_python_binding::PythonExtension; -use serde_json::json; - -fn a_request(source: &str) -> CompileRequest { - CompileRequest { - language_id: "python".into(), - sources: SourceSet { - root: None, - documents: vec![SourceDocument { - uri: "models.py".into(), - language_id: "python".into(), - version: 1, - text: source.into(), - }], - }, - package: CompilePackage { - name: "acme/example".into(), - exposed_modules: Some(vec!["Models".into()]), - }, - dependencies: vec![], - options: CompileOptions { - ir_version: "4".into(), - types_only: true, - ..Default::default() - }, - baseline: None, - } -} - -fn compile(source: &str) -> CompileResult { - PythonExtension.compile(a_request(source)).unwrap() -} - -#[test] -fn product_and_sum_have_the_expected_ir_and_roundtrip_through_mep() { - let extension = native::an_initialized_extension(); - let response = extension.protocol().handle( - ExtensionRequest::new( - methods::COMPILE, - a_request(include_str!("fixtures/models.py")), - 1, - ) - .unwrap(), - ); - let compiled: CompileResult = serde_json::from_value(response.result.unwrap()).unwrap(); - assert!(compiled.success, "{:?}", compiled.diagnostics); - let ir = compiled.ir.unwrap(); - let types = &ir["distribution"]["Library"]["def"]["modules"]["models"]["Public"]["types"]; - assert_eq!( - types["address"]["Public"]["TypeAliasDefinition"]["typeExp"]["Record"]["fields"]["zip-code"], - json!("morphir/SDK:basics#int") - ); - assert_eq!( - types["decision"]["Public"]["CustomTypeDefinition"]["constructors"]["pending"], - json!([]) - ); - assert!( - types.get("pending").is_none(), - "variants are constructors, not record aliases" - ); - let response = extension.protocol().handle( - ExtensionRequest::new( - methods::GENERATE, - GenerateRequest { - ir: ir.clone(), - target: "python".into(), - options: Default::default(), - }, - 2, - ) - .unwrap(), - ); - let generated: GenerateResult = serde_json::from_value(response.result.unwrap()).unwrap(); - assert!(generated.success, "{:?}", generated.diagnostics); - assert_eq!(generated.artifacts[0].path, "models.py"); - let recompiled = compile(&generated.artifacts[0].content); - assert!(recompiled.success, "{:?}", recompiled.diagnostics); - assert_eq!(recompiled.ir.unwrap(), ir); -} - -#[test] -fn rejects_unsupported_or_ambiguous_source_without_partial_ir() { - for source in [ - "def calculate():\n return 1\n", - "from dataclasses import dataclass\n@dataclass\nclass Mutable:\n x: int\n", - "from dataclasses import dataclass\n@dataclass(frozen=True)\nclass Default:\n x: int = 1\n", - "from dataclasses import dataclass\n@dataclass(frozen=True)\nclass Unknown:\n x: Missing\n", - "import os\n", - "type Generic[T] = T\n", - "from dataclasses import dataclass\nfrom __future__ import annotations\n", - "class Broken(:\n", - ] { - let result = compile(source); - assert!(!result.success, "accepted: {source}"); - assert!(result.ir.is_none()); - assert!(!result.diagnostics.is_empty()); - } -} - -#[test] -fn advertises_and_enforces_the_initial_contract() { - let capabilities = PythonExtension::capabilities(); - assert_eq!(capabilities.frontend.unwrap().ir_versions, ["3", "4"]); - assert_eq!(capabilities.backend.unwrap().targets, ["python"]); - let mut request = a_request(""); - request.options.ir_version = "2".into(); - assert!(!PythonExtension.compile(request).unwrap().success); - assert!( - !PythonExtension - .generate(GenerateRequest { - ir: json!({}), - target: "elm".into(), - options: Default::default() - }) - .unwrap() - .success - ); -} - -#[test] -fn accepts_cli_context_but_rejects_required_parse_stage_output() { - let mut request = a_request(include_str!("fixtures/models.py")); - request.options.extra = [ - ("outputDir".into(), json!("ignored/output")), - ("emitParseStage".into(), json!(true)), - ("emitParseStageFatal".into(), json!(false)), - ] - .into_iter() - .collect(); - let result = PythonExtension.compile(request.clone()).unwrap(); - assert!(result.success, "{:?}", result.diagnostics); - assert_eq!(result.diagnostics[0].severity, DiagnosticSeverity::Warning); - request - .options - .extra - .insert("emitParseStageFatal".into(), json!(true)); - assert!(!PythonExtension.compile(request).unwrap().success); -} - -/// A source root belongs in `sources.root`. The old options key is rejected -/// even when a caller constructs the request directly in Rust. -#[test] -fn legacy_source_root_uri_option_is_rejected_not_ignored() { - let mut request = a_request(include_str!("fixtures/models.py")); - request - .options - .extra - .insert("sourceRootUri".into(), json!("file:///project/src")); - let result = PythonExtension.compile(request).unwrap(); - assert!(!result.success, "{:?}", result.diagnostics); -} - -/// The modern CLI request carries the root with its documents and preserves -/// CLI context options through the protocol boundary. -#[test] -fn a_modern_compile_request_compiles_through_the_protocol() { - let extension = native::an_initialized_extension(); - let response = extension.protocol().handle( - ExtensionRequest::new( - methods::COMPILE, - json!({ - "languageId": "python", - "sources": { - "root": "file:///project/src", - "documents": [{ - "uri": "file:///project/src/domain/models.py", - "languageId": "python", - "version": 1, - "text": include_str!("fixtures/models.py"), - }], - }, - "package": {"name": "acme/example", "exposedModules": ["domain.models"]}, - "dependencies": [], - "options": { - "typesOnly": true, - "irVersion": "4", - "outputDir": "compiled", - "emitParseStage": false, - "emitParseStageFatal": false, - }, - }), - 1, - ) - .unwrap(), - ); - - assert!(response.error.is_none(), "{:?}", response.error); - let compiled: CompileResult = serde_json::from_value(response.result.unwrap()).unwrap(); - assert!(compiled.success, "{:?}", compiled.diagnostics); - // The root was applied, not dropped: dropping it would name the module - // after the bare file and the document would resolve to the same name for - // any directory it sat in. - assert_eq!(compiled.modules, ["domain/models"]); -} - -#[test] -fn rejects_name_collisions_shared_variants_and_constructor_references() { - for source in [ - "@dataclass(frozen=True)\nclass Thing:\n zip_code: int\n zipCode: int\n", - "@dataclass(frozen=True)\nclass A:\n pass\ntype Choice = A | A\n", - "@dataclass(frozen=True)\nclass A:\n pass\ntype Choice = A\ntype Other = A\n", - "@dataclass(frozen=True)\nclass A:\n pass\ntype Choice = A\n@dataclass(frozen=True)\nclass B:\n field: A\n", - "@dataclass(frozen=True)\nclass FooBar:\n pass\n@dataclass(frozen=True)\nclass Foo_Bar:\n pass\n", - "@dataclass(frozen=True)\nclass A(Base):\n pass\n", - "@dataclass(frozen=True)\nclass A:\n field: __import__('os').system('echo unsafe')\n", - ] { - let result = compile(&format!("from dataclasses import dataclass\n{source}")); - assert!(!result.success, "accepted: {source}"); - assert!(result.ir.is_none()); - } -} - -#[test] -fn rejects_names_whose_generated_spelling_is_reserved() { - for body in [ - "@dataclass(frozen=True)\nclass Product:\n Class: str\n", - "@dataclass(frozen=True)\nclass Variant:\n Int: int\ntype Choice = Variant\n", - "@dataclass(frozen=True)\nclass none:\n pass\n", - "@dataclass(frozen=True)\nclass Variant:\n pass\ntype true = Variant\n", - ] { - let result = compile(&format!("from dataclasses import dataclass\n{body}")); - assert!(!result.success, "accepted: {body}"); - assert!(result.ir.is_none()); - assert_eq!(result.diagnostics[0].code.as_deref(), Some("PY003")); - } -} - -fn a_library(definition: serde_json::Value) -> serde_json::Value { - json!({"formatVersion": 4, "distribution": {"Library": { - "packageName": "acme/example", "dependencies": {}, "def": {"modules": { - "models": {"Public": {"types": {"choice": {"Public": definition}}, "values": {}}} - }} - }}}) -} - -#[test] -fn backend_accepts_an_independently_authored_sum() { - let ir = a_library( - json!({"CustomTypeDefinition": {"typeParams": [], "access": "Public", "constructors": { - "absent": [], "present": [["value", "morphir/SDK:string#string"]] - }}}), - ); - let result = PythonExtension - .generate(GenerateRequest { - ir: ir.clone(), - target: "python".into(), - options: Default::default(), - }) - .unwrap(); - assert!(result.success, "{:?}", result.diagnostics); - assert_eq!(compile(&result.artifacts[0].content).ir.unwrap(), ir); -} - -#[test] -fn backend_refuses_unsupported_ir_without_partial_artifacts() { - for definition in [ - json!({"TypeAliasDefinition": {"typeParams": ["a"], "typeExp": "a"}}), - json!({"TypeAliasDefinition": {"typeParams": [], "typeExp": {"Function": {"parameterType": "morphir/SDK:basics#int", "returnType": "morphir/SDK:basics#int"}}}}), - json!({"TypeAliasDefinition": {"typeParams": [], "typeExp": {"Record": {"fields": {"value": "elsewhere/pkg:other#type"}}}}}), - json!({"CustomTypeDefinition": {"typeParams": [], "access": "Private", "constructors": {"hidden": []}}}), - json!({"CustomTypeDefinition": {"typeParams": [], "access": "Public", "constructors": {"choice": []}}}), - json!({"CustomTypeDefinition": {"typeParams": [], "access": "Public", "constructors": {}}}), - ] { - let result = PythonExtension - .generate(GenerateRequest { - ir: a_library(definition), - target: "python".into(), - options: Default::default(), - }) - .unwrap(); - assert!(!result.success); - assert!(result.artifacts.is_empty()); - assert!(!result.diagnostics.is_empty()); - } -} - -#[test] -fn recursive_types_and_initialisms_roundtrip() { - let result = compile( - "from __future__ import annotations\nfrom dataclasses import dataclass\n@dataclass(frozen=True)\nclass Empty:\n pass\n@dataclass(frozen=True)\nclass Link:\n value_in_USD: float\n rest: Chain\ntype Chain = Empty | Link\n", - ); - assert!(result.success, "{:?}", result.diagnostics); - let ir = result.ir.unwrap(); - let generated = PythonExtension - .generate(GenerateRequest { - ir: ir.clone(), - target: "python".into(), - options: Default::default(), - }) - .unwrap(); - assert!(generated.success, "{:?}", generated.diagnostics); - assert_eq!(compile(&generated.artifacts[0].content).ir.unwrap(), ir); -} - -#[test] -fn syntax_errors_identify_the_source_document() { - let result = compile("class Broken(:\n"); - assert_eq!(result.diagnostics[0].code.as_deref(), Some("PY002")); - assert_eq!( - result.diagnostics[0].location.as_ref().unwrap().uri, - "models.py" - ); -} - -#[test] -fn refuses_nonportable_output_module_names() { - for module in [ - "../models", - "nested/con", - "dataclasses", - "con", - "aux", - "COM1", - "lpt9", - ] { - let mut ir = a_library( - json!({"TypeAliasDefinition": {"typeParams": [], "typeExp": {"Record": {"fields": {}}}}}), - ); - let modules = ir["distribution"]["Library"]["def"]["modules"] - .as_object_mut() - .unwrap(); - let definition = modules.remove("models").unwrap(); - modules.insert(module.into(), definition); - let result = PythonExtension - .generate(GenerateRequest { - ir, - target: "python".into(), - options: Default::default(), - }) - .unwrap(); - assert!(!result.success, "accepted module {module}"); - assert!(result.artifacts.is_empty()); - } -} diff --git a/crates/morphir-python-binding/tests/support/native.rs b/crates/morphir-python-binding/tests/support/native.rs deleted file mode 100644 index 762d5a0cd..000000000 --- a/crates/morphir-python-binding/tests/support/native.rs +++ /dev/null @@ -1,29 +0,0 @@ -//! Native protocol fixture with a completed initialization handshake. - -use morphir_extension_sdk::{ - NativeExtension, - protocol::{ExtensionRequest, InitializeParams, MEP_VERSION, PeerInfo, methods}, -}; -use morphir_python_binding::PythonExtension; - -pub fn an_initialized_extension() -> NativeExtension { - let extension = NativeExtension::frontend_backend(PythonExtension).unwrap(); - let response = extension.protocol().handle( - ExtensionRequest::new( - methods::INITIALIZE, - InitializeParams { - model_transport: None, - protocol_versions: vec![MEP_VERSION.into()], - host: PeerInfo { - kind: Default::default(), - name: "binding-test".into(), - version: "1.0.0".into(), - }, - }, - 0, - ) - .unwrap(), - ); - assert!(response.error.is_none(), "{:?}", response.error); - extension -} diff --git a/tests/ci/ci_impact_test_support.py b/tests/ci/ci_impact_test_support.py index f2025ef41..11933ba3d 100644 --- a/tests/ci/ci_impact_test_support.py +++ b/tests/ci/ci_impact_test_support.py @@ -17,7 +17,7 @@ def fake_metadata() -> dict: """A small workspace shaped like morphir-rust for graph tests. - core <- sdk <- {daemon, gleam, python} + core <- sdk <- {daemon, gleam, rust} core <- projection <- avro daemon dev-depends on avro (installed wasm tests) ext-example is a member but not a default member @@ -40,7 +40,7 @@ def package(name: str, deps: list[tuple[str, str]]) -> dict: package("morphir-projection", [("morphir-core", "normal")]), package("morphir-daemon", [("morphir-extension-sdk", "normal"), ("morphir-avro-extension", "dev")]), package("morphir-gleam-binding", [("morphir-extension-sdk", "normal")]), - package("morphir-python-binding", [("morphir-extension-sdk", "normal")]), + package("morphir-rust-binding", [("morphir-extension-sdk", "normal")]), package("morphir-avro-extension", [("morphir-projection", "normal"), ("morphir-extension-sdk", "build")]), package("morphir-ext-example", [("morphir-extension-sdk", "normal")]), ] @@ -73,5 +73,5 @@ def fake_config_data() -> dict: def fake_extensions() -> tuple[classify.Extension, ...]: return ( classify.Extension(short_id="avro", package="morphir-avro-extension"), - classify.Extension(short_id="python", package="morphir-python-binding"), + classify.Extension(short_id="rust", package="morphir-rust-binding"), ) diff --git a/tests/ci/test_ci_change_detection.py b/tests/ci/test_ci_change_detection.py index 07286d582..057bf1e6a 100644 --- a/tests/ci/test_ci_change_detection.py +++ b/tests/ci/test_ci_change_detection.py @@ -52,10 +52,10 @@ def _seed_repository(repository: Path) -> None: encoding="utf-8", ) (repository / ".github" / "extensions.toml").write_text( - "[extensions.python]\npackage = 'morphir-python-binding'\n", encoding="utf-8" + "[extensions.rust]\npackage = 'morphir-rust-binding'\n", encoding="utf-8" ) - (repository / "crates" / "morphir-python-binding" / "src").mkdir(parents=True) - (repository / "crates" / "morphir-python-binding" / "src" / "lib.rs").write_text("", encoding="utf-8") + (repository / "crates" / "morphir-rust-binding" / "src").mkdir(parents=True) + (repository / "crates" / "morphir-rust-binding" / "src" / "lib.rs").write_text("", encoding="utf-8") (repository / "README.md").write_text("base\n", encoding="utf-8") (repository / "Cargo.lock").write_text("base\n", encoding="utf-8") @@ -124,7 +124,7 @@ def test_cli_writes_scoped_outputs_for_a_leaf_crate_change(self) -> None: _init_repository(repository) _seed_repository(repository) base = _commit(repository, "base") - (repository / "crates" / "morphir-python-binding" / "src" / "lib.rs").write_text("// change\n", encoding="utf-8") + (repository / "crates" / "morphir-rust-binding" / "src" / "lib.rs").write_text("// change\n", encoding="utf-8") head = _commit(repository, "head") output = Path(temporary) / "github-output" fake_cargo = _fake_cargo(Path(temporary) / "bin") @@ -134,9 +134,9 @@ def test_cli_writes_scoped_outputs_for_a_leaf_crate_change(self) -> None: self.assertEqual(0, result.returncode, result.stderr) lines = dict(line.split("=", 1) for line in output.read_text(encoding="utf-8").splitlines()) self.assertEqual("false", lines["all"]) - self.assertEqual("-p morphir-python-binding", lines["cargo_packages"]) + self.assertEqual("-p morphir-rust-binding", lines["cargo_packages"]) self.assertEqual("false", lines["job_kit_conformance"]) - self.assertEqual([{"id": "python", "package": "morphir-python-binding"}], json.loads(lines["extensions"])) + self.assertEqual([{"id": "rust", "package": "morphir-rust-binding"}], json.loads(lines["extensions"])) def test_cli_writes_all_true_for_readme_plus_lockfile(self) -> None: with tempfile.TemporaryDirectory() as temporary: diff --git a/tests/ci/test_ci_impact_classify.py b/tests/ci/test_ci_impact_classify.py index 18a021971..9e40ec7fe 100644 --- a/tests/ci/test_ci_impact_classify.py +++ b/tests/ci/test_ci_impact_classify.py @@ -24,14 +24,14 @@ def test_crate_for_path(self) -> None: self.assertIsNone(classify.crate_for_path("docs/crates/morphir-core/x.md")) def test_leaf_binding_change_affects_only_itself(self) -> None: - plan = self.plan("crates/morphir-python-binding/src/lib.rs") + plan = self.plan("crates/morphir-rust-binding/src/lib.rs") self.assertFalse(plan.all) - self.assertEqual(("morphir-python-binding",), plan.crates) - self.assertEqual("-p morphir-python-binding", plan.cargo_packages) + self.assertEqual(("morphir-rust-binding",), plan.crates) + self.assertEqual("-p morphir-rust-binding", plan.cargo_packages) self.assertTrue(plan.rust) self.assertFalse(plan.jobs["kit-conformance"]) self.assertFalse(plan.jobs["test-extism"]) - self.assertEqual(("python",), tuple(item.short_id for item in plan.extensions)) + self.assertEqual(("rust",), tuple(item.short_id for item in plan.extensions)) def test_core_change_affects_dependents_and_all_extensions(self) -> None: plan = self.plan("crates/morphir-core/src/lib.rs") @@ -41,11 +41,11 @@ def test_core_change_affects_dependents_and_all_extensions(self) -> None: self.assertIn("-p morphir-daemon", plan.cargo_packages) self.assertTrue(plan.jobs["kit-conformance"]) self.assertTrue(plan.jobs["test-extism"]) - self.assertEqual(("avro", "python"), tuple(item.short_id for item in plan.extensions)) + self.assertEqual(("avro", "rust"), tuple(item.short_id for item in plan.extensions)) def test_daemon_change_rebuilds_every_extension(self) -> None: plan = self.plan("crates/morphir-daemon/src/lib.rs") - self.assertEqual(("avro", "python"), tuple(item.short_id for item in plan.extensions)) + self.assertEqual(("avro", "rust"), tuple(item.short_id for item in plan.extensions)) def test_global_path_affects_everything(self) -> None: plan = self.plan("Cargo.lock") @@ -53,7 +53,7 @@ def test_global_path_affects_everything(self) -> None: self.assertEqual("", plan.cargo_packages) self.assertTrue(all(plan.jobs.values())) self.assertTrue(plan.rust) - self.assertEqual(("avro", "python"), tuple(item.short_id for item in plan.extensions)) + self.assertEqual(("avro", "rust"), tuple(item.short_id for item in plan.extensions)) self.assertTrue(any("Cargo.lock" in reason for reason in plan.reasons)) def test_unclaimed_path_affects_everything(self) -> None: @@ -106,7 +106,7 @@ def test_full_plan_enables_everything(self) -> None: def test_load_extensions_reads_the_repository_registry(self) -> None: loaded = classify.load_extensions(REPOSITORY_ROOT / ".github" / "extensions.toml") - self.assertIn(classify.Extension("python", "morphir-python-binding"), loaded) + self.assertIn(classify.Extension("rust", "morphir-rust-binding"), loaded) if __name__ == "__main__": diff --git a/tests/ci/test_ci_impact_consistency.py b/tests/ci/test_ci_impact_consistency.py index 2c0ab6542..e50947b4c 100644 --- a/tests/ci/test_ci_impact_consistency.py +++ b/tests/ci/test_ci_impact_consistency.py @@ -74,11 +74,11 @@ def test_rust_bundle_uses_selective_ci_routing(self) -> None: self.assertIn("rust", all_extensions) cases = [ ("crates/morphir-rust-binding/src/lib.rs", ("rust",)), - ("crates/morphir-python-binding/src/lib.rs", ("python",)), + ("crates/morphir-openapi-extension/src/lib.rs", ("openapi",)), ("crates/morphir-gleam-binding/src/lib.rs", ("gleam",)), ("README.md", ()), (".mise/tasks/extension/artifact/rust", ("rust",)), - (".mise/tasks/extension/artifact/python", ("python",)), + (".mise/tasks/extension/artifact/openapi", ("openapi",)), (".mise/tasks/extension/artifact/gleam", ("gleam",)), (".github/extensions.toml", all_extensions), ("scripts/package_extension.py", all_extensions), diff --git a/tests/ci/test_ci_impact_graph.py b/tests/ci/test_ci_impact_graph.py index 18d0e3a5b..9cccb0d2d 100644 --- a/tests/ci/test_ci_impact_graph.py +++ b/tests/ci/test_ci_impact_graph.py @@ -31,8 +31,8 @@ def test_external_dependencies_are_ignored(self) -> None: def test_leaf_change_affects_only_itself(self) -> None: self.assertEqual( - frozenset({"morphir-python-binding"}), - graph.affected_closure(self.workspace, ["morphir-python-binding"]), + frozenset({"morphir-rust-binding"}), + graph.affected_closure(self.workspace, ["morphir-rust-binding"]), ) def test_core_change_affects_every_dependent(self) -> None: diff --git a/tests/ci/test_ci_impact_outputs.py b/tests/ci/test_ci_impact_outputs.py index 237bd3141..cccf636e4 100644 --- a/tests/ci/test_ci_impact_outputs.py +++ b/tests/ci/test_ci_impact_outputs.py @@ -20,21 +20,21 @@ def test_output_key_uses_underscores(self) -> None: def test_render_github_for_a_leaf_change(self) -> None: plan = classify.plan_changes( - ["crates/morphir-python-binding/src/lib.rs"], self.config, self.workspace, self.extensions + ["crates/morphir-rust-binding/src/lib.rs"], self.config, self.workspace, self.extensions ) rendered = outputs.render_github(plan) lines = dict(line.split("=", 1) for line in rendered.splitlines()) self.assertEqual("false", lines["all"]) - self.assertEqual(["morphir-python-binding"], json.loads(lines["crates"])) - self.assertEqual("-p morphir-python-binding", lines["cargo_packages"]) + self.assertEqual(["morphir-rust-binding"], json.loads(lines["crates"])) + self.assertEqual("-p morphir-rust-binding", lines["cargo_packages"]) self.assertEqual("true", lines["job_rust"]) self.assertEqual("false", lines["job_kit_conformance"]) self.assertEqual("false", lines["job_test_extism"]) self.assertEqual("false", lines["job_lint_shell"]) self.assertEqual("false", lines["job_docs_generated"]) self.assertEqual( - [{"id": "python", "package": "morphir-python-binding"}], + [{"id": "rust", "package": "morphir-rust-binding"}], json.loads(lines["extensions"]), ) self.assertTrue(rendered.endswith("\n")) diff --git a/tests/ci/test_extension_release_routing.py b/tests/ci/test_extension_release_routing.py index e681a16f3..755c22bbb 100644 --- a/tests/ci/test_extension_release_routing.py +++ b/tests/ci/test_extension_release_routing.py @@ -121,7 +121,6 @@ def test_workspace_tag_selects_every_registered_extension(self) -> None: "morphir-elm-binding": "0.1.0", "morphir-gleam-binding": "0.2.0", "morphir-openapi-extension": "0.1.0", - "morphir-python-binding": "0.1.0", "morphir-rust-binding": "0.1.0", } @@ -130,7 +129,7 @@ def test_workspace_tag_selects_every_registered_extension(self) -> None: ) self.assertEqual( - ["avro", "elm-native", "gleam", "openapi", "python", "rust"], release.short_ids + ["avro", "elm-native", "gleam", "openapi", "rust"], release.short_ids ) def test_workspace_tag_selects_opted_in_extensions_in_sorted_order(self) -> None: diff --git a/tests/ci/test_frontend_extension_release.py b/tests/ci/test_frontend_extension_release.py new file mode 100644 index 000000000..214c39263 --- /dev/null +++ b/tests/ci/test_frontend_extension_release.py @@ -0,0 +1,50 @@ +"""Frontend release metadata keeps its languages through packaging and selection.""" + +from package_extension_test_support import * +from extension_release_test_support import AssetFixture, select_extension_assets + + +class FrontendExtensionReleaseTests(unittest.TestCase): + def test_frontend_descriptor_survives_selection_and_cannot_be_dropped(self) -> None: + with tempfile.TemporaryDirectory() as temporary: + fixture = AssetFixture(Path(temporary).resolve()) + _, checksum, descriptor_path = fixture.add_extension( + short_id="rust", package="morphir-rust-binding", + artifact_base="morphir-rust-binding", extension_id="morphir-rust", + version="0.1.0", name="Morphir Rust", + ) + # Checksums are an exact LF-delimited wire format on every platform. + checksum.write_bytes(checksum.read_bytes().replace(b"\r\n", b"\n")) + registry = fixture.root / ".github/extensions.toml" + with registry.open("a", encoding="utf-8", newline="\n") as stream: + stream.write('languages = [{ id = "rust", file_extensions = [".rs"] }]\n') + descriptor = json.loads(descriptor_path.read_text(encoding="utf-8")) + descriptor["artifacts"][0]["claims"] = claims_for_extension( + tomllib.loads(registry.read_text())["extensions"]["rust"], "0.1.0" + ) + before = (json.dumps(descriptor, indent=2) + "\n").encode("utf-8") + descriptor_path.write_bytes(before) + fixture.bundles = descriptor_path.parent + selection = fixture.select("extension/rust/v0.1.0") + uploaded = next(asset for asset in selection.uploads if asset.name.endswith(".release.json")) + self.assertEqual(before, uploaded.source.read_bytes()) + del descriptor["artifacts"][0]["claims"]["capabilities"]["frontend"]["languages"] + descriptor_path.write_bytes(json.dumps(descriptor).encode("utf-8")) + with self.assertRaisesRegex(select_extension_assets.AssetError, "languages"): + fixture.select("extension/rust/v0.1.0") + + def test_packaging_accepts_frontend_only_and_rejects_invalid_languages(self) -> None: + registry = tomllib.loads(AVRO_REGISTRY)["extensions"]["avro"] + registry["targets"] = [] + registry["languages"] = [{"id": "rust", "file_extensions": [".rs"]}] + descriptor = json.loads(descriptor_bytes("rust", registry, "0.1.0", "guest.wasm", "0" * 64, None)) + self.assertNotIn("backend", descriptor["artifacts"][0]["claims"]["capabilities"]) + for languages in [[], [{"id": "rust", "file_extensions": ["rs"]}], + [{"id": "rust", "file_extensions": [".rs", ".rs"]}], + [{"id": "rust", "file_extensions": [".rs"]}] * 2]: + with self.subTest(languages=languages), self.assertRaises(PackageError): + descriptor_bytes("rust", {**registry, "languages": languages}, "0.1.0", "guest.wasm", "0" * 64, None) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/ci/test_package_extension_packaging.py b/tests/ci/test_package_extension_packaging.py index 5f13d8438..1d74e1d91 100644 --- a/tests/ci/test_package_extension_packaging.py +++ b/tests/ci/test_package_extension_packaging.py @@ -73,10 +73,11 @@ def test_an_incremental_frontend_entry_reaches_the_release_descriptor(self) -> N def test_a_frontend_without_the_registry_flag_claims_no_incremental_support(self) -> None: """An absent registry flag means the guest must not claim this capability.""" registry = tomllib.loads(EXTENSIONS_TOML.read_text(encoding="utf-8")) - python = registry["extensions"]["python"] - self.assertNotIn("incremental", python) + rust = registry["extensions"]["rust"] + self.assertIn("languages", rust) + self.assertNotIn("incremental", rust) - descriptor = self._descriptor("python", python) + descriptor = self._descriptor("rust", rust) self.assertFalse(descriptor["artifacts"][0]["claims"]["capabilities"]["frontend"]["incremental"]) @@ -114,10 +115,11 @@ def test_the_discovery_providers_reach_the_release_descriptor(self) -> None: def test_an_extension_without_the_flag_writes_no_workspace_key(self) -> None: """An absent registry flag means the guest must not claim this capability.""" registry = tomllib.loads(EXTENSIONS_TOML.read_text(encoding="utf-8")) - python = registry["extensions"]["python"] - self.assertNotIn("workspace_discovery", python) + rust = registry["extensions"]["rust"] + self.assertIn("languages", rust) + self.assertNotIn("workspace_discovery", rust) - descriptor = self._descriptor("python", python) + descriptor = self._descriptor("rust", rust) self.assertNotIn("workspace", descriptor["artifacts"][0]["claims"]["capabilities"]) diff --git a/tests/ci/test_python_extension_release.py b/tests/ci/test_python_extension_release.py deleted file mode 100644 index 2da20e66a..000000000 --- a/tests/ci/test_python_extension_release.py +++ /dev/null @@ -1,72 +0,0 @@ -"""Python release metadata keeps both frontend and backend discovery.""" - -from package_extension_test_support import * -from extension_release_test_support import AssetFixture, extension_release, select_extension_assets -from ci_impact_test_support import classify - - -class PythonExtensionReleaseTests(unittest.TestCase): - def test_frontend_descriptor_survives_selection_and_cannot_be_dropped(self) -> None: - with tempfile.TemporaryDirectory() as temporary: - fixture = AssetFixture(Path(temporary).resolve()) - _, checksum, descriptor_path = fixture.add_extension( - short_id="python", package="morphir-python-binding", - artifact_base="morphir-python-binding", extension_id="morphir-python", - version="0.1.0", name="Morphir Python", - ) - # Checksums are an exact LF-delimited wire format on every platform. - checksum.write_bytes(checksum.read_bytes().replace(b"\r\n", b"\n")) - registry = fixture.root / ".github/extensions.toml" - with registry.open("a", encoding="utf-8", newline="\n") as stream: - stream.write('languages = [{ id = "python", file_extensions = [".py"] }]\n') - descriptor = json.loads(descriptor_path.read_text(encoding="utf-8")) - descriptor["artifacts"][0]["claims"] = claims_for_extension( - tomllib.loads(registry.read_text())["extensions"]["python"], "0.1.0" - ) - before = (json.dumps(descriptor, indent=2) + "\n").encode("utf-8") - descriptor_path.write_bytes(before) - fixture.bundles = descriptor_path.parent - selection = fixture.select("extension/python/v0.1.0") - uploaded = next(asset for asset in selection.uploads if asset.name.endswith(".release.json")) - self.assertEqual(before, uploaded.source.read_bytes()) - del descriptor["artifacts"][0]["claims"]["capabilities"]["frontend"]["languages"] - descriptor_path.write_bytes(json.dumps(descriptor).encode("utf-8")) - with self.assertRaisesRegex(select_extension_assets.AssetError, "languages"): - fixture.select("extension/python/v0.1.0") - - def test_python_tag_and_descriptor_preserve_both_capabilities(self) -> None: - registry = tomllib.loads(EXTENSIONS_TOML.read_text(encoding="utf-8")) - extension = registry["extensions"]["python"] - release = extension_release.resolve_release( - "extension/python/v0.1.0", registry, "0.2.0", - {"morphir-python-binding": "0.1.0"}, - ) - self.assertEqual(["python"], release.short_ids) - descriptor = json.loads(descriptor_bytes( - "python", extension, "0.1.0", "morphir-python-binding-0.1.0.wasm", "0" * 64, "a" * 40, - )) - self.assertEqual("Morphir Python", descriptor["artifacts"][0]["claims"]["extension"]["name"]) - self.assertEqual(["python"], descriptor["artifacts"][0]["claims"]["capabilities"]["backend"]["targets"]) - self.assertEqual([{"id": "python", "fileExtensions": [".py"]}], descriptor["artifacts"][0]["claims"]["capabilities"]["frontend"]["languages"]) - self.assertEqual(["3", "4"], descriptor["artifacts"][0]["claims"]["capabilities"]["backend"]["irVersions"]) - - def test_packaging_accepts_frontend_only_and_rejects_invalid_languages(self) -> None: - registry = tomllib.loads(AVRO_REGISTRY)["extensions"]["avro"] - registry["targets"] = [] - registry["languages"] = [{"id": "python", "file_extensions": [".py"]}] - descriptor = json.loads(descriptor_bytes("python", registry, "0.1.0", "guest.wasm", "0" * 64, None)) - self.assertNotIn("backend", descriptor["artifacts"][0]["claims"]["capabilities"]) - for languages in [[], [{"id": "python", "file_extensions": ["py"]}], - [{"id": "python", "file_extensions": [".py", ".py"]}], - [{"id": "python", "file_extensions": [".py"]}] * 2]: - with self.subTest(languages=languages), self.assertRaises(PackageError): - descriptor_bytes("python", {**registry, "languages": languages}, "0.1.0", "guest.wasm", "0" * 64, None) - - def test_python_ci_publishes_a_downloadable_bundle(self) -> None: - workflow = (REPOSITORY_ROOT / ".github/workflows/ci.yml").read_text(encoding="utf-8") - self.assertIn('mise run "extension:artifact:${{ matrix.id }}"', workflow) - self.assertIn("name: morphir-${{ matrix.id }}-extension-bundle", workflow) - self.assertIn("path: .morphir/build/extensions/${{ matrix.id }}/*", workflow) - self.assertIn("include: ${{ fromJSON(needs.changes.outputs.extensions) }}", workflow) - extensions = classify.load_extensions(REPOSITORY_ROOT / ".github/extensions.toml") - self.assertIn(classify.Extension("python", "morphir-python-binding"), extensions) diff --git a/tests/ci/test_windows_source_identity.py b/tests/ci/test_windows_source_identity.py index 06df1fbb4..79bf84513 100644 --- a/tests/ci/test_windows_source_identity.py +++ b/tests/ci/test_windows_source_identity.py @@ -89,7 +89,7 @@ def test_actual_source_tests_fixtures_and_dependencies_select_existing_windows_j self.assertFalse(plan.all) self.assertIn("morphir-common", plan.crates) self.assertTrue(plan.jobs["kit-conformance"]) - for path in ("README.md", "docs/example.md", "crates/morphir-python-binding/src/lib.rs", "crates/morphir-devkit/src/lib.rs"): + for path in ("README.md", "docs/example.md", "crates/morphir-rust-binding/src/lib.rs", "crates/morphir-devkit/src/lib.rs"): with self.subTest(path=path): plan = classify.plan_changes([path], self.impact, self.workspace, ()) self.assertFalse(plan.all) From a731a492cf096c4ea520dad7529f8f2898e46da5 Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 14:12:27 -0500 Subject: [PATCH 2/2] docs: point the Python extension tutorial to finos/morphir-python The tutorial page stays at its URL as a short pointer: the extension is now maintained and released from finos/morphir-python, and the historical extension/python/v0.1.0 to v0.4.0 bundles remain in morphir-rust releases. The extension tutorial now names the Rust binding as its frontend and backend example. docs/llms.txt and docs/llms-full.txt are regenerated with the docs:llms-txt generator. Refs: morphir-ukvn.1.6.7 --- docs/llms-full.txt | 211 ++------------------------- docs/llms.txt | 2 +- docs/tutorials/extension-tutorial.md | 5 +- docs/tutorials/python-extension.md | 208 ++------------------------ 4 files changed, 24 insertions(+), 402 deletions(-) diff --git a/docs/llms-full.txt b/docs/llms-full.txt index 36bcf26bd..356920423 100644 --- a/docs/llms-full.txt +++ b/docs/llms-full.txt @@ -810,8 +810,9 @@ feature is implemented for contributor testing but has not produced a public release. For an extension implementing both compilation and generation, see the -[Python ADT extension](python-extension.md). It uses the same SDK and MEP -transport for both capabilities. +[Rust binding](https://github.com/finos/morphir-rust/tree/main/crates/morphir-rust-binding). +It uses the same SDK and MEP transport for both capabilities. The Python +extension now lives in [finos/morphir-python](https://github.com/finos/morphir-python). ## 1. Start from the reference backend @@ -909,205 +910,15 @@ Source: https://finos.github.io/morphir-rust/tutorials/python-extension # Python ADT extension -The `morphir-python-binding` crate provides the `morphir-python` frontend and -backend. Start with its [modeling example and supported subset](../../crates/morphir-python-binding/README.md). -The extension compiles frozen dataclasses, named unions, fixed tuples and annotated conditional -functions, typed calls and unary lambdas to Morphir IR v3 or v4, and generates -Python 3.12 or later code from those definitions. +The Python extension (crate `morphir-python-binding`, bundle `morphir-python`) is +now maintained and released from +[finos/morphir-python](https://github.com/finos/morphir-python). Its source, +tests, examples and release bundles live in that repository. This repository no +longer builds or releases it. -The published Python `0.2.0` bundle also supports IR v3, private modules, multiple source modules -with imports, and higher-order functions using unary `Callable` annotations. The earlier `0.1.0` -bundle supports one public module per request and IR v4 only. - -Functions support returning `if`/`elif`/`else` branches and Python conditional -expressions. For example: - -```python -def choose(flag: bool, first: int, second: int) -> int: - if flag: - return first - else: - return second -``` - -This body maps to Morphir's `IfThenElse` value. Each path must return the declared -type, and the condition must be boolean. Compile functions with `typesOnly=false` -through MEP. Project compilation through the CLI includes function bodies. The crate README lists the -supported expressions and remaining limits. - -## Build and test - -Run these commands from the `morphir-rust` repository: - -```console -rustup target add wasm32-unknown-unknown -cargo test --locked -p morphir-python-binding -cargo run --locked -p morphir-python-binding --example python_adt -cargo run --locked -p morphir-python-binding --example python_modules -cargo build --locked --release -p morphir-python-binding --target wasm32-unknown-unknown -cargo test --locked -p morphir-host-native --test python_extension -- --ignored --exact python_adt_and_conditional_roundtrip_through_the_real_wasm_extension -``` - -The example returns JSON containing IR and generated Python. The WASM test -performs capability negotiation, compilation, generation and recompilation -through the Extism container in `morphir_host_native::extism`. CI runs it -with a release guest. - -## Install a release bundle - -The Python extension is released independently as `extension/python/v0.4.0` in -[finos/morphir-rust releases](https://github.com/finos/morphir-rust/releases). -It contains a WASM guest, SHA-256 checksum and release descriptor. It is a Morphir -extension, not a PyPI package or a CPython import module. - -Use Rust Morphir CLI `0.4.0-beta.7` or later with this bundle: its version-2 descriptor carries -capability claims, which earlier CLIs cannot read, so they cannot publish it. The npm `morphir-elm` -executable does not provide these installation commands. - -Download and install with PowerShell: - -```powershell -gh release download extension/python/v0.4.0 --repo finos/morphir-rust --dir python-bundle -Rename-Item -LiteralPath python-bundle/morphir-python-binding-0.4.0.release.json -NewName release.json -$pythonIndex = Join-Path (Get-Location) '.morphir/python-index' -$env:MORPHIR_HOME = Join-Path (Get-Location) '.morphir/python-home' -morphir extension repository init $pythonIndex -morphir extension repository add python-local --directory $pythonIndex -morphir extension repository publish python-local --bundle python-bundle -morphir extension repository verify python-local -morphir extension install --repository python-local morphir-python -``` - -Newly built bundles use version-2 descriptors containing the WASM guest's own -capability claims. Publishing requires Morphir CLI `0.4.0-beta.7` or later; -older CLIs cannot read them. The release asset filename is unchanged, so the -`Rename-Item` command above also applies to version-2 bundles. WASM installs -retain `claims` with `claimCheck: "unchecked"` and display `Claims: unchecked`. - -Keep the same `MORPHIR_HOME` for compilation and generation. Installation verifies -the artifact digest and records both capabilities. Once installed, compilation -and generation work offline without access to the original repository. - -## Build a development bundle - -From `morphir-rust`, run `mise run extension:artifact:python`. This builds and -validates the guest, packages it, and tests installation and offline roundtripping. -Use `.morphir/build/extensions/python` as the `--bundle` directory in the -publication command above; its descriptor is already named `release.json`. -CI also uploads this directory as the `morphir-python-extension-bundle` artifact. - -For a Windows build using the native toolchain, after the build commands above: - -```powershell -python -B scripts/package_extension.py --short-id python --wasm target/wasm32-unknown-unknown/release/morphir_python_binding.wasm --output .morphir/build/extensions/python -$env:MORPHIR_PYTHON_BUNDLE = (Resolve-Path .morphir/build/extensions/python).Path -cargo test --locked -p morphir-host-native --test python_extension -- --ignored -``` - -The release pipeline builds from a committed snapshot and verifies that the -bundle's `gitCommit` matches the release tag. Development bundles from modified -working trees omit that provenance claim. - -## Compile and generate - -Create a project directory with a `src/models.py` file using the example above, -and this `morphir.toml`: - -```toml -[project] -name = "acme/example" -version = "0.1.0" -source_directory = "src" - -[frontend] -language = "python" -emit_parse_stage = false - -[codegen] -targets = ["python"] -``` - -From that project directory, use the Rust CLI's installed provider path: - -```console -morphir compile --language python --input src/models.py --output compiled -morphir generate --target python --input compiled/morphir-ir.json --output generated -``` - -Compilation installs `morphir-ir.json` under the `compiled` directory. The -backend returns `models.py` as an artifact. The host writes it to the output -location according to the CLI's install/output settings. The extension itself -has no filesystem access. It fails with diagnostics if the request contains -unsupported imports, function behavior, or types outside the -documented subset. The [binding README](../../crates/morphir-python-binding/README.md) -lists the supported IR nodes, tuple forms, and the scope of its conformance evidence. - -With the development bundle installed, place related files under `src/` and -compile the directory to resolve their imports together: - -```console -morphir compile --language python --input src --output compiled -morphir generate --target python --input compiled/morphir-ir.json --output generated -morphir compile --language python --input generated --output recompiled -``` - -For example, `src/domain/models.py` may define `Decision`, and -`src/domain/rules.py` may use `from .models import Decision` in its annotations. -The resulting IR contains `domain/models` and `domain/rules`; generation returns -`domain/models.py` and `domain/rules.py`. Nested directories are namespace -packages, without `__init__.py` files. Keep all referenced modules in the source -set. Same-package function imports and typed calls work too; cross-package -dependencies remain unsupported. - -With a CLI and development bundle containing project-context support, the -configured source directory is sufficient: - -```console -morphir compile --project packages/orders -morphir generate --project packages/orders --target python --output generated -``` - -The project selector accepts a declared workspace-relative path or exact -project name. Compilation uses that member's source directory and package name; -generation reads its recorded compile artifact. An explicit `--input` replaces -the configured input and resolves from the invocation directory. - -Set `project.exposed_modules = ["domain.rules"]` to expose only that module. -`domain.models` remains in the IR as a private module and is still available to -sibling imports. Generation emits both files. Preserve the exposure configuration -when recompiling generated source: Python itself does not enforce Morphir module -privacy. Omitting exposure makes all modules public; `[]` makes all private. - -## Select the IR version - -Use a CLI and development bundle containing IR v3 support. The CLI defaults to v4; -select v3 in the project configuration: - -```toml -[ir] -format_version = 3 -layout = "single-file" -format = "json" -``` - -`format = "yaml"` also works for v3. Document-tree storage requires v4. -The CLI override applies to this invocation, without editing the project: - -```console -morphir compile --ir-version 3 --output compiled-v3 -morphir generate --target python --input compiled-v3/morphir-ir.json --output generated-v3 -morphir compile --ir-version 4 --output compiled-v4 -``` - -The backend detects the version from the input document. Through MEP, set -`options.irVersion` to `3` or `3.0.0`, or to `4` or `4.0.0`. Both capabilities -advertise versions `3` and `4`. The supported ADTs, tuples, conditional functions, -typed calls, unary `Callable` annotations and captured lambdas, -imports and private modules work with both versions. V3 functions carry typed -value annotations, which the backend checks before generating Python. The current -v3 codec limits whole-number literals to signed 64-bit integers; v4 retains -arbitrary precision. Neither version enables general Python execution or full -Morphir IR conformance. See the README for the complete boundaries. +The historical `extension/python/v0.1.0` to `extension/python/v0.4.0` bundles +remain available in the +[morphir-rust releases](https://github.com/finos/morphir-rust/releases). --- diff --git a/docs/llms.txt b/docs/llms.txt index 50449702c..70d9b7f5c 100644 --- a/docs/llms.txt +++ b/docs/llms.txt @@ -18,7 +18,7 @@ Morphir Rust provides tools for working with Morphir IR (Intermediate Representa - [Complete Workflow](https://finos.github.io/morphir-rust/tutorials/complete-workflow): This guide walks through a complete end-to-end workflow using Morphir with Gleam. - [Tutorials](https://finos.github.io/morphir-rust/tutorials/): Step-by-step guides for using Morphir Rust. - [Extension Development Tutorial](https://finos.github.io/morphir-rust/tutorials/extension-tutorial): This tutorial follows the current Extism and MEP JSON-RPC extension model. The feature is implemented for contributor testing but has not produced a public release. -- [Python ADT extension](https://finos.github.io/morphir-rust/tutorials/python-extension): The `morphir-python-binding` crate provides the `morphir-python` frontend and backend. Start with its [modeling example and supported subset](../../crates/morphir-python-binding/README.md). The ext... +- [Python ADT extension](https://finos.github.io/morphir-rust/tutorials/python-extension): The Python extension (crate `morphir-python-binding`, bundle `morphir-python`) is now maintained and released from [finos/morphir-python](https://github.com/finos/morphir-python). Its source, tests... ## For Contributors diff --git a/docs/tutorials/extension-tutorial.md b/docs/tutorials/extension-tutorial.md index 95e5c0f80..996f52caa 100644 --- a/docs/tutorials/extension-tutorial.md +++ b/docs/tutorials/extension-tutorial.md @@ -12,8 +12,9 @@ feature is implemented for contributor testing but has not produced a public release. For an extension implementing both compilation and generation, see the -[Python ADT extension](python-extension.md). It uses the same SDK and MEP -transport for both capabilities. +[Rust binding](https://github.com/finos/morphir-rust/tree/main/crates/morphir-rust-binding). +It uses the same SDK and MEP transport for both capabilities. The Python +extension now lives in [finos/morphir-python](https://github.com/finos/morphir-python). ## 1. Start from the reference backend diff --git a/docs/tutorials/python-extension.md b/docs/tutorials/python-extension.md index c22187abf..ad9279475 100644 --- a/docs/tutorials/python-extension.md +++ b/docs/tutorials/python-extension.md @@ -7,202 +7,12 @@ parent: Tutorials # Python ADT extension -The `morphir-python-binding` crate provides the `morphir-python` frontend and -backend. Start with its [modeling example and supported subset](../../crates/morphir-python-binding/README.md). -The extension compiles frozen dataclasses, named unions, fixed tuples and annotated conditional -functions, typed calls and unary lambdas to Morphir IR v3 or v4, and generates -Python 3.12 or later code from those definitions. - -The published Python `0.2.0` bundle also supports IR v3, private modules, multiple source modules -with imports, and higher-order functions using unary `Callable` annotations. The earlier `0.1.0` -bundle supports one public module per request and IR v4 only. - -Functions support returning `if`/`elif`/`else` branches and Python conditional -expressions. For example: - -```python -def choose(flag: bool, first: int, second: int) -> int: - if flag: - return first - else: - return second -``` - -This body maps to Morphir's `IfThenElse` value. Each path must return the declared -type, and the condition must be boolean. Compile functions with `typesOnly=false` -through MEP. Project compilation through the CLI includes function bodies. The crate README lists the -supported expressions and remaining limits. - -## Build and test - -Run these commands from the `morphir-rust` repository: - -```console -rustup target add wasm32-unknown-unknown -cargo test --locked -p morphir-python-binding -cargo run --locked -p morphir-python-binding --example python_adt -cargo run --locked -p morphir-python-binding --example python_modules -cargo build --locked --release -p morphir-python-binding --target wasm32-unknown-unknown -cargo test --locked -p morphir-host-native --test python_extension -- --ignored --exact python_adt_and_conditional_roundtrip_through_the_real_wasm_extension -``` - -The example returns JSON containing IR and generated Python. The WASM test -performs capability negotiation, compilation, generation and recompilation -through the Extism container in `morphir_host_native::extism`. CI runs it -with a release guest. - -## Install a release bundle - -The Python extension is released independently as `extension/python/v0.4.0` in -[finos/morphir-rust releases](https://github.com/finos/morphir-rust/releases). -It contains a WASM guest, SHA-256 checksum and release descriptor. It is a Morphir -extension, not a PyPI package or a CPython import module. - -Use Rust Morphir CLI `0.4.0-beta.7` or later with this bundle: its version-2 descriptor carries -capability claims, which earlier CLIs cannot read, so they cannot publish it. The npm `morphir-elm` -executable does not provide these installation commands. - -Download and install with PowerShell: - -```powershell -gh release download extension/python/v0.4.0 --repo finos/morphir-rust --dir python-bundle -Rename-Item -LiteralPath python-bundle/morphir-python-binding-0.4.0.release.json -NewName release.json -$pythonIndex = Join-Path (Get-Location) '.morphir/python-index' -$env:MORPHIR_HOME = Join-Path (Get-Location) '.morphir/python-home' -morphir extension repository init $pythonIndex -morphir extension repository add python-local --directory $pythonIndex -morphir extension repository publish python-local --bundle python-bundle -morphir extension repository verify python-local -morphir extension install --repository python-local morphir-python -``` - -Newly built bundles use version-2 descriptors containing the WASM guest's own -capability claims. Publishing requires Morphir CLI `0.4.0-beta.7` or later; -older CLIs cannot read them. The release asset filename is unchanged, so the -`Rename-Item` command above also applies to version-2 bundles. WASM installs -retain `claims` with `claimCheck: "unchecked"` and display `Claims: unchecked`. - -Keep the same `MORPHIR_HOME` for compilation and generation. Installation verifies -the artifact digest and records both capabilities. Once installed, compilation -and generation work offline without access to the original repository. - -## Build a development bundle - -From `morphir-rust`, run `mise run extension:artifact:python`. This builds and -validates the guest, packages it, and tests installation and offline roundtripping. -Use `.morphir/build/extensions/python` as the `--bundle` directory in the -publication command above; its descriptor is already named `release.json`. -CI also uploads this directory as the `morphir-python-extension-bundle` artifact. - -For a Windows build using the native toolchain, after the build commands above: - -```powershell -python -B scripts/package_extension.py --short-id python --wasm target/wasm32-unknown-unknown/release/morphir_python_binding.wasm --output .morphir/build/extensions/python -$env:MORPHIR_PYTHON_BUNDLE = (Resolve-Path .morphir/build/extensions/python).Path -cargo test --locked -p morphir-host-native --test python_extension -- --ignored -``` - -The release pipeline builds from a committed snapshot and verifies that the -bundle's `gitCommit` matches the release tag. Development bundles from modified -working trees omit that provenance claim. - -## Compile and generate - -Create a project directory with a `src/models.py` file using the example above, -and this `morphir.toml`: - -```toml -[project] -name = "acme/example" -version = "0.1.0" -source_directory = "src" - -[frontend] -language = "python" -emit_parse_stage = false - -[codegen] -targets = ["python"] -``` - -From that project directory, use the Rust CLI's installed provider path: - -```console -morphir compile --language python --input src/models.py --output compiled -morphir generate --target python --input compiled/morphir-ir.json --output generated -``` - -Compilation installs `morphir-ir.json` under the `compiled` directory. The -backend returns `models.py` as an artifact. The host writes it to the output -location according to the CLI's install/output settings. The extension itself -has no filesystem access. It fails with diagnostics if the request contains -unsupported imports, function behavior, or types outside the -documented subset. The [binding README](../../crates/morphir-python-binding/README.md) -lists the supported IR nodes, tuple forms, and the scope of its conformance evidence. - -With the development bundle installed, place related files under `src/` and -compile the directory to resolve their imports together: - -```console -morphir compile --language python --input src --output compiled -morphir generate --target python --input compiled/morphir-ir.json --output generated -morphir compile --language python --input generated --output recompiled -``` - -For example, `src/domain/models.py` may define `Decision`, and -`src/domain/rules.py` may use `from .models import Decision` in its annotations. -The resulting IR contains `domain/models` and `domain/rules`; generation returns -`domain/models.py` and `domain/rules.py`. Nested directories are namespace -packages, without `__init__.py` files. Keep all referenced modules in the source -set. Same-package function imports and typed calls work too; cross-package -dependencies remain unsupported. - -With a CLI and development bundle containing project-context support, the -configured source directory is sufficient: - -```console -morphir compile --project packages/orders -morphir generate --project packages/orders --target python --output generated -``` - -The project selector accepts a declared workspace-relative path or exact -project name. Compilation uses that member's source directory and package name; -generation reads its recorded compile artifact. An explicit `--input` replaces -the configured input and resolves from the invocation directory. - -Set `project.exposed_modules = ["domain.rules"]` to expose only that module. -`domain.models` remains in the IR as a private module and is still available to -sibling imports. Generation emits both files. Preserve the exposure configuration -when recompiling generated source: Python itself does not enforce Morphir module -privacy. Omitting exposure makes all modules public; `[]` makes all private. - -## Select the IR version - -Use a CLI and development bundle containing IR v3 support. The CLI defaults to v4; -select v3 in the project configuration: - -```toml -[ir] -format_version = 3 -layout = "single-file" -format = "json" -``` - -`format = "yaml"` also works for v3. Document-tree storage requires v4. -The CLI override applies to this invocation, without editing the project: - -```console -morphir compile --ir-version 3 --output compiled-v3 -morphir generate --target python --input compiled-v3/morphir-ir.json --output generated-v3 -morphir compile --ir-version 4 --output compiled-v4 -``` - -The backend detects the version from the input document. Through MEP, set -`options.irVersion` to `3` or `3.0.0`, or to `4` or `4.0.0`. Both capabilities -advertise versions `3` and `4`. The supported ADTs, tuples, conditional functions, -typed calls, unary `Callable` annotations and captured lambdas, -imports and private modules work with both versions. V3 functions carry typed -value annotations, which the backend checks before generating Python. The current -v3 codec limits whole-number literals to signed 64-bit integers; v4 retains -arbitrary precision. Neither version enables general Python execution or full -Morphir IR conformance. See the README for the complete boundaries. +The Python extension (crate `morphir-python-binding`, bundle `morphir-python`) is +now maintained and released from +[finos/morphir-python](https://github.com/finos/morphir-python). Its source, +tests, examples and release bundles live in that repository. This repository no +longer builds or releases it. + +The historical `extension/python/v0.1.0` to `extension/python/v0.4.0` bundles +remain available in the +[morphir-rust releases](https://github.com/finos/morphir-rust/releases).