Repository navigation
feat: maintain the Python compiler here with a native API, CLI and extension bundle - #41
Merged
Merged
Conversation
Link the A4 migration to native Python API, CLI round trips and coordinated provider cutover without assuming a Rust speed advantage. Refs: morphir-ukvn.1.6
Copy crates/morphir-python-binding from finos/morphir-rust main 0aa60eb799d46f8a317ec47abef4a7a4e61dc8ac (no binding changes since the inspected f4298d0 snapshot) into a new rust/ Cargo workspace. Resolve morphir-core =0.2.0 and morphir-extension-sdk =0.3.0 from crates.io instead of sibling paths and keep the four Ruff crates pinned together at =0.0.15. Pin Rust 1.98.1 and wasm-tools in mise. Update README links for the new repository and document external Cargo use.
The SDK refuses compile and generate requests before initialization, so both examples failed with RPC error -32014 at morphir-rust 0aa60eb too. Send the initialize handshake first, as the test support does.
Add rust/crates/morphir-python-native, a PyO3 0.30.1 bridge built as morphir_python_native._native. It extracts an owned request, detaches from the interpreter, calls the binding's Frontend and Backend traits, catches panics, and returns private JSON with IR as text. The compiler crate stays free of PyO3. Add the Maturin 1.15.0 distribution morphir-python-native 0.1.0 with stubs and py.typed, wired into the uv workspace. morphir-python pins it exactly. morphir_tools.python adds compile_sources, generate_python and native_compiler with frozen result unions, a severity enum, source locations separate from IR paths, centralized codes, and a lazy loader that rejects mismatched distribution, module or bridge versions.
compile reads one file or a sorted directory tree without following symbolic links, and writes a v3 Library by default or draft v4 with --ir-version v4. Repeated --expose options select public modules. A failure keeps an existing output file. generate decodes with the shared format reader and codecs, validates artifact paths and collisions, requires a new or empty real directory, and publishes a staged tree as a whole. Both commands load the compiler lazily, so ir inspect and ir convert never initialize it.
Document installation, the typed API, diagnostics, CLI options, the supported subset and source builds. Add behave scenarios for compile, generate and recompile in both IR versions, rejection, output protection and in-process API use, and run the guide's example as a test. Exclude rust/ compiler inputs from Ruff.
Add scripts/qualify_native.py. It checks a native wheel's cp314-abi3 name, module, stubs, py.typed, license and metadata against the crate version, the tools pin and EXPECTED_NATIVE_VERSION; builds a wheel from the source archive alone outside the checkout; and installs the exact core, native and tools wheels with Rust off PATH to exercise the API, the CLI round trip, a library-only install and refusal of a mismatched native install. Each check prints a digest receipt. The stub now declares __all__, so mypy.stubtest finds no difference.
rust:check formats, lints and tests the workspace, runs the examples, proves the compiler crate has no PyO3 and validates the release WASM imports only the Extism host. rust:consumer builds an independent Cargo consumer of HEAD. native:build and native:qualify build and qualify the native wheel and source archive. extension:bundle packages and checks the MEP bundle offline. install can reuse a prebuilt native wheel or skip it; typecheck covers the stubs and tests run stubtest.
The finos/morphir v4 JSON profile at efa36060 says that readers MUST accept
every spelling the v4 JSON Schema marks as valid. The draft v4 reader read
only the expanded spelling, so it refused the compact canonical output of
Rust morphir-core 0.2.0, which the migrated Python compiler writes.
The reader now accepts:
- bare FQName and lowercase-name strings and bare arrays at type position;
- bare booleans, numbers (integer or float by lexeme), names, FQNames and
arrays at value position (decision 0009);
- the compact payloads of Variable, Reference (string or [fqn, args...]),
Tuple, List, Literal, Constructor, FieldFunction, TuplePattern and
LiteralPattern, and the {"value": x} literal payload;
- legacy name, path and FQName arrays, legacy constructor and value
specification input arrays, and legacy tagged-array type definitions and
specifications;
- doc flattened beside a definition or specification (decision 0010);
- the Public/Private/public/private/pub access spellings, the legacy
{"access", "value"} form and the flattened {"access", ...} form.
Each spelling gives the same model as the expanded one, and the writer still
emits the expanded spelling. Spellings the schema does not allow still fail
with a located diagnostic. The reader now also refuses a DecimalLiteral that
is not a decimal lexeme, a non-finite FloatLiteral, a DocumentLiteral in a
LiteralPattern, and a doc on a let-bound value definition, which the Python
model cannot hold.
Tests decode the Python compiler output (morphir-python-binding 0.4.0) and
the finos/morphir v4 examples at efa36060, which are checked in as fixtures.
rust/extension.toml declares the morphir-python extension and pins the canonical WASM packager (finos/morphir-rust 0aa60eb) and the Morphir CLI 0.4.0-beta.10 by digest. The finos/morphir extension bundle command handles native process extensions only. scripts/extension_bundle.py runs the pinned packager unchanged on this repository's declaration, checks the bundle, then installs it with the pinned CLI, deletes the repository, blocks network access and round trips IR v3 and v4. The host's IR must equal the Python API's.
scripts/ci_changes.py maps changed paths to python, native, rust and bundle categories; unknown paths, workflow changes, pushes and release calls run everything. The All Checks aggregate fails for a failed, cancelled, missing or unexpectedly skipped job. CI builds one Linux x86_64 manylinux_2_28 wheel and source archive, audits it, and reuses it for tests, the bundle host check and clean-install qualification. release.yml adds native-v and extension/python/v tags: a five-platform wheel matrix that installs each exact wheel without Rust, candidate verification with a SHA256SUMS receipt, and approval-gated publication. A python-v release waits until its pinned native release is on PyPI. Document the extension, the release order and the new tasks.
PyO3's build script needs an interpreter, and mise points VIRTUAL_ENV at a .venv that the Rust CI job does not create. Maturin now enables a python feature for the module; Rust tests, examples and benchmarks of the bridge logic build without Python, and rust:check lints the module with the feature. Expose compile_request so benchmarks time the same request. The wheel audit takes ownership of the container's output.
mise run bench builds and installs the release native wheel, then scripts/bench.py times the Rust trait calls and the JSON bridge (via a release-build Rust example), the CPython module, the public API, the v3 and v4 codecs, the CLI in new processes, cold import, and ast.parse as a parsing-only baseline. It uses the checked-in sample and a generated multi-module input with scale parameters, records warmup, repetitions, input and IR sizes, toolchains, hardware and the native module digest, and writes raw samples plus an HTML report under .dev/bench.
Derive per-input statements from the medians and re-render a report from saved results with --report-only.
IR that morphir.ir.migrate.v3_to_v4 produces carries the empty documentation every v3 member has. The v4 backend rejected any documentation, although the v3 reader already treats empty text as absent. Treat empty v4 documentation the same way; text is still rejected. Bump morphir-python-binding to 0.4.1, the first version released from this repository, so it never shares a version with a different finos/morphir-rust bundle. Add cross-codec tests: arbitrary v4 integers and nested tuples, the v3 64-bit limit, compact and expanded v4 generating the same Python, explicit refusal of migrated typed values and of downgrades without documentation, warnings kept on success, and both type checkers on the documented example. Document the limits. Remove an invalid fmt comment.
…checks - generate and compile publish with the umask's mode, or keep an existing directory's or file's mode, instead of 0700 and 0600. - Non-UTF-8 sources and internal compiler errors end with a message, not a traceback. - v4 FQName strings need a package and a module path, as the schema's pattern requires. - The wheel audit matches auditwheel's verdict, not the file name, and runs with pipefail. - A python-v release tests the published native wheel and refuses if the compiler changed after its native-v tag. - Changes to extension_bundle.py run lint and type checks, and .gitignore changes run the packaging checks.
The CLI wheel now requires morphir-python-native, which is not on PyPI until its first release. release-check installs the wheel CI built, or for a python-v release resolves the published one, and leaves the first-party native package out of the third-party vulnerability audit. The Build job downloads the native artifact before this step.
The aarch64 audit now takes ownership of the container's dist output, the Rust-free PATH also drops directories that hold only rustup, and the Windows virtual environment interpreter is Scripts/python.exe. The first five-platform qualification run built and checked every wheel but failed in these steps.
DamianReeves
marked this pull request as ready for review
October 27, 2026 21:04
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Member
Author
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b96f63c0e5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+257
to
+260
| - name: Qualify the native archive and a clean install without Rust | ||
| run: mise run native:qualify | ||
| env: | ||
| NATIVE_DIST: ${{ github.workspace }}/dist/native |
There was a problem hiding this comment.
On a python-v* release, this step receives only NATIVE_DIST; RELEASE_TAG is set on the preceding release-check step, not this one. Consequently, the RELEASE_TAG guard in .config/mise/tasks/native/qualify:9 never activates: qualification tests the newly built native wheel instead of the published pinned wheel and skips the check for compiler changes since the native release. The release-check script verifies publication but only exercises ir inspect, which does not load the compiler, so the CLI can be published without testing compilation against the native bytes users will actually install. Pass ${{ inputs.release-tag }} to this step as well.
Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment.
Verified: the guard in .config/mise/tasks/native/qualify never saw RELEASE_TAG in CI. Fixed in f2bfd39: the native qualification step now receives RELEASE_TAG: ${{ inputs.release-tag }}. actionlint is clean.
| f"{payload!r} is not a decimal lexeme: " | ||
| "[+-]?(digits(.digits?)?|.digits)([eE][+-]?digits)?" | ||
| ) | ||
| return lit.DecimalLiteral(Decimal(payload)) |
There was a problem hiding this comment.
A decimal such as {"DecimalLiteral": "1e999999999999999999999999999"} passes the new lexical validation, but Decimal(payload) raises decimal.InvalidOperation because its exponent exceeds the implementation's range. The previous decoder caught this exception and returned an invalid_literal diagnostic; this change lets it escape instead, causing API callers and CLI operations that catch DiagnosticError to fail with an unexpected exception or traceback. Retain the exception-to-diagnostic conversion around the constructor even after validating the lexeme.
Useful? React with 👍 / 👎.
Member
Author
There was a problem hiding this comment.
Verified: decode_literal({"DecimalLiteral": "1e999999999999999999999999999"}) raised decimal.InvalidOperation. Fixed in f2bfd39: the constructor's InvalidOperation becomes an invalid_literal diagnostic again. A regression test was added (test_a_decimal_beyond_the_decimal_range_is_a_diagnostic); the full suite passes, 2466 tests.
- Pass RELEASE_TAG to native qualification in CI, so a python-v release qualifies the published native wheel and refuses a compiler change since its native-v tag. - Report a decimal lexeme outside Python's Decimal range as invalid_literal instead of raising InvalidOperation.
This was referenced Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements OpenSpec change
migrate-python-binding(epic beadmorphir-ukvn.1.6). This PR moves the Python language compiler from finos/morphir-rust into this repository. It also gives Python users the compiler without Rust:rust/crates/morphir-python-bindingcomes from finos/morphir-rust0aa60eb. The binding is unchanged since the inspectedf4298d0, and its 33-file inventory is identical exceptCargo.tomlandREADME.md. It now resolves the publishedmorphir-core =0.2.0andmorphir-extension-sdk =0.3.0, and keeps Ruff pinned at=0.0.15. The native Rust API and MEP identity are preserved, and the Rust tests match the upstream run exactly. The examples now send the MEPinitializehandshake; without it they failed at upstream too.rust/crates/morphir-python-nativeuses PyO3 0.30.1. It takes ownership of the request, runs the compiler outside the GIL (detach), turns panics into errors, and has no shared state. The compiler crate and the WASM guest never link CPython.morphir-python-native0.1.0 is a cp314-abi3 wheel with stubs andpy.typed.morphir-pythonpins it exactly. The compiler loads lazily and checks the distribution, module and bridge versions.morphir_tools.python.compile_sources/generate_pythonreturn frozen union results (CompiledV3 | CompiledV4 | CompileFailure,Generated | GenerationFailure). Models decode through the existing v3/v4 codecs. Diagnostics keep their codes, severities and source locations.morphir-python compile SOURCE --package P --output F [--ir-version v4] [--expose M]...andgenerate F --output-dir Dstage their output, validate paths, collisions and symlinks, and keep existing user files.ir inspectandir convertnever load the compiler.extension bundlecommand packages native process extensions only. The bundle is checked offline in the pinned Morphir CLI 0.4.0-beta.10.All Checksaggregate gates the result.release.ymladdsnative-v*(5-platform matrix) andextension/python/v*, with candidate verification and approval-gated publication in native-before-tools order.Evidence (local, macOS arm64)
mise run check: lint, format, mypy and pyright strict. 2445 tests, 98.3% coverage,mypy.stubtestclean, 34 behave scenarios.mise run rust:check: binding and bridge tests, Gherkin acceptance, examples, clippy-D warnings. A release WASM build imports onlyextism:host/env.scripts/qualify_native.py: the wheel contents match, and a wheel builds from the sdist alone outside the checkout. A clean install of the exact core, native and tools wheels with Rust offPATHpasses the API and CLI round trips in v3 and v4 and runs the generated code. A library-only install works, and a mismatched native install is refused.scripts/extension_bundle.py: publish, install, delete the repository, block the network, then compile, generate and recompile in v3 and v4 through Morphir CLI 0.4.0-beta.10. The host IR equals the Python API IR.Qualification since the first push
workflow_dispatch, run 40580312956, non-publishing): each manylinux_2_28 x86_64/aarch64, macOS arm64/x86_64 and win_amd64 wheel was installed with the exact core and CLI wheels without Rust and exercised. Candidate verification producedSHA256SUMS, and the source archive digest32dc497d…is reproducible. In the earlier run 40576693113, verification and publication refused to run after failed platform jobs.v3_to_v4) is now accepted. Documentation text and value attributes are still rejected explicitly. The version bump avoids reusing morphir-rust's 0.4.0 identity.ast.parseis reported only as a parsing baseline. No claim compares Rust and Python compiler speed.Approval gates
Merging, tagging
native-v0.1.0/python-v*/extension/python/v0.4.0(or later), PyPI and GitHub release publication, and the finos/morphir pin change all wait for maintainer approval. The coordinated morphir-rust removal and finos/morphir provider changes go in separate PRs.All 26 tasks in
openspec/changes/migrate-python-binding/tasks.mdare complete. The release and cutover follow-up is beadmorphir-ukvn.1.6.8, in this order: merge #41,native-v0.1.0,python-v<next>,extension/python/v0.4.1, the final pin in finos/morphir#1104, then merge finos/morphir-rust#314.