add-process-mep-extension
- +Context
See proposal.md for motivation and the two spec deltas for requirements. PythonExtension already implements native Rust frontend/backend traits. The published SDK 0.3.0 exposes NativeExtension::frontend_backend, open_protocol, model methods and admission helpers. Its wire request type still uses a numeric u64 ID, while the canonical contract admits numeric and string IDs; adopting helpers alone is not full draft conformance. The Python implementation does not opt into model transport and its manifest declares MEP 0.1 only. finos/morphir c552dac801f23f2b2b566d29e08780723498ce36 defines exact MEP 0.2.0-draft.1 and model envelope 0.1.0-draft.1. These versions are independent of SDK 0.3.0, compiler 0.4.2, claims format and IR version.
Goals and non-goals
Add a native Rust process guest and current-draft model adapters while retaining WASM and legacy behavior. Keep language semantics shared. Initial model tuples use morphir-ir/morphir-json and supported Library IR 3.0.0 and draft 4.0.0. Rich Ion profiles, unsupported distribution kinds, additional Python syntax, cancellation/progress and performance claims are not introduced. A Python CLI stdio wrapper can follow separately; the selected CLI installation remains mise/pipx.
Decisions
RNT-1. Add a thin native Rust guest
Add a process guest crate/entry point, depending on the maintained compiler and published SDK, with no PyO3 or CPython requirement. The proposed command is:
morphir-python-extension --stdioReuse the SDK's session endpoint and dispatch. Keep model adaptation in the shared PythonExtension so native, WASM and process paths agree. Put stdio effects in the guest layer. A Python CLI wrapper would reuse the engine but retain an interpreter/launcher dependency; a frozen Python CLI would bundle an unnecessary runtime for this guest.
| Boundary | Current release | Proposed change |
|---|---|---|
| MEP | Legacy 0.1 | Retain 0.1 and add exact 0.2.0-draft.1 |
| Model envelope | Not advertised | 0.1.0-draft.1, admitted JSON profile |
| Extension runtime | WASM | WASM and native process |
| SDK | 0.3.0, numeric-only wire IDs | Qualified published pin covering the canonical wire contract |
| Semantic IR | Supported v3 and draft v4 Library models | Same semantic scope, explicit admitted tuples |
SDK-1. Resolve concrete SDK wire gaps before claiming conformance
Pin a published SDK that supports canonical numeric and string request IDs and the required draft behavior. SDK 0.3.0 currently has numeric-only ExtensionRequest.id, so link and qualify an upstream SDK update if no suitable release exists. Do not vendor a sibling checkout or silently coerce wire IDs. Both WASM and process must pass the same ID-preservation vectors before the new draft is advertised. Related upstream work is MEP protocol type generation, issue 241; its existence is not evidence that the wire gap is fixed.
MEP-1. Negotiate the new draft explicitly
Retain legacy 0.1 operations and exact 0.2.0-draft.1 offers. Opt in only after supported model methods and executable conformance pass. Reuse SDK model admission and session rules; method constants alone are not support. Never enable model methods under legacy negotiation. Keep optional flags false until implemented.
MOD-1. Start with the explicit JSON model profile
Implement compile_model/generate_model over the existing compile/generate core. Admit complete operation/profile/codec/envelope/IR tuples and their bilateral limits. The initial profile is morphir-ir/morphir-json with envelope 0.1.0-draft.1. Use the canonical base64 payload, exact byte length/hash and critical-field rules. Reject unadvertised Ion profiles, guarantees and unsupported semantic forms. This achieves bounded new-draft support without implying rich Ion preservation or silently converting rich input.
IO-1. Treat framing and lifetime as a bounded server boundary
Use Content-Length JSON-RPC framing, byte-based lengths, an 8 KiB header ceiling and an initial 16 MiB complete-body ceiling. Advertised decoded/encoded model limits must fit those outer bounds, including envelope overhead. Enforce local bounds even if the peer advertises more. Check partial input, duplicate lengths, integer overflow, invalid UTF-8/JSON and EOF before unbounded allocation. Reserve stdout for frames and flush responses. Follow initialization, capability locks, shutdown and exit; do not respond to notifications. Host supervision owns deadlines and descendant cleanup.
RLS-1. Qualify dual-runtime distribution
Retain the portable WASM guest and publish native process executables for the advertised targets, with --stdio locked as an argument. Use a canonical host-readable descriptor with runtime/platform/mode/digest/claims bound per artifact. Validate the staged descriptor with pinned host tooling; do not assume that either the existing WASM-only packager or native-only bundle command can package a mixed release unchanged. Preserve historical assets. Release exact native/CLI package pairs if shared compiler changes affect them.
HST-1. Qualify hosts independently
The Rust host provides ProcessLaunch/ProcessChannel and verified-byte staging. It clears the child environment, favoring a directly executable Rust guest. Qualify a pinned conforming Rust host and test each documented MoonBit route independently. Use a pinned source build when a released host lacks the new draft, and state that prerequisite. If a route needs upstream work, record that dependency and label it unsupported until it passes. Existing MoonBit process wrappers demonstrate framing, not Python extension activation.
Decisions
RNT-1. Add a thin native Rust guest
Add a process guest crate/entry point, depending on the maintained compiler and published SDK, with no PyO3 or CPython requirement. The proposed command is:
morphir-python-extension --stdioReuse the SDK's session endpoint and dispatch. Keep model adaptation in the shared PythonExtension so native, WASM and process paths agree. Put stdio effects in the guest layer. A Python CLI wrapper would reuse the engine but retain an interpreter/launcher dependency; a frozen Python CLI would bundle an unnecessary runtime for this guest.
| Boundary | Current release | Proposed change |
|---|---|---|
| MEP | Legacy 0.1 | Retain 0.1 and add exact 0.2.0-draft.1 |
| Model envelope | Not advertised | 0.1.0-draft.1, admitted JSON profile |
| Extension runtime | WASM | WASM and native process |
| SDK | 0.3.0, numeric-only wire IDs | Qualified published pin covering the canonical wire contract |
| Semantic IR | Supported v3 and draft v4 Library models | Same semantic scope, explicit admitted tuples |
SDK-1. Resolve concrete SDK wire gaps before claiming conformance
Pin a published SDK that supports canonical numeric and string request IDs and the required draft behavior. SDK 0.3.0 currently has numeric-only ExtensionRequest.id, so link and qualify an upstream SDK update if no suitable release exists. Do not vendor a sibling checkout or silently coerce wire IDs. Both WASM and process must pass the same ID-preservation vectors before the new draft is advertised. The concrete prerequisite is request ID support, issue 316, tracked as morphir-4wrx. A candidate SDK 0.4.0 is under qualification and is not yet published. Related upstream work is MEP protocol type generation, issue 241; its existence is not evidence that the wire gap is fixed.
MEP-1. Negotiate the new draft explicitly
Retain legacy 0.1 operations and exact 0.2.0-draft.1 offers. Opt in only after supported model methods and executable conformance pass. Reuse SDK model admission and session rules; method constants alone are not support. Never enable model methods under legacy negotiation. Keep optional flags false until implemented.
MOD-1. Start with the explicit JSON model profile
Implement compile_model/generate_model over the existing compile/generate core. Admit complete operation/profile/codec/envelope/IR tuples and their bilateral limits. The initial profile is morphir-ir/morphir-json with envelope 0.1.0-draft.1. Use the canonical base64 payload, exact byte length/hash and critical-field rules. Reject unadvertised Ion profiles, guarantees and unsupported semantic forms. This achieves bounded new-draft support without implying rich Ion preservation or silently converting rich input.
IO-1. Treat framing and lifetime as a bounded server boundary
Use Content-Length JSON-RPC framing, byte-based lengths, an 8 KiB header ceiling and an initial 16 MiB complete-body ceiling. Advertised decoded/encoded model limits must fit those outer bounds, including envelope overhead. Enforce local bounds even if the peer advertises more. Check partial input, duplicate lengths, integer overflow, invalid UTF-8/JSON and EOF before unbounded allocation. Reserve stdout for frames and flush responses. Follow initialization, capability locks, shutdown and exit; do not respond to notifications. Host supervision owns deadlines and descendant cleanup.
RLS-1. Qualify dual-runtime distribution
Retain the portable WASM guest and publish native process executables for the advertised targets, with --stdio locked as an argument. Use a canonical host-readable descriptor with runtime/platform/mode/digest/claims bound per artifact. Validate the staged descriptor with pinned host tooling; do not assume that either the existing WASM-only packager or native-only bundle command can package a mixed release unchanged. Preserve historical assets. Release exact native/CLI package pairs if shared compiler changes affect them.
HST-1. Qualify hosts independently
The Rust host provides ProcessLaunch/ProcessChannel and verified-byte staging. It clears the child environment, favoring a directly executable Rust guest. Qualify a pinned conforming Rust host and test each documented MoonBit route independently. Use a pinned source build when a released host lacks the new draft, and state that prerequisite. If a route needs upstream work, record that dependency and label it unsupported until it passes. Existing MoonBit process wrappers demonstrate framing, not Python extension activation.
Risks and trade-offs
| Risk | Mitigation |
|---|---|
| Draft, SDK and runtime support are confused | Exact independent version table, live negotiation and candidate-bound conformance before claims. |
| Model content or guarantees are lost | Admit only explicit JSON tuples initially. Refuse unsupported profiles, critical fields and guarantees; no silent fallback. |
| Malformed input exhausts resources | Bound headers, wire bodies and decoded bytes before allocation; test boundary and partial cases. |
| Process mode is mistaken for a sandbox | Document ordinary OS privileges and the host's explicit supervision, distinct from WASM isolation. |
| Packaging or host support lags | Pin descriptor readers and host builds, qualify installed bytes, record upstream blockers and make only verified support claims. |
| Native execution is assumed faster | Measure startup and representative compile/generate workloads before making a performance claim. |
Migration and rollout
- Add negotiation and bounded JSON model adapters with tests and compatibility documentation.
- Add the Rust process server with framing and lifecycle tests; preserve the existing WASM path.
- Build and qualify runtime artifacts and descriptors with matching claims and source receipts.
- Run pinned MEP guest conformance and real host installation/operation cases, including legacy hosts and explicit refusals.
- After maintainer approval, publish a new extension version and document runtime selection. Existing users can stay on the prior WASM release; never replace its bytes.
The current proposal does not change protocol claims or release a guest. Every process task remains unstarted.
Repository evidence and references
- rust/crates/morphir-python-binding/src/lib.rs: shared PythonExtension traits and truthful current capabilities.
- rust/crates/morphir-python-native/src/lib.rs: current private two-function JSON bridge; it is not a full MEP transport.
- Published morphir-extension-sdk 0.3.0: native.rs, traits.rs and model admission helpers.
- finos/morphir-rust 26d5e707e5ec94d16421db6380edbdc865d85c04: process launch/channel and stdio frontend fixture. Fixture framing must not be copied as a hardened server.
- finos/morphir-moonbit bd3adc52e6b8f6368fdaee50f76535799732f8d6: portable MEP embedding wrapper. No claim of Python host compatibility follows.
- Canonical MEP contract and model transport rules.
Related work: morphir-bj9r owns the selected CLI installation route; morphir-ukvn.5.98 tracks the earlier MoonBit extension qualification context.
(or json, toml, bash, diff). Give sections an id and
list them in nav.toc.
-->
-
-
-
-
-
+
+
+
+
+
add-documentation-site
-
+
diff --git a/openspec/changes/add-process-mep-extension/proposal.md b/openspec/changes/add-process-mep-extension/proposal.md
index 9bb7cf18..6275950b 100644
--- a/openspec/changes/add-process-mep-extension/proposal.md
+++ b/openspec/changes/add-process-mep-extension/proposal.md
@@ -11,7 +11,6 @@ The Python compiler currently exposes a WASM MEP guest advertising legacy `0.1`,
- Add exact `0.2.0-draft.1` negotiation and bilateral model transport admission, initially for the `morphir-ir`/`morphir-json` profile with the compiler's supported IR v3/v4 Library models.
- Implement bounded, integrity-checked model envelopes and compileModel/generateModel adapters for the admitted tuples. Refuse unsupported profiles and guarantees explicitly; rich Ion transport is not advertised by this initial increment.
- Publish and qualify both runtime choices, with accurate platform/runtime descriptors and host compatibility documentation. Qualify each documented host route independently.
-- Keep this change proposed: no process guest or newer-MEP implementation has been added by PR #52.
## Capabilities
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index a7898a12..7df8a3f9 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -1465,6 +1465,7 @@ dependencies = [
"cucumber",
"morphir-core",
"morphir-extension-sdk",
+ "morphir-json",
"num-bigint 0.5.1",
"ruff_python_ast",
"ruff_python_codegen",
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index 60c90e72..afb5aa52 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -13,6 +13,7 @@ homepage = "https://github.com/finos/morphir-python"
[workspace.dependencies]
# Published Morphir crates. Never use sibling-checkout paths here.
morphir-core = "=0.2.0"
+morphir-json = "=0.2.0"
morphir-extension-sdk = "=0.3.0"
num-bigint = "0.5"
# Ruff component crates evolve together; update this group as one unit.
diff --git a/rust/crates/morphir-python-binding/Cargo.toml b/rust/crates/morphir-python-binding/Cargo.toml
index 81ff6904..5e04cc85 100644
--- a/rust/crates/morphir-python-binding/Cargo.toml
+++ b/rust/crates/morphir-python-binding/Cargo.toml
@@ -15,6 +15,7 @@ crate-type = ["rlib", "cdylib"]
[dependencies]
num-bigint.workspace = true
morphir-core.workspace = true
+morphir-json.workspace = true
morphir-extension-sdk.workspace = true
ruff_python_parser.workspace = true
ruff_python_ast.workspace = true
diff --git a/rust/crates/morphir-python-binding/src/lib.rs b/rust/crates/morphir-python-binding/src/lib.rs
index 77c08d98..46f3cbfd 100644
--- a/rust/crates/morphir-python-binding/src/lib.rs
+++ b/rust/crates/morphir-python-binding/src/lib.rs
@@ -26,6 +26,7 @@
mod backend;
mod frontend;
mod ir;
+mod model_transport;
mod modules;
mod names;
mod values;
@@ -72,6 +73,13 @@ impl Extension for PythonExtension {
}
impl Frontend for PythonExtension {
+ fn compile_model(
+ &self,
+ request: morphir_extension_sdk::model::CompileModelRequest,
+ ) -> Result {
+ model_transport::compile(self, request)
+ }
+
fn compile(&self, request: CompileRequest) -> Result {
let diagnostics = if frontend::parse_stage_requested(&request) {
vec![Diagnostic {
@@ -118,6 +126,13 @@ impl Frontend for PythonExtension {
}
impl Backend for PythonExtension {
+ fn generate_model(
+ &self,
+ request: morphir_extension_sdk::model::GenerateModelRequest,
+ ) -> Result {
+ model_transport::generate(self, request)
+ }
+
fn generate(&self, request: GenerateRequest) -> Result {
Ok(match backend::generate(&request) {
Ok(artifacts) => GenerateResult {
diff --git a/rust/crates/morphir-python-binding/src/model_transport.rs b/rust/crates/morphir-python-binding/src/model_transport.rs
new file mode 100644
index 00000000..a349eae5
--- /dev/null
+++ b/rust/crates/morphir-python-binding/src/model_transport.rs
@@ -0,0 +1,178 @@
+//! Explicit JSON-model admission around the shared compiler, without transcoding.
+use crate::{PythonExtension, error};
+use morphir_extension_sdk::model::*;
+use morphir_extension_sdk::prelude::*;
+
+pub(crate) fn support() -> ModelTransportSupport {
+ ModelTransportSupport {
+ tuples: [ModelOperation::CompileModel, ModelOperation::GenerateModel]
+ .into_iter()
+ .flat_map(|operation| {
+ ["3.0.0", "4.0.0"].map(|version| ModelTransportTuple {
+ operation,
+ profile: "morphir-ir".into(),
+ codec: ModelCodec::MorphirJson,
+ envelope_version: MODEL_ENVELOPE_VERSION.into(),
+ ir_version: version.into(),
+ })
+ })
+ .collect(),
+ // Base64 payloads leave at least 4 MiB for envelope/source metadata in
+ // the process transport's 16 MiB complete-message bound.
+ limits: ModelTransportLimits {
+ max_encoded_bytes: 12 * 1024 * 1024,
+ max_decoded_bytes: 8 * 1024 * 1024,
+ max_depth: 128,
+ max_nodes: 100_000,
+ },
+ guarantees: vec![],
+ }
+}
+
+fn invalid(error: impl std::fmt::Display) -> ExtensionError {
+ ExtensionError::InvalidParams(format!("MEP model admission: {error}"))
+}
+
+fn admit(method: &str, value: &serde_json::Value) -> Result<()> {
+ let support = support();
+ let admitted = NegotiatedModelTransport::select(&support, &support).map_err(invalid)?;
+ let mut capabilities = PythonExtension::capabilities();
+ capabilities.model_transport = Some(support);
+ validate_model_invocation(&admitted, &capabilities, method, value).map_err(invalid)
+}
+
+pub(crate) fn compile(
+ extension: &PythonExtension,
+ request: CompileModelRequest,
+) -> Result {
+ admit(
+ "morphir.frontend.compileModel",
+ &serde_json::to_value(&request)?,
+ )?;
+ if !request.dependencies.is_empty() {
+ return Ok(CompileModelResult::Failed {
+ diagnostics: vec![error("PY001", "Expected Python and no dependencies")],
+ });
+ }
+ let model_id = request.package.name.clone();
+ let result = extension.compile(CompileRequest {
+ language_id: request.language_id,
+ sources: request.sources,
+ package: request.package,
+ dependencies: vec![],
+ options: request.options,
+ baseline: None,
+ })?;
+ if !result.success {
+ return Ok(CompileModelResult::Failed {
+ diagnostics: result.diagnostics,
+ });
+ }
+ let ir = result
+ .ir
+ .ok_or_else(|| invalid("successful compiler result is missing IR"))?;
+ let bytes = bounded_json(&ir, support().limits.max_decoded_bytes)?;
+ let payload = ModelPayload::from_bytes(ModelCodec::MorphirJson, &bytes).map_err(invalid)?;
+ let model = MorphirModel {
+ envelope_version: MODEL_ENVELOPE_VERSION.into(),
+ model_id,
+ revision: format!("sha256:{}", payload.sha256),
+ ir_version: request.output.ir_version.clone(),
+ profile: request.output.profile.clone(),
+ payload,
+ evidence: vec![],
+ critical: vec![],
+ };
+ retain_model(&model, &request.output, &support().limits).map_err(invalid)?;
+ Ok(CompileModelResult::Compiled {
+ model: Box::new(model),
+ diagnostics: result.diagnostics,
+ modules: result.modules,
+ })
+}
+
+pub(crate) fn generate(
+ extension: &PythonExtension,
+ request: GenerateModelRequest,
+) -> Result {
+ admit(
+ "morphir.backend.generateModel",
+ &serde_json::to_value(&request)?,
+ )?;
+ let retained =
+ retain_model(&request.model, &request.input, &support().limits).map_err(invalid)?;
+ let limits = support().limits;
+ let source = std::str::from_utf8(retained.bytes()).map_err(invalid)?;
+ let ir = morphir_json::read(
+ source,
+ &mut JsonBudget {
+ remaining: limits.max_nodes,
+ depth: limits.max_depth as usize,
+ string_bytes: limits.max_decoded_bytes as usize,
+ },
+ )
+ .map_err(invalid)?;
+ extension.generate(GenerateRequest {
+ ir,
+ target: request.target,
+ options: request.options,
+ })
+}
+
+/// Stop JSON serialization before extending the payload buffer beyond its bound.
+fn bounded_json(value: &serde_json::Value, limit: u32) -> Result> {
+ struct Buffer {
+ bytes: Vec,
+ limit: usize,
+ }
+ impl std::io::Write for Buffer {
+ fn write(&mut self, bytes: &[u8]) -> std::io::Result {
+ if self.bytes.len().saturating_add(bytes.len()) > self.limit {
+ return Err(std::io::Error::new(
+ std::io::ErrorKind::InvalidData,
+ "model payload exceeds decoded limit",
+ ));
+ }
+ self.bytes.extend_from_slice(bytes);
+ Ok(bytes.len())
+ }
+ fn flush(&mut self) -> std::io::Result<()> {
+ Ok(())
+ }
+ }
+ let mut buffer = Buffer {
+ bytes: Vec::new(),
+ limit: limit as usize,
+ };
+ serde_json::to_writer(&mut buffer, value).map_err(invalid)?;
+ Ok(buffer.bytes)
+}
+
+// Materialize only after SDK admission, using the published literal-JSON reader
+// so serde-private-looking keys cannot acquire a different scalar meaning.
+struct JsonBudget {
+ remaining: u32,
+ depth: usize,
+ string_bytes: usize,
+}
+impl morphir_json::ReadBudget for JsonBudget {
+ type Error = &'static str;
+ fn before_value(
+ &mut self,
+ depth: usize,
+ _: morphir_json::ValueKind,
+ ) -> std::result::Result<(), Self::Error> {
+ if depth >= self.depth {
+ return Err("model depth limit");
+ }
+ self.remaining = self.remaining.checked_sub(1).ok_or("model node limit")?;
+ Ok(())
+ }
+ fn before_string(&mut self, value: &str) -> std::result::Result<(), Self::Error> {
+ if value.len() > self.string_bytes {
+ Err("model string limit")
+ } else {
+ Ok(())
+ }
+ }
+}
diff --git a/rust/crates/morphir-python-binding/tests/model_transport.rs b/rust/crates/morphir-python-binding/tests/model_transport.rs
new file mode 100644
index 00000000..b9066d2d
--- /dev/null
+++ b/rust/crates/morphir-python-binding/tests/model_transport.rs
@@ -0,0 +1,228 @@
+//! The model boundary retains supported semantics and refuses unsupported admission.
+use morphir_extension_sdk::model::*;
+use morphir_extension_sdk::prelude::*;
+use morphir_python_binding::PythonExtension;
+use serde_json::{Value, json};
+
+fn a_request(version: &str) -> CompileModelRequest {
+ CompileModelRequest {
+ language_id: "python".into(),
+ sources: SourceSet {
+ root: None,
+ documents: vec![SourceDocument {
+ uri: "model.py".into(), language_id: "python".into(), version: 1,
+ text: "from dataclasses import dataclass\n@dataclass(frozen=True)\nclass Box:\n value: int\n".into(),
+ }],
+ },
+ package: CompilePackage { name: "acme".into(), exposed_modules: None },
+ dependencies: vec![],
+ options: CompileOptions { ir_version: version.into(), ..Default::default() },
+ context_digest: "a".repeat(64),
+ output: ModelTransportTuple {
+ operation: ModelOperation::CompileModel, profile: "morphir-ir".into(),
+ codec: ModelCodec::MorphirJson, envelope_version: MODEL_ENVELOPE_VERSION.into(), ir_version: version.into(),
+ },
+ }
+}
+
+fn a_model(version: &str) -> MorphirModel {
+ match PythonExtension.compile_model(a_request(version)).unwrap() {
+ CompileModelResult::Compiled { model, .. } => *model,
+ CompileModelResult::Failed { diagnostics } => panic!("{diagnostics:?}"),
+ }
+}
+
+fn generation(model: MorphirModel) -> GenerateModelRequest {
+ GenerateModelRequest {
+ input: ModelTransportTuple {
+ operation: ModelOperation::GenerateModel,
+ profile: "morphir-ir".into(),
+ codec: ModelCodec::MorphirJson,
+ envelope_version: MODEL_ENVELOPE_VERSION.into(),
+ ir_version: model.ir_version.clone(),
+ },
+ model,
+ target: "python".into(),
+ context_digest: "a".repeat(64),
+ required_guarantees: vec![],
+ options: Default::default(),
+ }
+}
+
+#[test]
+fn json_models_compile_and_generate_with_the_legacy_semantics() {
+ for version in ["3.0.0", "4.0.0"] {
+ let model = a_model(version);
+ assert_eq!(model.envelope_version, MODEL_ENVELOPE_VERSION);
+ assert_eq!(model.ir_version, version);
+ assert_eq!(model.model_id, "acme");
+ let request = generation(model);
+ let retained = retain_model(
+ &request.model,
+ &request.input,
+ &ModelTransportLimits {
+ max_encoded_bytes: 1024 * 1024,
+ max_decoded_bytes: 1024 * 1024,
+ max_depth: 128,
+ max_nodes: 100_000,
+ },
+ )
+ .unwrap();
+ let ir: Value = serde_json::from_slice(retained.bytes()).unwrap();
+ let legacy = PythonExtension
+ .generate(GenerateRequest {
+ ir,
+ target: "python".into(),
+ options: Default::default(),
+ })
+ .unwrap();
+ let generated = PythonExtension.generate_model(request).unwrap();
+ assert!(generated.success, "{:?}", generated.diagnostics);
+ let mut recompile = a_request(version);
+ recompile.sources.documents = generated
+ .artifacts
+ .iter()
+ .map(|artifact| SourceDocument {
+ uri: artifact.path.clone(),
+ language_id: "python".into(),
+ version: 1,
+ text: artifact.content.clone(),
+ })
+ .collect();
+ let reparsed = PythonExtension.compile_model(recompile).unwrap();
+ let CompileModelResult::Compiled {
+ model: reparsed, ..
+ } = reparsed
+ else {
+ panic!("generated model should recompile")
+ };
+ assert_eq!(reparsed.payload, a_model(version).payload);
+ assert_eq!(
+ serde_json::to_value(generated).unwrap(),
+ serde_json::to_value(legacy).unwrap()
+ );
+ }
+}
+
+#[test]
+fn model_compilation_preserves_language_diagnostics() {
+ let mut request = a_request("4.0.0");
+ request.sources.documents[0].text = "import os\n".into();
+ let legacy = PythonExtension
+ .compile(CompileRequest {
+ language_id: request.language_id.clone(),
+ sources: request.sources.clone(),
+ package: request.package.clone(),
+ dependencies: vec![],
+ options: request.options.clone(),
+ baseline: None,
+ })
+ .unwrap();
+ assert!(!legacy.success);
+ let CompileModelResult::Failed { diagnostics } =
+ PythonExtension.compile_model(request).unwrap()
+ else {
+ panic!("unsupported source must fail")
+ };
+ assert_eq!(
+ serde_json::to_value(diagnostics).unwrap(),
+ serde_json::to_value(legacy.diagnostics).unwrap()
+ );
+}
+
+#[test]
+fn malformed_envelopes_guarantees_and_unadvertised_forms_are_refused() {
+ let baseline = generation(a_model("4.0.0"));
+ let mut value = serde_json::to_value(&baseline).unwrap();
+ value["model"]["critical"] = json!([]);
+ for (path, invalid) in [
+ ("/model/payload/sha256", json!("0".repeat(64))),
+ ("/model/payload/decodedByteLength", json!(1)),
+ ("/model/payload/content", json!("not canonical base64")),
+ ("/model/critical", json!(["future.required"])),
+ ("/model/irVersion", json!("3.0.0")),
+ ("/input/codec", json!("ion-binary")),
+ ("/input/profile", json!("rich-model")),
+ ("/input/operation", json!("morphir.frontend.compileModel")),
+ (
+ "/requiredGuarantees",
+ json!([{"guarantee":"example.typed","scope":"model:acme"}]),
+ ),
+ ] {
+ let mut invalid_request = value.clone();
+ *invalid_request.pointer_mut(path).unwrap() = invalid;
+ let request = serde_json::from_value(invalid_request).unwrap();
+ assert!(
+ PythonExtension.generate_model(request).is_err(),
+ "accepted {path}"
+ );
+ }
+ let mut request = a_request("4.0.0");
+ request.output.ir_version = "3.1.0".into();
+ assert!(PythonExtension.compile_model(request).is_err());
+}
+
+#[test]
+fn model_round_trip_retains_large_integer_precision_and_v3_refuses_it() {
+ let decimal = "1234567890123456789012345678901234567890";
+ for version in ["3.0.0", "4.0.0"] {
+ let mut request = a_request(version);
+ request.options.types_only = false;
+ request.sources.documents[0].text = format!("def number() -> int:\n return {decimal}\n");
+ match PythonExtension.compile_model(request).unwrap() {
+ CompileModelResult::Compiled { model, .. } => {
+ assert_eq!(version, "4.0.0");
+ let generated = PythonExtension.generate_model(generation(*model)).unwrap();
+ assert!(generated.success, "{:?}", generated.diagnostics);
+ assert!(
+ generated
+ .artifacts
+ .iter()
+ .any(|artifact| artifact.content.contains(decimal))
+ );
+ }
+ CompileModelResult::Failed { diagnostics } => {
+ assert_eq!(version, "3.0.0");
+ assert!(
+ diagnostics
+ .iter()
+ .any(|item| item.message.contains("64-bit"))
+ );
+ }
+ }
+ }
+}
+
+#[test]
+fn aggregate_nodes_and_dependency_support_are_bounded() {
+ let mut request = generation(a_model("4.0.0"));
+ let retained = retain_model(
+ &request.model,
+ &request.input,
+ &ModelTransportLimits {
+ max_encoded_bytes: 1024 * 1024,
+ max_decoded_bytes: 1024 * 1024,
+ max_depth: 128,
+ max_nodes: 100_000,
+ },
+ )
+ .unwrap();
+ let mut ir: Value = serde_json::from_slice(retained.bytes()).unwrap();
+ ir["extra"] = json!(vec![0; 100_001]);
+ request.model.payload =
+ ModelPayload::from_bytes(ModelCodec::MorphirJson, &serde_json::to_vec(&ir).unwrap())
+ .unwrap();
+ assert!(PythonExtension.generate_model(request).is_err());
+
+ let mut request = a_request("4.0.0");
+ request.dependencies.push(CompileModelDependency {
+ package_name: "dep".into(),
+ model: a_model("4.0.0"),
+ });
+ let CompileModelResult::Failed { diagnostics } =
+ PythonExtension.compile_model(request).unwrap()
+ else {
+ panic!("dependencies are unsupported")
+ };
+ assert_eq!(diagnostics[0].code.as_deref(), Some("PY001"));
+}
From 22c7a8311b274745f2f86ee6e2af7774c68ebddc Mon Sep 17 00:00:00 2001
From: Damian Reeves <957246+DamianReeves@users.noreply.github.com>
Date: Wed, 28 Oct 2026 05:04:20 -0500
Subject: [PATCH 04/26] feat(extension): serve standalone MEP sessions
---
.../add-process-mep-extension/tasks.md | 4 +-
rust/Cargo.lock | 6 +
rust/Cargo.toml | 1 +
.../morphir-python-extension/Cargo.toml | 6 +
.../crates/morphir-python-extension/README.md | 27 +++++
.../morphir-python-extension/src/lib.rs | 2 +
.../morphir-python-extension/src/main.rs | 24 ++++
.../morphir-python-extension/src/server.rs | 114 ++++++++++++++++++
.../morphir-python-extension/tests/framing.rs | 13 ++
.../morphir-python-extension/tests/session.rs | 109 +++++++++++++++++
10 files changed, 304 insertions(+), 2 deletions(-)
create mode 100644 rust/crates/morphir-python-extension/README.md
create mode 100644 rust/crates/morphir-python-extension/src/main.rs
create mode 100644 rust/crates/morphir-python-extension/src/server.rs
create mode 100644 rust/crates/morphir-python-extension/tests/session.rs
diff --git a/openspec/changes/add-process-mep-extension/tasks.md b/openspec/changes/add-process-mep-extension/tasks.md
index 871bce00..2ace45e8 100644
--- a/openspec/changes/add-process-mep-extension/tasks.md
+++ b/openspec/changes/add-process-mep-extension/tasks.md
@@ -16,8 +16,8 @@ This change is being applied. The published SDK prerequisite is tracked in morph
## 3. Serve native stdio sessions (morphir-5hxy.3)
-- [ ] 3.1 Add the thin Rust process guest and documented `morphir-python-extension --stdio` entry point; verify it builds and executes without CPython/PyO3 or a user Rust toolchain.
-- [ ] 3.2 Implement bounded Content-Length framing around the SDK session; test Unicode byte lengths, partial/truncated messages, duplicate/invalid lengths, overflow, invalid UTF-8/JSON and header/body limits.
+- [x] 3.1 Add the thin Rust process guest and documented `morphir-python-extension --stdio` entry point; verify it builds and executes without CPython/PyO3 or a user Rust toolchain.
+- [x] 3.2 Implement bounded Content-Length framing around the SDK session; test Unicode byte lengths, partial/truncated messages, duplicate/invalid lengths, overflow, invalid UTF-8/JSON and header/body limits.
- [ ] 3.3 Preserve request IDs, notification silence, fixed session capabilities, shutdown/exit and protocol-only stdout; verify a real multi-request process test and document stderr and supervision behavior.
## 4. Distribute both runtime choices (morphir-5hxy.4)
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 7df8a3f9..5fbc6a77 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -1479,6 +1479,12 @@ dependencies = [
[[package]]
name = "morphir-python-extension"
version = "0.1.0"
+dependencies = [
+ "morphir-extension-sdk",
+ "morphir-python-binding",
+ "serde",
+ "serde_json",
+]
[[package]]
name = "morphir-python-native"
diff --git a/rust/Cargo.toml b/rust/Cargo.toml
index afb5aa52..6c1c135c 100644
--- a/rust/Cargo.toml
+++ b/rust/Cargo.toml
@@ -22,6 +22,7 @@ ruff_python_ast = "=0.0.15"
ruff_python_codegen = "=0.0.15"
ruff_text_size = "=0.0.15"
serde_json = "1.0"
+serde = { version = "1.0", features = ["derive"] }
tempfile = "3"
[profile.release]
diff --git a/rust/crates/morphir-python-extension/Cargo.toml b/rust/crates/morphir-python-extension/Cargo.toml
index a7947eed..84009521 100644
--- a/rust/crates/morphir-python-extension/Cargo.toml
+++ b/rust/crates/morphir-python-extension/Cargo.toml
@@ -8,3 +8,9 @@ license.workspace = true
repository.workspace = true
description = "Native stdio MEP guest for the Morphir Python compiler"
publish = false
+
+[dependencies]
+morphir-python-binding = { path = "../morphir-python-binding", version = "0.4.2" }
+morphir-extension-sdk.workspace = true
+serde.workspace = true
+serde_json = { workspace = true, features = ["raw_value"] }
diff --git a/rust/crates/morphir-python-extension/README.md b/rust/crates/morphir-python-extension/README.md
new file mode 100644
index 00000000..7b094043
--- /dev/null
+++ b/rust/crates/morphir-python-extension/README.md
@@ -0,0 +1,27 @@
+# Morphir Python process guest
+
+This crate runs the shared Python compiler as a native MEP process. It has no
+CPython or PyO3 dependency, and the executable needs no user Rust toolchain.
+Source is parsed statically and is never imported or executed.
+
+Development build and entry point:
+
+```bash
+cargo build --manifest-path rust/Cargo.toml --release -p morphir-python-extension
+./rust/target/release/morphir-python-extension --stdio
+```
+
+In stdio mode stdout contains only Content-Length-framed UTF-8 JSON-RPC.
+Headers are bounded to 8 KiB and complete bodies to 16 MiB. Lengths count bytes;
+unknown headers are ignored and duplicate lengths are refused. A damaged frame
+ends the connection. Invalid JSON in an otherwise valid frame gets a parse-error
+response and the next framed request can recover. Notifications get no response;
+`morphir.exit` ends the process. Each process owns its SDK protocol session.
+
+The host owns deadlines and process cleanup. A native process has ordinary OS
+rights; choosing it does not provide a WASM sandbox.
+
+The current development build advertises legacy MEP 0.1. Exact-draft discovery
+and model transport wait for the qualified published SDK prerequisite and
+conformance checks. Platform archives and installation recipes are still under
+development. The retained WASM release remains the qualified installation route.
diff --git a/rust/crates/morphir-python-extension/src/lib.rs b/rust/crates/morphir-python-extension/src/lib.rs
index ea0e7d8b..aa2df2e5 100644
--- a/rust/crates/morphir-python-extension/src/lib.rs
+++ b/rust/crates/morphir-python-extension/src/lib.rs
@@ -1,2 +1,4 @@
//! Bounded stdio transport for the Morphir Python process guest.
pub mod framing;
+mod server;
+pub use server::serve;
diff --git a/rust/crates/morphir-python-extension/src/main.rs b/rust/crates/morphir-python-extension/src/main.rs
new file mode 100644
index 00000000..e4cc81c3
--- /dev/null
+++ b/rust/crates/morphir-python-extension/src/main.rs
@@ -0,0 +1,24 @@
+//! Standalone process entry point; the compiler never imports or executes Python.
+use morphir_python_extension::serve;
+
+fn main() {
+ let mut arguments = std::env::args_os().skip(1);
+ let argument = arguments.next();
+ if arguments.next().is_some() {
+ eprintln!("Usage: morphir-python-extension --stdio");
+ std::process::exit(2);
+ }
+ match argument.as_deref() {
+ Some(value) if value == "--stdio" => {
+ if let Err(error) = serve(&mut std::io::stdin().lock(), &mut std::io::stdout().lock()) {
+ eprintln!("morphir-python-extension: {error}");
+ std::process::exit(1);
+ }
+ }
+ Some(value) if value == "--help" => println!("Usage: morphir-python-extension --stdio"),
+ _ => {
+ eprintln!("Usage: morphir-python-extension --stdio");
+ std::process::exit(2);
+ }
+ }
+}
diff --git a/rust/crates/morphir-python-extension/src/server.rs b/rust/crates/morphir-python-extension/src/server.rs
new file mode 100644
index 00000000..1017d056
--- /dev/null
+++ b/rust/crates/morphir-python-extension/src/server.rs
@@ -0,0 +1,114 @@
+//! Framed stdio sessions around the SDK's native protocol endpoint.
+use crate::framing::{MAX_BODY_BYTES, read_frame, write_frame};
+use morphir_extension_sdk::{
+ NativeExtension,
+ protocol::{ExtensionRequest, error_codes, methods},
+};
+use morphir_python_binding::PythonExtension;
+use serde::{Deserialize, Deserializer, Serialize};
+use serde_json::value::RawValue;
+use std::io::{self, BufRead, Write};
+
+#[derive(Deserialize)]
+struct MessageHead {
+ jsonrpc: String,
+ method: String,
+ // Explicit null is a malformed request ID, not a notification.
+ #[serde(default, deserialize_with = "present_id")]
+ id: Option>,
+}
+
+fn present_id<'de, D: Deserializer<'de>>(reader: D) -> Result>, D::Error> {
+ Box::::deserialize(reader).map(Some)
+}
+
+fn write_error(writer: &mut impl Write, code: i32, message: &str) -> io::Result<()> {
+ write_response(
+ writer,
+ &serde_json::json!({
+ "jsonrpc": "2.0", "id": null, "error": { "code": code, "message": message },
+ }),
+ )
+}
+
+fn write_response(writer: &mut impl Write, value: &impl Serialize) -> io::Result<()> {
+ struct Body(Vec);
+ impl Write for Body {
+ fn write(&mut self, bytes: &[u8]) -> io::Result {
+ if self.0.len().saturating_add(bytes.len()) > MAX_BODY_BYTES {
+ return Err(io::Error::new(
+ io::ErrorKind::InvalidData,
+ "MEP response exceeds 16 MiB",
+ ));
+ }
+ self.0.extend_from_slice(bytes);
+ Ok(bytes.len())
+ }
+ fn flush(&mut self) -> io::Result<()> {
+ Ok(())
+ }
+ }
+ let mut body = Body(Vec::new());
+ serde_json::to_writer(&mut body, value).map_err(io::Error::other)?;
+ write_frame(writer, &body.0)
+}
+
+/// Serve a fresh SDK session, with protocol frames as the only output.
+///
+/// Invalid JSON or request envelopes receive JSON-RPC errors without entering
+/// the compiler. A damaged frame is fatal because its next boundary is unknown.
+/// Notifications receive no response; exit stops processing further messages.
+/// Every invocation owns an independent native SDK session.
+pub fn serve(reader: &mut impl BufRead, writer: &mut impl Write) -> io::Result<()> {
+ let extension = NativeExtension::frontend_backend(PythonExtension)
+ .map_err(|error| io::Error::other(error.to_string()))?;
+ let session = extension.open_protocol();
+ while let Some(body) = read_frame(reader)? {
+ // Borrow the raw document to validate syntax without materializing
+ // opaque parameters or giving serde-private-looking keys meaning.
+ if serde_json::from_slice::<&RawValue>(&body).is_err() {
+ write_error(writer, error_codes::PARSE_ERROR, "Invalid JSON")?;
+ continue;
+ }
+ let head = match serde_json::from_slice::(&body) {
+ Ok(head) if head.jsonrpc == "2.0" => head,
+ _ => {
+ write_error(
+ writer,
+ error_codes::INVALID_REQUEST,
+ "Invalid JSON-RPC request",
+ )?;
+ continue;
+ }
+ };
+ if head.id.is_none() {
+ if head.method == methods::EXIT {
+ break;
+ }
+ // Initialized acknowledges a handshake already held by the SDK.
+ // Unknown or unadvertised notifications do no work.
+ continue;
+ }
+ let request = match serde_json::from_slice::(&body) {
+ Ok(request) => request,
+ Err(_) => {
+ write_error(writer, error_codes::INVALID_REQUEST, "Invalid MEP request")?;
+ continue;
+ }
+ };
+ write_response(writer, &session.handle(request))?;
+ }
+ Ok(())
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ #[test]
+ fn oversized_response_is_refused_before_any_frame_is_emitted() {
+ let mut output = Vec::new();
+ let value = "x".repeat(MAX_BODY_BYTES);
+ assert!(write_response(&mut output, &value).is_err());
+ assert!(output.is_empty());
+ }
+}
diff --git a/rust/crates/morphir-python-extension/tests/framing.rs b/rust/crates/morphir-python-extension/tests/framing.rs
index 7cdc4b16..619ff1e2 100644
--- a/rust/crates/morphir-python-extension/tests/framing.rs
+++ b/rust/crates/morphir-python-extension/tests/framing.rs
@@ -60,3 +60,16 @@ fn content_length_is_case_insensitive_and_uses_bytes() {
write_frame(&mut wire, "λ".as_bytes()).unwrap();
assert_eq!(wire, "Content-Length: 2\r\n\r\nλ".as_bytes());
}
+
+#[test]
+fn a_complete_header_at_the_limit_is_accepted() {
+ let prefix = "Content-Length: 2\r\nOther: ";
+ let suffix = "\r\n\r\n";
+ let header = format!(
+ "{prefix}{}{suffix}",
+ "x".repeat(MAX_HEADER_BYTES - prefix.len() - suffix.len())
+ );
+ assert_eq!(header.len(), MAX_HEADER_BYTES);
+ let mut input = Cursor::new(format!("{header}{{}}"));
+ assert_eq!(read_frame(&mut input).unwrap().unwrap(), b"{}");
+}
diff --git a/rust/crates/morphir-python-extension/tests/session.rs b/rust/crates/morphir-python-extension/tests/session.rs
new file mode 100644
index 00000000..73d2fd60
--- /dev/null
+++ b/rust/crates/morphir-python-extension/tests/session.rs
@@ -0,0 +1,109 @@
+//! A real process and in-memory streams share the same SDK lifecycle.
+use morphir_python_extension::{
+ framing::{read_frame, write_frame},
+ serve,
+};
+use serde_json::{Value, json};
+use std::io::{Cursor, Write};
+use std::process::{Command, Stdio};
+
+fn wire(messages: &[Value]) -> Vec {
+ let mut bytes = Vec::new();
+ for message in messages {
+ write_frame(&mut bytes, &serde_json::to_vec(message).unwrap()).unwrap();
+ }
+ bytes
+}
+fn responses(bytes: Vec) -> Vec {
+ let mut input = Cursor::new(bytes);
+ let mut responses = Vec::new();
+ while let Some(body) = read_frame(&mut input).unwrap() {
+ responses.push(serde_json::from_slice(&body).unwrap());
+ }
+ responses
+}
+fn initialize() -> Value {
+ json!({"jsonrpc":"2.0","id":1,"method":"morphir.initialize","params":{"protocolVersions":["0.1"],"host":{"name":"test","version":"1"}}})
+}
+
+#[test]
+fn sessions_preserve_lifecycle_ids_and_notification_silence() {
+ let messages = [
+ json!({"jsonrpc":"2.0","id":0,"method":"morphir.extension.info","params":{}}),
+ initialize(),
+ json!({"jsonrpc":"2.0","method":"morphir.initialized"}),
+ json!({"jsonrpc":"2.0","method":"unknown.notification","params":{"text":"λ"}}),
+ json!({"jsonrpc":"2.0","id":2,"method":"morphir.ping","params":{}}),
+ json!({"jsonrpc":"2.0","id":3,"method":"morphir.shutdown","params":{}}),
+ json!({"jsonrpc":"2.0","id":4,"method":"morphir.extension.info","params":{}}),
+ json!({"jsonrpc":"2.0","method":"morphir.exit"}),
+ json!({"jsonrpc":"2.0","id":5,"method":"morphir.ping","params":{}}),
+ ];
+ let mut output = Vec::new();
+ serve(&mut Cursor::new(wire(&messages)), &mut output).unwrap();
+ let replies = responses(output);
+ assert_eq!(replies.len(), 5);
+ assert_eq!(
+ replies
+ .iter()
+ .map(|value| value["id"].as_u64().unwrap())
+ .collect::>(),
+ [0, 1, 2, 3, 4]
+ );
+ assert_eq!(replies[0]["error"]["code"], -32014);
+ assert_eq!(replies[1]["result"]["protocolVersion"], "0.1");
+ assert_eq!(replies[2]["result"]["ok"], true);
+ assert_eq!(replies[4]["error"]["code"], -32014);
+}
+
+#[test]
+fn invalid_json_is_refused_and_the_next_framed_request_can_recover() {
+ let mut input = Vec::new();
+ write_frame(&mut input, b"{invalid").unwrap();
+ input.extend(wire(&[
+ json!({"jsonrpc":"2.0","id":7,"method":"morphir.ping","params":{}}),
+ ]));
+ let mut output = Vec::new();
+ serve(&mut Cursor::new(input), &mut output).unwrap();
+ let replies = responses(output);
+ assert_eq!(replies.len(), 2);
+ assert_eq!(replies[0]["error"]["code"], -32700);
+ assert_eq!(replies[0]["id"], Value::Null);
+ assert_eq!(replies[1]["id"], 7);
+}
+
+#[test]
+fn malformed_frames_fail_without_writing_non_protocol_output() {
+ let mut output = Vec::new();
+ assert!(
+ serve(
+ &mut Cursor::new(b"Content-Length: 999999999\r\n\r\n"),
+ &mut output
+ )
+ .is_err()
+ );
+ assert!(output.is_empty());
+}
+
+#[test]
+fn standalone_process_needs_no_python_or_rust_environment() {
+ let mut child = Command::new(env!("CARGO_BIN_EXE_morphir-python-extension"))
+ .arg("--stdio")
+ .env_clear()
+ .stdin(Stdio::piped())
+ .stdout(Stdio::piped())
+ .stderr(Stdio::piped())
+ .spawn()
+ .unwrap();
+ let input = wire(&[
+ initialize(),
+ json!({"jsonrpc":"2.0","method":"morphir.exit"}),
+ ]);
+ child.stdin.take().unwrap().write_all(&input).unwrap();
+ let result = child.wait_with_output().unwrap();
+ assert!(result.status.success(), "{:?}", result.stderr);
+ assert!(result.stderr.is_empty());
+ let replies = responses(result.stdout);
+ assert_eq!(replies.len(), 1);
+ assert_eq!(replies[0]["result"]["extension"]["id"], "morphir-python");
+}
From 0df65849fb5e35343da3a98c20d6ecb86bd29504 Mon Sep 17 00:00:00 2001
From: Damian Reeves <957246+DamianReeves@users.noreply.github.com>
Date: Wed, 28 Oct 2026 05:08:06 -0500
Subject: [PATCH 05/26] fix(binding): enforce portable model admission
---
.../add-process-mep-extension/tasks.md | 6 +-
rust/crates/morphir-python-binding/README.md | 29 +++++++-
.../src/model_transport.rs | 37 +++++++++++
.../tests/model_transport.rs | 66 ++++++++++++++++++-
4 files changed, 133 insertions(+), 5 deletions(-)
diff --git a/openspec/changes/add-process-mep-extension/tasks.md b/openspec/changes/add-process-mep-extension/tasks.md
index 2ace45e8..9d7eea6c 100644
--- a/openspec/changes/add-process-mep-extension/tasks.md
+++ b/openspec/changes/add-process-mep-extension/tasks.md
@@ -10,9 +10,9 @@ This change is being applied. The published SDK prerequisite is tracked in morph
## 2. Adapt supported model envelopes (morphir-5hxy.2)
-- [ ] 2.1 Implement bounded canonical JSON model-envelope admission using SDK helpers; verify base64, byte length, hash, critical fields, semantic IR version and size-limit negative cases.
-- [ ] 2.2 Adapt compileModel/generateModel for admitted JSON-profile v3/v4 Library tuples through the shared compiler; verify compile/generate/recompile semantics and diagnostic parity with legacy operations.
-- [ ] 2.3 Refuse unadvertised Ion profiles, unsupported distributions and guarantees without lossy fallback; test these refusals and document the initial support matrix.
+- [x] 2.1 Implement bounded canonical JSON model-envelope admission using SDK helpers; verify base64, byte length, hash, critical fields, semantic IR version and size-limit negative cases.
+- [x] 2.2 Adapt compileModel/generateModel for admitted JSON-profile v3/v4 Library tuples through the shared compiler; verify compile/generate/recompile semantics and diagnostic parity with legacy operations.
+- [x] 2.3 Refuse unadvertised Ion profiles, unsupported distributions and guarantees without lossy fallback; test these refusals and document the initial support matrix.
## 3. Serve native stdio sessions (morphir-5hxy.3)
diff --git a/rust/crates/morphir-python-binding/README.md b/rust/crates/morphir-python-binding/README.md
index 1b67cebc..e52d8f7f 100644
--- a/rust/crates/morphir-python-binding/README.md
+++ b/rust/crates/morphir-python-binding/README.md
@@ -11,7 +11,8 @@ This is an initial implementation of a defined subset of Python 3.12 and later
and Morphir IR v3 and v4. It supports ADT declarations, fixed tuples and annotated pure conditional
functions, typed calls and unary lambdas. It does not implement all Python syntax or all Morphir IR nodes, and
does not claim full Morphir Compatibility Kit conformance for this extension.
-The crate depends on the published `morphir-core` and `morphir-extension-sdk` crates only.
+Morphir dependencies use published `morphir-core`, `morphir-json` and
+`morphir-extension-sdk` crates.
It does not link CPython; the separate `morphir-python-native` crate in this workspace
adapts it for the `morphir-python` Python package.
@@ -23,6 +24,32 @@ and releases it. Bundles up to `extension/python/v0.4.0` remain available from t
function support, and Morphir CLI `0.4.0-beta.1` and later cannot compile with it. Use the
[extension guide](../../../docs/python-extension.md) to build and install the bundle.
+## Development model adapter
+
+The direct Rust `Frontend::compile_model` and `Backend::generate_model` methods
+share the legacy compiler. Model transport is not yet advertised in MEP discovery;
+protocol qualification and the published SDK prerequisite remain in progress in
+`add-process-mep-extension`.
+
+| Boundary | Initial support |
+| --- | --- |
+| Operation | `compileModel`, `generateModel` |
+| Profile and codec | `morphir-ir`, `morphir-json` |
+| Envelope version | `0.1.0-draft.1` |
+| Semantic IR versions | `3.0.0`, draft `4.0.0` |
+| Distribution | `Library`, within the documented Python subset |
+| Evidence | JSON only; no preservation or typing guarantees established |
+| Encoded / decoded bytes | 12 MiB / 8 MiB, aggregated with dependencies and evidence |
+| Parse budget | 128 levels, 100,000 aggregate value nodes |
+
+Admission checks canonical padded base64, exact decoded length and SHA-256,
+semantic IR version, critical fields and resource bounds before generation.
+Compilation output has the same decoded-byte bound. Model compilation preserves
+legacy diagnostics and refuses dependency compilation. Generation refuses `Specs`
+and `Application` distributions with `PY004`. Ion payloads and evidence, other
+profiles, unsupported tuples and required guarantees are refused without
+conversion. Limits negotiated by a session can only narrow these static bounds.
+
## A complete model
Save this as `models.py`. It is valid Python and supported frontend input:
diff --git a/rust/crates/morphir-python-binding/src/model_transport.rs b/rust/crates/morphir-python-binding/src/model_transport.rs
index a349eae5..0243b950 100644
--- a/rust/crates/morphir-python-binding/src/model_transport.rs
+++ b/rust/crates/morphir-python-binding/src/model_transport.rs
@@ -41,10 +41,29 @@ fn admit(method: &str, value: &serde_json::Value) -> Result<()> {
validate_model_invocation(&admitted, &capabilities, method, value).map_err(invalid)
}
+// Native SDK builds can inspect Ion evidence, but WASM cannot. Keep this
+// adapter's accepted subset identical across runtimes, including sidecars.
+fn require_json(model: &MorphirModel) -> Result<()> {
+ if model.payload.codec != ModelCodec::MorphirJson
+ || model
+ .evidence
+ .iter()
+ .any(|item| item.evidence.codec != ModelCodec::MorphirJson)
+ {
+ return Err(invalid(
+ "Python model transport is JSON-only, including evidence",
+ ));
+ }
+ Ok(())
+}
+
pub(crate) fn compile(
extension: &PythonExtension,
request: CompileModelRequest,
) -> Result {
+ for dependency in &request.dependencies {
+ require_json(&dependency.model)?;
+ }
admit(
"morphir.frontend.compileModel",
&serde_json::to_value(&request)?,
@@ -95,6 +114,7 @@ pub(crate) fn generate(
extension: &PythonExtension,
request: GenerateModelRequest,
) -> Result {
+ require_json(&request.model)?;
admit(
"morphir.backend.generateModel",
&serde_json::to_value(&request)?,
@@ -176,3 +196,20 @@ impl morphir_json::ReadBudget for JsonBudget {
}
}
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn output_json_stops_at_its_decoded_budget() {
+ let value = serde_json::json!("λ");
+ assert_eq!(bounded_json(&value, 4).unwrap(), "\"λ\"".as_bytes());
+ assert!(
+ bounded_json(&value, 3)
+ .unwrap_err()
+ .to_string()
+ .contains("decoded limit")
+ );
+ }
+}
diff --git a/rust/crates/morphir-python-binding/tests/model_transport.rs b/rust/crates/morphir-python-binding/tests/model_transport.rs
index b9066d2d..9ee74cb4 100644
--- a/rust/crates/morphir-python-binding/tests/model_transport.rs
+++ b/rust/crates/morphir-python-binding/tests/model_transport.rs
@@ -212,7 +212,8 @@ fn aggregate_nodes_and_dependency_support_are_bounded() {
request.model.payload =
ModelPayload::from_bytes(ModelCodec::MorphirJson, &serde_json::to_vec(&ir).unwrap())
.unwrap();
- assert!(PythonExtension.generate_model(request).is_err());
+ let error = PythonExtension.generate_model(request).unwrap_err();
+ assert!(error.to_string().contains("node_limit"), "{error}");
let mut request = a_request("4.0.0");
request.dependencies.push(CompileModelDependency {
@@ -226,3 +227,66 @@ fn aggregate_nodes_and_dependency_support_are_bounded() {
};
assert_eq!(diagnostics[0].code.as_deref(), Some("PY001"));
}
+
+#[test]
+fn ion_evidence_is_refused_uniformly_across_native_and_wasm() {
+ let mut request = generation(a_model("4.0.0"));
+ request.model.evidence.push(ModelGuaranteeEvidence {
+ guarantee: "urn:example:optional".into(),
+ model_id: request.model.model_id.clone(),
+ model_revision: request.model.revision.clone(),
+ context_digest: request.context_digest.clone(),
+ scope: format!("model:{}", request.model.model_id),
+ producer: "fixture".into(),
+ evidence: ModelPayload::from_bytes(ModelCodec::IonBinary, &[0xe0, 1, 0, 0xea, 0x21, 1])
+ .unwrap(),
+ });
+ let error = PythonExtension.generate_model(request).unwrap_err();
+ assert!(error.to_string().contains("JSON-only"), "{error}");
+}
+
+#[test]
+fn unsupported_distribution_forms_are_explicit_language_refusals() {
+ for distribution in [
+ json!({"Specs":{"packageName":"acme","dependencies":{},"spec":{"modules":{}}}}),
+ json!({"Application":{"packageName":"acme","dependencies":{},"def":{"modules":{}},"entryPoints":{}}}),
+ ] {
+ let ir = json!({"formatVersion":4,"distribution":distribution});
+ // Prove these are complete valid IR, rather than malformed enum inputs.
+ serde_json::from_value::(ir.clone()).unwrap();
+ let mut request = generation(a_model("4.0.0"));
+ request.model.payload =
+ ModelPayload::from_bytes(ModelCodec::MorphirJson, &serde_json::to_vec(&ir).unwrap())
+ .unwrap();
+ let result = PythonExtension.generate_model(request).unwrap();
+ assert!(!result.success);
+ assert!(result.artifacts.is_empty());
+ assert_eq!(result.diagnostics[0].code.as_deref(), Some("PY004"));
+ assert!(result.diagnostics[0].message.contains("Only Library"));
+ }
+}
+
+#[test]
+fn semantic_versions_and_payload_resource_limits_are_checked_before_generation() {
+ let payloads = [
+ (
+ br#"{"formatVersion":3,"distribution":{}}"#.to_vec(),
+ "unsupported_ir_version",
+ ),
+ (
+ format!("{}0{}", "[".repeat(128), "]".repeat(128)).into_bytes(),
+ "depth_limit",
+ ),
+ (vec![b' '; 8 * 1024 * 1024 + 1], "decoded_limit"),
+ ];
+ for (bytes, reason) in payloads {
+ let mut request = generation(a_model("4.0.0"));
+ request.model.payload = ModelPayload::from_bytes(ModelCodec::MorphirJson, &bytes).unwrap();
+ let error = PythonExtension.generate_model(request).unwrap_err();
+ assert!(error.to_string().contains(reason), "{reason}: {error}");
+ }
+ let mut request = generation(a_model("4.0.0"));
+ request.model.payload.content = "A".repeat(12 * 1024 * 1024 + 4);
+ let error = PythonExtension.generate_model(request).unwrap_err();
+ assert!(error.to_string().contains("encoded_limit"), "{error}");
+}
From 30a216aed0cb60e55343601019769a7baf8f9583 Mon Sep 17 00:00:00 2001
From: Damian Reeves <957246+DamianReeves@users.noreply.github.com>
Date: Wed, 28 Oct 2026 05:16:40 -0500
Subject: [PATCH 06/26] build(extension): stage canonical process bundles
---
.../add-process-mep-extension/design.html | 4 +-
.../add-process-mep-extension/tasks.md | 2 +-
.../crates/morphir-python-extension/README.md | 32 ++++
.../morphir-python-extension/src/main.rs | 15 +-
.../morphir-python-extension/tests/session.rs | 7 +-
rust/extension.toml | 2 +
scripts/extension_bundle.py | 44 +++--
scripts/process_extension.py | 178 ++++++++++++++++++
tests/unit/test_extension_bundle.py | 45 +++++
tests/unit/test_process_extension.py | 114 +++++++++++
10 files changed, 419 insertions(+), 24 deletions(-)
create mode 100644 scripts/process_extension.py
create mode 100644 tests/unit/test_process_extension.py
diff --git a/openspec/changes/add-process-mep-extension/design.html b/openspec/changes/add-process-mep-extension/design.html
index 9d44561e..4bed3902 100644
--- a/openspec/changes/add-process-mep-extension/design.html
+++ b/openspec/changes/add-process-mep-extension/design.html
@@ -26,8 +26,8 @@ add-process-mep-extension
Context
See proposal.md for motivation and the two spec deltas for requirements. PythonExtension already implements native Rust frontend/backend traits. The published SDK 0.3.0 exposes NativeExtension::frontend_backend, open_protocol, model methods and admission helpers. Its wire request type still uses a numeric u64 ID, while the canonical contract admits numeric and string IDs; adopting helpers alone is not full draft conformance. The Python implementation does not opt into model transport and its manifest declares MEP 0.1 only. finos/morphir c552dac801f23f2b2b566d29e08780723498ce36 defines exact MEP 0.2.0-draft.1 and model envelope 0.1.0-draft.1. These versions are independent of SDK 0.3.0, compiler 0.4.2, claims format and IR version.
Goals and non-goals
Add a native Rust process guest and current-draft model adapters while retaining WASM and legacy behavior. Keep language semantics shared. Initial model tuples use morphir-ir/morphir-json and supported Library IR 3.0.0 and draft 4.0.0. Rich Ion profiles, unsupported distribution kinds, additional Python syntax, cancellation/progress and performance claims are not introduced. A Python CLI stdio wrapper can follow separately; the selected CLI installation remains mise/pipx.
Decisions
RNT-1. Add a thin native Rust guest
Add a process guest crate/entry point, depending on the maintained compiler and published SDK, with no PyO3 or CPython requirement. The proposed command is:
morphir-python-extension --stdio
Reuse the SDK's session endpoint and dispatch. Keep model adaptation in the shared PythonExtension so native, WASM and process paths agree. Put stdio effects in the guest layer. A Python CLI wrapper would reuse the engine but retain an interpreter/launcher dependency; a frozen Python CLI would bundle an unnecessary runtime for this guest.
Boundary Current release Proposed change MEP Legacy 0.1 Retain 0.1 and add exact 0.2.0-draft.1 Model envelope Not advertised 0.1.0-draft.1, admitted JSON profile Extension runtime WASM WASM and native process SDK 0.3.0, numeric-only wire IDs Qualified published pin covering the canonical wire contract Semantic IR Supported v3 and draft v4 Library models Same semantic scope, explicit admitted tuples
SDK-1. Resolve concrete SDK wire gaps before claiming conformance
Pin a published SDK that supports canonical numeric and string request IDs and the required draft behavior. SDK 0.3.0 currently has numeric-only ExtensionRequest.id, so link and qualify an upstream SDK update if no suitable release exists. Do not vendor a sibling checkout or silently coerce wire IDs. Both WASM and process must pass the same ID-preservation vectors before the new draft is advertised. The concrete prerequisite is request ID support, issue 316, tracked as morphir-4wrx. A candidate SDK 0.4.0 is under qualification and is not yet published. Related upstream work is MEP protocol type generation, issue 241; its existence is not evidence that the wire gap is fixed.
MEP-1. Negotiate the new draft explicitly
Retain legacy 0.1 operations and exact 0.2.0-draft.1 offers. Opt in only after supported model methods and executable conformance pass. Reuse SDK model admission and session rules; method constants alone are not support. Never enable model methods under legacy negotiation. Keep optional flags false until implemented.
MOD-1. Start with the explicit JSON model profile
Implement compile_model/generate_model over the existing compile/generate core. Admit complete operation/profile/codec/envelope/IR tuples and their bilateral limits. The initial profile is morphir-ir/morphir-json with envelope 0.1.0-draft.1. Use the canonical base64 payload, exact byte length/hash and critical-field rules. After SDK admission, materialize JSON through published morphir-json 0.2.0 with finite depth/node/string budgets, preserving numeric lexemes and literal private-looking keys. Bound outgoing JSON before extending payload buffers. Reject unadvertised Ion profiles, guarantees and unsupported semantic forms. This achieves bounded new-draft support without implying rich Ion preservation or silently converting rich input.
IO-1. Treat framing and lifetime as a bounded server boundary
Use Content-Length JSON-RPC framing, byte-based lengths, an 8 KiB header ceiling and an initial 16 MiB complete-body ceiling. Advertised decoded/encoded model limits must fit those outer bounds, including envelope overhead. Enforce local bounds even if the peer advertises more. Check partial input, duplicate lengths, integer overflow, invalid UTF-8/JSON and EOF before unbounded allocation. Reserve stdout for frames and flush responses. Follow initialization, capability locks, shutdown and exit; do not respond to notifications. Host supervision owns deadlines and descendant cleanup.
RLS-1. Qualify dual-runtime distribution
Retain the portable WASM guest and publish native process executables for the advertised targets, with --stdio locked as an argument. Use a canonical host-readable descriptor with runtime/platform/mode/digest/claims bound per artifact. Validate the staged descriptor with pinned host tooling; do not assume that either the existing WASM-only packager or native-only bundle command can package a mixed release unchanged. Preserve historical assets. Release exact native/CLI package pairs if shared compiler changes affect them.
HST-1. Qualify hosts independently
The Rust host provides ProcessLaunch/ProcessChannel and verified-byte staging. It clears the child environment, favoring a directly executable Rust guest. Qualify a pinned conforming Rust host and test each documented MoonBit route independently. Use a pinned source build when a released host lacks the new draft, and state that prerequisite. If a route needs upstream work, record that dependency and label it unsupported until it passes. Existing MoonBit process wrappers demonstrate framing, not Python extension activation.
Risks and trade-offs
Risk Mitigation Draft, SDK and runtime support are confused Exact independent version table, live negotiation and candidate-bound conformance before claims. Model content or guarantees are lost Admit only explicit JSON tuples initially. Refuse unsupported profiles, critical fields and guarantees; no silent fallback. Malformed input exhausts resources Bound headers, wire bodies and decoded bytes before allocation; test boundary and partial cases. Process mode is mistaken for a sandbox Document ordinary OS privileges and the host's explicit supervision, distinct from WASM isolation. Packaging or host support lags Pin descriptor readers and host builds, qualify installed bytes, record upstream blockers and make only verified support claims. Native execution is assumed faster Measure startup and representative compile/generate workloads before making a performance claim.
Decisions
RNT-1. Add a thin native Rust guest
Add a process guest crate/entry point, depending on the maintained compiler and published SDK, with no PyO3 or CPython requirement. The proposed command is:
morphir-python-extension --stdio
No-argument launches also select stdio, matching the canonical native bundler and installed process launcher. Reuse the SDK's session endpoint and dispatch. Keep model adaptation in the shared PythonExtension so native, WASM and process paths agree. Put stdio effects in the guest layer. A Python CLI wrapper would reuse the engine but retain an interpreter/launcher dependency; a frozen Python CLI would bundle an unnecessary runtime for this guest.
Boundary Current release Proposed change MEP Legacy 0.1 Retain 0.1 and add exact 0.2.0-draft.1 Model envelope Not advertised 0.1.0-draft.1, admitted JSON profile Extension runtime WASM WASM and native process SDK 0.3.0, numeric-only wire IDs Qualified published pin covering the canonical wire contract Semantic IR Supported v3 and draft v4 Library models Same semantic scope, explicit admitted tuples
SDK-1. Resolve concrete SDK wire gaps before claiming conformance
Pin a published SDK that supports canonical numeric and string request IDs and the required draft behavior. SDK 0.3.0 currently has numeric-only ExtensionRequest.id, so link and qualify an upstream SDK update if no suitable release exists. Do not vendor a sibling checkout or silently coerce wire IDs. Both WASM and process must pass the same ID-preservation vectors before the new draft is advertised. The concrete prerequisite is request ID support, issue 316, tracked as morphir-4wrx. SDK 0.4.0 is merged in PR 317 as d51daa8aea7ae09ae295ff254d72b27770c64104. All CI and the committed-source publication dry run pass. Crate-tag publication approval is pending; the SDK is not yet published. Related upstream work is MEP protocol type generation, issue 241; its existence is not evidence that the wire gap is fixed.
MEP-1. Negotiate the new draft explicitly
Retain legacy 0.1 operations and exact 0.2.0-draft.1 offers. Opt in only after supported model methods and executable conformance pass. Reuse SDK model admission and session rules; method constants alone are not support. Never enable model methods under legacy negotiation. Keep optional flags false until implemented.
MOD-1. Start with the explicit JSON model profile
Implement compile_model/generate_model over the existing compile/generate core. Admit complete operation/profile/codec/envelope/IR tuples and their bilateral limits. The initial profile is morphir-ir/morphir-json with envelope 0.1.0-draft.1. Use the canonical base64 payload, exact byte length/hash and critical-field rules. After SDK admission, materialize JSON through published morphir-json 0.2.0 with finite depth/node/string budgets, preserving numeric lexemes and literal private-looking keys. Bound outgoing JSON before extending payload buffers. Reject unadvertised Ion profiles, guarantees and unsupported semantic forms. This achieves bounded new-draft support without implying rich Ion preservation or silently converting rich input.
IO-1. Treat framing and lifetime as a bounded server boundary
Use Content-Length JSON-RPC framing, byte-based lengths, an 8 KiB header ceiling and an initial 16 MiB complete-body ceiling. Advertised decoded/encoded model limits must fit those outer bounds, including envelope overhead. Enforce local bounds even if the peer advertises more. Check partial input, duplicate lengths, integer overflow, invalid UTF-8/JSON and EOF before unbounded allocation. Reserve stdout for frames and flush responses. Follow initialization, capability locks, shutdown and exit; do not respond to notifications. Host supervision owns deadlines and descendant cleanup.
RLS-1. Qualify dual-runtime distribution
Retain the portable WASM guest and publish native process executables for the advertised targets, with --stdio locked as an argument. Use a canonical host-readable descriptor with runtime/platform/mode/digest/claims bound per artifact. Validate the staged descriptor with pinned host tooling; do not assume that either the existing WASM-only packager or native-only bundle command can package a mixed release unchanged. Preserve historical assets. Release exact native/CLI package pairs if shared compiler changes affect them.
HST-1. Qualify hosts independently
The Rust host provides ProcessLaunch/ProcessChannel and verified-byte staging. It clears the child environment, favoring a directly executable Rust guest. Qualify a pinned conforming Rust host and test each documented MoonBit route independently. Use a pinned source build when a released host lacks the new draft, and state that prerequisite. If a route needs upstream work, record that dependency and label it unsupported until it passes. Existing MoonBit process wrappers demonstrate framing, not Python extension activation.
Risks and trade-offs
Risk Mitigation Draft, SDK and runtime support are confused Exact independent version table, live negotiation and candidate-bound conformance before claims. Model content or guarantees are lost Admit only explicit JSON tuples initially. Refuse unsupported profiles, critical fields and guarantees; no silent fallback. Malformed input exhausts resources Bound headers, wire bodies and decoded bytes before allocation; test boundary and partial cases. Process mode is mistaken for a sandbox Document ordinary OS privileges and the host's explicit supervision, distinct from WASM isolation. Packaging or host support lags Pin descriptor readers and host builds, qualify installed bytes, record upstream blockers and make only verified support claims. Native execution is assumed faster Measure startup and representative compile/generate workloads before making a performance claim.
The pinned executable MCK currently accepts only moonbit-model-bool-library-v1, with mandatory rich Ion semantics and MoonBit artifact inventories. Python cannot qualify against that profile. Approval is pending for a separate upstream Python JSON profile; no existing MoonBit check will be relaxed and no unit-test result substitutes for executable qualification.
Migration and rollout
- Add negotiation and bounded JSON model adapters with tests and compatibility documentation.
- Add the Rust process server with framing and lifecycle tests; preserve the existing WASM path.
- Build and qualify runtime artifacts and descriptors with matching claims and source receipts.
- Run pinned MEP guest conformance and real host installation/operation cases, including legacy hosts and explicit refusals.
- After maintainer approval, publish a new extension version and document runtime selection. Existing users can stay on the prior WASM release; never replace its bytes.
Implementation is underway on the process delivery branch. Protocol discovery remains legacy-only until the published SDK prerequisite and candidate-bound conformance pass. All five task groups remain incomplete.
Repository evidence and references
- rust/crates/morphir-python-binding/src/lib.rs: shared PythonExtension traits and truthful current capabilities.
- rust/crates/morphir-python-native/src/lib.rs: current private two-function JSON bridge; it is not a full MEP transport.
- Published morphir-extension-sdk 0.3.0: native.rs, traits.rs and model admission helpers.
- finos/morphir-rust 26d5e707e5ec94d16421db6380edbdc865d85c04: process launch/channel and stdio frontend fixture. Fixture framing must not be copied as a hardened server.
- finos/morphir-moonbit bd3adc52e6b8f6368fdaee50f76535799732f8d6: portable MEP embedding wrapper. No claim of Python host compatibility follows.
- Canonical MEP contract and model transport rules.
Related work: morphir-bj9r owns the selected CLI installation route; morphir-ukvn.5.98 tracks the earlier MoonBit extension qualification context.