From 013e0e603d9c47e780aec12642111903c2eb5730 Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 02:00:37 -0500 Subject: [PATCH 1/9] docs(openspec): propose align-ir-v4-with-upstream Plan the realignment of the Python v4 model and draft codec with upstream finos/morphir v4 at the pinned commit (morphir-ukvn.8.14) and an explicit, reported drop of per-input value attributes (morphir-ukvn.8.15). --- .../align-ir-v4-with-upstream/.openspec.yaml | 2 + .../align-ir-v4-with-upstream/design.html | 134 ++++++++++++++++++ .../align-ir-v4-with-upstream/proposal.md | 40 ++++++ .../specs/ir/migrate/spec.md | 51 +++++++ .../specs/ir/v4/spec.md | 83 +++++++++++ .../specs/tools/ir-interchange/spec.md | 26 ++++ .../align-ir-v4-with-upstream/tasks.md | 48 +++++++ 7 files changed, 384 insertions(+) create mode 100644 openspec/changes/align-ir-v4-with-upstream/.openspec.yaml create mode 100644 openspec/changes/align-ir-v4-with-upstream/design.html create mode 100644 openspec/changes/align-ir-v4-with-upstream/proposal.md create mode 100644 openspec/changes/align-ir-v4-with-upstream/specs/ir/migrate/spec.md create mode 100644 openspec/changes/align-ir-v4-with-upstream/specs/ir/v4/spec.md create mode 100644 openspec/changes/align-ir-v4-with-upstream/specs/tools/ir-interchange/spec.md create mode 100644 openspec/changes/align-ir-v4-with-upstream/tasks.md diff --git a/openspec/changes/align-ir-v4-with-upstream/.openspec.yaml b/openspec/changes/align-ir-v4-with-upstream/.openspec.yaml new file mode 100644 index 00000000..dda53b8a --- /dev/null +++ b/openspec/changes/align-ir-v4-with-upstream/.openspec.yaml @@ -0,0 +1,2 @@ +schema: html-design +created: 2026-10-27 diff --git a/openspec/changes/align-ir-v4-with-upstream/design.html b/openspec/changes/align-ir-v4-with-upstream/design.html new file mode 100644 index 00000000..73e2ba95 --- /dev/null +++ b/openspec/changes/align-ir-v4-with-upstream/design.html @@ -0,0 +1,134 @@ + + + + + +align-ir-v4-with-upstream: design + + + + + + + + +
+

align-ir-v4-with-upstream: design

+

OpenSpec change · Bead morphir-ukvn.8.14 (group 1: morphir-ukvn.8.15) · 2026-10-27 · proposed

+ + + +

Context

+

See proposal.md for the motivation. The draft codec (morphir.ir.v4.codec) follows the finos/morphir v4 schema +at FINOS_MORPHIR_PIN = "efa36060", which is still finos/morphir main on 2026-10-27. It writes and reads only the +expanded spelling ({"Kind": {members}}); compact spellings are the later re-sync in morphir-ukvn.8.16.

+

The sources for the new shapes, in order of authority:

+ + + + + + + + + +
ConstructSource at the pin
Definition bodies, native hints, external bindingsschema ValueDefinition, NativeBody, ExternalBody, IncompleteBody; kit definitions-0007 to -0009, -0016, -0025, -0030
Incompleteness and hole reasonsschema Incompleteness, HoleReason; kit definitions-0014, -0015, -0024, -0026
Annotationsschema Annotation; kit definitions-0020 to -0023; morphir-rust ir/v4/annotation.rs (Compact and Structured)
Uniondocs/spec/draft/types.md (not yet in the schema); reference implementation finos/morphir-moonbit #155 (morphir-ukvn.7.9.1)
Per-input attributesreference model inputs: List(#(Name, Type)); kit definitions-0027 ("An input type is a bare type"); morphir-rust migration/classic_to_v4.rs, which drops v3 input attributes
+

The in-memory v3 to v4 to v3 round trip is a requirement (ir/migrate, "Shared subset round trips"), and the +morphir-elm reference model has 8 inputs whose resolved type differs from the declared type (alias expansion and +type-variable instances, for example float-alias resolved to Float).

+ +

Goals and non-goals

+

Goals: every construct in the pinned v4 schema and the Union draft has a Python model and an expanded +spelling; every new construct has an explicit downgrade failure; the reference model can reach v4 JSON through an +explicit choice; nothing that works today stops working, except the typed Hole.reason.

+

Non-goals: compact spellings, legacy one-release spellings and the legacy_spelling warning +(morphir-ukvn.8.16); the 4.1.0 linked annotations; removing the deprecated constructs; type checking +of unions.

+ +

Decisions

+ +

INP-1. Per-input attributes drop only on request

+

Upstream v4 has no slot for them, and adding one is not a spelling to "find": the reference model, the kit and the +reference implementation all agree that an input is a bare type. So no upstream proposal is needed. The Python side +gets an explicit drop: drop_input_attributes(lib) for any v4 Library, a drop_input_attributes=True +keyword on v3_to_v4, and ir convert --drop-input-attributes. An input loses information when its +attributes are neither empty nor exactly ValueAttributes(inferred_type=declared); only those inputs report +input_attributes_dropped. The drop walks let-bound definitions too.

+

Alternatives: drop always, as morphir-rust does. Rejected: it breaks the lossless in-memory round trip and the +package's rule that loss is reported. Drop in the codec. Rejected: the codec has no warning channel and stays strict. +Write the attributes into extensions. Rejected: an invented convention no other reader knows.

+ +

BODY-1. Four body classes, with the expression body keeping its name

+

value.Definition stays the expression body, with its (name, attributes, type) inputs, so existing +code and the v3 migration keep working. New frozen classes NativeBody, ExternalBody and +IncompleteBody hold (name, type) inputs as upstream does. ValueDefinitionBody is the union +of the four; module.Definition.values, LetDefinition.definition and LetRecursion.definitions +hold it, because the schema's let bindings reference ValueDefinition.

+

Alternative: one Definition with a body sum type, as morphir-rust does. Rejected: it changes the constructor +of every existing definition.

+ +

HOLE-1. Typed hole reasons, shared by values and incompleteness

+

UnresolvedReference(target), DeletedDuringRefactor(tx_id) and TypeMismatch(expected, found) +live in a new incompleteness module with HoleIncompleteness(reason, partial_body) and +DraftIncompleteness(). Hole.reason becomes HoleReason. The schema gives the hole's +partialBody the type Type, so HoleIncompleteness.partial_body is a type; an incomplete value +keeps its partial expression on IncompleteBody.partial_body, as kit definitions-0025 shows. The +reference model's Draft(notes) has no member in the schema; the pinned schema wins.

+ +

ANN-1. Annotations keep their spelling

+

CompactAnnotation(name, text) and StructuredAnnotation(name, arguments) mirror morphir-rust, because +the kit writes an argument-less structured annotation as an object and a compact one as a string, so the form is +content. The free text is everything after the first colon after #. Arguments are +PositionalArgument(value) or NamedArgument(name, value); a named argument is an object with exactly +name and value, anything else is a positional value. Specifications get an +annotations field with an empty default, placed last so positional construction still works.

+ +

UNI-1. Union is a type node with two or more alternatives

+

type.Union(attributes, alternatives), spelled {"Union": {"attributes"?, "alternatives": [...]}}. +Encode and decode refuse fewer than two alternatives with invalid_json at alternatives. Order, +duplicates and nesting are kept. map_attributes handles it.

+ +

DEP-1. Deprecate, do not remove, the Python-only constructs

+

The Native and External value nodes and TypeConstraints stay, with docstrings that say +they are deprecated and what replaces them. Removing them would break imports from morphir.ir; a follow-up +bead removes them in a later minor release.

+ +

MIG-1. Every new construct fails downgrade with unsupported_v4_downgrade

+

Format 3.0.0 has no home for any of them. Constructs: NativeBody, ExternalBody, +IncompleteBody, IncompleteTypeDefinition, Union, Annotations (on any +specification). The pass still collects every failure.

+ +

Risks and trade-offs

+ + + + + + + + +
RiskMitigation
Typed Hole.reason breaks callers that pass raw JSONDraft API; called out as breaking in the proposal and README
Upstream renames members before v4 settlesAll spellings stay in the codec's tables; morphir-ukvn.8.16 re-syncs them
Union is in the draft spec page but not the pinned schemaFollow the spec page and the morphir-moonbit implementation; the codec is draft
Callers that match on module values miss the new bodiesStrict type checkers report the non-exhaustive match
+ +

Migration plan

+

One release of morphir. No data migration. Rollback is a revert of the PR.

+ +
+ + diff --git a/openspec/changes/align-ir-v4-with-upstream/proposal.md b/openspec/changes/align-ir-v4-with-upstream/proposal.md new file mode 100644 index 00000000..0a7a4dd7 --- /dev/null +++ b/openspec/changes/align-ir-v4-with-upstream/proposal.md @@ -0,0 +1,40 @@ +# Proposal + +## Why + +The Python v4 model predates the upstream v4 profile. It cannot hold the native, external and incomplete definition bodies, incomplete type definitions, annotations, typed hole reasons or `Union` types that upstream v4 spells, so the draft codec refuses valid upstream documents with `unknown_node` or `unknown_member`. In the other direction, the morphir-elm reference model cannot be written as v4 JSON: its typed inputs carry a resolved type that differs from the declared type, and upstream v4 has no slot for per-input attributes, so encoding fails with `no_v4_spelling`. + +## What Changes + +- Add the upstream v4 value-definition bodies `NativeBody`, `ExternalBody` and `IncompleteBody` beside today's expression body (`Definition`). Module values and let bindings hold any of the four bodies. +- Add typed hole reasons (`UnresolvedReference`, `DeletedDuringRefactor`, `TypeMismatch`), the `Incompleteness` kinds `Hole` and `Draft`, and `IncompleteTypeDefinition`. **BREAKING** (draft API): `Hole.reason` is a typed `HoleReason`, not raw JSON. +- Add annotations (compact and structured, with positional and named arguments) to module, type and value specifications. +- Add the draft `Union` type expression (two or more alternatives). +- Extend the draft codec to read and write all of the above in the expanded spelling at the pinned commit, with the kit's refusals (duplicate external platforms, a platform-specific hint without its platform, a `Draft` hole reason, a union with fewer than two alternatives). +- Extend reverse migration so every new construct fails with `unsupported_v4_downgrade` at its location. Format 3.0.0 has no home for any of them. +- Add an explicit, reported drop of per-input value attributes: `drop_input_attributes` on `v3_to_v4`, a standalone function for any v4 Library, and `ir convert --drop-input-attributes`. Each input that loses information reports `input_attributes_dropped`. Without the opt-in, behaviour is unchanged: the in-memory migration stays lossless and encoding fails with `no_v4_spelling`. +- Keep the Python-only `Native` and `External` value nodes and the structured `TypeConstraints`, which upstream v4 cannot spell, and mark them deprecated. Their removal is follow-up work. + +## Capabilities + +### New Capabilities + +- (none) + +### Modified Capabilities + +- `ir/v4`: the model and draft codec cover the upstream definition bodies, incompleteness, annotations and `Union`; the Python-only constructs are deprecated. +- `ir/migrate`: the new v4 constructs fail reverse migration; per-input attributes can be dropped on request, with a warning. +- `tools/ir-interchange`: `ir convert` accepts `--drop-input-attributes` for upgrades. + +## Beads + +- **Tracked by**: `morphir-ukvn.8.14` (realign the Python v4 model with upstream finos/morphir v4) +- **Related**: `morphir-ukvn.8.15` (per-input attribute spelling; task group 1 of this change), `morphir-ukvn.8` (parent epic, IR v3 and v4 in morphir-python), `morphir-ukvn.8.16` (later re-sync of the draft codec when upstream v4 settles, including compact spellings), `morphir-diwy` (finos/morphir: v4 to v3 downgrade rules; codes may be renamed when it settles), `morphir-ukvn.7.9.1` (closed: the `Union` reference implementation in finos/morphir-moonbit) + +## Impact + +- `packages/morphir`: `morphir.ir.v4` modules `value`, `type`, `type_def`, `type_spec`, `module` and new `annotation` and `incompleteness` modules; `morphir.ir.v4.codec`; `morphir.ir.migrate`; `morphir.ir.diagnostics` (new warning code `input_attributes_dropped`). The core stays dependency-free. +- `packages/morphir-tools`: one new `ir convert` option. +- Callers that build `Hole` values with raw JSON reasons must pass a `HoleReason`. Callers that match on `module.Definition.values` entries must handle the three new bodies. +- No change to v3 or to format-3 encoding. diff --git a/openspec/changes/align-ir-v4-with-upstream/specs/ir/migrate/spec.md b/openspec/changes/align-ir-v4-with-upstream/specs/ir/migrate/spec.md new file mode 100644 index 00000000..8b6e93dc --- /dev/null +++ b/openspec/changes/align-ir-v4-with-upstream/specs/ir/migrate/spec.md @@ -0,0 +1,51 @@ +# Spec Delta + +## MODIFIED Requirements + +### Requirement: Reverse migrate v4 to v3 with known failure paths +The library SHALL migrate v4 to v3. When the input uses concepts v3 cannot represent, migration SHALL fail with structured errors and MUST NOT invent or drop IR content, except as allowed by the dropped-information requirement. (bead: morphir-ukvn.8) + +#### Scenario: Specs and Application do not downgrade +- **WHEN** a v4 Specs or Application distribution is migrated to v3 +- **THEN** migration fails with `unsupported_v4_downgrade` naming the distribution kind + +#### Scenario: V4-only nodes are reported +- **WHEN** a v4 Library contains a `Hole`, `Native`, or `External` value, or a `DocumentLiteral` +- **THEN** migration fails with `unsupported_v4_downgrade` naming the construct and its location + +#### Scenario: V4-only definitions and types are reported +- **WHEN** a v4 Library contains a native, external or incomplete value-definition body, an incomplete type definition, a `Union` type, or a specification with annotations +- **THEN** migration fails with `unsupported_v4_downgrade` naming the construct and its location + +#### Scenario: Missing types are reported +- **WHEN** a v4 value, pattern, or value-definition input has no `inferred_type` +- **THEN** migration fails with `missing_inferred_type` at that location + +#### Scenario: Semantic attributes are reported +- **WHEN** v4 type attributes carry a source, constraints, or extensions, or value attributes carry extensions +- **THEN** migration fails with `unsupported_attributes_downgrade` + +#### Scenario: Absent member documentation is reported +- **WHEN** a v4 type or value member has no documentation +- **THEN** migration fails with `unsupported_documentation_downgrade`, because v3 member documentation is a required string + +#### Scenario: Compatible Library downgrades +- **WHEN** a v4 Library uses only concepts v3 can represent +- **THEN** migration succeeds with a v3 Library + +## ADDED Requirements + +### Requirement: Per-input attributes drop on request +The library SHALL let a caller drop the value attributes of value-definition inputs, because v4 JSON spells an input as a bare type. On request, each input SHALL keep only its declared type as `inferred_type`, and each input that lost information SHALL report `input_attributes_dropped` at its location. Without the request, forward migration MUST keep input attributes unchanged. (bead: morphir-ukvn.8.15) + +#### Scenario: Reference model upgrades to encodable v4 +- **WHEN** the morphir-elm reference model, whose inputs carry resolved types that differ from their declared types, is migrated to v4 with the input-attribute drop requested +- **THEN** migration succeeds, reports one `input_attributes_dropped` warning per affected input, and the v4 Library encodes as v4 JSON + +#### Scenario: Default upgrade stays lossless +- **WHEN** the same model is migrated to v4 without the request +- **THEN** no input attribute changes and the v3 to v4 to v3 round trip is semantically equal + +#### Scenario: Inputs that lose nothing report nothing +- **WHEN** every input's attributes are empty or hold only its declared type +- **THEN** the drop reports no warning diff --git a/openspec/changes/align-ir-v4-with-upstream/specs/ir/v4/spec.md b/openspec/changes/align-ir-v4-with-upstream/specs/ir/v4/spec.md new file mode 100644 index 00000000..d839b273 --- /dev/null +++ b/openspec/changes/align-ir-v4-with-upstream/specs/ir/v4/spec.md @@ -0,0 +1,83 @@ +# Spec Delta + +## MODIFIED Requirements + +### Requirement: Draft v4 JSON codec +The library SHALL encode and decode v4 versioned distributions as `formatVersion` first and a distribution wrapped by its kind (`Library`, `Specs`, `Application`) using finos/morphir v4 JSON profile spellings at a pinned commit. The codec MUST be documented as draft and MUST NOT write morphir-elm format-3 shapes. (bead: morphir-ukvn.8) + +#### Scenario: V4 round trip +- **WHEN** a v4 Library built only from constructs the codec supports is encoded and decoded +- **THEN** the result equals the original, except that the package release version is absent + +#### Scenario: V4 encode is not format 3 +- **WHEN** a v4 distribution is encoded +- **THEN** the output declares `formatVersion` 4 and uses no format-3 tagged-array distribution shape + +#### Scenario: Unsupported spelling is refused +- **WHEN** v4 input contains a node kind or member the codec does not recognize +- **THEN** decoding fails with a structured error that names the node and its location, and does not guess + +#### Scenario: Construct with no draft spelling +- **WHEN** a caller encodes a v4 value that has no spelling in the pinned v4 JSON profile, such as the deprecated Python `Native` or `External` value nodes, structured type constraints, or a value-definition input whose attributes differ from its declared type +- **THEN** encoding fails with `no_v4_spelling` naming the construct rather than dropping it + +## ADDED Requirements + +### Requirement: Value definitions have upstream bodies +A v4 value definition, at module level or in a let binding, SHALL be one of four bodies: an expression body, a native body with a native hint, an external body with one or more platform bindings and an optional fallback expression, or an incomplete body with an incompleteness and an optional partial expression. Native, external and incomplete bodies MUST hold bare input types. (bead: morphir-ukvn.8.14) + +#### Scenario: Native body round trips +- **WHEN** a module value with a native body and an `Arithmetic` hint is encoded and decoded +- **THEN** the JSON is `{"NativeBody": {"inputTypes": ..., "outputType": ..., "nativeInfo": {"hint": {"Arithmetic": {}}}}}` and the decoded value equals the original + +#### Scenario: External platforms are unique +- **WHEN** an external body names the same target platform twice +- **THEN** encoding and decoding fail with `duplicate_member` + +#### Scenario: External body needs a binding +- **WHEN** an external body has no platform binding +- **THEN** encoding and decoding fail with a structured error + +#### Scenario: Platform-specific hint names its platform +- **WHEN** a native body's `PlatformSpecific` hint has no `platform` member +- **THEN** decoding fails with `missing_member` + +### Requirement: Incomplete definitions and typed hole reasons +The v4 model SHALL represent an incompleteness as a hole with a typed reason (`UnresolvedReference`, `DeletedDuringRefactor` or `TypeMismatch`) and an optional partial type, or as a draft. A type definition MAY be incomplete, with type parameters, an incompleteness and an optional partial type expression. A `Hole` value SHALL carry a typed reason. (bead: morphir-ukvn.8.14) + +#### Scenario: Incomplete type definition round trips +- **WHEN** an incomplete type definition whose hole reason is a type mismatch is encoded and decoded +- **THEN** the JSON is `{"IncompleteTypeDefinition": {"typeParams": [...], "incompleteness": {"Hole": {"reason": {"TypeMismatch": {"expected": ..., "found": ...}}}}}}` and the decoded definition equals the original + +#### Scenario: Draft is not a hole reason +- **WHEN** a hole reason is spelled `{"Draft": {}}` +- **THEN** decoding fails with `unknown_node` + +### Requirement: Specifications carry annotations +Module, type and value specifications SHALL carry an ordered list of annotations. An annotation SHALL be either compact (a name and optional free text) or structured (a name and positional or named value arguments). The codec SHALL write `annotations` only when the list is non-empty and SHALL omit an empty `arguments` member. Definitions MUST NOT carry annotations. (bead: morphir-ukvn.8.14) + +#### Scenario: Compact and structured annotations round trip +- **WHEN** a type alias specification with a compact annotation with text, a structured annotation with one positional and one named argument, and a structured annotation without arguments is encoded and decoded +- **THEN** the compact annotation is written as `"pkg:mod#name:text"`, the argument-less structured annotation as `{"name": ...}`, and the decoded specification equals the original + +#### Scenario: Definitions refuse annotations +- **WHEN** a type definition carries an `annotations` member +- **THEN** decoding fails with `unknown_member` + +### Requirement: Union type expression +The v4 model SHALL represent an untagged union as a type expression with attributes and an ordered list of two or more alternatives. The codec SHALL preserve alternative order, duplicates and nesting, and MUST refuse a union with fewer than two alternatives. (bead: morphir-ukvn.8.14) + +#### Scenario: Union round trips +- **WHEN** a union of `String` and `Int` is encoded and decoded +- **THEN** the JSON is `{"Union": {"alternatives": [..., ...]}}` and the decoded type equals the original + +#### Scenario: Singleton union is refused +- **WHEN** a union has one alternative +- **THEN** encoding and decoding fail with a structured error + +### Requirement: Python-only v4 constructs are deprecated +The Python `Native` and `External` value nodes and the structured type constraints SHALL stay constructible for compatibility, SHALL be documented as deprecated because upstream v4 has no spelling for them, and MUST keep failing encoding with `no_v4_spelling`. (bead: morphir-ukvn.8.14) + +#### Scenario: Deprecated value node still refuses to encode +- **WHEN** a caller encodes a value that contains a Python `Native` value node +- **THEN** encoding fails with `no_v4_spelling`, and the documentation directs the caller to a native definition body diff --git a/openspec/changes/align-ir-v4-with-upstream/specs/tools/ir-interchange/spec.md b/openspec/changes/align-ir-v4-with-upstream/specs/tools/ir-interchange/spec.md new file mode 100644 index 00000000..2260b07d --- /dev/null +++ b/openspec/changes/align-ir-v4-with-upstream/specs/tools/ir-interchange/spec.md @@ -0,0 +1,26 @@ +# Spec Delta + +## MODIFIED Requirements + +### Requirement: Convert between v3 and v4 +The `ir convert` command SHALL convert format 3.0.0 to v4 and v4 to format 3.0.0 through the library migrations. It SHALL accept an optional package version and an opt-in flag to drop per-input attributes for upgrades, and an opt-in flag to drop source locations on downgrade. Success writes the target file; failure exits non-zero, prints each diagnostic, and leaves no output file. (bead: morphir-ukvn.8) + +#### Scenario: Upgrade v3 file to v4 +- **WHEN** the user converts a valid format-3 Library file to v4 +- **THEN** the tool writes a `formatVersion` 4 file and exits 0 + +#### Scenario: Downgrade failure is visible +- **WHEN** the user converts a v4 file that cannot be represented in format 3.0.0 +- **THEN** the tool exits non-zero, prints each diagnostic code and location, and does not create the output file + +#### Scenario: Downgrade warnings are printed +- **WHEN** a downgrade succeeds with dropped information +- **THEN** the tool prints each warning code and still exits 0 + +#### Scenario: Upgrade with input attributes needs opt-in +- **WHEN** the user converts the morphir-elm reference model to v4 without `--drop-input-attributes` +- **THEN** the tool exits non-zero with `no_v4_spelling`, names the flag, and does not create the output file + +#### Scenario: Upgrade drops input attributes on request +- **WHEN** the user converts the same file with `--drop-input-attributes` +- **THEN** the tool writes a `formatVersion` 4 file, prints each `input_attributes_dropped` warning, and exits 0 diff --git a/openspec/changes/align-ir-v4-with-upstream/tasks.md b/openspec/changes/align-ir-v4-with-upstream/tasks.md new file mode 100644 index 00000000..f29b3b40 --- /dev/null +++ b/openspec/changes/align-ir-v4-with-upstream/tasks.md @@ -0,0 +1,48 @@ +# Tasks + +## 1. Per-input attribute drop (morphir-ukvn.8.15) + +- [ ] 1.1 Add `INPUT_ATTRIBUTES_DROPPED = "input_attributes_dropped"` to `morphir.ir.diagnostics` and `drop_input_attributes(lib)` to `morphir.ir.migrate`, which resets every module and let-bound expression-body input to its declared type and reports one warning per input that lost information; verify unit tests for a changed input, an unchanged input, a let-bound input, and the warning location +- [ ] 1.2 Add `drop_input_attributes: bool = False` to `v3_to_v4` and verify that the morphir-elm reference model migrates with 8 warnings and encodes as v4 JSON with the flag, and that without it the v3 to v4 to v3 round trip is still semantically equal +- [ ] 1.3 Add `ir convert --drop-input-attributes`, mention the flag in the `no_v4_spelling` input error, and verify CLI tests: without the flag the reference model fails with no output file; with it the tool writes v4 JSON, prints the warnings, and exits 0 +- [ ] 1.4 Document the drop in the `packages/morphir` README and the codec docstring's gap list, and verify the documented example runs + +## 2. Typed hole reasons, incompleteness and incomplete type definitions (morphir-ukvn.8.14.2) + +- [ ] 2.1 Add `morphir.ir.v4.incompleteness` with `UnresolvedReference`, `DeletedDuringRefactor`, `TypeMismatch`, `HoleReason`, `HoleIncompleteness`, `DraftIncompleteness` and `Incompleteness`, re-exported from `morphir.ir.v4`; verify construction and import-identity tests +- [ ] 2.2 Type `Hole.reason` as `HoleReason`, encode and decode reasons with their kit members, and verify that `{"Draft": {}}` as a reason fails with `unknown_node` and a non-object reason fails with a structured error +- [ ] 2.3 Add `type_def.IncompleteTypeDefinition(type_params, incompleteness, partial_type_exp)` and its codec spelling, and verify round trips for kit `definitions-0014`, `-0015` and `-0024` in expanded form +- [ ] 2.4 Make reverse migration fail with `unsupported_v4_downgrade` for an incomplete type definition and verify the construct name and location + +## 3. Value-definition bodies (morphir-ukvn.8.14.1) + +- [ ] 3.1 Add `NativeHint` kinds, `NativeInfo`, `ExternalBinding`, `NativeBody`, `ExternalBody`, `IncompleteBody` and `ValueDefinitionBody` to `morphir.ir.v4.value`, widen module values and let bindings to `ValueDefinitionBody`, and verify the type checkers pass +- [ ] 3.2 Encode and decode the three bodies with the kit members, and verify round trips for kit `definitions-0007` to `-0009`, `-0016` and `-0025` in expanded form +- [ ] 3.3 Refuse duplicate target platforms (`duplicate_member`), an empty `externals` list, and a `PlatformSpecific` hint without `platform` (`missing_member`), and verify one test each +- [ ] 3.4 Make reverse migration fail with `unsupported_v4_downgrade` for each new body at module level and in a let binding, and verify the locations +- [ ] 3.5 Make `drop_input_attributes` and the expression-body checks skip the new bodies, which hold bare input types, and verify a Library that mixes bodies encodes + +## 4. Annotations on specifications (morphir-ukvn.8.14.3) + +- [ ] 4.1 Add `morphir.ir.v4.annotation` with `CompactAnnotation`, `StructuredAnnotation`, `PositionalArgument`, `NamedArgument` and `Annotation`, and an `annotations` field (empty default, last) on module, type and value specifications; verify existing positional constructors still work +- [ ] 4.2 Encode and decode annotations (written first, only when non-empty; `arguments` omitted when empty; compact text after the first colon after `#`), and verify kit `definitions-0020` to `-0022` round trip and that `"arguments": []` reads as an argument-less structured annotation +- [ ] 4.3 Verify that a type definition with `annotations` fails decoding with `unknown_member` (kit `definitions-0023`) +- [ ] 4.4 Make reverse migration fail with `unsupported_v4_downgrade` for annotations on a specification in a v4 dependency, and verify the location + +## 5. Union type expression (morphir-ukvn.8.14.4) + +- [ ] 5.1 Add `type.Union(attributes, alternatives)` to the `Type` union and `map_attributes`, and verify a substitution-style attribute map keeps order and duplicates +- [ ] 5.2 Encode and decode `{"Union": {"attributes"?, "alternatives": [...]}}`, refuse fewer than two alternatives both ways, and verify round trips including a nested union +- [ ] 5.3 Make reverse migration fail with `unsupported_v4_downgrade` for a union and verify the location + +## 6. Deprecations, docs and integration checks (morphir-ukvn.8.14.5) + +- [ ] 6.1 Mark the `Native` and `External` value nodes and `TypeConstraints` as deprecated in docstrings, naming the replacement, and verify a test that they still fail encoding with `no_v4_spelling` +- [ ] 6.2 Update the codec module docstring's gap list and the README "Morphir IR versions" section for the new constructs and the typed hole reason, and verify the README examples run +- [ ] 6.3 File a follow-up bead to remove the deprecated constructs and verify it is a child of `morphir-ukvn.8` with label `repo:morphir-python` +- [ ] 6.4 Run `mise run check` and `mise run test-bdd` and verify lint, both type checkers, tests and coverage of at least 80% pass + +## Workflow follow-up + +- Archive the change in the same PR once all tasks are done, fix the design asset prefix, and relink `morphir-ukvn.8.14` to the archived path +- `morphir-ukvn.8.16` re-syncs the codec with upstream compact spellings when upstream v4 settles From 110707e60a8a681c23964db3d47269165e1e0f40 Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 02:02:49 -0500 Subject: [PATCH 2/9] feat(ir): drop per-input value attributes for v4 on request v4 JSON spells a value-definition input as a bare type (kit case definitions-0027; morphir-rust classic_to_v4 drops v3 input attributes), so the morphir-elm reference model, whose typed inputs carry resolved types, failed v4 encoding with no_v4_spelling. Add drop_input_attributes for any v4 distribution, a drop_input_attributes keyword on v3_to_v4, and ir convert --drop-input-attributes. Each input that loses information reports input_attributes_dropped. The default migration stays lossless. Refs: morphir-ukvn.8.15 --- .../align-ir-v4-with-upstream/design.html | 2 +- .../align-ir-v4-with-upstream/tasks.md | 8 +- .../morphir-tools/src/morphir_tools/cli.py | 23 +- packages/morphir/README.md | 25 ++ .../morphir/src/morphir/ir/diagnostics.py | 6 +- packages/morphir/src/morphir/ir/migrate.py | 216 +++++++++++++++++- packages/morphir/src/morphir/ir/v4/codec.py | 11 +- tests/tools/test_cli.py | 17 ++ tests/unit/ir/test_migrate.py | 71 +++++- 9 files changed, 350 insertions(+), 29 deletions(-) diff --git a/openspec/changes/align-ir-v4-with-upstream/design.html b/openspec/changes/align-ir-v4-with-upstream/design.html index 73e2ba95..34215633 100644 --- a/openspec/changes/align-ir-v4-with-upstream/design.html +++ b/openspec/changes/align-ir-v4-with-upstream/design.html @@ -51,7 +51,7 @@

Context

The in-memory v3 to v4 to v3 round trip is a requirement (ir/migrate, "Shared subset round trips"), and the -morphir-elm reference model has 8 inputs whose resolved type differs from the declared type (alias expansion and +morphir-elm reference model has 10 inputs (8 module-level, 2 let-bound) whose resolved type differs from the declared type (alias expansion and type-variable instances, for example float-alias resolved to Float).

Goals and non-goals

diff --git a/openspec/changes/align-ir-v4-with-upstream/tasks.md b/openspec/changes/align-ir-v4-with-upstream/tasks.md index f29b3b40..1ebb0cfa 100644 --- a/openspec/changes/align-ir-v4-with-upstream/tasks.md +++ b/openspec/changes/align-ir-v4-with-upstream/tasks.md @@ -2,10 +2,10 @@ ## 1. Per-input attribute drop (morphir-ukvn.8.15) -- [ ] 1.1 Add `INPUT_ATTRIBUTES_DROPPED = "input_attributes_dropped"` to `morphir.ir.diagnostics` and `drop_input_attributes(lib)` to `morphir.ir.migrate`, which resets every module and let-bound expression-body input to its declared type and reports one warning per input that lost information; verify unit tests for a changed input, an unchanged input, a let-bound input, and the warning location -- [ ] 1.2 Add `drop_input_attributes: bool = False` to `v3_to_v4` and verify that the morphir-elm reference model migrates with 8 warnings and encodes as v4 JSON with the flag, and that without it the v3 to v4 to v3 round trip is still semantically equal -- [ ] 1.3 Add `ir convert --drop-input-attributes`, mention the flag in the `no_v4_spelling` input error, and verify CLI tests: without the flag the reference model fails with no output file; with it the tool writes v4 JSON, prints the warnings, and exits 0 -- [ ] 1.4 Document the drop in the `packages/morphir` README and the codec docstring's gap list, and verify the documented example runs +- [x] 1.1 Add `INPUT_ATTRIBUTES_DROPPED = "input_attributes_dropped"` to `morphir.ir.diagnostics` and `drop_input_attributes(lib)` to `morphir.ir.migrate`, which resets every module and let-bound expression-body input to its declared type and reports one warning per input that lost information; verify unit tests for a changed input, an unchanged input, a let-bound input, and the warning location +- [x] 1.2 Add `drop_input_attributes: bool = False` to `v3_to_v4` and verify that the morphir-elm reference model migrates with 10 warnings (8 module-level and 2 let-bound inputs) and encodes as v4 JSON with the flag, and that without it the v3 to v4 to v3 round trip is still semantically equal +- [x] 1.3 Add `ir convert --drop-input-attributes`, mention the flag in the `no_v4_spelling` input error, and verify CLI tests: without the flag the reference model fails with no output file; with it the tool writes v4 JSON, prints the warnings, and exits 0 +- [x] 1.4 Document the drop in the `packages/morphir` README and the codec docstring's gap list, and verify the documented example runs ## 2. Typed hole reasons, incompleteness and incomplete type definitions (morphir-ukvn.8.14.2) diff --git a/packages/morphir-tools/src/morphir_tools/cli.py b/packages/morphir-tools/src/morphir_tools/cli.py index 1a44bd9b..d69a970e 100644 --- a/packages/morphir-tools/src/morphir_tools/cli.py +++ b/packages/morphir-tools/src/morphir_tools/cli.py @@ -20,7 +20,8 @@ from morphir.ir import format_version as fv from morphir.ir.diagnostics import Diagnostic, DiagnosticError -from morphir.ir.migrate import Err, v3_to_v4, v4_to_v3 +from morphir.ir.migrate import Err, Ok, v3_to_v4, v4_to_v3 +from morphir.ir.migrate import drop_input_attributes as drop_inputs from morphir.ir.v3 import codec as v3_codec from morphir.ir.v4 import codec as v4_codec from morphir.ir.v4 import distribution as v4_dist @@ -128,6 +129,13 @@ def convert( bool, typer.Option(help="Drop value source locations when downgrading to v3."), ] = False, + drop_input_attributes: Annotated[ + bool, + typer.Option( + help="Keep only the declared type of each value-definition input when " + "writing v4, which spells an input as a bare type." + ), + ] = False, ) -> None: """Convert between format 3.0.0 and draft IR v4. @@ -138,16 +146,19 @@ def convert( try: version, _ = fv.read_document(text) if to is Target.V4: + upgraded: Ok[v4_dist.Distribution] if version.major == 4: - dist = v4_codec.decode_versioned_distribution(text) + upgraded = Ok(v4_codec.decode_versioned_distribution(text)) else: - upgraded = v3_to_v4( + library_v4 = v3_to_v4( v3_codec.decode_versioned_distribution(text), package_version=package_version, ) - dist = upgraded.value - document = v4_codec.encode_versioned_distribution(dist) - warnings: tuple[Diagnostic, ...] = () + upgraded = Ok(library_v4.value, library_v4.warnings) + if drop_input_attributes: + upgraded = drop_inputs(upgraded.value) + document = v4_codec.encode_versioned_distribution(upgraded.value) + warnings: tuple[Diagnostic, ...] = upgraded.warnings else: if version.major == 3: library = v3_codec.decode_versioned_distribution(text) diff --git a/packages/morphir/README.md b/packages/morphir/README.md index 796eeaf3..d2fe1277 100644 --- a/packages/morphir/README.md +++ b/packages/morphir/README.md @@ -111,6 +111,31 @@ distributions, `Hole`, `Native`, `External`, `DocumentLiteral`, a missing `inferred_type`, semantic attributes, and missing member documentation. Value source locations fail unless you pass `drop_source_locations=True`. +v4 JSON spells a value-definition input as a bare type, so it has no place for +the per-input attributes of a typed v3 model. morphir-elm writes the resolved +type there, which can differ from the declared type (an alias expanded, a type +variable instantiated), and the draft codec then fails with `no_v4_spelling`. +Opt in to dropping them with `v3_to_v4(..., drop_input_attributes=True)`, +`drop_input_attributes(dist)` for any v4 distribution, or +`morphir-tools ir convert --to v4 --drop-input-attributes`. Each input that loses +information reports `input_attributes_dropped`; without the opt-in the in-memory +migration stays lossless. + +```python +from pathlib import Path + +from morphir.ir.migrate import v3_to_v4 +from morphir.ir.v3 import codec as v3_codec +from morphir.ir.v4 import codec as v4_codec + +text = Path("tests/fixtures/ir/v3/elm-reference-model.json").read_text() +upgraded = v3_to_v4( + v3_codec.decode_versioned_distribution(text), drop_input_attributes=True +) +assert {w.code for w in upgraded.warnings} == {"input_attributes_dropped"} +assert v4_codec.encode_versioned_distribution(upgraded.value)["formatVersion"] == 4 +``` + ## Morphir SDK `morphir.sdk` is the standard library that Morphir models compile against. The diff --git a/packages/morphir/src/morphir/ir/diagnostics.py b/packages/morphir/src/morphir/ir/diagnostics.py index 1a4212a8..0180111f 100644 --- a/packages/morphir/src/morphir/ir/diagnostics.py +++ b/packages/morphir/src/morphir/ir/diagnostics.py @@ -10,8 +10,9 @@ - downgrade codes come from the finos CLI and morphir-moonbit ``migration/v3/migration.mbt``, and may be renamed when finos bead ``morphir-diwy`` settles; -- ``package_version_dropped``, ``source_location_dropped``, and - ``no_v4_spelling`` are local to morphir-python. +- ``package_version_dropped``, ``source_location_dropped``, + ``input_attributes_dropped``, and ``no_v4_spelling`` are local to + morphir-python. """ from __future__ import annotations @@ -48,6 +49,7 @@ # Warnings (morphir-python). PACKAGE_VERSION_DROPPED = "package_version_dropped" SOURCE_LOCATION_DROPPED = "source_location_dropped" +INPUT_ATTRIBUTES_DROPPED = "input_attributes_dropped" Location = tuple[str | int, ...] diff --git a/packages/morphir/src/morphir/ir/migrate.py b/packages/morphir/src/morphir/ir/migrate.py index 50ec229a..2f465248 100644 --- a/packages/morphir/src/morphir/ir/migrate.py +++ b/packages/morphir/src/morphir/ir/migrate.py @@ -6,7 +6,11 @@ - the package release version, which v3 cannot hold (``package_version_dropped``); - value source locations, only when ``drop_source_locations=True`` - (``source_location_dropped``). + (``source_location_dropped``); +- value-definition input attributes, only when ``drop_input_attributes=True`` or + through `drop_input_attributes` (``input_attributes_dropped``). v4 JSON spells + an input as a bare type, so an input whose attributes hold more than its + declared type cannot be encoded without this drop. The mapping follows morphir-moonbit ``migration/v3/migration.mbt``, adjusted for the Python v4 model. Diagnostic codes are defined in `morphir.ir.diagnostics` and @@ -24,6 +28,7 @@ import re from dataclasses import dataclass, field +from typing import cast from . import diagnostics as d from .diagnostics import Diagnostic, Location @@ -334,7 +339,10 @@ def _package_def_up(defn: pkg3.Definition[None, t3.Type[None]]) -> pkg4.Definiti def v3_to_v4( - dist: dist3.Library, *, package_version: str | None = None + dist: dist3.Library, + *, + package_version: str | None = None, + drop_input_attributes: bool = False, ) -> Ok[dist4.LibraryDistribution]: """Migrate a format-3 Library to a v4 Library. @@ -345,19 +353,196 @@ def v3_to_v4( dist: The v3 Library. package_version: Optional release version for the v4 package identity. Without it the version is absent; nothing is invented. + drop_input_attributes: Reset each value-definition input's attributes to + its declared type, as `drop_input_attributes` does, so the result can + be encoded as v4 JSON. Each input that loses information is reported. Returns: - ``Ok`` with the v4 Library. Forward migration has no failure paths. + ``Ok`` with the v4 Library and any ``input_attributes_dropped`` warnings. + Forward migration has no failure paths. """ - return Ok( - dist4.LibraryDistribution( - package=dist4.PackageInfo(_path_up(dist.package_name), package_version), - definition=_package_def_up(dist.definition), - dependencies={ - _path_up(p): _package_spec_up(s) for p, s in dist.dependencies.items() - }, - ) + library = dist4.LibraryDistribution( + package=dist4.PackageInfo(_path_up(dist.package_name), package_version), + definition=_package_def_up(dist.definition), + dependencies={ + _path_up(p): _package_spec_up(s) for p, s in dist.dependencies.items() + }, ) + if drop_input_attributes: + return _drop_input_attributes(library) + return Ok(library) + + +# --- Per-input attributes --- + + +class _InputDrop: + """Resets value-definition input attributes, recording each loss.""" + + def __init__(self) -> None: + self.warnings: list[Diagnostic] = [] + + def definition(self, defn: v4.Definition, loc: Location) -> v4.Definition: + inputs: list[tuple[name4.Name, v4.ValueAttributes, t4.Type]] = [] + for n, attrs, tpe in defn.input_types: + declared = v4.ValueAttributes(inferred_type=tpe) + if attrs not in (v4.ValueAttributes(), declared): + self.warnings.append( + Diagnostic( + d.INPUT_ATTRIBUTES_DROPPED, + "ValueDefinitionInput", + (*loc, "input_types", name4.to_string(n)), + f"input {name4.to_string(n)!r} kept only its declared type; " + "v4 JSON spells an input as a bare type", + ) + ) + inputs.append((n, declared, tpe)) + return v4.Definition( + inputs, defn.output_type, self.value(defn.body, (*loc, "body")) + ) + + def value(self, val: v4.Value, loc: Location) -> v4.Value: + here = (*loc, type(val).__name__) + a = val.attributes + match val: + case v4.Tuple(_, elements): + return v4.Tuple( + a, [self.value(x, (*here, i)) for i, x in enumerate(elements)] + ) + case v4.List(_, items): + return v4.List( + a, [self.value(x, (*here, i)) for i, x in enumerate(items)] + ) + case v4.Record(_, fields): + return v4.Record(a, self.fields(fields, here)) + case v4.Field(_, subject, n): + return v4.Field(a, self.value(subject, (*here, "subject")), n) + case v4.Apply(_, function, argument): + return v4.Apply( + a, + self.value(function, (*here, "function")), + self.value(argument, (*here, "argument")), + ) + case v4.Lambda(_, pattern, body): + return v4.Lambda(a, pattern, self.value(body, (*here, "body"))) + case v4.LetDefinition(_, n, defn, in_value): + return v4.LetDefinition( + a, + n, + self.definition(defn, (*here, "definition")), + self.value(in_value, (*here, "in_value")), + ) + case v4.LetRecursion(_, defs, in_value): + return v4.LetRecursion( + a, + { + k: self.definition( + x, (*here, "definitions", name4.to_string(k)) + ) + for k, x in defs.items() + }, + self.value(in_value, (*here, "in_value")), + ) + case v4.Destructure(_, pattern, subject, in_value): + return v4.Destructure( + a, + pattern, + self.value(subject, (*here, "value_to_destructure")), + self.value(in_value, (*here, "in_value")), + ) + case v4.IfThenElse(_, cond, then_branch, else_branch): + return v4.IfThenElse( + a, + self.value(cond, (*here, "condition")), + self.value(then_branch, (*here, "then_branch")), + self.value(else_branch, (*here, "else_branch")), + ) + case v4.PatternMatch(_, subject, cases): + return v4.PatternMatch( + a, + self.value(subject, (*here, "branch_on")), + [ + (p, self.value(b, (*here, "cases", i))) + for i, (p, b) in enumerate(cases) + ], + ) + case v4.UpdateRecord(_, subject, fields): + return v4.UpdateRecord( + a, + self.value(subject, (*here, "value_to_update")), + self.fields(fields, here), + ) + case _: + return val + + def fields( + self, items: list[tuple[name4.Name, v4.Value]], loc: Location + ) -> list[tuple[name4.Name, v4.Value]]: + return [(k, self.value(x, (*loc, name4.to_string(k)))) for k, x in items] + + def package_def(self, defn: pkg4.Definition, loc: Location) -> pkg4.Definition: + modules: dict[path4.Path, ac4.AccessControlled[mod4.Definition]] = {} + for p, m in defn.modules.items(): + here = (*loc, path4.to_string(p)) + values = { + k: _with_access( + x, + doc4.Documented( + x.value.doc, + self.definition( + x.value.value, (*here, "values", name4.to_string(k)) + ), + ), + ) + for k, x in m.value.values.items() + } + modules[p] = _with_access( + m, mod4.Definition(m.value.types, values, m.value.doc) + ) + return pkg4.Definition(modules) + + +def _with_access[A, B]( + item: ac4.AccessControlled[A], value: B +) -> ac4.AccessControlled[B]: + return ac4.Public(value) if isinstance(item, ac4.Public) else ac4.Private(value) + + +def drop_input_attributes[D: dist4.Distribution](dist: D) -> Ok[D]: + """Reset every value-definition input's attributes to its declared type. + + v4 JSON spells an input as a bare type (kit case ``definitions-0027``), so + the draft codec refuses an input whose attributes hold anything else, such as + the resolved types morphir-elm writes for typed inputs. This drop makes such + a distribution encodable. Module-level and let-bound definitions are covered. + + Args: + dist: Any v4 distribution. A Specs distribution has no definitions and + is returned unchanged. + + Returns: + ``Ok`` with the distribution and one ``input_attributes_dropped`` warning + per input whose attributes were neither empty nor its declared type. + """ + return _drop_input_attributes(dist) + + +def _drop_input_attributes[D: dist4.Distribution](dist: D) -> Ok[D]: + job = _InputDrop() + loc: Location = ("distribution", "modules") + result: dist4.Distribution + match dist: + case dist4.SpecsDistribution(): + return Ok(dist) + case dist4.LibraryDistribution(package, definition, dependencies): + result = dist4.LibraryDistribution( + package, job.package_def(definition, loc), dependencies + ) + case dist4.ApplicationDistribution(package, definition, deps, entry_points): + result = dist4.ApplicationDistribution( + package, job.package_def(definition, loc), deps, entry_points + ) + return Ok(cast(D, result), tuple(job.warnings)) # --- v4 -> v3 --- @@ -896,4 +1081,11 @@ def v4_to_v3( return Ok(library, tuple(warnings)) -__all__ = ["Err", "MigrationResult", "Ok", "v3_to_v4", "v4_to_v3"] +__all__ = [ + "Err", + "MigrationResult", + "Ok", + "drop_input_attributes", + "v3_to_v4", + "v4_to_v3", +] diff --git a/packages/morphir/src/morphir/ir/v4/codec.py b/packages/morphir/src/morphir/ir/v4/codec.py index 8c26b4f3..8ec0b6b4 100644 --- a/packages/morphir/src/morphir/ir/v4/codec.py +++ b/packages/morphir/src/morphir/ir/v4/codec.py @@ -26,9 +26,12 @@ ``unknown_member``. - Records are objects keyed by field name (finos decision 0004); duplicate field names fail encoding with ``duplicate_member``. -- ``ExpressionBody.inputTypes`` has no attribute slot. Encoding accepts per-input +- ``ExpressionBody.inputTypes`` has no attribute slot: upstream spells an input as + a bare type (kit case ``definitions-0027``). Encoding accepts per-input attributes that are empty or whose only member is ``inferred_type`` equal to the - declared input type; decoding sets ``inferred_type`` to the declared type. + declared input type, and fails with ``no_v4_spelling`` otherwise; decoding sets + ``inferred_type`` to the declared type. `morphir.ir.migrate.drop_input_attributes` + resets the attributes on request and reports each input that lost information. - The package release version has no member (finos decision 0017); it is not written and decodes as absent. - Python names are word lists without an initialism marker. A run of two or more @@ -1106,7 +1109,9 @@ def encode_value_definition(defn: v.Definition, loc: Location = ()) -> Json: "ValueDefinition", (*here, "inputTypes", i), f"input {encode_name(name, here)!r} carries attributes other than its " - "declared type; ExpressionBody.inputTypes has no attribute slot", + "declared type; ExpressionBody.inputTypes has no attribute slot. " + "Drop them with morphir.ir.migrate.drop_input_attributes or " + "`morphir-tools ir convert --drop-input-attributes`", ) return { "ExpressionBody": { diff --git a/tests/tools/test_cli.py b/tests/tools/test_cli.py index 4871f2d6..8e6573a3 100644 --- a/tests/tools/test_cli.py +++ b/tests/tools/test_cli.py @@ -163,8 +163,25 @@ def test_encode_failure_leaves_no_output(self, tmp_path: Path) -> None: result = run("ir", "convert", REFERENCE, "--to", "v4", "-o", out) assert result.exit_code == 1 assert d.NO_V4_SPELLING in result.stderr + assert "--drop-input-attributes" in result.stderr assert not out.exists() + def test_drop_input_attributes_on_upgrade(self, tmp_path: Path) -> None: + out = tmp_path / "out.json" + result = run( + "ir", + "convert", + REFERENCE, + "--to", + "v4", + "-o", + out, + "--drop-input-attributes", + ) + assert result.exit_code == 0, result.output + assert result.stderr.count(d.INPUT_ATTRIBUTES_DROPPED) == 10 + assert json.loads(out.read_text())["formatVersion"] == 4 + def test_downgrade_warnings_exit_zero(self, tmp_path: Path) -> None: unit = {"Unit": {}} attrs = { diff --git a/tests/unit/ir/test_migrate.py b/tests/unit/ir/test_migrate.py index b2cd97b8..c0d4a937 100644 --- a/tests/unit/ir/test_migrate.py +++ b/tests/unit/ir/test_migrate.py @@ -8,12 +8,13 @@ from morphir.ir import diagnostics as d from morphir.ir import v3 -from morphir.ir.migrate import Err, Ok, v3_to_v4, v4_to_v3 +from morphir.ir.migrate import Err, Ok, drop_input_attributes, v3_to_v4, v4_to_v3 from morphir.ir.v3 import codec as v3_codec from morphir.ir.v3 import literal as lit3 from morphir.ir.v3 import package as pkg3 from morphir.ir.v3 import type as t3 from morphir.ir.v3 import value as val3 +from morphir.ir.v4 import codec as v4_codec from morphir.ir.v4 import document as doc4 from morphir.ir.v4 import literal as lit4 from morphir.ir.v4 import module as mod4 @@ -347,3 +348,71 @@ def test_hand_built_library_with_every_node_downgrades(self) -> None: again = v4_to_v3(up.value) assert isinstance(again, Ok) assert again.value == result.value + + +STRING = t4.Reference( + t4.TypeAttributes(), + FQName( + IRPath((n("morphir"), n("s", "d", "k"))), IRPath((n("string"),)), n("string") + ), +) +REFERENCE_MODEL = FIXTURES / "elm-reference-model.json" +INPUT_LOC = ("distribution", "modules", "app", "values", "main", "input_types") + + +class TestDropInputAttributes: + def test_input_with_resolved_type_is_reset_and_reported(self) -> None: + lib = library(inputs=[(n("x"), v4.ValueAttributes(inferred_type=STRING), INT)]) + result = drop_input_attributes(lib) + defn = result.value.definition.modules[IRPath((n("app"),))] + inputs = defn.value.values[n("main")].value.value.input_types # type: ignore[union-attr] + assert inputs == [(n("x"), TYPED, INT)] + assert [(w.code, w.construct, w.location) for w in result.warnings] == [ + (d.INPUT_ATTRIBUTES_DROPPED, "ValueDefinitionInput", (*INPUT_LOC, "x")) + ] + + @pytest.mark.parametrize( + "attrs", [v4.ValueAttributes(), TYPED], ids=["empty", "declared"] + ) + def test_input_that_loses_nothing_reports_nothing( + self, attrs: v4.ValueAttributes + ) -> None: + lib = library(inputs=[(n("x"), attrs, INT)]) + result = drop_input_attributes(lib) + assert result.warnings == () + + def test_let_bound_input_is_reset(self) -> None: + inner = v4.Definition( + [(n("y"), v4.ValueAttributes(inferred_type=STRING), INT)], + INT, + v4.Unit(TYPED), + ) + body = v4.LetDefinition(TYPED, n("f"), inner, v4.Unit(TYPED)) + result = drop_input_attributes(library(body=body)) + assert len(result.warnings) == 1 + assert result.warnings[0].location[-1] == "y" + assert "LetDefinition" in result.warnings[0].location + mod = result.value.definition.modules[IRPath((n("app"),))] + let = mod.value.values[n("main")].value.value.body # type: ignore[union-attr] + assert isinstance(let, v4.LetDefinition) + assert let.definition.input_types == [(n("y"), TYPED, INT)] # type: ignore[union-attr] + + def test_specs_have_no_inputs_to_drop(self) -> None: + specs = SpecsDistribution(PackageInfo(PKG), pkg4.Specification()) + assert drop_input_attributes(specs) == Ok(specs) + + def test_reference_model_upgrades_to_encodable_v4(self) -> None: + lib = load(REFERENCE_MODEL) + result = v3_to_v4(lib, drop_input_attributes=True) + assert len(result.warnings) == 10 + assert {w.code for w in result.warnings} == {d.INPUT_ATTRIBUTES_DROPPED} + document = v4_codec.encode_versioned_distribution(result.value) + assert document["formatVersion"] == 4 + + def test_default_upgrade_keeps_input_attributes(self) -> None: + lib = load(REFERENCE_MODEL) + result = v3_to_v4(lib) + assert result.warnings == () + back = v4_to_v3(result.value) + assert isinstance(back, Ok) + assert back.value == lib From bc2ef92e59b31032603a5d43aeb1034727870171 Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 02:04:35 -0500 Subject: [PATCH 3/9] feat(ir): add typed hole reasons and incomplete type definitions to v4 Add morphir.ir.v4.incompleteness (UnresolvedReference, DeletedDuringRefactor, TypeMismatch; Hole and Draft incompleteness) after the finos/morphir v4 schema at efa36060 and the morphir-rust v4 model. Hole.reason is now a typed HoleReason. Add IncompleteTypeDefinition with its expanded spelling (kit definitions-0014, -0015, -0024, -0026), and make reverse migration refuse it with unsupported_v4_downgrade. Refs: morphir-ukvn.8.14.2 --- .../align-ir-v4-with-upstream/tasks.md | 8 +- packages/morphir/src/morphir/ir/migrate.py | 7 +- .../morphir/src/morphir/ir/v4/__init__.py | 19 +- packages/morphir/src/morphir/ir/v4/codec.py | 108 ++++++++++- .../src/morphir/ir/v4/incompleteness.py | 78 ++++++++ .../morphir/src/morphir/ir/v4/type_def.py | 18 +- packages/morphir/src/morphir/ir/v4/value.py | 11 +- tests/unit/ir/test_migrate.py | 7 +- tests/unit/ir/test_v4_codec.py | 5 +- tests/unit/ir/test_v4_upstream.py | 168 ++++++++++++++++++ 10 files changed, 414 insertions(+), 15 deletions(-) create mode 100644 packages/morphir/src/morphir/ir/v4/incompleteness.py create mode 100644 tests/unit/ir/test_v4_upstream.py diff --git a/openspec/changes/align-ir-v4-with-upstream/tasks.md b/openspec/changes/align-ir-v4-with-upstream/tasks.md index 1ebb0cfa..e5d88ac3 100644 --- a/openspec/changes/align-ir-v4-with-upstream/tasks.md +++ b/openspec/changes/align-ir-v4-with-upstream/tasks.md @@ -9,10 +9,10 @@ ## 2. Typed hole reasons, incompleteness and incomplete type definitions (morphir-ukvn.8.14.2) -- [ ] 2.1 Add `morphir.ir.v4.incompleteness` with `UnresolvedReference`, `DeletedDuringRefactor`, `TypeMismatch`, `HoleReason`, `HoleIncompleteness`, `DraftIncompleteness` and `Incompleteness`, re-exported from `morphir.ir.v4`; verify construction and import-identity tests -- [ ] 2.2 Type `Hole.reason` as `HoleReason`, encode and decode reasons with their kit members, and verify that `{"Draft": {}}` as a reason fails with `unknown_node` and a non-object reason fails with a structured error -- [ ] 2.3 Add `type_def.IncompleteTypeDefinition(type_params, incompleteness, partial_type_exp)` and its codec spelling, and verify round trips for kit `definitions-0014`, `-0015` and `-0024` in expanded form -- [ ] 2.4 Make reverse migration fail with `unsupported_v4_downgrade` for an incomplete type definition and verify the construct name and location +- [x] 2.1 Add `morphir.ir.v4.incompleteness` with `UnresolvedReference`, `DeletedDuringRefactor`, `TypeMismatch`, `HoleReason`, `HoleIncompleteness`, `DraftIncompleteness` and `Incompleteness`, re-exported from `morphir.ir.v4`; verify construction and import-identity tests +- [x] 2.2 Type `Hole.reason` as `HoleReason`, encode and decode reasons with their kit members, and verify that `{"Draft": {}}` as a reason fails with `unknown_node` and a non-object reason fails with a structured error +- [x] 2.3 Add `type_def.IncompleteTypeDefinition(type_params, incompleteness, partial_type_exp)` and its codec spelling, and verify round trips for kit `definitions-0014`, `-0015` and `-0024` in expanded form +- [x] 2.4 Make reverse migration fail with `unsupported_v4_downgrade` for an incomplete type definition and verify the construct name and location ## 3. Value-definition bodies (morphir-ukvn.8.14.1) diff --git a/packages/morphir/src/morphir/ir/migrate.py b/packages/morphir/src/morphir/ir/migrate.py index 2f465248..bd6cea01 100644 --- a/packages/morphir/src/morphir/ir/migrate.py +++ b/packages/morphir/src/morphir/ir/migrate.py @@ -542,7 +542,7 @@ def _drop_input_attributes[D: dist4.Distribution](dist: D) -> Ok[D]: result = dist4.ApplicationDistribution( package, job.package_def(definition, loc), deps, entry_points ) - return Ok(cast(D, result), tuple(job.warnings)) + return Ok(cast("D", result), tuple(job.warnings)) # --- v4 -> v3 --- @@ -736,6 +736,11 @@ def type_def(self, defn: td4.Definition, loc: Location) -> t3.Definition[None]: return t3.CustomTypeDefinition( params, self.access(ctors, self.ctors(ctors.value, loc)) ) + case td4.IncompleteTypeDefinition(): + self.unsupported( + "IncompleteTypeDefinition", (*loc, "IncompleteTypeDefinition") + ) + return t3.TypeAliasDefinition(params, t3.Unit(None)) # Values diff --git a/packages/morphir/src/morphir/ir/v4/__init__.py b/packages/morphir/src/morphir/ir/v4/__init__.py index 58bd9b2c..4887f359 100644 --- a/packages/morphir/src/morphir/ir/v4/__init__.py +++ b/packages/morphir/src/morphir/ir/v4/__init__.py @@ -4,7 +4,7 @@ draft pinned to finos/morphir commit efa36060. """ -from . import module, package +from . import incompleteness, module, package from .access_controlled import AccessControlled from .decorations import ( DecorationFormat, @@ -33,6 +33,15 @@ ) from .documented import Documented from .fqname import FQName +from .incompleteness import ( + DeletedDuringRefactor, + DraftIncompleteness, + HoleIncompleteness, + HoleReason, + Incompleteness, + TypeMismatch, + UnresolvedReference, +) from .literal import ( BoolLiteral, CharLiteral, @@ -64,6 +73,7 @@ "DecorationFormat", "DecorationValuesFile", "DefRef", + "DeletedDuringRefactor", "Distribution", "DocArray", "DocBool", @@ -74,6 +84,7 @@ "DocString", "Document", "Documented", + "DraftIncompleteness", "EntryPoint", "EntryPointKind", "FQName", @@ -81,6 +92,9 @@ "FileMeta", "FileWithDefs", "FloatLiteral", + "HoleIncompleteness", + "HoleReason", + "Incompleteness", "IntegerLiteral", "LayerManifest", "LibraryDistribution", @@ -100,11 +114,14 @@ "TypeAttributes", "TypeConstraints", "TypeDefinition", + "TypeMismatch", "TypeSpecification", + "UnresolvedReference", "Value", "ValueAttributes", "ValueDefinition", "ValueSpecification", + "incompleteness", "module", "package", ] diff --git a/packages/morphir/src/morphir/ir/v4/codec.py b/packages/morphir/src/morphir/ir/v4/codec.py index 8ec0b6b4..cdad8bb1 100644 --- a/packages/morphir/src/morphir/ir/v4/codec.py +++ b/packages/morphir/src/morphir/ir/v4/codec.py @@ -54,6 +54,7 @@ from ..diagnostics import DiagnosticError, Location, fail from ..format_version import V4, V4_TABLE, read_document, write_format_version from . import document as doc +from . import incompleteness as inc from . import literal as lit from . import module, package from . import type as t @@ -147,6 +148,18 @@ def _m(required: str = "", optional: str = "attributes") -> Members: TYPE_DEFINITION_NODES: dict[str, Members] = { "TypeAliasDefinition": _m("typeParams typeExp", ""), "CustomTypeDefinition": _m("typeParams constructors", "access"), + "IncompleteTypeDefinition": _m("typeParams incompleteness", "partialTypeExp"), +} + +HOLE_REASON_NODES: dict[str, Members] = { + "UnresolvedReference": _m("target", ""), + "DeletedDuringRefactor": _m("tx-id", ""), + "TypeMismatch": _m("expected found", ""), +} + +INCOMPLETENESS_NODES: dict[str, Members] = { + "Hole": _m("reason", "partialBody"), + "Draft": _m("", ""), } VALUE_DEFINITION_NODES: dict[str, Members] = { @@ -982,7 +995,9 @@ def fields(items: list[tuple[Name, v.Value]], key: str = "fields") -> Json: case v.Unit(_): return node("Unit", {}) case v.Hole(_, reason, expected_type): - members: dict[str, Json] = {"reason": reason} + members: dict[str, Json] = { + "reason": encode_hole_reason(reason, (*here, "reason")) + } if expected_type is not None: members["expectedType"] = encode_type( expected_type, (*here, "expectedType") @@ -1085,13 +1100,77 @@ def fields() -> list[tuple[Name, v.Value]]: case _: return v.Hole( attrs, - body["reason"], + decode_hole_reason(body["reason"], (*here, "reason")), decode_type(body["expectedType"], (*here, "expectedType")) if "expectedType" in body else None, ) +# --- Hole reasons and incompleteness --- + + +def encode_hole_reason(reason: inc.HoleReason, loc: Location = ()) -> Json: + """Encode a hole reason as ``{"Kind": {members}}``.""" + match reason: + case inc.UnresolvedReference(target): + return _wrap( + "UnresolvedReference", + {"target": encode_fqname(target, (*loc, "UnresolvedReference"))}, + ) + case inc.DeletedDuringRefactor(tx_id): + return _wrap("DeletedDuringRefactor", {"tx-id": tx_id}) + case inc.TypeMismatch(expected, found): + return _wrap("TypeMismatch", {"expected": expected, "found": found}) + + +def decode_hole_reason(j: object, loc: Location = ()) -> inc.HoleReason: + """Decode a hole reason; ``Draft`` is an incompleteness, not a reason.""" + tag, body, here = _node(j, loc, "HoleReason", HOLE_REASON_NODES) + match tag: + case "UnresolvedReference": + return inc.UnresolvedReference( + decode_fqname(body["target"], (*here, "target")) + ) + case "DeletedDuringRefactor": + return inc.DeletedDuringRefactor( + _expect_str(body["tx-id"], (*here, "tx-id"), tag) + ) + case _: + return inc.TypeMismatch( + _expect_str(body["expected"], (*here, "expected"), tag), + _expect_str(body["found"], (*here, "found"), tag), + ) + + +def encode_incompleteness(item: inc.Incompleteness, loc: Location = ()) -> Json: + """Encode an incompleteness as ``{"Hole": {...}}`` or ``{"Draft": {}}``.""" + match item: + case inc.HoleIncompleteness(reason, partial): + here = (*loc, "Hole") + members: dict[str, Json] = { + "reason": encode_hole_reason(reason, (*here, "reason")) + } + if partial is not None: + members["partialBody"] = encode_type(partial, (*here, "partialBody")) + return _wrap("Hole", members) + case inc.DraftIncompleteness(): + return _wrap("Draft", {}) + + +def decode_incompleteness(j: object, loc: Location = ()) -> inc.Incompleteness: + """Decode ``{"Hole": {...}}`` or ``{"Draft": {}}``.""" + tag, body, here = _node(j, loc, "Incompleteness", INCOMPLETENESS_NODES) + if tag == "Draft": + return inc.DraftIncompleteness() + return inc.HoleIncompleteness( + decode_hole_reason(body["reason"], (*here, "reason")), + decode_type(body["partialBody"], (*here, "partialBody")) + if "partialBody" in body + else None, + ) + + def _encode_inputs( inputs: list[tuple[Name, t.Type]], loc: Location, construct: str ) -> dict[str, Json]: @@ -1282,6 +1361,19 @@ def encode_type_definition(defn: td.Definition, loc: Location = ()) -> Json: "constructors": _encode_constructors(ctors.value, here), }, ) + case td.IncompleteTypeDefinition(_, incompleteness, partial): + here = (*loc, "IncompleteTypeDefinition") + members: dict[str, Json] = { + "typeParams": params, + "incompleteness": encode_incompleteness( + incompleteness, (*here, "incompleteness") + ), + } + if partial is not None: + members["partialTypeExp"] = encode_type( + partial, (*here, "partialTypeExp") + ) + return _wrap("IncompleteTypeDefinition", members) def decode_type_definition(j: object, loc: Location = ()) -> td.Definition: @@ -1291,6 +1383,14 @@ def decode_type_definition(j: object, loc: Location = ()) -> td.Definition: return td.TypeAliasDefinition( params, decode_type(body["typeExp"], (*here, "typeExp")) ) + if tag == "IncompleteTypeDefinition": + return td.IncompleteTypeDefinition( + params, + decode_incompleteness(body["incompleteness"], (*here, "incompleteness")), + decode_type(body["partialTypeExp"], (*here, "partialTypeExp")) + if "partialTypeExp" in body + else None, + ) access = body.get("access", "Public") if access not in ACCESS_TAGS: raise fail( @@ -1713,6 +1813,8 @@ def decode_versioned_distribution(document: str | object) -> Distribution: "FINOS_MORPHIR_PIN", "decode_distribution", "decode_fqname", + "decode_hole_reason", + "decode_incompleteness", "decode_literal", "decode_name", "decode_path", @@ -1722,6 +1824,8 @@ def decode_versioned_distribution(document: str | object) -> Distribution: "decode_versioned_distribution", "encode_distribution", "encode_fqname", + "encode_hole_reason", + "encode_incompleteness", "encode_literal", "encode_name", "encode_path", diff --git a/packages/morphir/src/morphir/ir/v4/incompleteness.py b/packages/morphir/src/morphir/ir/v4/incompleteness.py new file mode 100644 index 00000000..bb802cf5 --- /dev/null +++ b/packages/morphir/src/morphir/ir/v4/incompleteness.py @@ -0,0 +1,78 @@ +"""Incompleteness: why a type or value definition is not complete yet. + +Follows the finos/morphir v4 schema (``Incompleteness``, ``HoleReason``) at the +pinned commit and the morphir-rust v4 model. A hole names its reason; a draft is +work in progress and names none. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Union + +from .fqname import FQName +from .type import Type + + +@dataclass(frozen=True) +class UnresolvedReference: + """A reference that could not be resolved. + + Attributes: + target: The name that did not resolve. + """ + + target: FQName + + +@dataclass(frozen=True) +class DeletedDuringRefactor: + """A reference removed by a refactoring. + + Attributes: + tx_id: The transaction id of the refactoring (JSON member ``tx-id``). + """ + + tx_id: str + + +@dataclass(frozen=True) +class TypeMismatch: + """Type checking failed. + + Attributes: + expected: A description of the expected type. + found: A description of the type found. + """ + + expected: str + found: str + + +HoleReason = Union[UnresolvedReference, DeletedDuringRefactor, TypeMismatch] +"""Why a hole exists. ``Draft`` is an incompleteness, not a hole reason.""" + + +@dataclass(frozen=True) +class HoleIncompleteness: + """A definition with a hole. + + Attributes: + reason: Why the hole exists. + partial_body: The type the author had written when the definition stopped + being complete, if any. The schema gives this member the type + ``Type``; an incomplete value keeps its partial expression on + `IncompleteBody.partial_body` instead. + """ + + reason: HoleReason + partial_body: Type | None = None + + +@dataclass(frozen=True) +class DraftIncompleteness: + """A definition the author marked as work in progress.""" + + +Incompleteness = Union[HoleIncompleteness, DraftIncompleteness] +"""Why a definition is incomplete: a hole or a draft.""" diff --git a/packages/morphir/src/morphir/ir/v4/type_def.py b/packages/morphir/src/morphir/ir/v4/type_def.py index 90c4ba58..d86d05b2 100644 --- a/packages/morphir/src/morphir/ir/v4/type_def.py +++ b/packages/morphir/src/morphir/ir/v4/type_def.py @@ -2,6 +2,7 @@ from typing import Union from .access_controlled import AccessControlled +from .incompleteness import Incompleteness from .name import Name from .type import Type from .type_spec import Constructors @@ -19,4 +20,19 @@ class CustomTypeDefinition: constructors: AccessControlled[Constructors] -Definition = Union[TypeAliasDefinition, CustomTypeDefinition] +@dataclass(frozen=True) +class IncompleteTypeDefinition: + """A type definition that is not complete yet. + + Attributes: + type_params: The declared type parameters. + incompleteness: Why the definition is incomplete. + partial_type_exp: The type expression written so far, if any. + """ + + type_params: list[Name] + incompleteness: Incompleteness + partial_type_exp: Type | None = None + + +Definition = Union[TypeAliasDefinition, CustomTypeDefinition, IncompleteTypeDefinition] diff --git a/packages/morphir/src/morphir/ir/v4/value.py b/packages/morphir/src/morphir/ir/v4/value.py index fc07831d..0899e3ab 100644 --- a/packages/morphir/src/morphir/ir/v4/value.py +++ b/packages/morphir/src/morphir/ir/v4/value.py @@ -6,6 +6,7 @@ from typing import Tuple as TTuple from .fqname import FQName +from .incompleteness import HoleReason from .literal import Literal from .name import Name from .type import Type @@ -266,10 +267,16 @@ class Unit: @dataclass(frozen=True) class Hole: - """Represents a hole in the AST.""" + """A hole in an expression: a part that is missing, with the reason why. + + Attributes: + attributes: The value attributes. + reason: Why the hole exists. + expected_type: The type the missing part should have, if known. + """ attributes: ValueAttributes - reason: Any + reason: HoleReason expected_type: Type | None diff --git a/tests/unit/ir/test_migrate.py b/tests/unit/ir/test_migrate.py index c0d4a937..18f1c5ca 100644 --- a/tests/unit/ir/test_migrate.py +++ b/tests/unit/ir/test_migrate.py @@ -16,6 +16,7 @@ from morphir.ir.v3 import value as val3 from morphir.ir.v4 import codec as v4_codec from morphir.ir.v4 import document as doc4 +from morphir.ir.v4 import incompleteness as inc4 from morphir.ir.v4 import literal as lit4 from morphir.ir.v4 import module as mod4 from morphir.ir.v4 import package as pkg4 @@ -166,7 +167,7 @@ def test_specs_and_application_do_not_downgrade( @pytest.mark.parametrize( "node", [ - v4.Hole(TYPED, None, None), + v4.Hole(TYPED, inc4.DeletedDuringRefactor("tx"), None), v4.Native(TYPED, FQName(PKG, IRPath(()), n("f")), None), v4.External(TYPED, "f", "js"), v4.LiteralValue(TYPED, lit4.DocumentLiteral(doc4.DocNull())), @@ -226,7 +227,9 @@ def test_missing_member_documentation(self) -> None: def test_every_failure_is_collected_in_one_pass(self) -> None: body = v4.Apply( - TYPED, v4.Hole(TYPED, None, None), v4.Unit(v4.ValueAttributes()) + TYPED, + v4.Hole(TYPED, inc4.DeletedDuringRefactor("tx"), None), + v4.Unit(v4.ValueAttributes()), ) found = errors(v4_to_v3(library(body, value_doc=None))) assert {code for code, _ in found} == { diff --git a/tests/unit/ir/test_v4_codec.py b/tests/unit/ir/test_v4_codec.py index af9481f9..01484bfb 100644 --- a/tests/unit/ir/test_v4_codec.py +++ b/tests/unit/ir/test_v4_codec.py @@ -12,6 +12,7 @@ from morphir.ir.diagnostics import DiagnosticError from morphir.ir.v4 import codec, module, package from morphir.ir.v4 import document as doc +from morphir.ir.v4 import incompleteness as inc from morphir.ir.v4 import literal as lit from morphir.ir.v4 import type as t from morphir.ir.v4 import type_def as td @@ -303,9 +304,9 @@ def test_attributes_and_hole(self) -> None: tpe = t.Unit(t.TypeAttributes(source=src_t, extensions=ext)) assert codec.decode_type(codec.encode_type(tpe)) == tpe attrs = v.ValueAttributes(source=src_v, inferred_type=tpe, extensions=ext) - hole = v.Hole(attrs, {"TypeMismatch": {"expected": "a", "found": "b"}}, INT) + hole = v.Hole(attrs, inc.TypeMismatch("a", "b"), INT) assert codec.decode_value(codec.encode_value(hole)) == hole - bare = v.Hole(v.ValueAttributes(), {"Draft": {}}, None) + bare = v.Hole(v.ValueAttributes(), inc.DeletedDuringRefactor("tx"), None) assert codec.decode_value(codec.encode_value(bare)) == bare def test_document_literal(self) -> None: diff --git a/tests/unit/ir/test_v4_upstream.py b/tests/unit/ir/test_v4_upstream.py new file mode 100644 index 00000000..6e51634c --- /dev/null +++ b/tests/unit/ir/test_v4_upstream.py @@ -0,0 +1,168 @@ +"""Tests for the upstream v4 constructs (change align-ir-v4-with-upstream). + +Expected JSON is the expanded form of the finos/morphir kit cases at the pinned +commit; the case id is named in each test. +""" + +from __future__ import annotations + +import pytest + +from morphir.ir import diagnostics as d +from morphir.ir import v4 +from morphir.ir.diagnostics import DiagnosticError +from morphir.ir.migrate import Err, v4_to_v3 +from morphir.ir.v4 import codec, module, package +from morphir.ir.v4 import incompleteness as inc +from morphir.ir.v4 import type as t +from morphir.ir.v4 import type_def as td +from morphir.ir.v4 import value as v +from morphir.ir.v4.access_controlled import Public +from morphir.ir.v4.distribution import LibraryDistribution, PackageInfo +from morphir.ir.v4.documented import Documented +from morphir.ir.v4.fqname import FQName +from morphir.ir.v4.name import Name +from morphir.ir.v4.path import Path + + +def n(*words: str) -> Name: + return Name(words) + + +def p(*names: Name) -> Path: + return Path(names) + + +TA = t.TypeAttributes() +SDK = p(n("morphir"), n("s", "d", "k")) +INT = t.Reference(TA, FQName(SDK, p(n("basics")), n("int"))) +STRING = t.Reference(TA, FQName(SDK, p(n("string")), n("string"))) +INT_JSON = {"Reference": {"fqname": "morphir/SDK:basics#int"}} +STRING_JSON = {"Reference": {"fqname": "morphir/SDK:string#string"}} +MISSING = FQName(p(n("my", "org"), n("project")), p(n("module")), n("missing")) +TYPED = v.ValueAttributes(inferred_type=INT) + + +def fail_codes(fn, *args) -> tuple[str, ...]: # type: ignore[no-untyped-def] + with pytest.raises(DiagnosticError) as info: + fn(*args) + return info.value.codes + + +def library( + *, + types: dict[Name, td.Definition] | None = None, + values: dict[Name, v.ValueDefinitionBody] | None = None, +) -> LibraryDistribution: + mod = module.Definition( + types={k: Public(Documented("doc", x)) for k, x in (types or {}).items()}, + values={k: Public(Documented("doc", x)) for k, x in (values or {}).items()}, + ) + return LibraryDistribution( + PackageInfo(p(n("my"), n("pkg"))), + package.Definition({p(n("app")): Public(mod)}), + ) + + +def downgrade_errors(lib: LibraryDistribution) -> list[tuple[str, str, str]]: + result = v4_to_v3(lib) + assert isinstance(result, Err) + return [ + (x.code, x.construct, "/".join(map(str, x.location))) + for x in result.diagnostics + ] + + +class TestIncompleteness: + def test_exported_from_v4(self) -> None: + assert v4.Incompleteness is inc.Incompleteness + assert v4.HoleReason is inc.HoleReason + + def test_type_mismatch_hole_definitions_0014(self) -> None: + defn = td.IncompleteTypeDefinition( + [n("a")], + inc.HoleIncompleteness(inc.TypeMismatch("morphir/SDK:basics#int", "s")), + ) + expected = { + "IncompleteTypeDefinition": { + "typeParams": ["a"], + "incompleteness": { + "Hole": { + "reason": { + "TypeMismatch": { + "expected": "morphir/SDK:basics#int", + "found": "s", + } + } + } + }, + } + } + assert codec.encode_type_definition(defn) == expected + assert codec.decode_type_definition(expected) == defn + + def test_draft_with_partial_type_definitions_0015(self) -> None: + defn = td.IncompleteTypeDefinition([], inc.DraftIncompleteness(), INT) + expected = { + "IncompleteTypeDefinition": { + "typeParams": [], + "incompleteness": {"Draft": {}}, + "partialTypeExp": INT_JSON, + } + } + assert codec.encode_type_definition(defn) == expected + assert codec.decode_type_definition(expected) == defn + + def test_hole_keeps_partial_body_definitions_0024(self) -> None: + defn = td.IncompleteTypeDefinition( + [], inc.HoleIncompleteness(inc.UnresolvedReference(MISSING), INT) + ) + expected = { + "IncompleteTypeDefinition": { + "typeParams": [], + "incompleteness": { + "Hole": { + "reason": { + "UnresolvedReference": { + "target": "my-org/project:module#missing" + } + }, + "partialBody": INT_JSON, + } + }, + } + } + assert codec.encode_type_definition(defn) == expected + assert codec.decode_type_definition(expected) == defn + + def test_deleted_during_refactor_reason(self) -> None: + hole = v.Hole(TYPED, inc.DeletedDuringRefactor("tx-42"), INT) + encoded = codec.encode_value(hole) + assert encoded["Hole"]["reason"] == { + "DeletedDuringRefactor": {"tx-id": "tx-42"} + } + assert codec.decode_value(encoded) == hole + + def test_draft_is_not_a_hole_reason_definitions_0026(self) -> None: + bad = { + "IncompleteTypeDefinition": { + "typeParams": [], + "incompleteness": {"Hole": {"reason": {"Draft": {}}}}, + } + } + assert fail_codes(codec.decode_type_definition, bad) == (d.UNKNOWN_NODE,) + + def test_reason_must_be_an_object(self) -> None: + bad = {"Hole": {"reason": "Draft"}} + assert fail_codes(codec.decode_value, bad) == (d.INVALID_JSON,) + + def test_incomplete_type_definition_does_not_downgrade(self) -> None: + defn = td.IncompleteTypeDefinition([], inc.DraftIncompleteness()) + errors = downgrade_errors(library(types={n("t"): defn})) + assert errors == [ + ( + d.UNSUPPORTED_V4_DOWNGRADE, + "IncompleteTypeDefinition", + "distribution/modules/app/types/t/IncompleteTypeDefinition", + ) + ] From c0ff980c1bab2cd85121297d6f3d6bf676e5a05d Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 02:06:24 -0500 Subject: [PATCH 4/9] feat(ir): add native, external and incomplete value-definition bodies to v4 Add NativeBody (with NativeInfo and the five native hints), ExternalBody (platform bindings and an optional fallback) and IncompleteBody beside the expression body, after the finos/morphir v4 schema at efa36060. Module values and let bindings hold any of the four (ValueDefinitionBody). The codec reads and writes them in expanded form with the kit refusals (definitions-0007 to -0009, -0016, -0025, -0030); reverse migration refuses them with unsupported_v4_downgrade, and drop_input_attributes walks their expressions. Refs: morphir-ukvn.8.14.1 --- .../align-ir-v4-with-upstream/tasks.md | 10 +- packages/morphir/src/morphir/ir/migrate.py | 35 ++- .../morphir/src/morphir/ir/v4/__init__.py | 20 +- packages/morphir/src/morphir/ir/v4/codec.py | 199 +++++++++++++++++- packages/morphir/src/morphir/ir/v4/module.py | 2 +- packages/morphir/src/morphir/ir/v4/value.py | 132 +++++++++++- tests/unit/ir/test_v4_codec.py | 6 +- tests/unit/ir/test_v4_upstream.py | 194 +++++++++++++++++ 8 files changed, 573 insertions(+), 25 deletions(-) diff --git a/openspec/changes/align-ir-v4-with-upstream/tasks.md b/openspec/changes/align-ir-v4-with-upstream/tasks.md index e5d88ac3..2f66834e 100644 --- a/openspec/changes/align-ir-v4-with-upstream/tasks.md +++ b/openspec/changes/align-ir-v4-with-upstream/tasks.md @@ -16,11 +16,11 @@ ## 3. Value-definition bodies (morphir-ukvn.8.14.1) -- [ ] 3.1 Add `NativeHint` kinds, `NativeInfo`, `ExternalBinding`, `NativeBody`, `ExternalBody`, `IncompleteBody` and `ValueDefinitionBody` to `morphir.ir.v4.value`, widen module values and let bindings to `ValueDefinitionBody`, and verify the type checkers pass -- [ ] 3.2 Encode and decode the three bodies with the kit members, and verify round trips for kit `definitions-0007` to `-0009`, `-0016` and `-0025` in expanded form -- [ ] 3.3 Refuse duplicate target platforms (`duplicate_member`), an empty `externals` list, and a `PlatformSpecific` hint without `platform` (`missing_member`), and verify one test each -- [ ] 3.4 Make reverse migration fail with `unsupported_v4_downgrade` for each new body at module level and in a let binding, and verify the locations -- [ ] 3.5 Make `drop_input_attributes` and the expression-body checks skip the new bodies, which hold bare input types, and verify a Library that mixes bodies encodes +- [x] 3.1 Add `NativeHint` kinds, `NativeInfo`, `ExternalBinding`, `NativeBody`, `ExternalBody`, `IncompleteBody` and `ValueDefinitionBody` to `morphir.ir.v4.value`, widen module values and let bindings to `ValueDefinitionBody`, and verify the type checkers pass +- [x] 3.2 Encode and decode the three bodies with the kit members, and verify round trips for kit `definitions-0007` to `-0009`, `-0016` and `-0025` in expanded form +- [x] 3.3 Refuse duplicate target platforms (`duplicate_member`), an empty `externals` list, and a `PlatformSpecific` hint without `platform` (`missing_member`), and verify one test each +- [x] 3.4 Make reverse migration fail with `unsupported_v4_downgrade` for each new body at module level and in a let binding, and verify the locations +- [x] 3.5 Make `drop_input_attributes` and the expression-body checks skip the new bodies, which hold bare input types, and verify a Library that mixes bodies encodes ## 4. Annotations on specifications (morphir-ukvn.8.14.3) diff --git a/packages/morphir/src/morphir/ir/migrate.py b/packages/morphir/src/morphir/ir/migrate.py index bd6cea01..bcd92281 100644 --- a/packages/morphir/src/morphir/ir/migrate.py +++ b/packages/morphir/src/morphir/ir/migrate.py @@ -382,7 +382,32 @@ class _InputDrop: def __init__(self) -> None: self.warnings: list[Diagnostic] = [] - def definition(self, defn: v4.Definition, loc: Location) -> v4.Definition: + def definition( + self, defn: v4.ValueDefinitionBody, loc: Location + ) -> v4.ValueDefinitionBody: + match defn: + case v4.Definition(): + return self.expression(defn, loc) + case v4.ExternalBody(inputs, output, externals, fallback): + here = (*loc, "ExternalBody", "body") + return v4.ExternalBody( + inputs, + output, + externals, + None if fallback is None else self.value(fallback, here), + ) + case v4.IncompleteBody(inputs, incompleteness, output, partial): + here = (*loc, "IncompleteBody", "partial_body") + return v4.IncompleteBody( + inputs, + incompleteness, + output, + None if partial is None else self.value(partial, here), + ) + case v4.NativeBody(): + return defn + + def expression(self, defn: v4.Definition, loc: Location) -> v4.Definition: inputs: list[tuple[name4.Name, v4.ValueAttributes, t4.Type]] = [] for n, attrs, tpe in defn.input_types: declared = v4.ValueAttributes(inferred_type=tpe) @@ -919,7 +944,13 @@ def value(self, val: v4.Value, loc: Location) -> TypedValue3: case _: return v3.Unit(a) - def value_def(self, defn: v4.Definition, loc: Location) -> TypedDefinition3: + def value_def( + self, defn: v4.ValueDefinitionBody, loc: Location + ) -> TypedDefinition3: + if not isinstance(defn, v4.Definition): + construct = type(defn).__name__ + self.unsupported(construct, (*loc, construct)) + return v3.Definition((), t3.Unit(None), v3.Unit(t3.Unit(None))) inputs = [] for n, va, tpe in defn.input_types: here = (*loc, "input_types", name4.to_string(n)) diff --git a/packages/morphir/src/morphir/ir/v4/__init__.py b/packages/morphir/src/morphir/ir/v4/__init__.py index 4887f359..a47328a3 100644 --- a/packages/morphir/src/morphir/ir/v4/__init__.py +++ b/packages/morphir/src/morphir/ir/v4/__init__.py @@ -61,7 +61,18 @@ from .type_def import Definition as TypeDefinition from .type_spec import Specification as TypeSpecification from .value import Definition as ValueDefinition -from .value import Pattern, Value, ValueAttributes +from .value import ( + ExternalBinding, + ExternalBody, + IncompleteBody, + NativeBody, + NativeHint, + NativeInfo, + Pattern, + Value, + ValueAttributes, + ValueDefinitionBody, +) from .value import Specification as ValueSpecification __all__ = [ @@ -87,6 +98,8 @@ "DraftIncompleteness", "EntryPoint", "EntryPointKind", + "ExternalBinding", + "ExternalBody", "FQName", "Field", "FileMeta", @@ -94,12 +107,16 @@ "FloatLiteral", "HoleIncompleteness", "HoleReason", + "IncompleteBody", "Incompleteness", "IntegerLiteral", "LayerManifest", "LibraryDistribution", "Literal", "Name", + "NativeBody", + "NativeHint", + "NativeInfo", "PackageInfo", "Path", "Pattern", @@ -120,6 +137,7 @@ "Value", "ValueAttributes", "ValueDefinition", + "ValueDefinitionBody", "ValueSpecification", "incompleteness", "module", diff --git a/packages/morphir/src/morphir/ir/v4/codec.py b/packages/morphir/src/morphir/ir/v4/codec.py index cdad8bb1..fbdf36df 100644 --- a/packages/morphir/src/morphir/ir/v4/codec.py +++ b/packages/morphir/src/morphir/ir/v4/codec.py @@ -164,6 +164,17 @@ def _m(required: str = "", optional: str = "attributes") -> Members: VALUE_DEFINITION_NODES: dict[str, Members] = { "ExpressionBody": _m("inputTypes outputType body", ""), + "NativeBody": _m("inputTypes outputType nativeInfo", ""), + "ExternalBody": _m("inputTypes outputType externals", "body"), + "IncompleteBody": _m("inputTypes incompleteness", "outputType partialBody"), +} + +NATIVE_HINT_NODES: dict[str, Members] = { + "Arithmetic": _m("", ""), + "Comparison": _m("", ""), + "StringOp": _m("", ""), + "CollectionOp": _m("", ""), + "PlatformSpecific": _m("platform", ""), } DISTRIBUTION_NODES: dict[str, Members] = { @@ -1177,8 +1188,7 @@ def _encode_inputs( return _keyed(inputs, _name_key, encode_type, loc, construct) -def encode_value_definition(defn: v.Definition, loc: Location = ()) -> Json: - """Encode a value definition as ``{"ExpressionBody": {...}}``.""" +def _encode_expression_body(defn: v.Definition, loc: Location) -> Json: here = (*loc, "ExpressionBody") for i, (name, attrs, tpe) in enumerate(defn.input_types): derived = v.ValueAttributes(inferred_type=tpe) @@ -1205,9 +1215,144 @@ def encode_value_definition(defn: v.Definition, loc: Location = ()) -> Json: } -def decode_value_definition(j: object, loc: Location = ()) -> v.Definition: - """Decode ``{"ExpressionBody": {...}}``; other bodies are not supported.""" - _, body, here = _node(j, loc, "ValueDefinition", VALUE_DEFINITION_NODES) +def _encode_native_hint(hint: v.NativeHint) -> Json: + if isinstance(hint, v.PlatformSpecific): + return _wrap("PlatformSpecific", {"platform": hint.platform}) + return _wrap(type(hint).__name__, {}) + + +def _decode_native_hint(j: object, loc: Location) -> v.NativeHint: + tag, body, here = _node(j, loc, "NativeHint", NATIVE_HINT_NODES) + match tag: + case "Arithmetic": + return v.Arithmetic() + case "Comparison": + return v.Comparison() + case "StringOp": + return v.StringOp() + case "CollectionOp": + return v.CollectionOp() + case _: + return v.PlatformSpecific( + _expect_str(body["platform"], (*here, "platform"), tag) + ) + + +def _encode_externals(bindings: list[v.ExternalBinding], loc: Location) -> list[Json]: + if not bindings: + raise fail( + d.INVALID_JSON, + "ExternalBody", + loc, + "an external body needs at least one platform binding", + ) + seen: set[str] = set() + result: list[Json] = [] + for i, binding in enumerate(bindings): + if binding.target_platform in seen: + raise fail( + d.DUPLICATE_MEMBER, + "ExternalBinding", + (*loc, i, "targetPlatform"), + f"target platform {binding.target_platform!r} is bound more than once", + ) + seen.add(binding.target_platform) + result.append( + { + "targetPlatform": binding.target_platform, + "externalName": binding.external_name, + } + ) + return result + + +def _decode_externals(j: object, loc: Location) -> list[v.ExternalBinding]: + items = _expect_list(j, loc, "ExternalBody") + if not items: + raise fail( + d.INVALID_JSON, + "ExternalBody", + loc, + "an external body needs at least one platform binding", + ) + bindings: list[v.ExternalBinding] = [] + for i, item in enumerate(items): + here = (*loc, i) + obj = _expect_object(item, here, "ExternalBinding") + _check_members( + obj, _m("targetPlatform externalName", ""), here, "ExternalBinding" + ) + bindings.append( + v.ExternalBinding( + _expect_str( + obj["targetPlatform"], (*here, "targetPlatform"), "ExternalBinding" + ), + _expect_str( + obj["externalName"], (*here, "externalName"), "ExternalBinding" + ), + ) + ) + # Re-encode to apply the same uniqueness rule as the writer. + _encode_externals(bindings, loc) + return bindings + + +def encode_value_definition(defn: v.ValueDefinitionBody, loc: Location = ()) -> Json: + """Encode a value definition as ``{"Body": {...}}``. + + Raises: + DiagnosticError: With ``no_v4_spelling`` when an expression-body input + carries attributes other than its declared type, ``duplicate_member`` + when an external body binds a platform twice, and ``invalid_json`` + when it binds none. + """ + match defn: + case v.Definition(): + return _encode_expression_body(defn, loc) + case v.NativeBody(inputs, output, info): + here = (*loc, "NativeBody") + return _wrap( + "NativeBody", + { + "inputTypes": _encode_inputs( + inputs, (*here, "inputTypes"), "NativeBody" + ), + "outputType": encode_type(output, (*here, "outputType")), + "nativeInfo": {"hint": _encode_native_hint(info.hint)}, + }, + ) + case v.ExternalBody(inputs, output, externals, fallback): + here = (*loc, "ExternalBody") + members: dict[str, Json] = { + "inputTypes": _encode_inputs( + inputs, (*here, "inputTypes"), "ExternalBody" + ), + "outputType": encode_type(output, (*here, "outputType")), + "externals": _encode_externals(externals, (*here, "externals")), + } + if fallback is not None: + members["body"] = encode_value(fallback, (*here, "body")) + return _wrap("ExternalBody", members) + case v.IncompleteBody(inputs, incompleteness, output, partial): + here = (*loc, "IncompleteBody") + members = { + "inputTypes": _encode_inputs( + inputs, (*here, "inputTypes"), "IncompleteBody" + ) + } + if output is not None: + members["outputType"] = encode_type(output, (*here, "outputType")) + members["incompleteness"] = encode_incompleteness( + incompleteness, (*here, "incompleteness") + ) + if partial is not None: + members["partialBody"] = encode_value(partial, (*here, "partialBody")) + return _wrap("IncompleteBody", members) + + +def decode_value_definition(j: object, loc: Location = ()) -> v.ValueDefinitionBody: + """Decode an expression, native, external or incomplete body.""" + tag, body, here = _node(j, loc, "ValueDefinition", VALUE_DEFINITION_NODES) inputs = _unkeyed( body["inputTypes"], decode_name, @@ -1215,11 +1360,45 @@ def decode_value_definition(j: object, loc: Location = ()) -> v.Definition: (*here, "inputTypes"), "inputTypes", ) - return v.Definition( - [(n, v.ValueAttributes(inferred_type=tp), tp) for n, tp in inputs], - decode_type(body["outputType"], (*here, "outputType")), - decode_value(body["body"], (*here, "body")), - ) + + def optional_type(key: str) -> t.Type | None: + return decode_type(body[key], (*here, key)) if key in body else None + + def optional_value(key: str) -> v.Value | None: + return decode_value(body[key], (*here, key)) if key in body else None + + match tag: + case "ExpressionBody": + return v.Definition( + [(n, v.ValueAttributes(inferred_type=tp), tp) for n, tp in inputs], + decode_type(body["outputType"], (*here, "outputType")), + decode_value(body["body"], (*here, "body")), + ) + case "NativeBody": + info_loc = (*here, "nativeInfo") + info = _expect_object(body["nativeInfo"], info_loc, "NativeInfo") + _check_members(info, _m("hint", ""), info_loc, "NativeInfo") + return v.NativeBody( + inputs, + decode_type(body["outputType"], (*here, "outputType")), + v.NativeInfo(_decode_native_hint(info["hint"], (*info_loc, "hint"))), + ) + case "ExternalBody": + return v.ExternalBody( + inputs, + decode_type(body["outputType"], (*here, "outputType")), + _decode_externals(body["externals"], (*here, "externals")), + optional_value("body"), + ) + case _: + return v.IncompleteBody( + inputs, + decode_incompleteness( + body["incompleteness"], (*here, "incompleteness") + ), + optional_type("outputType"), + optional_value("partialBody"), + ) def encode_value_specification(spec: v.Specification, loc: Location = ()) -> Json: diff --git a/packages/morphir/src/morphir/ir/v4/module.py b/packages/morphir/src/morphir/ir/v4/module.py index c076e504..e2b5475d 100644 --- a/packages/morphir/src/morphir/ir/v4/module.py +++ b/packages/morphir/src/morphir/ir/v4/module.py @@ -22,7 +22,7 @@ class Definition: types: dict[Name, AccessControlled[Documented[type_def.Definition]]] = field( default_factory=dict ) - values: dict[Name, AccessControlled[Documented[value.Definition]]] = field( + values: dict[Name, AccessControlled[Documented[value.ValueDefinitionBody]]] = field( default_factory=dict ) doc: str | None = None diff --git a/packages/morphir/src/morphir/ir/v4/value.py b/packages/morphir/src/morphir/ir/v4/value.py index 0899e3ab..cda41351 100644 --- a/packages/morphir/src/morphir/ir/v4/value.py +++ b/packages/morphir/src/morphir/ir/v4/value.py @@ -6,7 +6,7 @@ from typing import Tuple as TTuple from .fqname import FQName -from .incompleteness import HoleReason +from .incompleteness import HoleReason, Incompleteness from .literal import Literal from .name import Name from .type import Type @@ -207,7 +207,7 @@ class LetDefinition: attributes: ValueAttributes name: Name - definition: Definition + definition: ValueDefinitionBody in_value: Value @@ -216,7 +216,7 @@ class LetRecursion: """Represents a recursive let binding.""" attributes: ValueAttributes - definitions: dict[Name, Definition] + definitions: dict[Name, ValueDefinitionBody] in_value: Value @@ -335,8 +335,132 @@ class Specification: @dataclass(frozen=True) class Definition: - """Value definition.""" + """A value definition with an expression body (``ExpressionBody``). + + Attributes: + input_types: Each input's name, value attributes and declared type. v4 + JSON spells an input as a bare type, so attributes other than the + declared type cannot be encoded; see + `morphir.ir.migrate.drop_input_attributes`. + output_type: The declared output type. + body: The expression. + """ input_types: TList[TTuple[Name, ValueAttributes, Type]] output_type: Type body: Value + + +# --- Native and external definition bodies --- + + +@dataclass(frozen=True) +class Arithmetic: + """Native hint: a basic arithmetic or logic operation.""" + + +@dataclass(frozen=True) +class Comparison: + """Native hint: a comparison operation.""" + + +@dataclass(frozen=True) +class StringOp: + """Native hint: a string operation.""" + + +@dataclass(frozen=True) +class CollectionOp: + """Native hint: a collection operation such as map, filter or fold.""" + + +@dataclass(frozen=True) +class PlatformSpecific: + """Native hint: an operation specific to one platform. + + Attributes: + platform: The platform, such as ``"wasm"`` or ``"javascript"``. + """ + + platform: str + + +NativeHint = Union[Arithmetic, Comparison, StringOp, CollectionOp, PlatformSpecific] +"""What kind of operation a native definition performs.""" + + +@dataclass(frozen=True) +class NativeInfo: + """Information about a native definition. + + Attributes: + hint: The kind of operation. + """ + + hint: NativeHint + + +@dataclass(frozen=True) +class ExternalBinding: + """One target platform's binding for an external definition. + + Attributes: + target_platform: The platform. Unique within an external body. + external_name: The name of the operation on that platform. + """ + + target_platform: str + external_name: str + + +@dataclass(frozen=True) +class NativeBody: + """A value definition implemented natively by each runtime (``NativeBody``). + + Attributes: + input_types: Each input's name and declared type. + output_type: The declared output type. + native_info: What kind of operation it is. + """ + + input_types: TList[TTuple[Name, Type]] + output_type: Type + native_info: NativeInfo + + +@dataclass(frozen=True) +class ExternalBody: + """A value definition bound to external operations (``ExternalBody``). + + Attributes: + input_types: Each input's name and declared type. + output_type: The declared output type. + externals: One or more platform bindings, unique by platform. + body: An optional fallback expression for other platforms. + """ + + input_types: TList[TTuple[Name, Type]] + output_type: Type + externals: TList[ExternalBinding] + body: Value | None = None + + +@dataclass(frozen=True) +class IncompleteBody: + """A value definition that is not complete yet (``IncompleteBody``). + + Attributes: + input_types: Each input's name and declared type. + incompleteness: Why the definition is incomplete. + output_type: The declared output type, if known yet. + partial_body: The expression written so far, if any. + """ + + input_types: TList[TTuple[Name, Type]] + incompleteness: Incompleteness + output_type: Type | None = None + partial_body: Value | None = None + + +ValueDefinitionBody = Union[Definition, NativeBody, ExternalBody, IncompleteBody] +"""A v4 value definition: an expression, native, external or incomplete body.""" diff --git a/tests/unit/ir/test_v4_codec.py b/tests/unit/ir/test_v4_codec.py index 01484bfb..a355027e 100644 --- a/tests/unit/ir/test_v4_codec.py +++ b/tests/unit/ir/test_v4_codec.py @@ -403,9 +403,11 @@ def test_v3_is_refused(self) -> None: codec.decode_versioned_distribution, {"formatVersion": 3} ) == (d.UNSUPPORTED_FORMAT_VERSION_MAJOR,) - def test_other_definition_bodies_are_unknown(self) -> None: + def test_unknown_definition_body(self) -> None: + other = {"MacroBody": {"inputTypes": {}, "outputType": {"Unit": {}}}} + assert fail_codes(codec.decode_value_definition, other) == (d.UNKNOWN_NODE,) native = {"NativeBody": {"inputTypes": {}, "outputType": {"Unit": {}}}} - assert fail_codes(codec.decode_value_definition, native) == (d.UNKNOWN_NODE,) + assert fail_codes(codec.decode_value_definition, native) == (d.MISSING_MEMBER,) @pytest.mark.parametrize( "bad", diff --git a/tests/unit/ir/test_v4_upstream.py b/tests/unit/ir/test_v4_upstream.py index 6e51634c..2af6fa1b 100644 --- a/tests/unit/ir/test_v4_upstream.py +++ b/tests/unit/ir/test_v4_upstream.py @@ -166,3 +166,197 @@ def test_incomplete_type_definition_does_not_downgrade(self) -> None: "distribution/modules/app/types/t/IncompleteTypeDefinition", ) ] + + +INPUTS_INT = [(n("a"), INT), (n("b"), INT)] +X_JSON = {"Variable": {"attributes": {"inferredType": INT_JSON}, "name": "x"}} + + +class TestDefinitionBodies: + def test_exported_from_v4(self) -> None: + assert v4.ValueDefinitionBody is v.ValueDefinitionBody + assert v4.NativeBody is v.NativeBody + + def test_native_body_definitions_0009(self) -> None: + body = v.NativeBody(INPUTS_INT, INT, v.NativeInfo(v.Arithmetic())) + expected = { + "NativeBody": { + "inputTypes": {"a": INT_JSON, "b": INT_JSON}, + "outputType": INT_JSON, + "nativeInfo": {"hint": {"Arithmetic": {}}}, + } + } + assert codec.encode_value_definition(body) == expected + assert codec.decode_value_definition(expected) == body + + @pytest.mark.parametrize( + "hint", + [ + v.Comparison(), + v.StringOp(), + v.CollectionOp(), + v.PlatformSpecific("wasm"), + ], + ids=lambda h: type(h).__name__, + ) + def test_native_hints_round_trip(self, hint: v.NativeHint) -> None: + body = v.NativeBody([], INT, v.NativeInfo(hint)) + assert ( + codec.decode_value_definition(codec.encode_value_definition(body)) == body + ) + + def test_platform_specific_names_its_platform_definitions_0030(self) -> None: + bad = { + "NativeBody": { + "inputTypes": {}, + "outputType": INT_JSON, + "nativeInfo": {"hint": {"PlatformSpecific": {}}}, + } + } + assert fail_codes(codec.decode_value_definition, bad) == (d.MISSING_MEMBER,) + + def test_external_body_definitions_0007(self) -> None: + body = v.ExternalBody( + [(n("msg"), STRING)], + INT, + [v.ExternalBinding("javascript", "console.log")], + ) + expected = { + "ExternalBody": { + "inputTypes": {"msg": STRING_JSON}, + "outputType": INT_JSON, + "externals": [ + {"targetPlatform": "javascript", "externalName": "console.log"} + ], + } + } + assert codec.encode_value_definition(body) == expected + assert codec.decode_value_definition(expected) == body + + def test_external_body_with_fallback_definitions_0008(self) -> None: + body = v.ExternalBody( + [(n("x"), INT)], + INT, + [ + v.ExternalBinding("erlang", "math:abs"), + v.ExternalBinding("javascript", "Math.abs"), + ], + v.Variable(TYPED, n("x")), + ) + encoded = codec.encode_value_definition(body) + assert encoded["ExternalBody"]["body"] == X_JSON + assert codec.decode_value_definition(encoded) == body + + def test_duplicate_platform_is_refused(self) -> None: + bindings = [v.ExternalBinding("js", "a"), v.ExternalBinding("js", "b")] + body = v.ExternalBody([], INT, bindings) + assert fail_codes(codec.encode_value_definition, body) == (d.DUPLICATE_MEMBER,) + bad = { + "ExternalBody": { + "inputTypes": {}, + "outputType": INT_JSON, + "externals": [ + {"targetPlatform": "js", "externalName": "a"}, + {"targetPlatform": "js", "externalName": "b"}, + ], + } + } + assert fail_codes(codec.decode_value_definition, bad) == (d.DUPLICATE_MEMBER,) + + def test_external_body_needs_a_binding(self) -> None: + body = v.ExternalBody([], INT, []) + assert fail_codes(codec.encode_value_definition, body) == (d.INVALID_JSON,) + bad = { + "ExternalBody": {"inputTypes": {}, "outputType": INT_JSON, "externals": []} + } + assert fail_codes(codec.decode_value_definition, bad) == (d.INVALID_JSON,) + + def test_incomplete_body_definitions_0016(self) -> None: + body = v.IncompleteBody( + [(n("x"), INT)], + inc.HoleIncompleteness(inc.DeletedDuringRefactor("tx-42")), + INT, + ) + expected = { + "IncompleteBody": { + "inputTypes": {"x": INT_JSON}, + "outputType": INT_JSON, + "incompleteness": { + "Hole": {"reason": {"DeletedDuringRefactor": {"tx-id": "tx-42"}}} + }, + } + } + assert codec.encode_value_definition(body) == expected + assert codec.decode_value_definition(expected) == body + + def test_incomplete_body_keeps_partial_value_definitions_0025(self) -> None: + body = v.IncompleteBody( + [], inc.DraftIncompleteness(), INT, v.Variable(TYPED, n("x")) + ) + encoded = codec.encode_value_definition(body) + assert encoded["IncompleteBody"]["partialBody"] == X_JSON + assert codec.decode_value_definition(encoded) == body + no_output = v.IncompleteBody([], inc.DraftIncompleteness()) + encoded = codec.encode_value_definition(no_output) + assert "outputType" not in encoded["IncompleteBody"] + assert codec.decode_value_definition(encoded) == no_output + + def test_let_binding_holds_any_body(self) -> None: + native = v.NativeBody([], INT, v.NativeInfo(v.Arithmetic())) + let = v.LetDefinition(TYPED, n("f"), native, v.Variable(TYPED, n("f"))) + assert codec.decode_value(codec.encode_value(let)) == let + + def test_library_mixing_bodies_round_trips(self) -> None: + lib = library( + values={ + n("native"): v.NativeBody([], INT, v.NativeInfo(v.Arithmetic())), + n("expr"): v.Definition([], INT, v.Unit(TYPED)), + } + ) + document = codec.encode_versioned_distribution(lib) + assert codec.decode_versioned_distribution(document) == lib + + def test_new_bodies_do_not_downgrade(self) -> None: + native = v.NativeBody([], INT, v.NativeInfo(v.Arithmetic())) + external = v.ExternalBody([], INT, [v.ExternalBinding("js", "f")]) + incomplete = v.IncompleteBody([], inc.DraftIncompleteness()) + let = v.Definition( + [], INT, v.LetDefinition(TYPED, n("g"), native, v.Unit(TYPED)) + ) + lib = library( + values={ + n("n"): native, + n("e"): external, + n("i"): incomplete, + n("l"): let, + } + ) + values = "distribution/modules/app/values" + assert downgrade_errors(lib) == [ + (d.UNSUPPORTED_V4_DOWNGRADE, "NativeBody", f"{values}/n/NativeBody"), + (d.UNSUPPORTED_V4_DOWNGRADE, "ExternalBody", f"{values}/e/ExternalBody"), + ( + d.UNSUPPORTED_V4_DOWNGRADE, + "IncompleteBody", + f"{values}/i/IncompleteBody", + ), + ( + d.UNSUPPORTED_V4_DOWNGRADE, + "NativeBody", + f"{values}/l/body/LetDefinition/definition/NativeBody", + ), + ] + + def test_input_drop_walks_external_fallback(self) -> None: + from morphir.ir.migrate import drop_input_attributes + + inner = v.Definition( + [(n("y"), v.ValueAttributes(inferred_type=STRING), INT)], + INT, + v.Unit(TYPED), + ) + fallback = v.LetDefinition(TYPED, n("g"), inner, v.Unit(TYPED)) + external = v.ExternalBody([], INT, [v.ExternalBinding("js", "f")], fallback) + result = drop_input_attributes(library(values={n("e"): external})) + assert [w.location[-1] for w in result.warnings] == ["y"] + codec.encode_versioned_distribution(result.value) From 18f056e54f68bca0ffe9f68a1b072cd515cd5249 Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 02:08:18 -0500 Subject: [PATCH 5/9] feat(ir): add annotations to v4 module, type and value specifications Add morphir.ir.v4.annotation (compact and structured annotations, positional and named arguments) after the finos/morphir v4 schema at efa36060 and the morphir-rust v4 model. Specifications get an annotations field with an empty default. The codec writes it first and only when non-empty, omits empty arguments, and keeps refusing annotations on definitions (kit definitions-0020 to -0023). Reverse migration refuses annotations with unsupported_v4_downgrade. Refs: morphir-ukvn.8.14.3 --- .../align-ir-v4-with-upstream/tasks.md | 8 +- packages/morphir/src/morphir/ir/migrate.py | 8 + .../morphir/src/morphir/ir/v4/__init__.py | 17 +- .../morphir/src/morphir/ir/v4/annotation.py | 75 ++++++++ packages/morphir/src/morphir/ir/v4/codec.py | 158 ++++++++++++--- packages/morphir/src/morphir/ir/v4/module.py | 2 + .../morphir/src/morphir/ir/v4/type_spec.py | 7 +- packages/morphir/src/morphir/ir/v4/value.py | 8 +- tests/unit/ir/test_v4_upstream.py | 181 +++++++++++++++++- 9 files changed, 433 insertions(+), 31 deletions(-) create mode 100644 packages/morphir/src/morphir/ir/v4/annotation.py diff --git a/openspec/changes/align-ir-v4-with-upstream/tasks.md b/openspec/changes/align-ir-v4-with-upstream/tasks.md index 2f66834e..dcba179c 100644 --- a/openspec/changes/align-ir-v4-with-upstream/tasks.md +++ b/openspec/changes/align-ir-v4-with-upstream/tasks.md @@ -24,10 +24,10 @@ ## 4. Annotations on specifications (morphir-ukvn.8.14.3) -- [ ] 4.1 Add `morphir.ir.v4.annotation` with `CompactAnnotation`, `StructuredAnnotation`, `PositionalArgument`, `NamedArgument` and `Annotation`, and an `annotations` field (empty default, last) on module, type and value specifications; verify existing positional constructors still work -- [ ] 4.2 Encode and decode annotations (written first, only when non-empty; `arguments` omitted when empty; compact text after the first colon after `#`), and verify kit `definitions-0020` to `-0022` round trip and that `"arguments": []` reads as an argument-less structured annotation -- [ ] 4.3 Verify that a type definition with `annotations` fails decoding with `unknown_member` (kit `definitions-0023`) -- [ ] 4.4 Make reverse migration fail with `unsupported_v4_downgrade` for annotations on a specification in a v4 dependency, and verify the location +- [x] 4.1 Add `morphir.ir.v4.annotation` with `CompactAnnotation`, `StructuredAnnotation`, `PositionalArgument`, `NamedArgument` and `Annotation`, and an `annotations` field (empty default, last) on module, type and value specifications; verify existing positional constructors still work +- [x] 4.2 Encode and decode annotations (written first, only when non-empty; `arguments` omitted when empty; compact text after the first colon after `#`), and verify kit `definitions-0020` to `-0022` round trip and that `"arguments": []` reads as an argument-less structured annotation +- [x] 4.3 Verify that a type definition with `annotations` fails decoding with `unknown_member` (kit `definitions-0023`) +- [x] 4.4 Make reverse migration fail with `unsupported_v4_downgrade` for annotations on a specification in a v4 dependency, and verify the location ## 5. Union type expression (morphir-ukvn.8.14.4) diff --git a/packages/morphir/src/morphir/ir/migrate.py b/packages/morphir/src/morphir/ir/migrate.py index bcd92281..cc387eb9 100644 --- a/packages/morphir/src/morphir/ir/migrate.py +++ b/packages/morphir/src/morphir/ir/migrate.py @@ -44,6 +44,7 @@ from .v3 import type as t3 from .v3 import value as v3 from .v4 import access_controlled as ac4 +from .v4 import annotation as ann4 from .v4 import distribution as dist4 from .v4 import documented as doc4 from .v4 import fqname as fq4 @@ -733,9 +734,14 @@ def ctors(self, ctors: ts4.Constructors, loc: Location) -> t3.Constructors[None] def params(self, params: list[name4.Name], loc: Location) -> tuple[name3.Name, ...]: return tuple(self.name(p, loc) for p in params) + def annotations(self, items: list[ann4.Annotation], loc: Location) -> None: + if items: + self.unsupported("Annotations", (*loc, "annotations")) + def type_spec( self, spec: ts4.Specification, loc: Location ) -> t3.Specification[None]: + self.annotations(spec.annotations, loc) params = self.params(spec.type_params, loc) match spec: case ts4.TypeAliasSpecification(_, tpe): @@ -970,6 +976,7 @@ def value_def( def value_spec( self, spec: v4.Specification, loc: Location ) -> v3.Specification[None]: + self.annotations(spec.annotations, loc) return v3.Specification( tuple( (self.name(n, loc), self.type(tp, (*loc, "inputs", name4.to_string(n)))) @@ -983,6 +990,7 @@ def value_spec( def module_spec( self, spec: mod4.Specification, loc: Location ) -> mod3.Specification[None]: + self.annotations(spec.annotations, loc) types = {} for n, x in spec.types.items(): here = (*loc, "types", name4.to_string(n)) diff --git a/packages/morphir/src/morphir/ir/v4/__init__.py b/packages/morphir/src/morphir/ir/v4/__init__.py index a47328a3..3bbb8df9 100644 --- a/packages/morphir/src/morphir/ir/v4/__init__.py +++ b/packages/morphir/src/morphir/ir/v4/__init__.py @@ -4,8 +4,16 @@ draft pinned to finos/morphir commit efa36060. """ -from . import incompleteness, module, package +from . import annotation, incompleteness, module, package from .access_controlled import AccessControlled +from .annotation import ( + Annotation, + AnnotationArgument, + CompactAnnotation, + NamedArgument, + PositionalArgument, + StructuredAnnotation, +) from .decorations import ( DecorationFormat, DecorationValuesFile, @@ -77,9 +85,12 @@ __all__ = [ "AccessControlled", + "Annotation", + "AnnotationArgument", "ApplicationDistribution", "BoolLiteral", "CharLiteral", + "CompactAnnotation", "DecimalLiteral", "DecorationFormat", "DecorationValuesFile", @@ -114,6 +125,7 @@ "LibraryDistribution", "Literal", "Name", + "NamedArgument", "NativeBody", "NativeHint", "NativeInfo", @@ -121,12 +133,14 @@ "Path", "Pattern", "PointerRef", + "PositionalArgument", "QName", "Ref", "SchemaRef", "SourceRange", "SpecsDistribution", "StringLiteral", + "StructuredAnnotation", "Type", "TypeAttributes", "TypeConstraints", @@ -139,6 +153,7 @@ "ValueDefinition", "ValueDefinitionBody", "ValueSpecification", + "annotation", "incompleteness", "module", "package", diff --git a/packages/morphir/src/morphir/ir/v4/annotation.py b/packages/morphir/src/morphir/ir/v4/annotation.py new file mode 100644 index 00000000..e7ea176c --- /dev/null +++ b/packages/morphir/src/morphir/ir/v4/annotation.py @@ -0,0 +1,75 @@ +"""Annotations on v4 specifications. + +Follows the finos/morphir v4 schema (``Annotation``, ``AnnotationArgument``) at +the pinned commit and the morphir-rust v4 model. An annotation keeps the form it +was written in, because the kit writes an argument-less structured annotation as +an object and a compact one as a string. Annotations belong to specifications, +the public face; definitions carry none. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Union + +from .fqname import FQName +from .name import Name +from .value import Value + + +@dataclass(frozen=True) +class PositionalArgument: + """An annotation argument written on its own. + + Attributes: + value: The argument expression. + """ + + value: Value + + +@dataclass(frozen=True) +class NamedArgument: + """An annotation argument written under the name it answers to. + + Attributes: + name: The parameter name. + value: The argument expression. + """ + + name: Name + value: Value + + +AnnotationArgument = Union[PositionalArgument, NamedArgument] +"""A positional or named annotation argument.""" + + +@dataclass(frozen=True) +class CompactAnnotation: + """The compact spelling: ``"pkg:mod#name"`` or ``"pkg:mod#name:text"``. + + Attributes: + name: The annotation type. + text: The free text after the first colon after ``#``, if any. + """ + + name: FQName + text: str | None = None + + +@dataclass(frozen=True) +class StructuredAnnotation: + """The structured spelling: ``{"name": ..., "arguments": [...]}``. + + Attributes: + name: The annotation type. + arguments: Positional and named arguments, in order. + """ + + name: FQName + arguments: list[AnnotationArgument] = field(default_factory=list) + + +Annotation = Union[CompactAnnotation, StructuredAnnotation] +"""An annotation on a module, type or value specification.""" diff --git a/packages/morphir/src/morphir/ir/v4/codec.py b/packages/morphir/src/morphir/ir/v4/codec.py index fbdf36df..7f4f191f 100644 --- a/packages/morphir/src/morphir/ir/v4/codec.py +++ b/packages/morphir/src/morphir/ir/v4/codec.py @@ -53,6 +53,7 @@ from .. import diagnostics as d from ..diagnostics import DiagnosticError, Location, fail from ..format_version import V4, V4_TABLE, read_document, write_format_version +from . import annotation as ann from . import document as doc from . import incompleteness as inc from . import literal as lit @@ -139,10 +140,12 @@ def _m(required: str = "", optional: str = "attributes") -> Members: } TYPE_SPECIFICATION_NODES: dict[str, Members] = { - "TypeAliasSpecification": _m("typeParams typeExp", ""), - "OpaqueTypeSpecification": _m("", "typeParams"), - "CustomTypeSpecification": _m("typeParams constructors", ""), - "DerivedTypeSpecification": _m("typeParams baseType fromBaseType toBaseType", ""), + "TypeAliasSpecification": _m("typeParams typeExp", "annotations"), + "OpaqueTypeSpecification": _m("", "annotations typeParams"), + "CustomTypeSpecification": _m("typeParams constructors", "annotations"), + "DerivedTypeSpecification": _m( + "typeParams baseType fromBaseType toBaseType", "annotations" + ), } TYPE_DEFINITION_NODES: dict[str, Members] = { @@ -1402,18 +1405,113 @@ def optional_value(key: str) -> v.Value | None: def encode_value_specification(spec: v.Specification, loc: Location = ()) -> Json: - return { - "inputs": _encode_inputs(spec.inputs, (*loc, "inputs"), "ValueSpecification"), - "output": encode_type(spec.output, (*loc, "output")), - } + return _with_annotations( + spec.annotations, + loc, + { + "inputs": _encode_inputs( + spec.inputs, (*loc, "inputs"), "ValueSpecification" + ), + "output": encode_type(spec.output, (*loc, "output")), + }, + ) def decode_value_specification(j: object, loc: Location = ()) -> v.Specification: obj = _expect_object(j, loc, "ValueSpecification") - _check_members(obj, _m("inputs output", ""), loc, "ValueSpecification") + _check_members(obj, _m("inputs output", "annotations"), loc, "ValueSpecification") return v.Specification( _unkeyed(obj["inputs"], decode_name, decode_type, (*loc, "inputs"), "inputs"), decode_type(obj["output"], (*loc, "output")), + _decode_annotations(obj, loc), + ) + + +# --- Annotations --- + + +def encode_annotation(item: ann.Annotation, loc: Location = ()) -> Json: + """Encode an annotation in the form it was written in.""" + match item: + case ann.CompactAnnotation(name, text): + fqn = encode_fqname(name, loc) + return fqn if text is None else f"{fqn}:{text}" + case ann.StructuredAnnotation(name, arguments): + members: dict[str, Json] = {"name": encode_fqname(name, loc)} + if arguments: + members["arguments"] = [ + _encode_annotation_argument(x, (*loc, "arguments", i)) + for i, x in enumerate(arguments) + ] + return members + + +def _encode_annotation_argument(arg: ann.AnnotationArgument, loc: Location) -> Json: + match arg: + case ann.PositionalArgument(value): + return encode_value(value, loc) + case ann.NamedArgument(name, value): + return { + "name": encode_name(name, loc), + "value": encode_value(value, (*loc, "value")), + } + + +def decode_annotation(j: object, loc: Location = ()) -> ann.Annotation: + """Decode a compact string or a structured annotation object. + + In the compact form the free text starts after the first colon after ``#``. + """ + if isinstance(j, str): + hash_at = j.find("#") + colon_at = j.find(":", hash_at) if hash_at >= 0 else -1 + if colon_at < 0: + return ann.CompactAnnotation(decode_fqname(j, loc)) + return ann.CompactAnnotation( + decode_fqname(j[:colon_at], loc), j[colon_at + 1 :] + ) + obj = _expect_object(j, loc, "Annotation") + _check_members(obj, _m("name", "arguments"), loc, "Annotation") + arguments = ( + _decode_list( + _decode_annotation_argument, + obj["arguments"], + (*loc, "arguments"), + "Annotation", + ) + if "arguments" in obj + else [] + ) + return ann.StructuredAnnotation(decode_fqname(obj["name"], loc), arguments) + + +def _decode_annotation_argument(j: object, loc: Location) -> ann.AnnotationArgument: + if isinstance(j, dict) and set(j) == {"name", "value"}: + return ann.NamedArgument( + decode_name(j["name"], (*loc, "name")), + decode_value(j["value"], (*loc, "value")), + ) + return ann.PositionalArgument(decode_value(j, loc)) + + +def _with_annotations( + annotations: list[ann.Annotation], loc: Location, members: dict[str, Json] +) -> dict[str, Json]: + """Put a non-empty ``annotations`` member first, as the kit writes it.""" + if not annotations: + return members + encoded = [ + encode_annotation(x, (*loc, "annotations", i)) + for i, x in enumerate(annotations) + ] + return {"annotations": encoded, **members} + + +def _decode_annotations(obj: dict[str, object], loc: Location) -> list[ann.Annotation]: + if "annotations" not in obj: + return [] + return _decode_list( + decode_annotation, obj["annotations"], (*loc, "annotations"), "annotations" ) @@ -1459,6 +1557,12 @@ def decode_args(j: object, here: Location) -> list[tuple[Name, t.Type]]: def encode_type_specification(spec: ts.Specification, loc: Location = ()) -> Json: + tag = type(spec).__name__ + encoded = _encode_type_specification(spec, loc) + return _wrap(tag, _with_annotations(spec.annotations, (*loc, tag), encoded[tag])) + + +def _encode_type_specification(spec: ts.Specification, loc: Location) -> Json: params = _encode_params(spec.type_params, loc) match spec: case ts.TypeAliasSpecification(_, tpe): @@ -1498,17 +1602,19 @@ def decode_type_specification(j: object, loc: Location = ()) -> ts.Specification if "typeParams" in body else [] ) + annotations = _decode_annotations(body, here) match tag: case "TypeAliasSpecification": return ts.TypeAliasSpecification( - params, decode_type(body["typeExp"], (*here, "typeExp")) + params, decode_type(body["typeExp"], (*here, "typeExp")), annotations ) case "OpaqueTypeSpecification": - return ts.OpaqueTypeSpecification(params) + return ts.OpaqueTypeSpecification(params, annotations) case "CustomTypeSpecification": return ts.CustomTypeSpecification( params, _decode_constructors(body["constructors"], (*here, "constructors")), + annotations, ) case _: return ts.DerivedTypeSpecification( @@ -1518,6 +1624,7 @@ def decode_type_specification(j: object, loc: Location = ()) -> ts.Specification decode_fqname(body["fromBaseType"], here), decode_fqname(body["toBaseType"], here), ), + annotations, ) @@ -1667,22 +1774,28 @@ def _decode_doc(obj: dict[str, object], loc: Location) -> str | None: def encode_module_specification(spec: module.Specification, loc: Location = ()) -> Json: - return _with_doc( - { - "types": _keyed( - spec.types, _name_key, _enc_type_spec, (*loc, "types"), "types" - ), - "values": _keyed( - spec.values, _name_key, _enc_value_spec, (*loc, "values"), "values" - ), - }, - spec.doc, + return _with_annotations( + spec.annotations, + loc, + _with_doc( + { + "types": _keyed( + spec.types, _name_key, _enc_type_spec, (*loc, "types"), "types" + ), + "values": _keyed( + spec.values, _name_key, _enc_value_spec, (*loc, "values"), "values" + ), + }, + spec.doc, + ), ) def decode_module_specification(j: object, loc: Location = ()) -> module.Specification: obj = _expect_object(j, loc, "ModuleSpecification") - _check_members(obj, _m("", "types values doc"), loc, "ModuleSpecification") + _check_members( + obj, _m("", "annotations types values doc"), loc, "ModuleSpecification" + ) return module.Specification( types=dict( _unkeyed( @@ -1703,6 +1816,7 @@ def decode_module_specification(j: object, loc: Location = ()) -> module.Specifi ) ), doc=_decode_doc(obj, loc), + annotations=_decode_annotations(obj, loc), ) diff --git a/packages/morphir/src/morphir/ir/v4/module.py b/packages/morphir/src/morphir/ir/v4/module.py index e2b5475d..adc8d627 100644 --- a/packages/morphir/src/morphir/ir/v4/module.py +++ b/packages/morphir/src/morphir/ir/v4/module.py @@ -2,6 +2,7 @@ from . import type_def, type_spec, value from .access_controlled import AccessControlled +from .annotation import Annotation from .documented import Documented from .name import Name from .path import Path @@ -15,6 +16,7 @@ class Specification: types: dict[Name, Documented[type_spec.Specification]] = field(default_factory=dict) values: dict[Name, Documented[value.Specification]] = field(default_factory=dict) doc: str | None = None + annotations: list[Annotation] = field(default_factory=list) @dataclass(frozen=True) diff --git a/packages/morphir/src/morphir/ir/v4/type_spec.py b/packages/morphir/src/morphir/ir/v4/type_spec.py index 49377e0f..e6fa263a 100644 --- a/packages/morphir/src/morphir/ir/v4/type_spec.py +++ b/packages/morphir/src/morphir/ir/v4/type_spec.py @@ -1,8 +1,9 @@ from __future__ import annotations -from dataclasses import dataclass +from dataclasses import dataclass, field from typing import Union +from .annotation import Annotation from .fqname import FQName from .name import Name from .type import Type @@ -17,6 +18,7 @@ class TypeAliasSpecification: type_params: list[Name] tpe: Type + annotations: list[Annotation] = field(default_factory=list) @dataclass(frozen=True) @@ -24,6 +26,7 @@ class OpaqueTypeSpecification: """Specification for an opaque type.""" type_params: list[Name] + annotations: list[Annotation] = field(default_factory=list) @dataclass(frozen=True) @@ -32,6 +35,7 @@ class CustomTypeSpecification: type_params: list[Name] constructors: Constructors + annotations: list[Annotation] = field(default_factory=list) @dataclass(frozen=True) @@ -49,6 +53,7 @@ class DerivedTypeSpecification: type_params: list[Name] details: DerivedTypeSpecificationDetails + annotations: list[Annotation] = field(default_factory=list) Specification = Union[ diff --git a/packages/morphir/src/morphir/ir/v4/value.py b/packages/morphir/src/morphir/ir/v4/value.py index cda41351..cfaf663c 100644 --- a/packages/morphir/src/morphir/ir/v4/value.py +++ b/packages/morphir/src/morphir/ir/v4/value.py @@ -1,7 +1,7 @@ from __future__ import annotations from dataclasses import dataclass, field -from typing import Any, Union +from typing import TYPE_CHECKING, Any, Union from typing import List as TList from typing import Tuple as TTuple @@ -11,6 +11,9 @@ from .name import Name from .type import Type +if TYPE_CHECKING: + from .annotation import Annotation + @dataclass(frozen=True) class SourceLocation: @@ -327,10 +330,11 @@ class External: @dataclass(frozen=True) class Specification: - """Value specification.""" + """Value specification: the signature of a value, and its annotations.""" inputs: TList[TTuple[Name, Type]] output: Type + annotations: TList[Annotation] = field(default_factory=lambda: []) @dataclass(frozen=True) diff --git a/tests/unit/ir/test_v4_upstream.py b/tests/unit/ir/test_v4_upstream.py index 2af6fa1b..7873b365 100644 --- a/tests/unit/ir/test_v4_upstream.py +++ b/tests/unit/ir/test_v4_upstream.py @@ -12,13 +12,19 @@ from morphir.ir import v4 from morphir.ir.diagnostics import DiagnosticError from morphir.ir.migrate import Err, v4_to_v3 +from morphir.ir.v4 import annotation as ann from morphir.ir.v4 import codec, module, package from morphir.ir.v4 import incompleteness as inc from morphir.ir.v4 import type as t from morphir.ir.v4 import type_def as td +from morphir.ir.v4 import type_spec as ts from morphir.ir.v4 import value as v from morphir.ir.v4.access_controlled import Public -from morphir.ir.v4.distribution import LibraryDistribution, PackageInfo +from morphir.ir.v4.distribution import ( + LibraryDistribution, + PackageInfo, + SpecsDistribution, +) from morphir.ir.v4.documented import Documented from morphir.ir.v4.fqname import FQName from morphir.ir.v4.name import Name @@ -360,3 +366,176 @@ def test_input_drop_walks_external_fallback(self) -> None: result = drop_input_attributes(library(values={n("e"): external})) assert [w.location[-1] for w in result.warnings] == ["y"] codec.encode_versioned_distribution(result.value) + + +ANNOTATIONS = p(n("annotations")) +ORG = p(n("my", "org"), n("project")) + + +def anno(local: str) -> FQName: + return FQName(ORG, ANNOTATIONS, n(local)) + + +def lit_string(text: str) -> v.Value: + from morphir.ir.v4 import literal + + return v.LiteralValue(v.ValueAttributes(), literal.StringLiteral(text)) + + +class TestAnnotations: + def test_exported_from_v4(self) -> None: + assert v4.Annotation is ann.Annotation + + def test_type_specification_annotations_definitions_0020(self) -> None: + spec = ts.TypeAliasSpecification( + [], + STRING, + [ + ann.CompactAnnotation(anno("deprecated"), "Use user-v2 instead"), + ann.StructuredAnnotation( + anno("since"), + [ + ann.PositionalArgument(lit_string("1.2.0")), + ann.NamedArgument(n("reason"), lit_string("renamed")), + ], + ), + ann.StructuredAnnotation(anno("internal")), + ], + ) + string_literal = {"literal": {"StringLiteral": "1.2.0"}} + expected = { + "TypeAliasSpecification": { + "annotations": [ + "my-org/project:annotations#deprecated:Use user-v2 instead", + { + "name": "my-org/project:annotations#since", + "arguments": [ + {"Literal": string_literal}, + { + "name": "reason", + "value": { + "Literal": {"literal": {"StringLiteral": "renamed"}} + }, + }, + ], + }, + {"name": "my-org/project:annotations#internal"}, + ], + "typeParams": [], + "typeExp": STRING_JSON, + } + } + encoded = codec.encode_type_specification(spec) + assert encoded == expected + assert next(iter(encoded["TypeAliasSpecification"])) == "annotations" + assert codec.decode_type_specification(expected) == spec + + def test_empty_arguments_read_as_structured(self) -> None: + j = { + "OpaqueTypeSpecification": { + "annotations": [ + {"name": "my-org/project:annotations#internal", "arguments": []} + ] + } + } + spec = codec.decode_type_specification(j) + assert spec == ts.OpaqueTypeSpecification( + [], [ann.StructuredAnnotation(anno("internal"))] + ) + + def test_compact_text_starts_after_first_colon_after_hash(self) -> None: + j = { + "OpaqueTypeSpecification": { + "annotations": ["my-org/project:annotations#version:1.0.0:rc"] + } + } + spec = codec.decode_type_specification(j) + assert spec.annotations == [ann.CompactAnnotation(anno("version"), "1.0.0:rc")] + + def test_value_specification_definitions_0021(self) -> None: + spec = v.Specification( + [(n("x"), INT)], INT, [ann.CompactAnnotation(anno("pure"))] + ) + expected = { + "annotations": ["my-org/project:annotations#pure"], + "inputs": {"x": INT_JSON}, + "output": INT_JSON, + } + assert codec.encode_value_specification(spec) == expected + assert codec.decode_value_specification(expected) == spec + + def test_module_specification_definitions_0022(self) -> None: + spec = module.Specification( + doc="The domain module", + annotations=[ann.CompactAnnotation(anno("stable"))], + ) + expected = { + "annotations": ["my-org/project:annotations#stable"], + "types": {}, + "values": {}, + "doc": "The domain module", + } + encoded = codec.encode_module_specification(spec) + assert encoded == expected + assert list(encoded) == ["annotations", "types", "values", "doc"] + assert codec.decode_module_specification(expected) == spec + + def test_no_annotations_writes_no_member(self) -> None: + spec = ts.OpaqueTypeSpecification([]) + assert codec.encode_type_specification(spec) == { + "OpaqueTypeSpecification": {"typeParams": []} + } + + def test_definition_carries_no_annotations_definitions_0023(self) -> None: + bad = { + "TypeAliasDefinition": { + "annotations": [], + "typeParams": [], + "typeExp": STRING_JSON, + } + } + assert fail_codes(codec.decode_type_definition, bad) == (d.UNKNOWN_MEMBER,) + + def test_annotations_do_not_downgrade(self) -> None: + spec = module.Specification( + types={ + n("t"): Documented( + "doc", + ts.OpaqueTypeSpecification( + [], [ann.CompactAnnotation(anno("internal"))] + ), + ) + }, + values={ + n("f"): Documented( + "doc", + v.Specification([], INT, [ann.CompactAnnotation(anno("pure"))]), + ) + }, + annotations=[ann.CompactAnnotation(anno("stable"))], + ) + lib = library() + lib = LibraryDistribution( + lib.package, + lib.definition, + {p(n("dep")): package.Specification({p(n("m")): spec})}, + ) + deps = "distribution/dependencies/dep/m" + assert downgrade_errors(lib) == [ + (d.UNSUPPORTED_V4_DOWNGRADE, "Annotations", f"{deps}/annotations"), + (d.UNSUPPORTED_V4_DOWNGRADE, "Annotations", f"{deps}/types/t/annotations"), + ( + d.UNSUPPORTED_V4_DOWNGRADE, + "Annotations", + f"{deps}/values/f/annotations", + ), + ] + + def test_specs_distribution_with_annotations_round_trips(self) -> None: + spec = module.Specification(annotations=[ann.CompactAnnotation(anno("x"))]) + specs = SpecsDistribution( + PackageInfo(p(n("my"), n("pkg"))), + package.Specification({p(n("m")): spec}), + ) + document = codec.encode_versioned_distribution(specs) + assert codec.decode_versioned_distribution(document) == specs From 0b86c3520f476ca385cacab408cdf4c36c383176 Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 02:09:03 -0500 Subject: [PATCH 6/9] feat(ir): add the draft Union type expression to v4 Add type.Union (two or more alternatives, order, duplicates and nesting kept) after finos/morphir docs/spec/draft/types.md and the morphir-moonbit reference implementation (morphir-ukvn.7.9.1). The codec spells it {"Union": {"attributes"?, "alternatives": [...]}} and refuses fewer than two alternatives; map_attributes and the morphir.ir.type shim cover it; reverse migration refuses it with unsupported_v4_downgrade. Refs: morphir-ukvn.8.14.4 --- .../align-ir-v4-with-upstream/tasks.md | 6 +-- packages/morphir/src/morphir/ir/migrate.py | 3 ++ packages/morphir/src/morphir/ir/type.py | 2 + packages/morphir/src/morphir/ir/v4/codec.py | 28 +++++++++++ packages/morphir/src/morphir/ir/v4/type.py | 26 +++++++++- tests/unit/ir/test_v4_codec.py | 5 +- tests/unit/ir/test_v4_upstream.py | 47 +++++++++++++++++++ 7 files changed, 110 insertions(+), 7 deletions(-) diff --git a/openspec/changes/align-ir-v4-with-upstream/tasks.md b/openspec/changes/align-ir-v4-with-upstream/tasks.md index dcba179c..ef1f34ea 100644 --- a/openspec/changes/align-ir-v4-with-upstream/tasks.md +++ b/openspec/changes/align-ir-v4-with-upstream/tasks.md @@ -31,9 +31,9 @@ ## 5. Union type expression (morphir-ukvn.8.14.4) -- [ ] 5.1 Add `type.Union(attributes, alternatives)` to the `Type` union and `map_attributes`, and verify a substitution-style attribute map keeps order and duplicates -- [ ] 5.2 Encode and decode `{"Union": {"attributes"?, "alternatives": [...]}}`, refuse fewer than two alternatives both ways, and verify round trips including a nested union -- [ ] 5.3 Make reverse migration fail with `unsupported_v4_downgrade` for a union and verify the location +- [x] 5.1 Add `type.Union(attributes, alternatives)` to the `Type` union and `map_attributes`, and verify a substitution-style attribute map keeps order and duplicates +- [x] 5.2 Encode and decode `{"Union": {"attributes"?, "alternatives": [...]}}`, refuse fewer than two alternatives both ways, and verify round trips including a nested union +- [x] 5.3 Make reverse migration fail with `unsupported_v4_downgrade` for a union and verify the location ## 6. Deprecations, docs and integration checks (morphir-ukvn.8.14.5) diff --git a/packages/morphir/src/morphir/ir/migrate.py b/packages/morphir/src/morphir/ir/migrate.py index cc387eb9..145e2d20 100644 --- a/packages/morphir/src/morphir/ir/migrate.py +++ b/packages/morphir/src/morphir/ir/migrate.py @@ -713,6 +713,9 @@ def type(self, tpe: t4.Type, loc: Location) -> t3.Type[None]: ) case t4.Unit(_): return t3.Unit(None) + case t4.Union(): + self.unsupported("Union", here) + return t3.Unit(None) def fields( self, fields: list[t4.Field], loc: Location diff --git a/packages/morphir/src/morphir/ir/type.py b/packages/morphir/src/morphir/ir/type.py index f2fd35fa..dbe02310 100644 --- a/packages/morphir/src/morphir/ir/type.py +++ b/packages/morphir/src/morphir/ir/type.py @@ -11,6 +11,7 @@ Tuple, Type, TypeAttributes, + Union, Unit, Variable, get_attributes, @@ -28,6 +29,7 @@ "Tuple", "Type", "TypeAttributes", + "Union", "Unit", "Variable", "get_attributes", diff --git a/packages/morphir/src/morphir/ir/v4/codec.py b/packages/morphir/src/morphir/ir/v4/codec.py index 7f4f191f..e8699e3b 100644 --- a/packages/morphir/src/morphir/ir/v4/codec.py +++ b/packages/morphir/src/morphir/ir/v4/codec.py @@ -104,6 +104,7 @@ def _m(required: str = "", optional: str = "attributes") -> Members: "ExtensibleRecord": _m("variable fields"), "Function": _m("parameterType returnType"), "Unit": _m(), + "Union": _m("alternatives"), } VALUE_NODES: dict[str, Members] = { @@ -613,6 +614,27 @@ def node(tag: str, members: dict[str, Json]) -> Json: ) case t.Unit(_): return node("Unit", {}) + case t.Union(_, alternatives): + _check_alternatives(len(alternatives), (*here, "alternatives")) + return node( + "Union", + { + "alternatives": [ + encode_type(x, (*here, "alternatives", i)) + for i, x in enumerate(alternatives) + ] + }, + ) + + +def _check_alternatives(count: int, loc: Location) -> None: + if count < 2: + raise fail( + d.INVALID_JSON, + "Union", + loc, + f"a union needs two or more alternatives, found {count}", + ) def _encode_type_fields(fields: list[t.Field], loc: Location) -> dict[str, Json]: @@ -667,6 +689,12 @@ def fields() -> list[t.Field]: decode_type(body["parameterType"], (*here, "parameterType")), decode_type(body["returnType"], (*here, "returnType")), ) + case "Union": + alternatives = _decode_list( + decode_type, body["alternatives"], (*here, "alternatives"), "Union" + ) + _check_alternatives(len(alternatives), (*here, "alternatives")) + return t.Union(attrs, alternatives) case _: return t.Unit(attrs) diff --git a/packages/morphir/src/morphir/ir/v4/type.py b/packages/morphir/src/morphir/ir/v4/type.py index 5cb42871..7aa3f876 100644 --- a/packages/morphir/src/morphir/ir/v4/type.py +++ b/packages/morphir/src/morphir/ir/v4/type.py @@ -1,7 +1,8 @@ from __future__ import annotations from dataclasses import dataclass, field -from typing import Any, Union +from typing import Any +from typing import Union as TUnion from .fqname import FQName from .name import Name @@ -72,7 +73,26 @@ class Unit: attributes: TypeAttributes -Type = Union[Variable, Reference, Tuple, Record, ExtensibleRecord, Function, Unit] +@dataclass(frozen=True) +class Union: + """An untagged union: values of at least one of the alternative types. + + Draft v4 type expression (finos/morphir ``docs/spec/draft/types.md``). It + introduces no constructor or tag. Alternative order, duplicates and nesting + are content and are kept. + + Attributes: + attributes: The type attributes. + alternatives: Two or more alternative types. + """ + + attributes: TypeAttributes + alternatives: list[Type] = field(default_factory=list) + + +Type = TUnion[ + Variable, Reference, Tuple, Record, ExtensibleRecord, Function, Unit, Union +] @dataclass(frozen=True) @@ -114,5 +134,7 @@ def map_attributes(tpe: Type, f: Any) -> Type: map_attributes(tpe.argument_type, f), map_attributes(tpe.return_type, f), ) + elif isinstance(tpe, Union): + return Union(new_ta, [map_attributes(a, f) for a in tpe.alternatives]) else: return Unit(new_ta) diff --git a/tests/unit/ir/test_v4_codec.py b/tests/unit/ir/test_v4_codec.py index a355027e..fd720cad 100644 --- a/tests/unit/ir/test_v4_codec.py +++ b/tests/unit/ir/test_v4_codec.py @@ -358,9 +358,10 @@ def test_draft_notice_names_pin(self) -> None: class TestDecodeFailures: def test_unknown_node(self) -> None: - err = pytest.raises(DiagnosticError, codec.decode_type, {"Union": {}}).value + bad = {"Intersection": {}} + err = pytest.raises(DiagnosticError, codec.decode_type, bad).value assert err.codes == (d.UNKNOWN_NODE,) - assert err.diagnostics[0].location == ("Union",) + assert err.diagnostics[0].location == ("Intersection",) def test_compact_spelling_is_not_guessed(self) -> None: assert fail_codes(codec.decode_type, "morphir/SDK:basics#int") == ( diff --git a/tests/unit/ir/test_v4_upstream.py b/tests/unit/ir/test_v4_upstream.py index 7873b365..b0ebb444 100644 --- a/tests/unit/ir/test_v4_upstream.py +++ b/tests/unit/ir/test_v4_upstream.py @@ -539,3 +539,50 @@ def test_specs_distribution_with_annotations_round_trips(self) -> None: ) document = codec.encode_versioned_distribution(specs) assert codec.decode_versioned_distribution(document) == specs + + +class TestUnion: + def test_exported_from_v4(self) -> None: + assert t.Union in v4.Type.__args__ # type: ignore[attr-defined] + + def test_union_round_trips(self) -> None: + union = t.Union(TA, [STRING, INT]) + expected = {"Union": {"alternatives": [STRING_JSON, INT_JSON]}} + assert codec.encode_type(union) == expected + assert codec.decode_type(expected) == union + + def test_order_duplicates_nesting_and_attributes_are_kept(self) -> None: + attrs = t.TypeAttributes(source=t.SourceLocation(1, 1, 1, 9)) + union = t.Union(attrs, [INT, t.Union(TA, [STRING, INT]), INT]) + encoded = codec.encode_type(union) + assert "attributes" in encoded["Union"] + assert codec.decode_type(encoded) == union + + @pytest.mark.parametrize("alternatives", [[], [INT]], ids=["empty", "singleton"]) + def test_fewer_than_two_alternatives_are_refused( + self, alternatives: list[t.Type] + ) -> None: + union = t.Union(TA, alternatives) + assert fail_codes(codec.encode_type, union) == (d.INVALID_JSON,) + bad = {"Union": {"alternatives": [codec.encode_type(x) for x in alternatives]}} + assert fail_codes(codec.decode_type, bad) == (d.INVALID_JSON,) + + def test_map_attributes_keeps_order_and_duplicates(self) -> None: + union = t.Union(TA, [INT, STRING, INT]) + src = t.TypeAttributes(source=t.SourceLocation(1, 1, 1, 1)) + mapped = t.map_attributes(union, lambda _: src) + assert isinstance(mapped, t.Union) + assert mapped.attributes == src + assert [type(x).__name__ for x in mapped.alternatives] == ["Reference"] * 3 + assert all(x.attributes == src for x in mapped.alternatives) + + def test_union_does_not_downgrade(self) -> None: + defn = td.TypeAliasDefinition([], t.Union(TA, [STRING, INT])) + errors = downgrade_errors(library(types={n("t"): defn})) + assert errors == [ + ( + d.UNSUPPORTED_V4_DOWNGRADE, + "Union", + "distribution/modules/app/types/t/Union", + ) + ] From 703799bc53bba5bf4681fbc4aba660e2f4787119 Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 02:10:19 -0500 Subject: [PATCH 7/9] docs(ir): deprecate the Python-only v4 constructs and document the realigned model Mark the Native and External value nodes and TypeConstraints as deprecated, naming their replacements; update the codec's gap list and the package README for the new bodies, incompleteness, annotations and Union. Follow-up removal is morphir-ukvn.8.18. Refs: morphir-ukvn.8.14.5 --- .../align-ir-v4-with-upstream/tasks.md | 8 +- packages/morphir/README.md | 36 ++++- packages/morphir/src/morphir/ir/v4/codec.py | 20 +-- .../src/morphir/ir/v4/type_constraints.py | 8 ++ packages/morphir/src/morphir/ir/v4/value.py | 14 +- pretty.output | 127 ++++++++++++++++++ tests/unit/ir/test_v4_upstream.py | 52 +++++++ 7 files changed, 249 insertions(+), 16 deletions(-) create mode 100644 pretty.output diff --git a/openspec/changes/align-ir-v4-with-upstream/tasks.md b/openspec/changes/align-ir-v4-with-upstream/tasks.md index ef1f34ea..d9b44620 100644 --- a/openspec/changes/align-ir-v4-with-upstream/tasks.md +++ b/openspec/changes/align-ir-v4-with-upstream/tasks.md @@ -37,10 +37,10 @@ ## 6. Deprecations, docs and integration checks (morphir-ukvn.8.14.5) -- [ ] 6.1 Mark the `Native` and `External` value nodes and `TypeConstraints` as deprecated in docstrings, naming the replacement, and verify a test that they still fail encoding with `no_v4_spelling` -- [ ] 6.2 Update the codec module docstring's gap list and the README "Morphir IR versions" section for the new constructs and the typed hole reason, and verify the README examples run -- [ ] 6.3 File a follow-up bead to remove the deprecated constructs and verify it is a child of `morphir-ukvn.8` with label `repo:morphir-python` -- [ ] 6.4 Run `mise run check` and `mise run test-bdd` and verify lint, both type checkers, tests and coverage of at least 80% pass +- [x] 6.1 Mark the `Native` and `External` value nodes and `TypeConstraints` as deprecated in docstrings, naming the replacement, and verify a test that they still fail encoding with `no_v4_spelling` +- [x] 6.2 Update the codec module docstring's gap list and the README "Morphir IR versions" section for the new constructs and the typed hole reason, and verify the README examples run +- [x] 6.3 File a follow-up bead to remove the deprecated constructs and verify it is a child of `morphir-ukvn.8` with label `repo:morphir-python` +- [x] 6.4 Run `mise run check` and `mise run test-bdd` and verify lint, both type checkers, tests and coverage of at least 80% pass ## Workflow follow-up diff --git a/packages/morphir/README.md b/packages/morphir/README.md index d2fe1277..1fc7fa9a 100644 --- a/packages/morphir/README.md +++ b/packages/morphir/README.md @@ -97,6 +97,37 @@ the distribution wrapped by its kind, never a format-3 shape. The module docstring lists the gaps between the Python v4 model and upstream v4, each of which fails with a diagnostic instead of guessing or dropping data. +The v4 model follows the upstream reference model at that commit: + +- A value definition is one of four bodies (`ValueDefinitionBody`): the + expression body `ValueDefinition`, `NativeBody`, `ExternalBody` and + `IncompleteBody`. Module values and let bindings hold any of them. +- `IncompleteTypeDefinition` and `IncompleteBody` carry an `Incompleteness`: a + hole with a typed `HoleReason` (`UnresolvedReference`, `DeletedDuringRefactor`, + `TypeMismatch`) or a draft. A `Hole` value's `reason` is a `HoleReason` too. +- Module, type and value specifications carry `annotations`, compact + (`CompactAnnotation`) or structured (`StructuredAnnotation`). +- `morphir.ir.v4.type.Union` is the draft untagged union, with two or more + alternatives. +- The `Native` and `External` value nodes and `TypeConstraints` are deprecated: + upstream v4 cannot spell them. + +```python +from morphir.ir.v4 import Name, codec +from morphir.ir.v4 import type as t +from morphir.ir.v4 import value as v + +int_type = t.Reference( + t.TypeAttributes(), codec.decode_fqname("morphir/SDK:basics#int") +) +add = v.NativeBody( + [(Name(("a",)), int_type), (Name(("b",)), int_type)], + int_type, + v.NativeInfo(v.Arithmetic()), +) +assert codec.decode_value_definition(codec.encode_value_definition(add)) == add +``` + `PackageInfo.version` is optional (`str | None`). Neither format 3 nor v4 JSON has a package release version, so the draft codec does not write it and decodes it as absent. `v3_to_v4` sets it only from `package_version`, and `v4_to_v3` drops it @@ -107,8 +138,9 @@ with the warning `package_version_dropped`. `morphir.ir.migrate.v3_to_v4` always succeeds for a v3 Library. `v4_to_v3` returns `Ok(value, warnings)` or `Err(diagnostics)` and reports every construct format 3.0.0 cannot hold in one pass: Specs and Application -distributions, `Hole`, `Native`, `External`, `DocumentLiteral`, a missing -`inferred_type`, semantic attributes, and missing member documentation. Value +distributions, `Hole`, `Native`, `External`, `DocumentLiteral`, native, external +and incomplete bodies, incomplete type definitions, `Union`, annotations, a +missing `inferred_type`, semantic attributes, and missing member documentation. Value source locations fail unless you pass `drop_source_locations=True`. v4 JSON spells a value-definition input as a bare type, so it has no place for diff --git a/packages/morphir/src/morphir/ir/v4/codec.py b/packages/morphir/src/morphir/ir/v4/codec.py index e8699e3b..bd670de8 100644 --- a/packages/morphir/src/morphir/ir/v4/codec.py +++ b/packages/morphir/src/morphir/ir/v4/codec.py @@ -14,16 +14,20 @@ in `TYPE_NODES`, `VALUE_NODES`, `PATTERN_NODES`, and the other tables below, and fails with a located diagnostic on anything else; it does not guess. +The codec covers the value-definition bodies (``ExpressionBody``, ``NativeBody``, +``ExternalBody``, ``IncompleteBody``), ``IncompleteTypeDefinition`` with typed hole +reasons, annotations on specifications, and the draft ``Union`` type (from +``docs/spec/draft/types.md``, not yet in the pinned schema). + Known gaps between the Python v4 model and upstream v4, each handled explicitly: -- ``Native`` and ``External`` value nodes have no spelling (upstream makes them - definition bodies); encoding fails with ``no_v4_spelling``. -- Structured ``TypeConstraints`` have no spelling; encoding fails with - ``no_v4_spelling``. -- Upstream ``Union`` types, ``IncompleteTypeDefinition``, annotations, and the - ``NativeBody``, ``ExternalBody``, and ``IncompleteBody`` definition bodies are - not in the Python model; decoding fails with ``unknown_node`` or - ``unknown_member``. +- The deprecated Python ``Native`` and ``External`` value nodes have no spelling + (upstream makes them definition bodies); encoding fails with ``no_v4_spelling``. +- Structured ``TypeConstraints`` (deprecated) have no spelling; encoding fails + with ``no_v4_spelling``. +- Only the expanded spelling is read and written. Compact spellings, the legacy + spellings readers accept for one release, and the 4.1.0 linked annotations + wait for the re-sync when upstream v4 settles. - Records are objects keyed by field name (finos decision 0004); duplicate field names fail encoding with ``duplicate_member``. - ``ExpressionBody.inputTypes`` has no attribute slot: upstream spells an input as diff --git a/packages/morphir/src/morphir/ir/v4/type_constraints.py b/packages/morphir/src/morphir/ir/v4/type_constraints.py index d0e0e24e..19e4ce55 100644 --- a/packages/morphir/src/morphir/ir/v4/type_constraints.py +++ b/packages/morphir/src/morphir/ir/v4/type_constraints.py @@ -77,6 +77,14 @@ class CustomConstraint: @dataclass(frozen=True) class TypeConstraints: + """Structured constraints on a type. + + Deprecated: the upstream v4 schema keeps ``TypeAttributes.constraints`` as a + placeholder with no defined members, so these constraints have no v4 spelling + and encoding fails with ``no_v4_spelling``. Keep constraints in + ``TypeAttributes.extensions`` until upstream defines them. + """ + numeric: NumericConstraint | None = None string: StringConstraint | None = None collection: CollectionConstraint | None = None diff --git a/packages/morphir/src/morphir/ir/v4/value.py b/packages/morphir/src/morphir/ir/v4/value.py index cfaf663c..a9ee73cf 100644 --- a/packages/morphir/src/morphir/ir/v4/value.py +++ b/packages/morphir/src/morphir/ir/v4/value.py @@ -285,7 +285,12 @@ class Hole: @dataclass(frozen=True) class Native: - """Represents a native reference.""" + """A native reference as a value node. + + Deprecated: upstream v4 makes native operations definition bodies, not + expressions (finos/morphir decision 0008), so this node has no v4 spelling and + encoding fails with ``no_v4_spelling``. Use `NativeBody` instead. + """ attributes: ValueAttributes fqname: FQName @@ -294,7 +299,12 @@ class Native: @dataclass(frozen=True) class External: - """Represents an external reference.""" + """An external reference as a value node. + + Deprecated: upstream v4 makes external operations definition bodies, not + expressions (finos/morphir decision 0008), so this node has no v4 spelling and + encoding fails with ``no_v4_spelling``. Use `ExternalBody` instead. + """ attributes: ValueAttributes external_name: str diff --git a/pretty.output b/pretty.output new file mode 100644 index 00000000..c20f07c1 --- /dev/null +++ b/pretty.output @@ -0,0 +1,127 @@ +Feature: IR interchange # features/ir_interchange.feature:1 + As a Morphir tooling user + I want to inspect and convert versioned morphir-ir.json files + So that I can move models between IR format 3.0.0 and draft IR v4 + Scenario: Inspect a morphir-elm format 3 file # features/ir_interchange.feature:6 + Given the golden fixture "elm-simpleValueTree.json" # features/steps/ir_interchange_steps.py:31 + When I run "ir inspect" on it # features/steps/ir_interchange_steps.py:46 + Then the command exits with code 0 # features/steps/ir_interchange_steps.py:81 + And the output contains "format: 3.0.0" # features/steps/ir_interchange_steps.py:87 + And the output contains "kind: Library" # features/steps/ir_interchange_steps.py:87 + And the output contains "package: morphir/example/app" # features/steps/ir_interchange_steps.py:87 + + Scenario: Upgrade a format 3 file to draft v4 and back # features/ir_interchange.feature:14 + Given the golden fixture "elm-simpleValueTree.json" # features/steps/ir_interchange_steps.py:31 + When I convert it to "v4" # features/steps/ir_interchange_steps.py:52 + Then the command exits with code 0 # features/steps/ir_interchange_steps.py:81 + And the converted file declares formatVersion 4 # features/steps/ir_interchange_steps.py:99 + When I inspect the converted file # features/steps/ir_interchange_steps.py:68 + Then the output contains "draft" # features/steps/ir_interchange_steps.py:87 + When I convert the converted file to "v3" # features/steps/ir_interchange_steps.py:74 + Then the command exits with code 0 # features/steps/ir_interchange_steps.py:81 + And the converted file has the same JSON as the fixture # features/steps/ir_interchange_steps.py:105 + + Scenario: Unversioned IR is rejected # features/ir_interchange.feature:25 + Given a file with the JSON '{"distribution": []}' # features/steps/ir_interchange_steps.py:38 + When I run "ir inspect" on it # features/steps/ir_interchange_steps.py:46 + Then the command exits with code 1 # features/steps/ir_interchange_steps.py:81 + And the errors contain "missing_format_version" # features/steps/ir_interchange_steps.py:93 + + Scenario: Specs cannot be downgraded # features/ir_interchange.feature:31 + Given a file with the JSON '{"formatVersion": 4, "distribution": {"Specs": {"packageName": "a"}}}' # features/steps/ir_interchange_steps.py:38 + When I convert it to "v3" # features/steps/ir_interchange_steps.py:52 + Then the command exits with code 1 # features/steps/ir_interchange_steps.py:81 + And the errors contain "unsupported_v4_downgrade" # features/steps/ir_interchange_steps.py:93 + And no converted file exists # features/steps/ir_interchange_steps.py:113 + +Feature: Morphir SDK # features/morphir_sdk.feature:1 + As a developer of Morphir models + I want a Python runtime for the Morphir SDK + So that compiled models behave as they do in Elm + Scenario Outline: Every core SDK module can be imported -- @1.1 # features/morphir_sdk.feature:12 + When I import the SDK module "basics" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.2 # features/morphir_sdk.feature:13 + When I import the SDK module "char" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.3 # features/morphir_sdk.feature:14 + When I import the SDK module "string" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.4 # features/morphir_sdk.feature:15 + When I import the SDK module "list" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.5 # features/morphir_sdk.feature:16 + When I import the SDK module "dict" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.6 # features/morphir_sdk.feature:17 + When I import the SDK module "set" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.7 # features/morphir_sdk.feature:18 + When I import the SDK module "maybe" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.8 # features/morphir_sdk.feature:19 + When I import the SDK module "result" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.9 # features/morphir_sdk.feature:20 + When I import the SDK module "tuple" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.10 # features/morphir_sdk.feature:21 + When I import the SDK module "decimal" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.11 # features/morphir_sdk.feature:22 + When I import the SDK module "int" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.12 # features/morphir_sdk.feature:23 + When I import the SDK module "number" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario: A dictionary lists its entries in key order # features/morphir_sdk.feature:25 + Given an SDK dictionary built from the pairs "b=2, a=1, c=3" # features/steps/sdk_steps.py:36 + When I list the keys of the dictionary # features/steps/sdk_steps.py:48 + Then the keys should be "a, b, c" # features/steps/sdk_steps.py:54 + + Scenario: Two dictionaries built in a different order are equal # features/morphir_sdk.feature:30 + Given an SDK dictionary built from the pairs "b=2, a=1" # features/steps/sdk_steps.py:36 + And another SDK dictionary built from the pairs "a=1, b=2" # features/steps/sdk_steps.py:42 + Then the two dictionaries should be equal # features/steps/sdk_steps.py:60 + + Scenario: Rounding follows Elm and not Python # features/morphir_sdk.feature:35 + When I round the float 2.5 with the SDK # features/steps/sdk_steps.py:67 + Then the rounded value should be 3 # features/steps/sdk_steps.py:73 + + Scenario: Decimal arithmetic is exact # features/morphir_sdk.feature:39 + When I add the decimals "0.1" and "0.2" with the SDK # features/steps/sdk_steps.py:79 + Then the decimal result should print as "0.3" # features/steps/sdk_steps.py:87 + +Feature: Morphir Python Setup # features/morphir_setup.feature:1 + As a developer + I want to verify the project is set up correctly + So that I can start developing Morphir features + Scenario: Morphir core library can be imported # features/morphir_setup.feature:6 + Given the Python environment is set up # features/steps/setup_steps.py:13 + When I import the morphir module # features/steps/setup_steps.py:22 + Then the module should be available # features/steps/setup_steps.py:46 + And it should have a version attribute # features/steps/setup_steps.py:52 + + Scenario: Morphir tools can be imported # features/morphir_setup.feature:12 + Given the Python environment is set up # features/steps/setup_steps.py:13 + When I import the morphir_tools module # features/steps/setup_steps.py:30 + Then the module should be available # features/steps/setup_steps.py:46 + And it should have a version attribute # features/steps/setup_steps.py:52 + + Scenario: Morphir IR module can be imported # features/morphir_setup.feature:18 + Given the Python environment is set up # features/steps/setup_steps.py:13 + When I import the morphir.ir module # features/steps/setup_steps.py:38 + Then the module should be available # features/steps/setup_steps.py:46 + diff --git a/tests/unit/ir/test_v4_upstream.py b/tests/unit/ir/test_v4_upstream.py index b0ebb444..f8a4a9e0 100644 --- a/tests/unit/ir/test_v4_upstream.py +++ b/tests/unit/ir/test_v4_upstream.py @@ -586,3 +586,55 @@ def test_union_does_not_downgrade(self) -> None: "distribution/modules/app/types/t/Union", ) ] + + +class TestDeprecated: + @pytest.mark.parametrize( + ("cls", "replacement"), + [(v.Native, "NativeBody"), (v.External, "ExternalBody")], + ) + def test_value_nodes_name_their_replacement( + self, cls: type, replacement: str + ) -> None: + doc = cls.__doc__ or "" + assert "Deprecated" in doc + assert replacement in doc + + def test_type_constraints_are_deprecated(self) -> None: + from morphir.ir.v4.type_constraints import TypeConstraints + + assert "Deprecated" in (TypeConstraints.__doc__ or "") + + +class TestInputDropCoverage: + def test_walks_incomplete_partial_body_and_keeps_native(self) -> None: + from morphir.ir.migrate import drop_input_attributes + + inner = v.Definition( + [(n("y"), v.ValueAttributes(inferred_type=STRING), INT)], + INT, + v.Unit(TYPED), + ) + partial = v.LetDefinition(TYPED, n("g"), inner, v.Unit(TYPED)) + incomplete = v.IncompleteBody([], inc.DraftIncompleteness(), None, partial) + native = v.NativeBody([], INT, v.NativeInfo(v.Arithmetic())) + lib = library(values={n("i"): incomplete, n("n"): native}) + result = drop_input_attributes(lib) + assert [w.location[-1] for w in result.warnings] == ["y"] + mod = result.value.definition.modules[p(n("app"))] + assert mod.value.values[n("n")].value.value == native + + def test_application_distribution(self) -> None: + from morphir.ir.migrate import drop_input_attributes + from morphir.ir.v4.distribution import ApplicationDistribution + + defn = v.Definition( + [(n("x"), v.ValueAttributes(inferred_type=STRING), INT)], + INT, + v.Unit(TYPED), + ) + lib = library(values={n("main"): defn}) + app = ApplicationDistribution(lib.package, lib.definition) + result = drop_input_attributes(app) + assert isinstance(result.value, ApplicationDistribution) + assert len(result.warnings) == 1 From 14b63098ca93c89947c3e553824b145807e96bbd Mon Sep 17 00:00:00 2001 From: Damian Reeves <957246+DamianReeves@users.noreply.github.com> Date: Tue, 27 Oct 2026 02:10:28 -0500 Subject: [PATCH 8/9] chore(openspec): archive align-ir-v4-with-upstream Merge the change's deltas into openspec/specs (ir/v4, ir/migrate, tools/ir-interchange) and fix the archived design's asset paths. Refs: morphir-ukvn.8.14, morphir-ukvn.8.15 --- .../.openspec.yaml | 0 .../design.html | 12 ++-- .../proposal.md | 0 .../specs/ir/migrate/spec.md | 0 .../specs/ir/v4/spec.md | 0 .../specs/tools/ir-interchange/spec.md | 0 .../tasks.md | 0 openspec/specs/ir/migrate/spec.md | 19 ++++++ openspec/specs/ir/v4/spec.md | 63 ++++++++++++++++++- openspec/specs/tools/ir-interchange/spec.md | 10 ++- 10 files changed, 95 insertions(+), 9 deletions(-) rename openspec/changes/{align-ir-v4-with-upstream => archive/2026-10-27-align-ir-v4-with-upstream}/.openspec.yaml (100%) rename openspec/changes/{align-ir-v4-with-upstream => archive/2026-10-27-align-ir-v4-with-upstream}/design.html (94%) rename openspec/changes/{align-ir-v4-with-upstream => archive/2026-10-27-align-ir-v4-with-upstream}/proposal.md (100%) rename openspec/changes/{align-ir-v4-with-upstream => archive/2026-10-27-align-ir-v4-with-upstream}/specs/ir/migrate/spec.md (100%) rename openspec/changes/{align-ir-v4-with-upstream => archive/2026-10-27-align-ir-v4-with-upstream}/specs/ir/v4/spec.md (100%) rename openspec/changes/{align-ir-v4-with-upstream => archive/2026-10-27-align-ir-v4-with-upstream}/specs/tools/ir-interchange/spec.md (100%) rename openspec/changes/{align-ir-v4-with-upstream => archive/2026-10-27-align-ir-v4-with-upstream}/tasks.md (100%) diff --git a/openspec/changes/align-ir-v4-with-upstream/.openspec.yaml b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/.openspec.yaml similarity index 100% rename from openspec/changes/align-ir-v4-with-upstream/.openspec.yaml rename to openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/.openspec.yaml diff --git a/openspec/changes/align-ir-v4-with-upstream/design.html b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/design.html similarity index 94% rename from openspec/changes/align-ir-v4-with-upstream/design.html rename to openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/design.html index 34215633..c8d39aa1 100644 --- a/openspec/changes/align-ir-v4-with-upstream/design.html +++ b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/design.html @@ -13,16 +13,16 @@ Code goes in 
 (or json, toml, bash, diff). Give sections an id and
   list them in nav.toc.
 -->
-
-
-
-
-
+
+
+
+
+
 
 
 

 
align-ir-v4-with-upstream: design

-
OpenSpec change · Bead morphir-ukvn.8.14 (group 1: morphir-ukvn.8.15) · 2026-10-27 · proposed

+
OpenSpec change · Bead morphir-ukvn.8.14 (group 1: morphir-ukvn.8.15) · 2026-10-27 · implemented