diff --git a/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/.openspec.yaml b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/.openspec.yaml new file mode 100644 index 00000000..dda53b8a --- /dev/null +++ b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/.openspec.yaml @@ -0,0 +1,2 @@ +schema: html-design +created: 2026-10-27 diff --git a/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/design.html b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/design.html new file mode 100644 index 00000000..c8d39aa1 --- /dev/null +++ b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/design.html @@ -0,0 +1,134 @@ + + + + + +align-ir-v4-with-upstream: design + + + + + + + + +
+

align-ir-v4-with-upstream: design

+

OpenSpec change · Bead morphir-ukvn.8.14 (group 1: morphir-ukvn.8.15) · 2026-10-27 · implemented

+ + + +

Context

+

See proposal.md for the motivation. The draft codec (morphir.ir.v4.codec) follows the finos/morphir v4 schema +at FINOS_MORPHIR_PIN = "efa36060", which is still finos/morphir main on 2026-10-27. It writes and reads only the +expanded spelling ({"Kind": {members}}); compact spellings are the later re-sync in morphir-ukvn.8.16.

+

The sources for the new shapes, in order of authority:

+ + + + + + + + + +
ConstructSource at the pin
Definition bodies, native hints, external bindingsschema ValueDefinition, NativeBody, ExternalBody, IncompleteBody; kit definitions-0007 to -0009, -0016, -0025, -0030
Incompleteness and hole reasonsschema Incompleteness, HoleReason; kit definitions-0014, -0015, -0024, -0026
Annotationsschema Annotation; kit definitions-0020 to -0023; morphir-rust ir/v4/annotation.rs (Compact and Structured)
Uniondocs/spec/draft/types.md (not yet in the schema); reference implementation finos/morphir-moonbit #155 (morphir-ukvn.7.9.1)
Per-input attributesreference model inputs: List(#(Name, Type)); kit definitions-0027 ("An input type is a bare type"); morphir-rust migration/classic_to_v4.rs, which drops v3 input attributes
+

The in-memory v3 to v4 to v3 round trip is a requirement (ir/migrate, "Shared subset round trips"), and the +morphir-elm reference model has 10 inputs (8 module-level, 2 let-bound) whose resolved type differs from the declared type (alias expansion and +type-variable instances, for example float-alias resolved to Float).

+ +

Goals and non-goals

+

Goals: every construct in the pinned v4 schema and the Union draft has a Python model and an expanded +spelling; every new construct has an explicit downgrade failure; the reference model can reach v4 JSON through an +explicit choice; nothing that works today stops working, except the typed Hole.reason.

+

Non-goals: compact spellings, legacy one-release spellings and the legacy_spelling warning +(morphir-ukvn.8.16); the 4.1.0 linked annotations; removing the deprecated constructs; type checking +of unions.

+ +

Decisions

+ +

INP-1. Per-input attributes drop only on request

+

Upstream v4 has no slot for them, and adding one is not a spelling to "find": the reference model, the kit and the +reference implementation all agree that an input is a bare type. So no upstream proposal is needed. The Python side +gets an explicit drop: drop_input_attributes(lib) for any v4 Library, a drop_input_attributes=True +keyword on v3_to_v4, and ir convert --drop-input-attributes. An input loses information when its +attributes are neither empty nor exactly ValueAttributes(inferred_type=declared); only those inputs report +input_attributes_dropped. The drop walks let-bound definitions too.

+

Alternatives: drop always, as morphir-rust does. Rejected: it breaks the lossless in-memory round trip and the +package's rule that loss is reported. Drop in the codec. Rejected: the codec has no warning channel and stays strict. +Write the attributes into extensions. Rejected: an invented convention no other reader knows.

+ +

BODY-1. Four body classes, with the expression body keeping its name

+

value.Definition stays the expression body, with its (name, attributes, type) inputs, so existing +code and the v3 migration keep working. New frozen classes NativeBody, ExternalBody and +IncompleteBody hold (name, type) inputs as upstream does. ValueDefinitionBody is the union +of the four; module.Definition.values, LetDefinition.definition and LetRecursion.definitions +hold it, because the schema's let bindings reference ValueDefinition.

+

Alternative: one Definition with a body sum type, as morphir-rust does. Rejected: it changes the constructor +of every existing definition.

+ +

HOLE-1. Typed hole reasons, shared by values and incompleteness

+

UnresolvedReference(target), DeletedDuringRefactor(tx_id) and TypeMismatch(expected, found) +live in a new incompleteness module with HoleIncompleteness(reason, partial_body) and +DraftIncompleteness(). Hole.reason becomes HoleReason. The schema gives the hole's +partialBody the type Type, so HoleIncompleteness.partial_body is a type; an incomplete value +keeps its partial expression on IncompleteBody.partial_body, as kit definitions-0025 shows. The +reference model's Draft(notes) has no member in the schema; the pinned schema wins.

+ +

ANN-1. Annotations keep their spelling

+

CompactAnnotation(name, text) and StructuredAnnotation(name, arguments) mirror morphir-rust, because +the kit writes an argument-less structured annotation as an object and a compact one as a string, so the form is +content. The free text is everything after the first colon after #. Arguments are +PositionalArgument(value) or NamedArgument(name, value); a named argument is an object with exactly +name and value, anything else is a positional value. Specifications get an +annotations field with an empty default, placed last so positional construction still works.

+ +

UNI-1. Union is a type node with two or more alternatives

+

type.Union(attributes, alternatives), spelled {"Union": {"attributes"?, "alternatives": [...]}}. +Encode and decode refuse fewer than two alternatives with invalid_json at alternatives. Order, +duplicates and nesting are kept. map_attributes handles it.

+ +

DEP-1. Deprecate, do not remove, the Python-only constructs

+

The Native and External value nodes and TypeConstraints stay, with docstrings that say +they are deprecated and what replaces them. Removing them would break imports from morphir.ir; a follow-up +bead removes them in a later minor release.

+ +

MIG-1. Every new construct fails downgrade with unsupported_v4_downgrade

+

Format 3.0.0 has no home for any of them. Constructs: NativeBody, ExternalBody, +IncompleteBody, IncompleteTypeDefinition, Union, Annotations (on any +specification). The pass still collects every failure.

+ +

Risks and trade-offs

+ + + + + + + + +
RiskMitigation
Typed Hole.reason breaks callers that pass raw JSONDraft API; called out as breaking in the proposal and README
Upstream renames members before v4 settlesAll spellings stay in the codec's tables; morphir-ukvn.8.16 re-syncs them
Union is in the draft spec page but not the pinned schemaFollow the spec page and the morphir-moonbit implementation; the codec is draft
Callers that match on module values miss the new bodiesStrict type checkers report the non-exhaustive match
+ +

Migration plan

+

One release of morphir. No data migration. Rollback is a revert of the PR.

+ +
+ + diff --git a/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/proposal.md b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/proposal.md new file mode 100644 index 00000000..0a7a4dd7 --- /dev/null +++ b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/proposal.md @@ -0,0 +1,40 @@ +# Proposal + +## Why + +The Python v4 model predates the upstream v4 profile. It cannot hold the native, external and incomplete definition bodies, incomplete type definitions, annotations, typed hole reasons or `Union` types that upstream v4 spells, so the draft codec refuses valid upstream documents with `unknown_node` or `unknown_member`. In the other direction, the morphir-elm reference model cannot be written as v4 JSON: its typed inputs carry a resolved type that differs from the declared type, and upstream v4 has no slot for per-input attributes, so encoding fails with `no_v4_spelling`. + +## What Changes + +- Add the upstream v4 value-definition bodies `NativeBody`, `ExternalBody` and `IncompleteBody` beside today's expression body (`Definition`). Module values and let bindings hold any of the four bodies. +- Add typed hole reasons (`UnresolvedReference`, `DeletedDuringRefactor`, `TypeMismatch`), the `Incompleteness` kinds `Hole` and `Draft`, and `IncompleteTypeDefinition`. **BREAKING** (draft API): `Hole.reason` is a typed `HoleReason`, not raw JSON. +- Add annotations (compact and structured, with positional and named arguments) to module, type and value specifications. +- Add the draft `Union` type expression (two or more alternatives). +- Extend the draft codec to read and write all of the above in the expanded spelling at the pinned commit, with the kit's refusals (duplicate external platforms, a platform-specific hint without its platform, a `Draft` hole reason, a union with fewer than two alternatives). +- Extend reverse migration so every new construct fails with `unsupported_v4_downgrade` at its location. Format 3.0.0 has no home for any of them. +- Add an explicit, reported drop of per-input value attributes: `drop_input_attributes` on `v3_to_v4`, a standalone function for any v4 Library, and `ir convert --drop-input-attributes`. Each input that loses information reports `input_attributes_dropped`. Without the opt-in, behaviour is unchanged: the in-memory migration stays lossless and encoding fails with `no_v4_spelling`. +- Keep the Python-only `Native` and `External` value nodes and the structured `TypeConstraints`, which upstream v4 cannot spell, and mark them deprecated. Their removal is follow-up work. + +## Capabilities + +### New Capabilities + +- (none) + +### Modified Capabilities + +- `ir/v4`: the model and draft codec cover the upstream definition bodies, incompleteness, annotations and `Union`; the Python-only constructs are deprecated. +- `ir/migrate`: the new v4 constructs fail reverse migration; per-input attributes can be dropped on request, with a warning. +- `tools/ir-interchange`: `ir convert` accepts `--drop-input-attributes` for upgrades. + +## Beads + +- **Tracked by**: `morphir-ukvn.8.14` (realign the Python v4 model with upstream finos/morphir v4) +- **Related**: `morphir-ukvn.8.15` (per-input attribute spelling; task group 1 of this change), `morphir-ukvn.8` (parent epic, IR v3 and v4 in morphir-python), `morphir-ukvn.8.16` (later re-sync of the draft codec when upstream v4 settles, including compact spellings), `morphir-diwy` (finos/morphir: v4 to v3 downgrade rules; codes may be renamed when it settles), `morphir-ukvn.7.9.1` (closed: the `Union` reference implementation in finos/morphir-moonbit) + +## Impact + +- `packages/morphir`: `morphir.ir.v4` modules `value`, `type`, `type_def`, `type_spec`, `module` and new `annotation` and `incompleteness` modules; `morphir.ir.v4.codec`; `morphir.ir.migrate`; `morphir.ir.diagnostics` (new warning code `input_attributes_dropped`). The core stays dependency-free. +- `packages/morphir-tools`: one new `ir convert` option. +- Callers that build `Hole` values with raw JSON reasons must pass a `HoleReason`. Callers that match on `module.Definition.values` entries must handle the three new bodies. +- No change to v3 or to format-3 encoding. diff --git a/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/specs/ir/migrate/spec.md b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/specs/ir/migrate/spec.md new file mode 100644 index 00000000..8b6e93dc --- /dev/null +++ b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/specs/ir/migrate/spec.md @@ -0,0 +1,51 @@ +# Spec Delta + +## MODIFIED Requirements + +### Requirement: Reverse migrate v4 to v3 with known failure paths +The library SHALL migrate v4 to v3. When the input uses concepts v3 cannot represent, migration SHALL fail with structured errors and MUST NOT invent or drop IR content, except as allowed by the dropped-information requirement. (bead: morphir-ukvn.8) + +#### Scenario: Specs and Application do not downgrade +- **WHEN** a v4 Specs or Application distribution is migrated to v3 +- **THEN** migration fails with `unsupported_v4_downgrade` naming the distribution kind + +#### Scenario: V4-only nodes are reported +- **WHEN** a v4 Library contains a `Hole`, `Native`, or `External` value, or a `DocumentLiteral` +- **THEN** migration fails with `unsupported_v4_downgrade` naming the construct and its location + +#### Scenario: V4-only definitions and types are reported +- **WHEN** a v4 Library contains a native, external or incomplete value-definition body, an incomplete type definition, a `Union` type, or a specification with annotations +- **THEN** migration fails with `unsupported_v4_downgrade` naming the construct and its location + +#### Scenario: Missing types are reported +- **WHEN** a v4 value, pattern, or value-definition input has no `inferred_type` +- **THEN** migration fails with `missing_inferred_type` at that location + +#### Scenario: Semantic attributes are reported +- **WHEN** v4 type attributes carry a source, constraints, or extensions, or value attributes carry extensions +- **THEN** migration fails with `unsupported_attributes_downgrade` + +#### Scenario: Absent member documentation is reported +- **WHEN** a v4 type or value member has no documentation +- **THEN** migration fails with `unsupported_documentation_downgrade`, because v3 member documentation is a required string + +#### Scenario: Compatible Library downgrades +- **WHEN** a v4 Library uses only concepts v3 can represent +- **THEN** migration succeeds with a v3 Library + +## ADDED Requirements + +### Requirement: Per-input attributes drop on request +The library SHALL let a caller drop the value attributes of value-definition inputs, because v4 JSON spells an input as a bare type. On request, each input SHALL keep only its declared type as `inferred_type`, and each input that lost information SHALL report `input_attributes_dropped` at its location. Without the request, forward migration MUST keep input attributes unchanged. (bead: morphir-ukvn.8.15) + +#### Scenario: Reference model upgrades to encodable v4 +- **WHEN** the morphir-elm reference model, whose inputs carry resolved types that differ from their declared types, is migrated to v4 with the input-attribute drop requested +- **THEN** migration succeeds, reports one `input_attributes_dropped` warning per affected input, and the v4 Library encodes as v4 JSON + +#### Scenario: Default upgrade stays lossless +- **WHEN** the same model is migrated to v4 without the request +- **THEN** no input attribute changes and the v3 to v4 to v3 round trip is semantically equal + +#### Scenario: Inputs that lose nothing report nothing +- **WHEN** every input's attributes are empty or hold only its declared type +- **THEN** the drop reports no warning diff --git a/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/specs/ir/v4/spec.md b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/specs/ir/v4/spec.md new file mode 100644 index 00000000..d839b273 --- /dev/null +++ b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/specs/ir/v4/spec.md @@ -0,0 +1,83 @@ +# Spec Delta + +## MODIFIED Requirements + +### Requirement: Draft v4 JSON codec +The library SHALL encode and decode v4 versioned distributions as `formatVersion` first and a distribution wrapped by its kind (`Library`, `Specs`, `Application`) using finos/morphir v4 JSON profile spellings at a pinned commit. The codec MUST be documented as draft and MUST NOT write morphir-elm format-3 shapes. (bead: morphir-ukvn.8) + +#### Scenario: V4 round trip +- **WHEN** a v4 Library built only from constructs the codec supports is encoded and decoded +- **THEN** the result equals the original, except that the package release version is absent + +#### Scenario: V4 encode is not format 3 +- **WHEN** a v4 distribution is encoded +- **THEN** the output declares `formatVersion` 4 and uses no format-3 tagged-array distribution shape + +#### Scenario: Unsupported spelling is refused +- **WHEN** v4 input contains a node kind or member the codec does not recognize +- **THEN** decoding fails with a structured error that names the node and its location, and does not guess + +#### Scenario: Construct with no draft spelling +- **WHEN** a caller encodes a v4 value that has no spelling in the pinned v4 JSON profile, such as the deprecated Python `Native` or `External` value nodes, structured type constraints, or a value-definition input whose attributes differ from its declared type +- **THEN** encoding fails with `no_v4_spelling` naming the construct rather than dropping it + +## ADDED Requirements + +### Requirement: Value definitions have upstream bodies +A v4 value definition, at module level or in a let binding, SHALL be one of four bodies: an expression body, a native body with a native hint, an external body with one or more platform bindings and an optional fallback expression, or an incomplete body with an incompleteness and an optional partial expression. Native, external and incomplete bodies MUST hold bare input types. (bead: morphir-ukvn.8.14) + +#### Scenario: Native body round trips +- **WHEN** a module value with a native body and an `Arithmetic` hint is encoded and decoded +- **THEN** the JSON is `{"NativeBody": {"inputTypes": ..., "outputType": ..., "nativeInfo": {"hint": {"Arithmetic": {}}}}}` and the decoded value equals the original + +#### Scenario: External platforms are unique +- **WHEN** an external body names the same target platform twice +- **THEN** encoding and decoding fail with `duplicate_member` + +#### Scenario: External body needs a binding +- **WHEN** an external body has no platform binding +- **THEN** encoding and decoding fail with a structured error + +#### Scenario: Platform-specific hint names its platform +- **WHEN** a native body's `PlatformSpecific` hint has no `platform` member +- **THEN** decoding fails with `missing_member` + +### Requirement: Incomplete definitions and typed hole reasons +The v4 model SHALL represent an incompleteness as a hole with a typed reason (`UnresolvedReference`, `DeletedDuringRefactor` or `TypeMismatch`) and an optional partial type, or as a draft. A type definition MAY be incomplete, with type parameters, an incompleteness and an optional partial type expression. A `Hole` value SHALL carry a typed reason. (bead: morphir-ukvn.8.14) + +#### Scenario: Incomplete type definition round trips +- **WHEN** an incomplete type definition whose hole reason is a type mismatch is encoded and decoded +- **THEN** the JSON is `{"IncompleteTypeDefinition": {"typeParams": [...], "incompleteness": {"Hole": {"reason": {"TypeMismatch": {"expected": ..., "found": ...}}}}}}` and the decoded definition equals the original + +#### Scenario: Draft is not a hole reason +- **WHEN** a hole reason is spelled `{"Draft": {}}` +- **THEN** decoding fails with `unknown_node` + +### Requirement: Specifications carry annotations +Module, type and value specifications SHALL carry an ordered list of annotations. An annotation SHALL be either compact (a name and optional free text) or structured (a name and positional or named value arguments). The codec SHALL write `annotations` only when the list is non-empty and SHALL omit an empty `arguments` member. Definitions MUST NOT carry annotations. (bead: morphir-ukvn.8.14) + +#### Scenario: Compact and structured annotations round trip +- **WHEN** a type alias specification with a compact annotation with text, a structured annotation with one positional and one named argument, and a structured annotation without arguments is encoded and decoded +- **THEN** the compact annotation is written as `"pkg:mod#name:text"`, the argument-less structured annotation as `{"name": ...}`, and the decoded specification equals the original + +#### Scenario: Definitions refuse annotations +- **WHEN** a type definition carries an `annotations` member +- **THEN** decoding fails with `unknown_member` + +### Requirement: Union type expression +The v4 model SHALL represent an untagged union as a type expression with attributes and an ordered list of two or more alternatives. The codec SHALL preserve alternative order, duplicates and nesting, and MUST refuse a union with fewer than two alternatives. (bead: morphir-ukvn.8.14) + +#### Scenario: Union round trips +- **WHEN** a union of `String` and `Int` is encoded and decoded +- **THEN** the JSON is `{"Union": {"alternatives": [..., ...]}}` and the decoded type equals the original + +#### Scenario: Singleton union is refused +- **WHEN** a union has one alternative +- **THEN** encoding and decoding fail with a structured error + +### Requirement: Python-only v4 constructs are deprecated +The Python `Native` and `External` value nodes and the structured type constraints SHALL stay constructible for compatibility, SHALL be documented as deprecated because upstream v4 has no spelling for them, and MUST keep failing encoding with `no_v4_spelling`. (bead: morphir-ukvn.8.14) + +#### Scenario: Deprecated value node still refuses to encode +- **WHEN** a caller encodes a value that contains a Python `Native` value node +- **THEN** encoding fails with `no_v4_spelling`, and the documentation directs the caller to a native definition body diff --git a/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/specs/tools/ir-interchange/spec.md b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/specs/tools/ir-interchange/spec.md new file mode 100644 index 00000000..2260b07d --- /dev/null +++ b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/specs/tools/ir-interchange/spec.md @@ -0,0 +1,26 @@ +# Spec Delta + +## MODIFIED Requirements + +### Requirement: Convert between v3 and v4 +The `ir convert` command SHALL convert format 3.0.0 to v4 and v4 to format 3.0.0 through the library migrations. It SHALL accept an optional package version and an opt-in flag to drop per-input attributes for upgrades, and an opt-in flag to drop source locations on downgrade. Success writes the target file; failure exits non-zero, prints each diagnostic, and leaves no output file. (bead: morphir-ukvn.8) + +#### Scenario: Upgrade v3 file to v4 +- **WHEN** the user converts a valid format-3 Library file to v4 +- **THEN** the tool writes a `formatVersion` 4 file and exits 0 + +#### Scenario: Downgrade failure is visible +- **WHEN** the user converts a v4 file that cannot be represented in format 3.0.0 +- **THEN** the tool exits non-zero, prints each diagnostic code and location, and does not create the output file + +#### Scenario: Downgrade warnings are printed +- **WHEN** a downgrade succeeds with dropped information +- **THEN** the tool prints each warning code and still exits 0 + +#### Scenario: Upgrade with input attributes needs opt-in +- **WHEN** the user converts the morphir-elm reference model to v4 without `--drop-input-attributes` +- **THEN** the tool exits non-zero with `no_v4_spelling`, names the flag, and does not create the output file + +#### Scenario: Upgrade drops input attributes on request +- **WHEN** the user converts the same file with `--drop-input-attributes` +- **THEN** the tool writes a `formatVersion` 4 file, prints each `input_attributes_dropped` warning, and exits 0 diff --git a/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/tasks.md b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/tasks.md new file mode 100644 index 00000000..d9b44620 --- /dev/null +++ b/openspec/changes/archive/2026-10-27-align-ir-v4-with-upstream/tasks.md @@ -0,0 +1,48 @@ +# Tasks + +## 1. Per-input attribute drop (morphir-ukvn.8.15) + +- [x] 1.1 Add `INPUT_ATTRIBUTES_DROPPED = "input_attributes_dropped"` to `morphir.ir.diagnostics` and `drop_input_attributes(lib)` to `morphir.ir.migrate`, which resets every module and let-bound expression-body input to its declared type and reports one warning per input that lost information; verify unit tests for a changed input, an unchanged input, a let-bound input, and the warning location +- [x] 1.2 Add `drop_input_attributes: bool = False` to `v3_to_v4` and verify that the morphir-elm reference model migrates with 10 warnings (8 module-level and 2 let-bound inputs) and encodes as v4 JSON with the flag, and that without it the v3 to v4 to v3 round trip is still semantically equal +- [x] 1.3 Add `ir convert --drop-input-attributes`, mention the flag in the `no_v4_spelling` input error, and verify CLI tests: without the flag the reference model fails with no output file; with it the tool writes v4 JSON, prints the warnings, and exits 0 +- [x] 1.4 Document the drop in the `packages/morphir` README and the codec docstring's gap list, and verify the documented example runs + +## 2. Typed hole reasons, incompleteness and incomplete type definitions (morphir-ukvn.8.14.2) + +- [x] 2.1 Add `morphir.ir.v4.incompleteness` with `UnresolvedReference`, `DeletedDuringRefactor`, `TypeMismatch`, `HoleReason`, `HoleIncompleteness`, `DraftIncompleteness` and `Incompleteness`, re-exported from `morphir.ir.v4`; verify construction and import-identity tests +- [x] 2.2 Type `Hole.reason` as `HoleReason`, encode and decode reasons with their kit members, and verify that `{"Draft": {}}` as a reason fails with `unknown_node` and a non-object reason fails with a structured error +- [x] 2.3 Add `type_def.IncompleteTypeDefinition(type_params, incompleteness, partial_type_exp)` and its codec spelling, and verify round trips for kit `definitions-0014`, `-0015` and `-0024` in expanded form +- [x] 2.4 Make reverse migration fail with `unsupported_v4_downgrade` for an incomplete type definition and verify the construct name and location + +## 3. Value-definition bodies (morphir-ukvn.8.14.1) + +- [x] 3.1 Add `NativeHint` kinds, `NativeInfo`, `ExternalBinding`, `NativeBody`, `ExternalBody`, `IncompleteBody` and `ValueDefinitionBody` to `morphir.ir.v4.value`, widen module values and let bindings to `ValueDefinitionBody`, and verify the type checkers pass +- [x] 3.2 Encode and decode the three bodies with the kit members, and verify round trips for kit `definitions-0007` to `-0009`, `-0016` and `-0025` in expanded form +- [x] 3.3 Refuse duplicate target platforms (`duplicate_member`), an empty `externals` list, and a `PlatformSpecific` hint without `platform` (`missing_member`), and verify one test each +- [x] 3.4 Make reverse migration fail with `unsupported_v4_downgrade` for each new body at module level and in a let binding, and verify the locations +- [x] 3.5 Make `drop_input_attributes` and the expression-body checks skip the new bodies, which hold bare input types, and verify a Library that mixes bodies encodes + +## 4. Annotations on specifications (morphir-ukvn.8.14.3) + +- [x] 4.1 Add `morphir.ir.v4.annotation` with `CompactAnnotation`, `StructuredAnnotation`, `PositionalArgument`, `NamedArgument` and `Annotation`, and an `annotations` field (empty default, last) on module, type and value specifications; verify existing positional constructors still work +- [x] 4.2 Encode and decode annotations (written first, only when non-empty; `arguments` omitted when empty; compact text after the first colon after `#`), and verify kit `definitions-0020` to `-0022` round trip and that `"arguments": []` reads as an argument-less structured annotation +- [x] 4.3 Verify that a type definition with `annotations` fails decoding with `unknown_member` (kit `definitions-0023`) +- [x] 4.4 Make reverse migration fail with `unsupported_v4_downgrade` for annotations on a specification in a v4 dependency, and verify the location + +## 5. Union type expression (morphir-ukvn.8.14.4) + +- [x] 5.1 Add `type.Union(attributes, alternatives)` to the `Type` union and `map_attributes`, and verify a substitution-style attribute map keeps order and duplicates +- [x] 5.2 Encode and decode `{"Union": {"attributes"?, "alternatives": [...]}}`, refuse fewer than two alternatives both ways, and verify round trips including a nested union +- [x] 5.3 Make reverse migration fail with `unsupported_v4_downgrade` for a union and verify the location + +## 6. Deprecations, docs and integration checks (morphir-ukvn.8.14.5) + +- [x] 6.1 Mark the `Native` and `External` value nodes and `TypeConstraints` as deprecated in docstrings, naming the replacement, and verify a test that they still fail encoding with `no_v4_spelling` +- [x] 6.2 Update the codec module docstring's gap list and the README "Morphir IR versions" section for the new constructs and the typed hole reason, and verify the README examples run +- [x] 6.3 File a follow-up bead to remove the deprecated constructs and verify it is a child of `morphir-ukvn.8` with label `repo:morphir-python` +- [x] 6.4 Run `mise run check` and `mise run test-bdd` and verify lint, both type checkers, tests and coverage of at least 80% pass + +## Workflow follow-up + +- Archive the change in the same PR once all tasks are done, fix the design asset prefix, and relink `morphir-ukvn.8.14` to the archived path +- `morphir-ukvn.8.16` re-syncs the codec with upstream compact spellings when upstream v4 settles diff --git a/openspec/specs/ir/migrate/spec.md b/openspec/specs/ir/migrate/spec.md index 2e7f66e6..2f1a0c91 100644 --- a/openspec/specs/ir/migrate/spec.md +++ b/openspec/specs/ir/migrate/spec.md @@ -27,6 +27,10 @@ The library SHALL migrate v4 to v3. When the input uses concepts v3 cannot repre - **WHEN** a v4 Library contains a `Hole`, `Native`, or `External` value, or a `DocumentLiteral` - **THEN** migration fails with `unsupported_v4_downgrade` naming the construct and its location +#### Scenario: V4-only definitions and types are reported +- **WHEN** a v4 Library contains a native, external or incomplete value-definition body, an incomplete type definition, a `Union` type, or a specification with annotations +- **THEN** migration fails with `unsupported_v4_downgrade` naming the construct and its location + #### Scenario: Missing types are reported - **WHEN** a v4 value, pattern, or value-definition input has no `inferred_type` - **THEN** migration fails with `missing_inferred_type` at that location @@ -67,3 +71,18 @@ Migration APIs SHALL return a success value with any warnings, or a failure hold #### Scenario: Failure is machine-readable - **WHEN** reverse migration fails - **THEN** callers can read each diagnostic's code, construct, and location without parsing message text + +### Requirement: Per-input attributes drop on request +The library SHALL let a caller drop the value attributes of value-definition inputs, because v4 JSON spells an input as a bare type. On request, each input SHALL keep only its declared type as `inferred_type`, and each input that lost information SHALL report `input_attributes_dropped` at its location. Without the request, forward migration MUST keep input attributes unchanged. (bead: morphir-ukvn.8.15) + +#### Scenario: Reference model upgrades to encodable v4 +- **WHEN** the morphir-elm reference model, whose inputs carry resolved types that differ from their declared types, is migrated to v4 with the input-attribute drop requested +- **THEN** migration succeeds, reports one `input_attributes_dropped` warning per affected input, and the v4 Library encodes as v4 JSON + +#### Scenario: Default upgrade stays lossless +- **WHEN** the same model is migrated to v4 without the request +- **THEN** no input attribute changes and the v3 to v4 to v3 round trip is semantically equal + +#### Scenario: Inputs that lose nothing report nothing +- **WHEN** every input's attributes are empty or hold only its declared type +- **THEN** the drop reports no warning diff --git a/openspec/specs/ir/v4/spec.md b/openspec/specs/ir/v4/spec.md index f616ab0d..d84dba10 100644 --- a/openspec/specs/ir/v4/spec.md +++ b/openspec/specs/ir/v4/spec.md @@ -42,8 +42,8 @@ The library SHALL encode and decode v4 versioned distributions as `formatVersion - **THEN** decoding fails with a structured error that names the node and its location, and does not guess #### Scenario: Construct with no draft spelling -- **WHEN** a caller encodes a v4 value that has no spelling in the pinned v4 JSON profile, such as the Python `Native` or `External` value nodes -- **THEN** encoding fails with a structured error naming the construct rather than dropping it +- **WHEN** a caller encodes a v4 value that has no spelling in the pinned v4 JSON profile, such as the deprecated Python `Native` or `External` value nodes, structured type constraints, or a value-definition input whose attributes differ from its declared type +- **THEN** encoding fails with `no_v4_spelling` naming the construct rather than dropping it ### Requirement: Draft status is explicit Documentation for the v4 model and codec SHALL state that IR v4 is a draft upstream, name the pinned finos/morphir commit, and state that spellings may change in later minor releases of this package. (bead: morphir-ukvn.8) @@ -51,3 +51,62 @@ Documentation for the v4 model and codec SHALL state that IR v4 is a draft upstr #### Scenario: Draft notice is present - **WHEN** a user reads the v4 codec documentation - **THEN** it identifies v4 as draft and names the pinned upstream commit + +### Requirement: Value definitions have upstream bodies +A v4 value definition, at module level or in a let binding, SHALL be one of four bodies: an expression body, a native body with a native hint, an external body with one or more platform bindings and an optional fallback expression, or an incomplete body with an incompleteness and an optional partial expression. Native, external and incomplete bodies MUST hold bare input types. (bead: morphir-ukvn.8.14) + +#### Scenario: Native body round trips +- **WHEN** a module value with a native body and an `Arithmetic` hint is encoded and decoded +- **THEN** the JSON is `{"NativeBody": {"inputTypes": ..., "outputType": ..., "nativeInfo": {"hint": {"Arithmetic": {}}}}}` and the decoded value equals the original + +#### Scenario: External platforms are unique +- **WHEN** an external body names the same target platform twice +- **THEN** encoding and decoding fail with `duplicate_member` + +#### Scenario: External body needs a binding +- **WHEN** an external body has no platform binding +- **THEN** encoding and decoding fail with a structured error + +#### Scenario: Platform-specific hint names its platform +- **WHEN** a native body's `PlatformSpecific` hint has no `platform` member +- **THEN** decoding fails with `missing_member` + +### Requirement: Incomplete definitions and typed hole reasons +The v4 model SHALL represent an incompleteness as a hole with a typed reason (`UnresolvedReference`, `DeletedDuringRefactor` or `TypeMismatch`) and an optional partial type, or as a draft. A type definition MAY be incomplete, with type parameters, an incompleteness and an optional partial type expression. A `Hole` value SHALL carry a typed reason. (bead: morphir-ukvn.8.14) + +#### Scenario: Incomplete type definition round trips +- **WHEN** an incomplete type definition whose hole reason is a type mismatch is encoded and decoded +- **THEN** the JSON is `{"IncompleteTypeDefinition": {"typeParams": [...], "incompleteness": {"Hole": {"reason": {"TypeMismatch": {"expected": ..., "found": ...}}}}}}` and the decoded definition equals the original + +#### Scenario: Draft is not a hole reason +- **WHEN** a hole reason is spelled `{"Draft": {}}` +- **THEN** decoding fails with `unknown_node` + +### Requirement: Specifications carry annotations +Module, type and value specifications SHALL carry an ordered list of annotations. An annotation SHALL be either compact (a name and optional free text) or structured (a name and positional or named value arguments). The codec SHALL write `annotations` only when the list is non-empty and SHALL omit an empty `arguments` member. Definitions MUST NOT carry annotations. (bead: morphir-ukvn.8.14) + +#### Scenario: Compact and structured annotations round trip +- **WHEN** a type alias specification with a compact annotation with text, a structured annotation with one positional and one named argument, and a structured annotation without arguments is encoded and decoded +- **THEN** the compact annotation is written as `"pkg:mod#name:text"`, the argument-less structured annotation as `{"name": ...}`, and the decoded specification equals the original + +#### Scenario: Definitions refuse annotations +- **WHEN** a type definition carries an `annotations` member +- **THEN** decoding fails with `unknown_member` + +### Requirement: Union type expression +The v4 model SHALL represent an untagged union as a type expression with attributes and an ordered list of two or more alternatives. The codec SHALL preserve alternative order, duplicates and nesting, and MUST refuse a union with fewer than two alternatives. (bead: morphir-ukvn.8.14) + +#### Scenario: Union round trips +- **WHEN** a union of `String` and `Int` is encoded and decoded +- **THEN** the JSON is `{"Union": {"alternatives": [..., ...]}}` and the decoded type equals the original + +#### Scenario: Singleton union is refused +- **WHEN** a union has one alternative +- **THEN** encoding and decoding fail with a structured error + +### Requirement: Python-only v4 constructs are deprecated +The Python `Native` and `External` value nodes and the structured type constraints SHALL stay constructible for compatibility, SHALL be documented as deprecated because upstream v4 has no spelling for them, and MUST keep failing encoding with `no_v4_spelling`. (bead: morphir-ukvn.8.14) + +#### Scenario: Deprecated value node still refuses to encode +- **WHEN** a caller encodes a value that contains a Python `Native` value node +- **THEN** encoding fails with `no_v4_spelling`, and the documentation directs the caller to a native definition body diff --git a/openspec/specs/tools/ir-interchange/spec.md b/openspec/specs/tools/ir-interchange/spec.md index 9c97f64f..77091c3f 100644 --- a/openspec/specs/tools/ir-interchange/spec.md +++ b/openspec/specs/tools/ir-interchange/spec.md @@ -28,7 +28,7 @@ The tools MUST NOT parse IR JSON with ad-hoc CLI code; reading, decoding, encodi - **THEN** the tool reports the same diagnostic codes the library returns ### Requirement: Convert between v3 and v4 -The `ir convert` command SHALL convert format 3.0.0 to v4 and v4 to format 3.0.0 through the library migrations. It SHALL accept an optional package version for upgrades and an opt-in flag to drop source locations on downgrade. Success writes the target file; failure exits non-zero, prints each diagnostic, and leaves no output file. (bead: morphir-ukvn.8) +The `ir convert` command SHALL convert format 3.0.0 to v4 and v4 to format 3.0.0 through the library migrations. It SHALL accept an optional package version and an opt-in flag to drop per-input attributes for upgrades, and an opt-in flag to drop source locations on downgrade. Success writes the target file; failure exits non-zero, prints each diagnostic, and leaves no output file. (bead: morphir-ukvn.8) #### Scenario: Upgrade v3 file to v4 - **WHEN** the user converts a valid format-3 Library file to v4 @@ -42,6 +42,14 @@ The `ir convert` command SHALL convert format 3.0.0 to v4 and v4 to format 3.0.0 - **WHEN** a downgrade succeeds with dropped information - **THEN** the tool prints each warning code and still exits 0 +#### Scenario: Upgrade with input attributes needs opt-in +- **WHEN** the user converts the morphir-elm reference model to v4 without `--drop-input-attributes` +- **THEN** the tool exits non-zero with `no_v4_spelling`, names the flag, and does not create the output file + +#### Scenario: Upgrade drops input attributes on request +- **WHEN** the user converts the same file with `--drop-input-attributes` +- **THEN** the tool writes a `formatVersion` 4 file, prints each `input_attributes_dropped` warning, and exits 0 + ### Requirement: Scope excludes formats 1, 2, and 3.1 For this change, the tools MUST reject formats 1, 2, and 3.1.x rather than load or convert them. This differs from morphir-elm, which still reads format 1. (bead: morphir-ukvn.8) diff --git a/packages/morphir-tools/src/morphir_tools/cli.py b/packages/morphir-tools/src/morphir_tools/cli.py index 1a44bd9b..d69a970e 100644 --- a/packages/morphir-tools/src/morphir_tools/cli.py +++ b/packages/morphir-tools/src/morphir_tools/cli.py @@ -20,7 +20,8 @@ from morphir.ir import format_version as fv from morphir.ir.diagnostics import Diagnostic, DiagnosticError -from morphir.ir.migrate import Err, v3_to_v4, v4_to_v3 +from morphir.ir.migrate import Err, Ok, v3_to_v4, v4_to_v3 +from morphir.ir.migrate import drop_input_attributes as drop_inputs from morphir.ir.v3 import codec as v3_codec from morphir.ir.v4 import codec as v4_codec from morphir.ir.v4 import distribution as v4_dist @@ -128,6 +129,13 @@ def convert( bool, typer.Option(help="Drop value source locations when downgrading to v3."), ] = False, + drop_input_attributes: Annotated[ + bool, + typer.Option( + help="Keep only the declared type of each value-definition input when " + "writing v4, which spells an input as a bare type." + ), + ] = False, ) -> None: """Convert between format 3.0.0 and draft IR v4. @@ -138,16 +146,19 @@ def convert( try: version, _ = fv.read_document(text) if to is Target.V4: + upgraded: Ok[v4_dist.Distribution] if version.major == 4: - dist = v4_codec.decode_versioned_distribution(text) + upgraded = Ok(v4_codec.decode_versioned_distribution(text)) else: - upgraded = v3_to_v4( + library_v4 = v3_to_v4( v3_codec.decode_versioned_distribution(text), package_version=package_version, ) - dist = upgraded.value - document = v4_codec.encode_versioned_distribution(dist) - warnings: tuple[Diagnostic, ...] = () + upgraded = Ok(library_v4.value, library_v4.warnings) + if drop_input_attributes: + upgraded = drop_inputs(upgraded.value) + document = v4_codec.encode_versioned_distribution(upgraded.value) + warnings: tuple[Diagnostic, ...] = upgraded.warnings else: if version.major == 3: library = v3_codec.decode_versioned_distribution(text) diff --git a/packages/morphir/README.md b/packages/morphir/README.md index 796eeaf3..1fc7fa9a 100644 --- a/packages/morphir/README.md +++ b/packages/morphir/README.md @@ -97,6 +97,37 @@ the distribution wrapped by its kind, never a format-3 shape. The module docstring lists the gaps between the Python v4 model and upstream v4, each of which fails with a diagnostic instead of guessing or dropping data. +The v4 model follows the upstream reference model at that commit: + +- A value definition is one of four bodies (`ValueDefinitionBody`): the + expression body `ValueDefinition`, `NativeBody`, `ExternalBody` and + `IncompleteBody`. Module values and let bindings hold any of them. +- `IncompleteTypeDefinition` and `IncompleteBody` carry an `Incompleteness`: a + hole with a typed `HoleReason` (`UnresolvedReference`, `DeletedDuringRefactor`, + `TypeMismatch`) or a draft. A `Hole` value's `reason` is a `HoleReason` too. +- Module, type and value specifications carry `annotations`, compact + (`CompactAnnotation`) or structured (`StructuredAnnotation`). +- `morphir.ir.v4.type.Union` is the draft untagged union, with two or more + alternatives. +- The `Native` and `External` value nodes and `TypeConstraints` are deprecated: + upstream v4 cannot spell them. + +```python +from morphir.ir.v4 import Name, codec +from morphir.ir.v4 import type as t +from morphir.ir.v4 import value as v + +int_type = t.Reference( + t.TypeAttributes(), codec.decode_fqname("morphir/SDK:basics#int") +) +add = v.NativeBody( + [(Name(("a",)), int_type), (Name(("b",)), int_type)], + int_type, + v.NativeInfo(v.Arithmetic()), +) +assert codec.decode_value_definition(codec.encode_value_definition(add)) == add +``` + `PackageInfo.version` is optional (`str | None`). Neither format 3 nor v4 JSON has a package release version, so the draft codec does not write it and decodes it as absent. `v3_to_v4` sets it only from `package_version`, and `v4_to_v3` drops it @@ -107,10 +138,36 @@ with the warning `package_version_dropped`. `morphir.ir.migrate.v3_to_v4` always succeeds for a v3 Library. `v4_to_v3` returns `Ok(value, warnings)` or `Err(diagnostics)` and reports every construct format 3.0.0 cannot hold in one pass: Specs and Application -distributions, `Hole`, `Native`, `External`, `DocumentLiteral`, a missing -`inferred_type`, semantic attributes, and missing member documentation. Value +distributions, `Hole`, `Native`, `External`, `DocumentLiteral`, native, external +and incomplete bodies, incomplete type definitions, `Union`, annotations, a +missing `inferred_type`, semantic attributes, and missing member documentation. Value source locations fail unless you pass `drop_source_locations=True`. +v4 JSON spells a value-definition input as a bare type, so it has no place for +the per-input attributes of a typed v3 model. morphir-elm writes the resolved +type there, which can differ from the declared type (an alias expanded, a type +variable instantiated), and the draft codec then fails with `no_v4_spelling`. +Opt in to dropping them with `v3_to_v4(..., drop_input_attributes=True)`, +`drop_input_attributes(dist)` for any v4 distribution, or +`morphir-tools ir convert --to v4 --drop-input-attributes`. Each input that loses +information reports `input_attributes_dropped`; without the opt-in the in-memory +migration stays lossless. + +```python +from pathlib import Path + +from morphir.ir.migrate import v3_to_v4 +from morphir.ir.v3 import codec as v3_codec +from morphir.ir.v4 import codec as v4_codec + +text = Path("tests/fixtures/ir/v3/elm-reference-model.json").read_text() +upgraded = v3_to_v4( + v3_codec.decode_versioned_distribution(text), drop_input_attributes=True +) +assert {w.code for w in upgraded.warnings} == {"input_attributes_dropped"} +assert v4_codec.encode_versioned_distribution(upgraded.value)["formatVersion"] == 4 +``` + ## Morphir SDK `morphir.sdk` is the standard library that Morphir models compile against. The diff --git a/packages/morphir/src/morphir/ir/diagnostics.py b/packages/morphir/src/morphir/ir/diagnostics.py index 1a4212a8..0180111f 100644 --- a/packages/morphir/src/morphir/ir/diagnostics.py +++ b/packages/morphir/src/morphir/ir/diagnostics.py @@ -10,8 +10,9 @@ - downgrade codes come from the finos CLI and morphir-moonbit ``migration/v3/migration.mbt``, and may be renamed when finos bead ``morphir-diwy`` settles; -- ``package_version_dropped``, ``source_location_dropped``, and - ``no_v4_spelling`` are local to morphir-python. +- ``package_version_dropped``, ``source_location_dropped``, + ``input_attributes_dropped``, and ``no_v4_spelling`` are local to + morphir-python. """ from __future__ import annotations @@ -48,6 +49,7 @@ # Warnings (morphir-python). PACKAGE_VERSION_DROPPED = "package_version_dropped" SOURCE_LOCATION_DROPPED = "source_location_dropped" +INPUT_ATTRIBUTES_DROPPED = "input_attributes_dropped" Location = tuple[str | int, ...] diff --git a/packages/morphir/src/morphir/ir/migrate.py b/packages/morphir/src/morphir/ir/migrate.py index 50ec229a..2097cc63 100644 --- a/packages/morphir/src/morphir/ir/migrate.py +++ b/packages/morphir/src/morphir/ir/migrate.py @@ -6,7 +6,11 @@ - the package release version, which v3 cannot hold (``package_version_dropped``); - value source locations, only when ``drop_source_locations=True`` - (``source_location_dropped``). + (``source_location_dropped``); +- value-definition input attributes, only when ``drop_input_attributes=True`` or + through `drop_input_attributes` (``input_attributes_dropped``). v4 JSON spells + an input as a bare type, so an input whose attributes hold more than its + declared type cannot be encoded without this drop. The mapping follows morphir-moonbit ``migration/v3/migration.mbt``, adjusted for the Python v4 model. Diagnostic codes are defined in `morphir.ir.diagnostics` and @@ -22,8 +26,10 @@ from __future__ import annotations +import dataclasses import re from dataclasses import dataclass, field +from typing import cast from . import diagnostics as d from .diagnostics import Diagnostic, Location @@ -39,6 +45,7 @@ from .v3 import type as t3 from .v3 import value as v3 from .v4 import access_controlled as ac4 +from .v4 import annotation as ann4 from .v4 import distribution as dist4 from .v4 import documented as doc4 from .v4 import fqname as fq4 @@ -334,7 +341,10 @@ def _package_def_up(defn: pkg3.Definition[None, t3.Type[None]]) -> pkg4.Definiti def v3_to_v4( - dist: dist3.Library, *, package_version: str | None = None + dist: dist3.Library, + *, + package_version: str | None = None, + drop_input_attributes: bool = False, ) -> Ok[dist4.LibraryDistribution]: """Migrate a format-3 Library to a v4 Library. @@ -345,19 +355,299 @@ def v3_to_v4( dist: The v3 Library. package_version: Optional release version for the v4 package identity. Without it the version is absent; nothing is invented. + drop_input_attributes: Reset each value-definition input's attributes to + its declared type, as `drop_input_attributes` does, so the result can + be encoded as v4 JSON. Each input that loses information is reported. Returns: - ``Ok`` with the v4 Library. Forward migration has no failure paths. + ``Ok`` with the v4 Library and any ``input_attributes_dropped`` warnings. + Forward migration has no failure paths. """ - return Ok( - dist4.LibraryDistribution( - package=dist4.PackageInfo(_path_up(dist.package_name), package_version), - definition=_package_def_up(dist.definition), - dependencies={ - _path_up(p): _package_spec_up(s) for p, s in dist.dependencies.items() - }, - ) + library = dist4.LibraryDistribution( + package=dist4.PackageInfo(_path_up(dist.package_name), package_version), + definition=_package_def_up(dist.definition), + dependencies={ + _path_up(p): _package_spec_up(s) for p, s in dist.dependencies.items() + }, ) + if drop_input_attributes: + return _drop_input_attributes(library) + return Ok(library) + + +# --- Per-input attributes --- + + +class _InputDrop: + """Resets value-definition input attributes, recording each loss.""" + + def __init__(self) -> None: + self.warnings: list[Diagnostic] = [] + + def definition( + self, defn: v4.ValueDefinitionBody, loc: Location + ) -> v4.ValueDefinitionBody: + match defn: + case v4.Definition(): + return self.expression(defn, loc) + case v4.ExternalBody(inputs, output, externals, fallback): + here = (*loc, "ExternalBody", "body") + return v4.ExternalBody( + inputs, + output, + externals, + None if fallback is None else self.value(fallback, here), + ) + case v4.IncompleteBody(inputs, incompleteness, output, partial): + here = (*loc, "IncompleteBody", "partial_body") + return v4.IncompleteBody( + inputs, + incompleteness, + output, + None if partial is None else self.value(partial, here), + ) + case v4.NativeBody(): + return defn + + def expression(self, defn: v4.Definition, loc: Location) -> v4.Definition: + inputs: list[tuple[name4.Name, v4.ValueAttributes, t4.Type]] = [] + for n, attrs, tpe in defn.input_types: + declared = v4.ValueAttributes(inferred_type=tpe) + if attrs not in (v4.ValueAttributes(), declared): + self.warnings.append( + Diagnostic( + d.INPUT_ATTRIBUTES_DROPPED, + "ValueDefinitionInput", + (*loc, "input_types", name4.to_string(n)), + f"input {name4.to_string(n)!r} kept only its declared type; " + "v4 JSON spells an input as a bare type", + ) + ) + inputs.append((n, declared, tpe)) + return v4.Definition( + inputs, defn.output_type, self.value(defn.body, (*loc, "body")) + ) + + def value(self, val: v4.Value, loc: Location) -> v4.Value: + here = (*loc, type(val).__name__) + a = val.attributes + match val: + case v4.Tuple(_, elements): + return v4.Tuple( + a, [self.value(x, (*here, i)) for i, x in enumerate(elements)] + ) + case v4.List(_, items): + return v4.List( + a, [self.value(x, (*here, i)) for i, x in enumerate(items)] + ) + case v4.Record(_, fields): + return v4.Record(a, self.fields(fields, here)) + case v4.Field(_, subject, n): + return v4.Field(a, self.value(subject, (*here, "subject")), n) + case v4.Apply(_, function, argument): + return v4.Apply( + a, + self.value(function, (*here, "function")), + self.value(argument, (*here, "argument")), + ) + case v4.Lambda(_, pattern, body): + return v4.Lambda(a, pattern, self.value(body, (*here, "body"))) + case v4.LetDefinition(_, n, defn, in_value): + return v4.LetDefinition( + a, + n, + self.definition(defn, (*here, "definition")), + self.value(in_value, (*here, "in_value")), + ) + case v4.LetRecursion(_, defs, in_value): + return v4.LetRecursion( + a, + { + k: self.definition( + x, (*here, "definitions", name4.to_string(k)) + ) + for k, x in defs.items() + }, + self.value(in_value, (*here, "in_value")), + ) + case v4.Destructure(_, pattern, subject, in_value): + return v4.Destructure( + a, + pattern, + self.value(subject, (*here, "value_to_destructure")), + self.value(in_value, (*here, "in_value")), + ) + case v4.IfThenElse(_, cond, then_branch, else_branch): + return v4.IfThenElse( + a, + self.value(cond, (*here, "condition")), + self.value(then_branch, (*here, "then_branch")), + self.value(else_branch, (*here, "else_branch")), + ) + case v4.PatternMatch(_, subject, cases): + return v4.PatternMatch( + a, + self.value(subject, (*here, "branch_on")), + [ + (p, self.value(b, (*here, "cases", i))) + for i, (p, b) in enumerate(cases) + ], + ) + case v4.UpdateRecord(_, subject, fields): + return v4.UpdateRecord( + a, + self.value(subject, (*here, "value_to_update")), + self.fields(fields, here), + ) + case _: + return val + + def fields( + self, items: list[tuple[name4.Name, v4.Value]], loc: Location + ) -> list[tuple[name4.Name, v4.Value]]: + return [(k, self.value(x, (*loc, name4.to_string(k)))) for k, x in items] + + def package_def(self, defn: pkg4.Definition, loc: Location) -> pkg4.Definition: + modules: dict[path4.Path, ac4.AccessControlled[mod4.Definition]] = {} + for p, m in defn.modules.items(): + here = (*loc, path4.to_string(p)) + values = { + k: _with_access( + x, + doc4.Documented( + x.value.doc, + self.definition( + x.value.value, (*here, "values", name4.to_string(k)) + ), + ), + ) + for k, x in m.value.values.items() + } + modules[p] = _with_access( + m, mod4.Definition(m.value.types, values, m.value.doc) + ) + return pkg4.Definition(modules) + + # Specifications: annotation arguments are expressions too. + + def annotations( + self, items: list[ann4.Annotation], loc: Location + ) -> list[ann4.Annotation]: + result: list[ann4.Annotation] = [] + for i, item in enumerate(items): + here = (*loc, "annotations", i) + if isinstance(item, ann4.CompactAnnotation): + result.append(item) + continue + arguments: list[ann4.AnnotationArgument] = [] + for j, arg in enumerate(item.arguments): + at = (*here, "arguments", j) + match arg: + case ann4.PositionalArgument(value): + arguments.append(ann4.PositionalArgument(self.value(value, at))) + case ann4.NamedArgument(name, value): + arguments.append( + ann4.NamedArgument(name, self.value(value, (*at, "value"))) + ) + result.append(ann4.StructuredAnnotation(item.name, arguments)) + return result + + def module_spec( + self, spec: mod4.Specification, loc: Location + ) -> mod4.Specification: + types = { + k: doc4.Documented( + x.doc, + dataclasses.replace( + x.value, + annotations=self.annotations( + x.value.annotations, (*loc, "types", name4.to_string(k)) + ), + ), + ) + for k, x in spec.types.items() + } + values = { + k: doc4.Documented( + x.doc, + dataclasses.replace( + x.value, + annotations=self.annotations( + x.value.annotations, (*loc, "values", name4.to_string(k)) + ), + ), + ) + for k, x in spec.values.items() + } + return mod4.Specification( + types, values, spec.doc, self.annotations(spec.annotations, loc) + ) + + def package_spec( + self, spec: pkg4.Specification, loc: Location + ) -> pkg4.Specification: + return pkg4.Specification( + { + p: self.module_spec(m, (*loc, path4.to_string(p))) + for p, m in spec.modules.items() + } + ) + + +def _with_access[A, B]( + item: ac4.AccessControlled[A], value: B +) -> ac4.AccessControlled[B]: + return ac4.Public(value) if isinstance(item, ac4.Public) else ac4.Private(value) + + +def drop_input_attributes[D: dist4.Distribution](dist: D) -> Ok[D]: + """Reset every value-definition input's attributes to its declared type. + + v4 JSON spells an input as a bare type (kit case ``definitions-0027``), so + the draft codec refuses an input whose attributes hold anything else, such as + the resolved types morphir-elm writes for typed inputs. This drop makes such + a distribution encodable. Module-level and let-bound definitions are covered. + + Args: + dist: Any v4 distribution. Dependencies and the expressions in + specification annotation arguments are covered too. + + Returns: + ``Ok`` with the distribution and one ``input_attributes_dropped`` warning + per input whose attributes were neither empty nor its declared type. + """ + return _drop_input_attributes(dist) + + +def _drop_input_attributes[D: dist4.Distribution](dist: D) -> Ok[D]: + job = _InputDrop() + loc: Location = ("distribution", "modules") + + def dep(p: path4.Path) -> Location: + return ("distribution", "dependencies", path4.to_string(p)) + + result: dist4.Distribution + match dist: + case dist4.SpecsDistribution(package, specification, deps): + result = dist4.SpecsDistribution( + package, + job.package_spec(specification, loc), + {p: job.package_spec(s, dep(p)) for p, s in deps.items()}, + ) + case dist4.LibraryDistribution(package, definition, deps): + result = dist4.LibraryDistribution( + package, + job.package_def(definition, loc), + {p: job.package_spec(s, dep(p)) for p, s in deps.items()}, + ) + case dist4.ApplicationDistribution(package, definition, deps, entry_points): + result = dist4.ApplicationDistribution( + package, + job.package_def(definition, loc), + {p: job.package_def(x, dep(p)) for p, x in deps.items()}, + entry_points, + ) + return Ok(cast("D", result), tuple(job.warnings)) # --- v4 -> v3 --- @@ -502,6 +792,9 @@ def type(self, tpe: t4.Type, loc: Location) -> t3.Type[None]: ) case t4.Unit(_): return t3.Unit(None) + case t4.Union(): + self.unsupported("Union", here) + return t3.Unit(None) def fields( self, fields: list[t4.Field], loc: Location @@ -523,9 +816,14 @@ def ctors(self, ctors: ts4.Constructors, loc: Location) -> t3.Constructors[None] def params(self, params: list[name4.Name], loc: Location) -> tuple[name3.Name, ...]: return tuple(self.name(p, loc) for p in params) + def annotations(self, items: list[ann4.Annotation], loc: Location) -> None: + if items: + self.unsupported("Annotations", (*loc, "annotations")) + def type_spec( self, spec: ts4.Specification, loc: Location ) -> t3.Specification[None]: + self.annotations(spec.annotations, loc) params = self.params(spec.type_params, loc) match spec: case ts4.TypeAliasSpecification(_, tpe): @@ -551,6 +849,11 @@ def type_def(self, defn: td4.Definition, loc: Location) -> t3.Definition[None]: return t3.CustomTypeDefinition( params, self.access(ctors, self.ctors(ctors.value, loc)) ) + case td4.IncompleteTypeDefinition(): + self.unsupported( + "IncompleteTypeDefinition", (*loc, "IncompleteTypeDefinition") + ) + return t3.TypeAliasDefinition(params, t3.Unit(None)) # Values @@ -729,7 +1032,13 @@ def value(self, val: v4.Value, loc: Location) -> TypedValue3: case _: return v3.Unit(a) - def value_def(self, defn: v4.Definition, loc: Location) -> TypedDefinition3: + def value_def( + self, defn: v4.ValueDefinitionBody, loc: Location + ) -> TypedDefinition3: + if not isinstance(defn, v4.Definition): + construct = type(defn).__name__ + self.unsupported(construct, (*loc, construct)) + return v3.Definition((), t3.Unit(None), v3.Unit(t3.Unit(None))) inputs = [] for n, va, tpe in defn.input_types: here = (*loc, "input_types", name4.to_string(n)) @@ -749,6 +1058,7 @@ def value_def(self, defn: v4.Definition, loc: Location) -> TypedDefinition3: def value_spec( self, spec: v4.Specification, loc: Location ) -> v3.Specification[None]: + self.annotations(spec.annotations, loc) return v3.Specification( tuple( (self.name(n, loc), self.type(tp, (*loc, "inputs", name4.to_string(n)))) @@ -762,6 +1072,7 @@ def value_spec( def module_spec( self, spec: mod4.Specification, loc: Location ) -> mod3.Specification[None]: + self.annotations(spec.annotations, loc) types = {} for n, x in spec.types.items(): here = (*loc, "types", name4.to_string(n)) @@ -896,4 +1207,11 @@ def v4_to_v3( return Ok(library, tuple(warnings)) -__all__ = ["Err", "MigrationResult", "Ok", "v3_to_v4", "v4_to_v3"] +__all__ = [ + "Err", + "MigrationResult", + "Ok", + "drop_input_attributes", + "v3_to_v4", + "v4_to_v3", +] diff --git a/packages/morphir/src/morphir/ir/type.py b/packages/morphir/src/morphir/ir/type.py index f2fd35fa..dbe02310 100644 --- a/packages/morphir/src/morphir/ir/type.py +++ b/packages/morphir/src/morphir/ir/type.py @@ -11,6 +11,7 @@ Tuple, Type, TypeAttributes, + Union, Unit, Variable, get_attributes, @@ -28,6 +29,7 @@ "Tuple", "Type", "TypeAttributes", + "Union", "Unit", "Variable", "get_attributes", diff --git a/packages/morphir/src/morphir/ir/v4/__init__.py b/packages/morphir/src/morphir/ir/v4/__init__.py index 58bd9b2c..3bbb8df9 100644 --- a/packages/morphir/src/morphir/ir/v4/__init__.py +++ b/packages/morphir/src/morphir/ir/v4/__init__.py @@ -4,8 +4,16 @@ draft pinned to finos/morphir commit efa36060. """ -from . import module, package +from . import annotation, incompleteness, module, package from .access_controlled import AccessControlled +from .annotation import ( + Annotation, + AnnotationArgument, + CompactAnnotation, + NamedArgument, + PositionalArgument, + StructuredAnnotation, +) from .decorations import ( DecorationFormat, DecorationValuesFile, @@ -33,6 +41,15 @@ ) from .documented import Documented from .fqname import FQName +from .incompleteness import ( + DeletedDuringRefactor, + DraftIncompleteness, + HoleIncompleteness, + HoleReason, + Incompleteness, + TypeMismatch, + UnresolvedReference, +) from .literal import ( BoolLiteral, CharLiteral, @@ -52,18 +69,33 @@ from .type_def import Definition as TypeDefinition from .type_spec import Specification as TypeSpecification from .value import Definition as ValueDefinition -from .value import Pattern, Value, ValueAttributes +from .value import ( + ExternalBinding, + ExternalBody, + IncompleteBody, + NativeBody, + NativeHint, + NativeInfo, + Pattern, + Value, + ValueAttributes, + ValueDefinitionBody, +) from .value import Specification as ValueSpecification __all__ = [ "AccessControlled", + "Annotation", + "AnnotationArgument", "ApplicationDistribution", "BoolLiteral", "CharLiteral", + "CompactAnnotation", "DecimalLiteral", "DecorationFormat", "DecorationValuesFile", "DefRef", + "DeletedDuringRefactor", "Distribution", "DocArray", "DocBool", @@ -74,37 +106,55 @@ "DocString", "Document", "Documented", + "DraftIncompleteness", "EntryPoint", "EntryPointKind", + "ExternalBinding", + "ExternalBody", "FQName", "Field", "FileMeta", "FileWithDefs", "FloatLiteral", + "HoleIncompleteness", + "HoleReason", + "IncompleteBody", + "Incompleteness", "IntegerLiteral", "LayerManifest", "LibraryDistribution", "Literal", "Name", + "NamedArgument", + "NativeBody", + "NativeHint", + "NativeInfo", "PackageInfo", "Path", "Pattern", "PointerRef", + "PositionalArgument", "QName", "Ref", "SchemaRef", "SourceRange", "SpecsDistribution", "StringLiteral", + "StructuredAnnotation", "Type", "TypeAttributes", "TypeConstraints", "TypeDefinition", + "TypeMismatch", "TypeSpecification", + "UnresolvedReference", "Value", "ValueAttributes", "ValueDefinition", + "ValueDefinitionBody", "ValueSpecification", + "annotation", + "incompleteness", "module", "package", ] diff --git a/packages/morphir/src/morphir/ir/v4/annotation.py b/packages/morphir/src/morphir/ir/v4/annotation.py new file mode 100644 index 00000000..e7ea176c --- /dev/null +++ b/packages/morphir/src/morphir/ir/v4/annotation.py @@ -0,0 +1,75 @@ +"""Annotations on v4 specifications. + +Follows the finos/morphir v4 schema (``Annotation``, ``AnnotationArgument``) at +the pinned commit and the morphir-rust v4 model. An annotation keeps the form it +was written in, because the kit writes an argument-less structured annotation as +an object and a compact one as a string. Annotations belong to specifications, +the public face; definitions carry none. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Union + +from .fqname import FQName +from .name import Name +from .value import Value + + +@dataclass(frozen=True) +class PositionalArgument: + """An annotation argument written on its own. + + Attributes: + value: The argument expression. + """ + + value: Value + + +@dataclass(frozen=True) +class NamedArgument: + """An annotation argument written under the name it answers to. + + Attributes: + name: The parameter name. + value: The argument expression. + """ + + name: Name + value: Value + + +AnnotationArgument = Union[PositionalArgument, NamedArgument] +"""A positional or named annotation argument.""" + + +@dataclass(frozen=True) +class CompactAnnotation: + """The compact spelling: ``"pkg:mod#name"`` or ``"pkg:mod#name:text"``. + + Attributes: + name: The annotation type. + text: The free text after the first colon after ``#``, if any. + """ + + name: FQName + text: str | None = None + + +@dataclass(frozen=True) +class StructuredAnnotation: + """The structured spelling: ``{"name": ..., "arguments": [...]}``. + + Attributes: + name: The annotation type. + arguments: Positional and named arguments, in order. + """ + + name: FQName + arguments: list[AnnotationArgument] = field(default_factory=list) + + +Annotation = Union[CompactAnnotation, StructuredAnnotation] +"""An annotation on a module, type or value specification.""" diff --git a/packages/morphir/src/morphir/ir/v4/codec.py b/packages/morphir/src/morphir/ir/v4/codec.py index 8c26b4f3..bd670de8 100644 --- a/packages/morphir/src/morphir/ir/v4/codec.py +++ b/packages/morphir/src/morphir/ir/v4/codec.py @@ -14,21 +14,28 @@ in `TYPE_NODES`, `VALUE_NODES`, `PATTERN_NODES`, and the other tables below, and fails with a located diagnostic on anything else; it does not guess. +The codec covers the value-definition bodies (``ExpressionBody``, ``NativeBody``, +``ExternalBody``, ``IncompleteBody``), ``IncompleteTypeDefinition`` with typed hole +reasons, annotations on specifications, and the draft ``Union`` type (from +``docs/spec/draft/types.md``, not yet in the pinned schema). + Known gaps between the Python v4 model and upstream v4, each handled explicitly: -- ``Native`` and ``External`` value nodes have no spelling (upstream makes them - definition bodies); encoding fails with ``no_v4_spelling``. -- Structured ``TypeConstraints`` have no spelling; encoding fails with - ``no_v4_spelling``. -- Upstream ``Union`` types, ``IncompleteTypeDefinition``, annotations, and the - ``NativeBody``, ``ExternalBody``, and ``IncompleteBody`` definition bodies are - not in the Python model; decoding fails with ``unknown_node`` or - ``unknown_member``. +- The deprecated Python ``Native`` and ``External`` value nodes have no spelling + (upstream makes them definition bodies); encoding fails with ``no_v4_spelling``. +- Structured ``TypeConstraints`` (deprecated) have no spelling; encoding fails + with ``no_v4_spelling``. +- Only the expanded spelling is read and written. Compact spellings, the legacy + spellings readers accept for one release, and the 4.1.0 linked annotations + wait for the re-sync when upstream v4 settles. - Records are objects keyed by field name (finos decision 0004); duplicate field names fail encoding with ``duplicate_member``. -- ``ExpressionBody.inputTypes`` has no attribute slot. Encoding accepts per-input +- ``ExpressionBody.inputTypes`` has no attribute slot: upstream spells an input as + a bare type (kit case ``definitions-0027``). Encoding accepts per-input attributes that are empty or whose only member is ``inferred_type`` equal to the - declared input type; decoding sets ``inferred_type`` to the declared type. + declared input type, and fails with ``no_v4_spelling`` otherwise; decoding sets + ``inferred_type`` to the declared type. `morphir.ir.migrate.drop_input_attributes` + resets the attributes on request and reports each input that lost information. - The package release version has no member (finos decision 0017); it is not written and decodes as absent. - Python names are word lists without an initialism marker. A run of two or more @@ -50,7 +57,9 @@ from .. import diagnostics as d from ..diagnostics import DiagnosticError, Location, fail from ..format_version import V4, V4_TABLE, read_document, write_format_version +from . import annotation as ann from . import document as doc +from . import incompleteness as inc from . import literal as lit from . import module, package from . import type as t @@ -99,6 +108,7 @@ def _m(required: str = "", optional: str = "attributes") -> Members: "ExtensibleRecord": _m("variable fields"), "Function": _m("parameterType returnType"), "Unit": _m(), + "Union": _m("alternatives"), } VALUE_NODES: dict[str, Members] = { @@ -135,19 +145,44 @@ def _m(required: str = "", optional: str = "attributes") -> Members: } TYPE_SPECIFICATION_NODES: dict[str, Members] = { - "TypeAliasSpecification": _m("typeParams typeExp", ""), - "OpaqueTypeSpecification": _m("", "typeParams"), - "CustomTypeSpecification": _m("typeParams constructors", ""), - "DerivedTypeSpecification": _m("typeParams baseType fromBaseType toBaseType", ""), + "TypeAliasSpecification": _m("typeParams typeExp", "annotations"), + "OpaqueTypeSpecification": _m("", "annotations typeParams"), + "CustomTypeSpecification": _m("typeParams constructors", "annotations"), + "DerivedTypeSpecification": _m( + "typeParams baseType fromBaseType toBaseType", "annotations" + ), } TYPE_DEFINITION_NODES: dict[str, Members] = { "TypeAliasDefinition": _m("typeParams typeExp", ""), "CustomTypeDefinition": _m("typeParams constructors", "access"), + "IncompleteTypeDefinition": _m("typeParams incompleteness", "partialTypeExp"), +} + +HOLE_REASON_NODES: dict[str, Members] = { + "UnresolvedReference": _m("target", ""), + "DeletedDuringRefactor": _m("tx-id", ""), + "TypeMismatch": _m("expected found", ""), +} + +INCOMPLETENESS_NODES: dict[str, Members] = { + "Hole": _m("reason", "partialBody"), + "Draft": _m("", ""), } VALUE_DEFINITION_NODES: dict[str, Members] = { "ExpressionBody": _m("inputTypes outputType body", ""), + "NativeBody": _m("inputTypes outputType nativeInfo", ""), + "ExternalBody": _m("inputTypes outputType externals", "body"), + "IncompleteBody": _m("inputTypes incompleteness", "outputType partialBody"), +} + +NATIVE_HINT_NODES: dict[str, Members] = { + "Arithmetic": _m("", ""), + "Comparison": _m("", ""), + "StringOp": _m("", ""), + "CollectionOp": _m("", ""), + "PlatformSpecific": _m("platform", ""), } DISTRIBUTION_NODES: dict[str, Members] = { @@ -583,6 +618,27 @@ def node(tag: str, members: dict[str, Json]) -> Json: ) case t.Unit(_): return node("Unit", {}) + case t.Union(_, alternatives): + _check_alternatives(len(alternatives), (*here, "alternatives")) + return node( + "Union", + { + "alternatives": [ + encode_type(x, (*here, "alternatives", i)) + for i, x in enumerate(alternatives) + ] + }, + ) + + +def _check_alternatives(count: int, loc: Location) -> None: + if count < 2: + raise fail( + d.INVALID_JSON, + "Union", + loc, + f"a union needs two or more alternatives, found {count}", + ) def _encode_type_fields(fields: list[t.Field], loc: Location) -> dict[str, Json]: @@ -637,6 +693,12 @@ def fields() -> list[t.Field]: decode_type(body["parameterType"], (*here, "parameterType")), decode_type(body["returnType"], (*here, "returnType")), ) + case "Union": + alternatives = _decode_list( + decode_type, body["alternatives"], (*here, "alternatives"), "Union" + ) + _check_alternatives(len(alternatives), (*here, "alternatives")) + return t.Union(attrs, alternatives) case _: return t.Unit(attrs) @@ -979,7 +1041,9 @@ def fields(items: list[tuple[Name, v.Value]], key: str = "fields") -> Json: case v.Unit(_): return node("Unit", {}) case v.Hole(_, reason, expected_type): - members: dict[str, Json] = {"reason": reason} + members: dict[str, Json] = { + "reason": encode_hole_reason(reason, (*here, "reason")) + } if expected_type is not None: members["expectedType"] = encode_type( expected_type, (*here, "expectedType") @@ -1082,21 +1146,84 @@ def fields() -> list[tuple[Name, v.Value]]: case _: return v.Hole( attrs, - body["reason"], + decode_hole_reason(body["reason"], (*here, "reason")), decode_type(body["expectedType"], (*here, "expectedType")) if "expectedType" in body else None, ) +# --- Hole reasons and incompleteness --- + + +def encode_hole_reason(reason: inc.HoleReason, loc: Location = ()) -> Json: + """Encode a hole reason as ``{"Kind": {members}}``.""" + match reason: + case inc.UnresolvedReference(target): + return _wrap( + "UnresolvedReference", + {"target": encode_fqname(target, (*loc, "UnresolvedReference"))}, + ) + case inc.DeletedDuringRefactor(tx_id): + return _wrap("DeletedDuringRefactor", {"tx-id": tx_id}) + case inc.TypeMismatch(expected, found): + return _wrap("TypeMismatch", {"expected": expected, "found": found}) + + +def decode_hole_reason(j: object, loc: Location = ()) -> inc.HoleReason: + """Decode a hole reason; ``Draft`` is an incompleteness, not a reason.""" + tag, body, here = _node(j, loc, "HoleReason", HOLE_REASON_NODES) + match tag: + case "UnresolvedReference": + return inc.UnresolvedReference( + decode_fqname(body["target"], (*here, "target")) + ) + case "DeletedDuringRefactor": + return inc.DeletedDuringRefactor( + _expect_str(body["tx-id"], (*here, "tx-id"), tag) + ) + case _: + return inc.TypeMismatch( + _expect_str(body["expected"], (*here, "expected"), tag), + _expect_str(body["found"], (*here, "found"), tag), + ) + + +def encode_incompleteness(item: inc.Incompleteness, loc: Location = ()) -> Json: + """Encode an incompleteness as ``{"Hole": {...}}`` or ``{"Draft": {}}``.""" + match item: + case inc.HoleIncompleteness(reason, partial): + here = (*loc, "Hole") + members: dict[str, Json] = { + "reason": encode_hole_reason(reason, (*here, "reason")) + } + if partial is not None: + members["partialBody"] = encode_type(partial, (*here, "partialBody")) + return _wrap("Hole", members) + case inc.DraftIncompleteness(): + return _wrap("Draft", {}) + + +def decode_incompleteness(j: object, loc: Location = ()) -> inc.Incompleteness: + """Decode ``{"Hole": {...}}`` or ``{"Draft": {}}``.""" + tag, body, here = _node(j, loc, "Incompleteness", INCOMPLETENESS_NODES) + if tag == "Draft": + return inc.DraftIncompleteness() + return inc.HoleIncompleteness( + decode_hole_reason(body["reason"], (*here, "reason")), + decode_type(body["partialBody"], (*here, "partialBody")) + if "partialBody" in body + else None, + ) + + def _encode_inputs( inputs: list[tuple[Name, t.Type]], loc: Location, construct: str ) -> dict[str, Json]: return _keyed(inputs, _name_key, encode_type, loc, construct) -def encode_value_definition(defn: v.Definition, loc: Location = ()) -> Json: - """Encode a value definition as ``{"ExpressionBody": {...}}``.""" +def _encode_expression_body(defn: v.Definition, loc: Location) -> Json: here = (*loc, "ExpressionBody") for i, (name, attrs, tpe) in enumerate(defn.input_types): derived = v.ValueAttributes(inferred_type=tpe) @@ -1106,7 +1233,9 @@ def encode_value_definition(defn: v.Definition, loc: Location = ()) -> Json: "ValueDefinition", (*here, "inputTypes", i), f"input {encode_name(name, here)!r} carries attributes other than its " - "declared type; ExpressionBody.inputTypes has no attribute slot", + "declared type; ExpressionBody.inputTypes has no attribute slot. " + "Drop them with morphir.ir.migrate.drop_input_attributes or " + "`morphir-tools ir convert --drop-input-attributes`", ) return { "ExpressionBody": { @@ -1121,9 +1250,144 @@ def encode_value_definition(defn: v.Definition, loc: Location = ()) -> Json: } -def decode_value_definition(j: object, loc: Location = ()) -> v.Definition: - """Decode ``{"ExpressionBody": {...}}``; other bodies are not supported.""" - _, body, here = _node(j, loc, "ValueDefinition", VALUE_DEFINITION_NODES) +def _encode_native_hint(hint: v.NativeHint) -> Json: + if isinstance(hint, v.PlatformSpecific): + return _wrap("PlatformSpecific", {"platform": hint.platform}) + return _wrap(type(hint).__name__, {}) + + +def _decode_native_hint(j: object, loc: Location) -> v.NativeHint: + tag, body, here = _node(j, loc, "NativeHint", NATIVE_HINT_NODES) + match tag: + case "Arithmetic": + return v.Arithmetic() + case "Comparison": + return v.Comparison() + case "StringOp": + return v.StringOp() + case "CollectionOp": + return v.CollectionOp() + case _: + return v.PlatformSpecific( + _expect_str(body["platform"], (*here, "platform"), tag) + ) + + +def _encode_externals(bindings: list[v.ExternalBinding], loc: Location) -> list[Json]: + if not bindings: + raise fail( + d.INVALID_JSON, + "ExternalBody", + loc, + "an external body needs at least one platform binding", + ) + seen: set[str] = set() + result: list[Json] = [] + for i, binding in enumerate(bindings): + if binding.target_platform in seen: + raise fail( + d.DUPLICATE_MEMBER, + "ExternalBinding", + (*loc, i, "targetPlatform"), + f"target platform {binding.target_platform!r} is bound more than once", + ) + seen.add(binding.target_platform) + result.append( + { + "targetPlatform": binding.target_platform, + "externalName": binding.external_name, + } + ) + return result + + +def _decode_externals(j: object, loc: Location) -> list[v.ExternalBinding]: + items = _expect_list(j, loc, "ExternalBody") + if not items: + raise fail( + d.INVALID_JSON, + "ExternalBody", + loc, + "an external body needs at least one platform binding", + ) + bindings: list[v.ExternalBinding] = [] + for i, item in enumerate(items): + here = (*loc, i) + obj = _expect_object(item, here, "ExternalBinding") + _check_members( + obj, _m("targetPlatform externalName", ""), here, "ExternalBinding" + ) + bindings.append( + v.ExternalBinding( + _expect_str( + obj["targetPlatform"], (*here, "targetPlatform"), "ExternalBinding" + ), + _expect_str( + obj["externalName"], (*here, "externalName"), "ExternalBinding" + ), + ) + ) + # Re-encode to apply the same uniqueness rule as the writer. + _encode_externals(bindings, loc) + return bindings + + +def encode_value_definition(defn: v.ValueDefinitionBody, loc: Location = ()) -> Json: + """Encode a value definition as ``{"Body": {...}}``. + + Raises: + DiagnosticError: With ``no_v4_spelling`` when an expression-body input + carries attributes other than its declared type, ``duplicate_member`` + when an external body binds a platform twice, and ``invalid_json`` + when it binds none. + """ + match defn: + case v.Definition(): + return _encode_expression_body(defn, loc) + case v.NativeBody(inputs, output, info): + here = (*loc, "NativeBody") + return _wrap( + "NativeBody", + { + "inputTypes": _encode_inputs( + inputs, (*here, "inputTypes"), "NativeBody" + ), + "outputType": encode_type(output, (*here, "outputType")), + "nativeInfo": {"hint": _encode_native_hint(info.hint)}, + }, + ) + case v.ExternalBody(inputs, output, externals, fallback): + here = (*loc, "ExternalBody") + members: dict[str, Json] = { + "inputTypes": _encode_inputs( + inputs, (*here, "inputTypes"), "ExternalBody" + ), + "outputType": encode_type(output, (*here, "outputType")), + "externals": _encode_externals(externals, (*here, "externals")), + } + if fallback is not None: + members["body"] = encode_value(fallback, (*here, "body")) + return _wrap("ExternalBody", members) + case v.IncompleteBody(inputs, incompleteness, output, partial): + here = (*loc, "IncompleteBody") + members = { + "inputTypes": _encode_inputs( + inputs, (*here, "inputTypes"), "IncompleteBody" + ) + } + if output is not None: + members["outputType"] = encode_type(output, (*here, "outputType")) + members["incompleteness"] = encode_incompleteness( + incompleteness, (*here, "incompleteness") + ) + if partial is not None: + members["partialBody"] = encode_value(partial, (*here, "partialBody")) + return _wrap("IncompleteBody", members) + + +def decode_value_definition(j: object, loc: Location = ()) -> v.ValueDefinitionBody: + """Decode an expression, native, external or incomplete body.""" + tag, body, here = _node(j, loc, "ValueDefinition", VALUE_DEFINITION_NODES) inputs = _unkeyed( body["inputTypes"], decode_name, @@ -1131,26 +1395,155 @@ def decode_value_definition(j: object, loc: Location = ()) -> v.Definition: (*here, "inputTypes"), "inputTypes", ) - return v.Definition( - [(n, v.ValueAttributes(inferred_type=tp), tp) for n, tp in inputs], - decode_type(body["outputType"], (*here, "outputType")), - decode_value(body["body"], (*here, "body")), - ) + + def optional_type(key: str) -> t.Type | None: + return decode_type(body[key], (*here, key)) if key in body else None + + def optional_value(key: str) -> v.Value | None: + return decode_value(body[key], (*here, key)) if key in body else None + + match tag: + case "ExpressionBody": + return v.Definition( + [(n, v.ValueAttributes(inferred_type=tp), tp) for n, tp in inputs], + decode_type(body["outputType"], (*here, "outputType")), + decode_value(body["body"], (*here, "body")), + ) + case "NativeBody": + info_loc = (*here, "nativeInfo") + info = _expect_object(body["nativeInfo"], info_loc, "NativeInfo") + _check_members(info, _m("hint", ""), info_loc, "NativeInfo") + return v.NativeBody( + inputs, + decode_type(body["outputType"], (*here, "outputType")), + v.NativeInfo(_decode_native_hint(info["hint"], (*info_loc, "hint"))), + ) + case "ExternalBody": + return v.ExternalBody( + inputs, + decode_type(body["outputType"], (*here, "outputType")), + _decode_externals(body["externals"], (*here, "externals")), + optional_value("body"), + ) + case _: + return v.IncompleteBody( + inputs, + decode_incompleteness( + body["incompleteness"], (*here, "incompleteness") + ), + optional_type("outputType"), + optional_value("partialBody"), + ) def encode_value_specification(spec: v.Specification, loc: Location = ()) -> Json: - return { - "inputs": _encode_inputs(spec.inputs, (*loc, "inputs"), "ValueSpecification"), - "output": encode_type(spec.output, (*loc, "output")), - } + return _with_annotations( + spec.annotations, + loc, + { + "inputs": _encode_inputs( + spec.inputs, (*loc, "inputs"), "ValueSpecification" + ), + "output": encode_type(spec.output, (*loc, "output")), + }, + ) def decode_value_specification(j: object, loc: Location = ()) -> v.Specification: obj = _expect_object(j, loc, "ValueSpecification") - _check_members(obj, _m("inputs output", ""), loc, "ValueSpecification") + _check_members(obj, _m("inputs output", "annotations"), loc, "ValueSpecification") return v.Specification( _unkeyed(obj["inputs"], decode_name, decode_type, (*loc, "inputs"), "inputs"), decode_type(obj["output"], (*loc, "output")), + _decode_annotations(obj, loc), + ) + + +# --- Annotations --- + + +def encode_annotation(item: ann.Annotation, loc: Location = ()) -> Json: + """Encode an annotation in the form it was written in.""" + match item: + case ann.CompactAnnotation(name, text): + fqn = encode_fqname(name, loc) + return fqn if text is None else f"{fqn}:{text}" + case ann.StructuredAnnotation(name, arguments): + members: dict[str, Json] = {"name": encode_fqname(name, loc)} + if arguments: + members["arguments"] = [ + _encode_annotation_argument(x, (*loc, "arguments", i)) + for i, x in enumerate(arguments) + ] + return members + + +def _encode_annotation_argument(arg: ann.AnnotationArgument, loc: Location) -> Json: + match arg: + case ann.PositionalArgument(value): + return encode_value(value, loc) + case ann.NamedArgument(name, value): + return { + "name": encode_name(name, loc), + "value": encode_value(value, (*loc, "value")), + } + + +def decode_annotation(j: object, loc: Location = ()) -> ann.Annotation: + """Decode a compact string or a structured annotation object. + + In the compact form the free text starts after the first colon after ``#``. + """ + if isinstance(j, str): + hash_at = j.find("#") + colon_at = j.find(":", hash_at) if hash_at >= 0 else -1 + if colon_at < 0: + return ann.CompactAnnotation(decode_fqname(j, loc)) + return ann.CompactAnnotation( + decode_fqname(j[:colon_at], loc), j[colon_at + 1 :] + ) + obj = _expect_object(j, loc, "Annotation") + _check_members(obj, _m("name", "arguments"), loc, "Annotation") + arguments = ( + _decode_list( + _decode_annotation_argument, + obj["arguments"], + (*loc, "arguments"), + "Annotation", + ) + if "arguments" in obj + else [] + ) + return ann.StructuredAnnotation(decode_fqname(obj["name"], loc), arguments) + + +def _decode_annotation_argument(j: object, loc: Location) -> ann.AnnotationArgument: + if isinstance(j, dict) and set(j) == {"name", "value"}: + return ann.NamedArgument( + decode_name(j["name"], (*loc, "name")), + decode_value(j["value"], (*loc, "value")), + ) + return ann.PositionalArgument(decode_value(j, loc)) + + +def _with_annotations( + annotations: list[ann.Annotation], loc: Location, members: dict[str, Json] +) -> dict[str, Json]: + """Put a non-empty ``annotations`` member first, as the kit writes it.""" + if not annotations: + return members + encoded = [ + encode_annotation(x, (*loc, "annotations", i)) + for i, x in enumerate(annotations) + ] + return {"annotations": encoded, **members} + + +def _decode_annotations(obj: dict[str, object], loc: Location) -> list[ann.Annotation]: + if "annotations" not in obj: + return [] + return _decode_list( + decode_annotation, obj["annotations"], (*loc, "annotations"), "annotations" ) @@ -1196,6 +1589,12 @@ def decode_args(j: object, here: Location) -> list[tuple[Name, t.Type]]: def encode_type_specification(spec: ts.Specification, loc: Location = ()) -> Json: + tag = type(spec).__name__ + encoded = _encode_type_specification(spec, loc) + return _wrap(tag, _with_annotations(spec.annotations, (*loc, tag), encoded[tag])) + + +def _encode_type_specification(spec: ts.Specification, loc: Location) -> Json: params = _encode_params(spec.type_params, loc) match spec: case ts.TypeAliasSpecification(_, tpe): @@ -1235,17 +1634,19 @@ def decode_type_specification(j: object, loc: Location = ()) -> ts.Specification if "typeParams" in body else [] ) + annotations = _decode_annotations(body, here) match tag: case "TypeAliasSpecification": return ts.TypeAliasSpecification( - params, decode_type(body["typeExp"], (*here, "typeExp")) + params, decode_type(body["typeExp"], (*here, "typeExp")), annotations ) case "OpaqueTypeSpecification": - return ts.OpaqueTypeSpecification(params) + return ts.OpaqueTypeSpecification(params, annotations) case "CustomTypeSpecification": return ts.CustomTypeSpecification( params, _decode_constructors(body["constructors"], (*here, "constructors")), + annotations, ) case _: return ts.DerivedTypeSpecification( @@ -1255,6 +1656,7 @@ def decode_type_specification(j: object, loc: Location = ()) -> ts.Specification decode_fqname(body["fromBaseType"], here), decode_fqname(body["toBaseType"], here), ), + annotations, ) @@ -1277,6 +1679,19 @@ def encode_type_definition(defn: td.Definition, loc: Location = ()) -> Json: "constructors": _encode_constructors(ctors.value, here), }, ) + case td.IncompleteTypeDefinition(_, incompleteness, partial): + here = (*loc, "IncompleteTypeDefinition") + members: dict[str, Json] = { + "typeParams": params, + "incompleteness": encode_incompleteness( + incompleteness, (*here, "incompleteness") + ), + } + if partial is not None: + members["partialTypeExp"] = encode_type( + partial, (*here, "partialTypeExp") + ) + return _wrap("IncompleteTypeDefinition", members) def decode_type_definition(j: object, loc: Location = ()) -> td.Definition: @@ -1286,6 +1701,14 @@ def decode_type_definition(j: object, loc: Location = ()) -> td.Definition: return td.TypeAliasDefinition( params, decode_type(body["typeExp"], (*here, "typeExp")) ) + if tag == "IncompleteTypeDefinition": + return td.IncompleteTypeDefinition( + params, + decode_incompleteness(body["incompleteness"], (*here, "incompleteness")), + decode_type(body["partialTypeExp"], (*here, "partialTypeExp")) + if "partialTypeExp" in body + else None, + ) access = body.get("access", "Public") if access not in ACCESS_TAGS: raise fail( @@ -1383,22 +1806,28 @@ def _decode_doc(obj: dict[str, object], loc: Location) -> str | None: def encode_module_specification(spec: module.Specification, loc: Location = ()) -> Json: - return _with_doc( - { - "types": _keyed( - spec.types, _name_key, _enc_type_spec, (*loc, "types"), "types" - ), - "values": _keyed( - spec.values, _name_key, _enc_value_spec, (*loc, "values"), "values" - ), - }, - spec.doc, + return _with_annotations( + spec.annotations, + loc, + _with_doc( + { + "types": _keyed( + spec.types, _name_key, _enc_type_spec, (*loc, "types"), "types" + ), + "values": _keyed( + spec.values, _name_key, _enc_value_spec, (*loc, "values"), "values" + ), + }, + spec.doc, + ), ) def decode_module_specification(j: object, loc: Location = ()) -> module.Specification: obj = _expect_object(j, loc, "ModuleSpecification") - _check_members(obj, _m("", "types values doc"), loc, "ModuleSpecification") + _check_members( + obj, _m("", "annotations types values doc"), loc, "ModuleSpecification" + ) return module.Specification( types=dict( _unkeyed( @@ -1419,6 +1848,7 @@ def decode_module_specification(j: object, loc: Location = ()) -> module.Specifi ) ), doc=_decode_doc(obj, loc), + annotations=_decode_annotations(obj, loc), ) @@ -1708,6 +2138,8 @@ def decode_versioned_distribution(document: str | object) -> Distribution: "FINOS_MORPHIR_PIN", "decode_distribution", "decode_fqname", + "decode_hole_reason", + "decode_incompleteness", "decode_literal", "decode_name", "decode_path", @@ -1717,6 +2149,8 @@ def decode_versioned_distribution(document: str | object) -> Distribution: "decode_versioned_distribution", "encode_distribution", "encode_fqname", + "encode_hole_reason", + "encode_incompleteness", "encode_literal", "encode_name", "encode_path", diff --git a/packages/morphir/src/morphir/ir/v4/incompleteness.py b/packages/morphir/src/morphir/ir/v4/incompleteness.py new file mode 100644 index 00000000..bb802cf5 --- /dev/null +++ b/packages/morphir/src/morphir/ir/v4/incompleteness.py @@ -0,0 +1,78 @@ +"""Incompleteness: why a type or value definition is not complete yet. + +Follows the finos/morphir v4 schema (``Incompleteness``, ``HoleReason``) at the +pinned commit and the morphir-rust v4 model. A hole names its reason; a draft is +work in progress and names none. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Union + +from .fqname import FQName +from .type import Type + + +@dataclass(frozen=True) +class UnresolvedReference: + """A reference that could not be resolved. + + Attributes: + target: The name that did not resolve. + """ + + target: FQName + + +@dataclass(frozen=True) +class DeletedDuringRefactor: + """A reference removed by a refactoring. + + Attributes: + tx_id: The transaction id of the refactoring (JSON member ``tx-id``). + """ + + tx_id: str + + +@dataclass(frozen=True) +class TypeMismatch: + """Type checking failed. + + Attributes: + expected: A description of the expected type. + found: A description of the type found. + """ + + expected: str + found: str + + +HoleReason = Union[UnresolvedReference, DeletedDuringRefactor, TypeMismatch] +"""Why a hole exists. ``Draft`` is an incompleteness, not a hole reason.""" + + +@dataclass(frozen=True) +class HoleIncompleteness: + """A definition with a hole. + + Attributes: + reason: Why the hole exists. + partial_body: The type the author had written when the definition stopped + being complete, if any. The schema gives this member the type + ``Type``; an incomplete value keeps its partial expression on + `IncompleteBody.partial_body` instead. + """ + + reason: HoleReason + partial_body: Type | None = None + + +@dataclass(frozen=True) +class DraftIncompleteness: + """A definition the author marked as work in progress.""" + + +Incompleteness = Union[HoleIncompleteness, DraftIncompleteness] +"""Why a definition is incomplete: a hole or a draft.""" diff --git a/packages/morphir/src/morphir/ir/v4/module.py b/packages/morphir/src/morphir/ir/v4/module.py index c076e504..adc8d627 100644 --- a/packages/morphir/src/morphir/ir/v4/module.py +++ b/packages/morphir/src/morphir/ir/v4/module.py @@ -2,6 +2,7 @@ from . import type_def, type_spec, value from .access_controlled import AccessControlled +from .annotation import Annotation from .documented import Documented from .name import Name from .path import Path @@ -15,6 +16,7 @@ class Specification: types: dict[Name, Documented[type_spec.Specification]] = field(default_factory=dict) values: dict[Name, Documented[value.Specification]] = field(default_factory=dict) doc: str | None = None + annotations: list[Annotation] = field(default_factory=list) @dataclass(frozen=True) @@ -22,7 +24,7 @@ class Definition: types: dict[Name, AccessControlled[Documented[type_def.Definition]]] = field( default_factory=dict ) - values: dict[Name, AccessControlled[Documented[value.Definition]]] = field( + values: dict[Name, AccessControlled[Documented[value.ValueDefinitionBody]]] = field( default_factory=dict ) doc: str | None = None diff --git a/packages/morphir/src/morphir/ir/v4/type.py b/packages/morphir/src/morphir/ir/v4/type.py index 5cb42871..7aa3f876 100644 --- a/packages/morphir/src/morphir/ir/v4/type.py +++ b/packages/morphir/src/morphir/ir/v4/type.py @@ -1,7 +1,8 @@ from __future__ import annotations from dataclasses import dataclass, field -from typing import Any, Union +from typing import Any +from typing import Union as TUnion from .fqname import FQName from .name import Name @@ -72,7 +73,26 @@ class Unit: attributes: TypeAttributes -Type = Union[Variable, Reference, Tuple, Record, ExtensibleRecord, Function, Unit] +@dataclass(frozen=True) +class Union: + """An untagged union: values of at least one of the alternative types. + + Draft v4 type expression (finos/morphir ``docs/spec/draft/types.md``). It + introduces no constructor or tag. Alternative order, duplicates and nesting + are content and are kept. + + Attributes: + attributes: The type attributes. + alternatives: Two or more alternative types. + """ + + attributes: TypeAttributes + alternatives: list[Type] = field(default_factory=list) + + +Type = TUnion[ + Variable, Reference, Tuple, Record, ExtensibleRecord, Function, Unit, Union +] @dataclass(frozen=True) @@ -114,5 +134,7 @@ def map_attributes(tpe: Type, f: Any) -> Type: map_attributes(tpe.argument_type, f), map_attributes(tpe.return_type, f), ) + elif isinstance(tpe, Union): + return Union(new_ta, [map_attributes(a, f) for a in tpe.alternatives]) else: return Unit(new_ta) diff --git a/packages/morphir/src/morphir/ir/v4/type_constraints.py b/packages/morphir/src/morphir/ir/v4/type_constraints.py index d0e0e24e..19e4ce55 100644 --- a/packages/morphir/src/morphir/ir/v4/type_constraints.py +++ b/packages/morphir/src/morphir/ir/v4/type_constraints.py @@ -77,6 +77,14 @@ class CustomConstraint: @dataclass(frozen=True) class TypeConstraints: + """Structured constraints on a type. + + Deprecated: the upstream v4 schema keeps ``TypeAttributes.constraints`` as a + placeholder with no defined members, so these constraints have no v4 spelling + and encoding fails with ``no_v4_spelling``. Keep constraints in + ``TypeAttributes.extensions`` until upstream defines them. + """ + numeric: NumericConstraint | None = None string: StringConstraint | None = None collection: CollectionConstraint | None = None diff --git a/packages/morphir/src/morphir/ir/v4/type_def.py b/packages/morphir/src/morphir/ir/v4/type_def.py index 90c4ba58..d86d05b2 100644 --- a/packages/morphir/src/morphir/ir/v4/type_def.py +++ b/packages/morphir/src/morphir/ir/v4/type_def.py @@ -2,6 +2,7 @@ from typing import Union from .access_controlled import AccessControlled +from .incompleteness import Incompleteness from .name import Name from .type import Type from .type_spec import Constructors @@ -19,4 +20,19 @@ class CustomTypeDefinition: constructors: AccessControlled[Constructors] -Definition = Union[TypeAliasDefinition, CustomTypeDefinition] +@dataclass(frozen=True) +class IncompleteTypeDefinition: + """A type definition that is not complete yet. + + Attributes: + type_params: The declared type parameters. + incompleteness: Why the definition is incomplete. + partial_type_exp: The type expression written so far, if any. + """ + + type_params: list[Name] + incompleteness: Incompleteness + partial_type_exp: Type | None = None + + +Definition = Union[TypeAliasDefinition, CustomTypeDefinition, IncompleteTypeDefinition] diff --git a/packages/morphir/src/morphir/ir/v4/type_spec.py b/packages/morphir/src/morphir/ir/v4/type_spec.py index 49377e0f..e6fa263a 100644 --- a/packages/morphir/src/morphir/ir/v4/type_spec.py +++ b/packages/morphir/src/morphir/ir/v4/type_spec.py @@ -1,8 +1,9 @@ from __future__ import annotations -from dataclasses import dataclass +from dataclasses import dataclass, field from typing import Union +from .annotation import Annotation from .fqname import FQName from .name import Name from .type import Type @@ -17,6 +18,7 @@ class TypeAliasSpecification: type_params: list[Name] tpe: Type + annotations: list[Annotation] = field(default_factory=list) @dataclass(frozen=True) @@ -24,6 +26,7 @@ class OpaqueTypeSpecification: """Specification for an opaque type.""" type_params: list[Name] + annotations: list[Annotation] = field(default_factory=list) @dataclass(frozen=True) @@ -32,6 +35,7 @@ class CustomTypeSpecification: type_params: list[Name] constructors: Constructors + annotations: list[Annotation] = field(default_factory=list) @dataclass(frozen=True) @@ -49,6 +53,7 @@ class DerivedTypeSpecification: type_params: list[Name] details: DerivedTypeSpecificationDetails + annotations: list[Annotation] = field(default_factory=list) Specification = Union[ diff --git a/packages/morphir/src/morphir/ir/v4/value.py b/packages/morphir/src/morphir/ir/v4/value.py index fc07831d..a9ee73cf 100644 --- a/packages/morphir/src/morphir/ir/v4/value.py +++ b/packages/morphir/src/morphir/ir/v4/value.py @@ -1,15 +1,19 @@ from __future__ import annotations from dataclasses import dataclass, field -from typing import Any, Union +from typing import TYPE_CHECKING, Any, Union from typing import List as TList from typing import Tuple as TTuple from .fqname import FQName +from .incompleteness import HoleReason, Incompleteness from .literal import Literal from .name import Name from .type import Type +if TYPE_CHECKING: + from .annotation import Annotation + @dataclass(frozen=True) class SourceLocation: @@ -206,7 +210,7 @@ class LetDefinition: attributes: ValueAttributes name: Name - definition: Definition + definition: ValueDefinitionBody in_value: Value @@ -215,7 +219,7 @@ class LetRecursion: """Represents a recursive let binding.""" attributes: ValueAttributes - definitions: dict[Name, Definition] + definitions: dict[Name, ValueDefinitionBody] in_value: Value @@ -266,16 +270,27 @@ class Unit: @dataclass(frozen=True) class Hole: - """Represents a hole in the AST.""" + """A hole in an expression: a part that is missing, with the reason why. + + Attributes: + attributes: The value attributes. + reason: Why the hole exists. + expected_type: The type the missing part should have, if known. + """ attributes: ValueAttributes - reason: Any + reason: HoleReason expected_type: Type | None @dataclass(frozen=True) class Native: - """Represents a native reference.""" + """A native reference as a value node. + + Deprecated: upstream v4 makes native operations definition bodies, not + expressions (finos/morphir decision 0008), so this node has no v4 spelling and + encoding fails with ``no_v4_spelling``. Use `NativeBody` instead. + """ attributes: ValueAttributes fqname: FQName @@ -284,7 +299,12 @@ class Native: @dataclass(frozen=True) class External: - """Represents an external reference.""" + """An external reference as a value node. + + Deprecated: upstream v4 makes external operations definition bodies, not + expressions (finos/morphir decision 0008), so this node has no v4 spelling and + encoding fails with ``no_v4_spelling``. Use `ExternalBody` instead. + """ attributes: ValueAttributes external_name: str @@ -320,16 +340,141 @@ class External: @dataclass(frozen=True) class Specification: - """Value specification.""" + """Value specification: the signature of a value, and its annotations.""" inputs: TList[TTuple[Name, Type]] output: Type + annotations: TList[Annotation] = field(default_factory=lambda: []) @dataclass(frozen=True) class Definition: - """Value definition.""" + """A value definition with an expression body (``ExpressionBody``). + + Attributes: + input_types: Each input's name, value attributes and declared type. v4 + JSON spells an input as a bare type, so attributes other than the + declared type cannot be encoded; see + `morphir.ir.migrate.drop_input_attributes`. + output_type: The declared output type. + body: The expression. + """ input_types: TList[TTuple[Name, ValueAttributes, Type]] output_type: Type body: Value + + +# --- Native and external definition bodies --- + + +@dataclass(frozen=True) +class Arithmetic: + """Native hint: a basic arithmetic or logic operation.""" + + +@dataclass(frozen=True) +class Comparison: + """Native hint: a comparison operation.""" + + +@dataclass(frozen=True) +class StringOp: + """Native hint: a string operation.""" + + +@dataclass(frozen=True) +class CollectionOp: + """Native hint: a collection operation such as map, filter or fold.""" + + +@dataclass(frozen=True) +class PlatformSpecific: + """Native hint: an operation specific to one platform. + + Attributes: + platform: The platform, such as ``"wasm"`` or ``"javascript"``. + """ + + platform: str + + +NativeHint = Union[Arithmetic, Comparison, StringOp, CollectionOp, PlatformSpecific] +"""What kind of operation a native definition performs.""" + + +@dataclass(frozen=True) +class NativeInfo: + """Information about a native definition. + + Attributes: + hint: The kind of operation. + """ + + hint: NativeHint + + +@dataclass(frozen=True) +class ExternalBinding: + """One target platform's binding for an external definition. + + Attributes: + target_platform: The platform. Unique within an external body. + external_name: The name of the operation on that platform. + """ + + target_platform: str + external_name: str + + +@dataclass(frozen=True) +class NativeBody: + """A value definition implemented natively by each runtime (``NativeBody``). + + Attributes: + input_types: Each input's name and declared type. + output_type: The declared output type. + native_info: What kind of operation it is. + """ + + input_types: TList[TTuple[Name, Type]] + output_type: Type + native_info: NativeInfo + + +@dataclass(frozen=True) +class ExternalBody: + """A value definition bound to external operations (``ExternalBody``). + + Attributes: + input_types: Each input's name and declared type. + output_type: The declared output type. + externals: One or more platform bindings, unique by platform. + body: An optional fallback expression for other platforms. + """ + + input_types: TList[TTuple[Name, Type]] + output_type: Type + externals: TList[ExternalBinding] + body: Value | None = None + + +@dataclass(frozen=True) +class IncompleteBody: + """A value definition that is not complete yet (``IncompleteBody``). + + Attributes: + input_types: Each input's name and declared type. + incompleteness: Why the definition is incomplete. + output_type: The declared output type, if known yet. + partial_body: The expression written so far, if any. + """ + + input_types: TList[TTuple[Name, Type]] + incompleteness: Incompleteness + output_type: Type | None = None + partial_body: Value | None = None + + +ValueDefinitionBody = Union[Definition, NativeBody, ExternalBody, IncompleteBody] +"""A v4 value definition: an expression, native, external or incomplete body.""" diff --git a/pretty.output b/pretty.output new file mode 100644 index 00000000..c20f07c1 --- /dev/null +++ b/pretty.output @@ -0,0 +1,127 @@ +Feature: IR interchange # features/ir_interchange.feature:1 + As a Morphir tooling user + I want to inspect and convert versioned morphir-ir.json files + So that I can move models between IR format 3.0.0 and draft IR v4 + Scenario: Inspect a morphir-elm format 3 file # features/ir_interchange.feature:6 + Given the golden fixture "elm-simpleValueTree.json" # features/steps/ir_interchange_steps.py:31 + When I run "ir inspect" on it # features/steps/ir_interchange_steps.py:46 + Then the command exits with code 0 # features/steps/ir_interchange_steps.py:81 + And the output contains "format: 3.0.0" # features/steps/ir_interchange_steps.py:87 + And the output contains "kind: Library" # features/steps/ir_interchange_steps.py:87 + And the output contains "package: morphir/example/app" # features/steps/ir_interchange_steps.py:87 + + Scenario: Upgrade a format 3 file to draft v4 and back # features/ir_interchange.feature:14 + Given the golden fixture "elm-simpleValueTree.json" # features/steps/ir_interchange_steps.py:31 + When I convert it to "v4" # features/steps/ir_interchange_steps.py:52 + Then the command exits with code 0 # features/steps/ir_interchange_steps.py:81 + And the converted file declares formatVersion 4 # features/steps/ir_interchange_steps.py:99 + When I inspect the converted file # features/steps/ir_interchange_steps.py:68 + Then the output contains "draft" # features/steps/ir_interchange_steps.py:87 + When I convert the converted file to "v3" # features/steps/ir_interchange_steps.py:74 + Then the command exits with code 0 # features/steps/ir_interchange_steps.py:81 + And the converted file has the same JSON as the fixture # features/steps/ir_interchange_steps.py:105 + + Scenario: Unversioned IR is rejected # features/ir_interchange.feature:25 + Given a file with the JSON '{"distribution": []}' # features/steps/ir_interchange_steps.py:38 + When I run "ir inspect" on it # features/steps/ir_interchange_steps.py:46 + Then the command exits with code 1 # features/steps/ir_interchange_steps.py:81 + And the errors contain "missing_format_version" # features/steps/ir_interchange_steps.py:93 + + Scenario: Specs cannot be downgraded # features/ir_interchange.feature:31 + Given a file with the JSON '{"formatVersion": 4, "distribution": {"Specs": {"packageName": "a"}}}' # features/steps/ir_interchange_steps.py:38 + When I convert it to "v3" # features/steps/ir_interchange_steps.py:52 + Then the command exits with code 1 # features/steps/ir_interchange_steps.py:81 + And the errors contain "unsupported_v4_downgrade" # features/steps/ir_interchange_steps.py:93 + And no converted file exists # features/steps/ir_interchange_steps.py:113 + +Feature: Morphir SDK # features/morphir_sdk.feature:1 + As a developer of Morphir models + I want a Python runtime for the Morphir SDK + So that compiled models behave as they do in Elm + Scenario Outline: Every core SDK module can be imported -- @1.1 # features/morphir_sdk.feature:12 + When I import the SDK module "basics" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.2 # features/morphir_sdk.feature:13 + When I import the SDK module "char" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.3 # features/morphir_sdk.feature:14 + When I import the SDK module "string" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.4 # features/morphir_sdk.feature:15 + When I import the SDK module "list" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.5 # features/morphir_sdk.feature:16 + When I import the SDK module "dict" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.6 # features/morphir_sdk.feature:17 + When I import the SDK module "set" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.7 # features/morphir_sdk.feature:18 + When I import the SDK module "maybe" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.8 # features/morphir_sdk.feature:19 + When I import the SDK module "result" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.9 # features/morphir_sdk.feature:20 + When I import the SDK module "tuple" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.10 # features/morphir_sdk.feature:21 + When I import the SDK module "decimal" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.11 # features/morphir_sdk.feature:22 + When I import the SDK module "int" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario Outline: Every core SDK module can be imported -- @1.12 # features/morphir_sdk.feature:23 + When I import the SDK module "number" # features/steps/sdk_steps.py:23 + Then the SDK module should be available # features/steps/sdk_steps.py:29 + + Scenario: A dictionary lists its entries in key order # features/morphir_sdk.feature:25 + Given an SDK dictionary built from the pairs "b=2, a=1, c=3" # features/steps/sdk_steps.py:36 + When I list the keys of the dictionary # features/steps/sdk_steps.py:48 + Then the keys should be "a, b, c" # features/steps/sdk_steps.py:54 + + Scenario: Two dictionaries built in a different order are equal # features/morphir_sdk.feature:30 + Given an SDK dictionary built from the pairs "b=2, a=1" # features/steps/sdk_steps.py:36 + And another SDK dictionary built from the pairs "a=1, b=2" # features/steps/sdk_steps.py:42 + Then the two dictionaries should be equal # features/steps/sdk_steps.py:60 + + Scenario: Rounding follows Elm and not Python # features/morphir_sdk.feature:35 + When I round the float 2.5 with the SDK # features/steps/sdk_steps.py:67 + Then the rounded value should be 3 # features/steps/sdk_steps.py:73 + + Scenario: Decimal arithmetic is exact # features/morphir_sdk.feature:39 + When I add the decimals "0.1" and "0.2" with the SDK # features/steps/sdk_steps.py:79 + Then the decimal result should print as "0.3" # features/steps/sdk_steps.py:87 + +Feature: Morphir Python Setup # features/morphir_setup.feature:1 + As a developer + I want to verify the project is set up correctly + So that I can start developing Morphir features + Scenario: Morphir core library can be imported # features/morphir_setup.feature:6 + Given the Python environment is set up # features/steps/setup_steps.py:13 + When I import the morphir module # features/steps/setup_steps.py:22 + Then the module should be available # features/steps/setup_steps.py:46 + And it should have a version attribute # features/steps/setup_steps.py:52 + + Scenario: Morphir tools can be imported # features/morphir_setup.feature:12 + Given the Python environment is set up # features/steps/setup_steps.py:13 + When I import the morphir_tools module # features/steps/setup_steps.py:30 + Then the module should be available # features/steps/setup_steps.py:46 + And it should have a version attribute # features/steps/setup_steps.py:52 + + Scenario: Morphir IR module can be imported # features/morphir_setup.feature:18 + Given the Python environment is set up # features/steps/setup_steps.py:13 + When I import the morphir.ir module # features/steps/setup_steps.py:38 + Then the module should be available # features/steps/setup_steps.py:46 + diff --git a/tests/tools/test_cli.py b/tests/tools/test_cli.py index 4871f2d6..8e6573a3 100644 --- a/tests/tools/test_cli.py +++ b/tests/tools/test_cli.py @@ -163,8 +163,25 @@ def test_encode_failure_leaves_no_output(self, tmp_path: Path) -> None: result = run("ir", "convert", REFERENCE, "--to", "v4", "-o", out) assert result.exit_code == 1 assert d.NO_V4_SPELLING in result.stderr + assert "--drop-input-attributes" in result.stderr assert not out.exists() + def test_drop_input_attributes_on_upgrade(self, tmp_path: Path) -> None: + out = tmp_path / "out.json" + result = run( + "ir", + "convert", + REFERENCE, + "--to", + "v4", + "-o", + out, + "--drop-input-attributes", + ) + assert result.exit_code == 0, result.output + assert result.stderr.count(d.INPUT_ATTRIBUTES_DROPPED) == 10 + assert json.loads(out.read_text())["formatVersion"] == 4 + def test_downgrade_warnings_exit_zero(self, tmp_path: Path) -> None: unit = {"Unit": {}} attrs = { diff --git a/tests/unit/ir/test_migrate.py b/tests/unit/ir/test_migrate.py index b2cd97b8..18f1c5ca 100644 --- a/tests/unit/ir/test_migrate.py +++ b/tests/unit/ir/test_migrate.py @@ -8,13 +8,15 @@ from morphir.ir import diagnostics as d from morphir.ir import v3 -from morphir.ir.migrate import Err, Ok, v3_to_v4, v4_to_v3 +from morphir.ir.migrate import Err, Ok, drop_input_attributes, v3_to_v4, v4_to_v3 from morphir.ir.v3 import codec as v3_codec from morphir.ir.v3 import literal as lit3 from morphir.ir.v3 import package as pkg3 from morphir.ir.v3 import type as t3 from morphir.ir.v3 import value as val3 +from morphir.ir.v4 import codec as v4_codec from morphir.ir.v4 import document as doc4 +from morphir.ir.v4 import incompleteness as inc4 from morphir.ir.v4 import literal as lit4 from morphir.ir.v4 import module as mod4 from morphir.ir.v4 import package as pkg4 @@ -165,7 +167,7 @@ def test_specs_and_application_do_not_downgrade( @pytest.mark.parametrize( "node", [ - v4.Hole(TYPED, None, None), + v4.Hole(TYPED, inc4.DeletedDuringRefactor("tx"), None), v4.Native(TYPED, FQName(PKG, IRPath(()), n("f")), None), v4.External(TYPED, "f", "js"), v4.LiteralValue(TYPED, lit4.DocumentLiteral(doc4.DocNull())), @@ -225,7 +227,9 @@ def test_missing_member_documentation(self) -> None: def test_every_failure_is_collected_in_one_pass(self) -> None: body = v4.Apply( - TYPED, v4.Hole(TYPED, None, None), v4.Unit(v4.ValueAttributes()) + TYPED, + v4.Hole(TYPED, inc4.DeletedDuringRefactor("tx"), None), + v4.Unit(v4.ValueAttributes()), ) found = errors(v4_to_v3(library(body, value_doc=None))) assert {code for code, _ in found} == { @@ -347,3 +351,71 @@ def test_hand_built_library_with_every_node_downgrades(self) -> None: again = v4_to_v3(up.value) assert isinstance(again, Ok) assert again.value == result.value + + +STRING = t4.Reference( + t4.TypeAttributes(), + FQName( + IRPath((n("morphir"), n("s", "d", "k"))), IRPath((n("string"),)), n("string") + ), +) +REFERENCE_MODEL = FIXTURES / "elm-reference-model.json" +INPUT_LOC = ("distribution", "modules", "app", "values", "main", "input_types") + + +class TestDropInputAttributes: + def test_input_with_resolved_type_is_reset_and_reported(self) -> None: + lib = library(inputs=[(n("x"), v4.ValueAttributes(inferred_type=STRING), INT)]) + result = drop_input_attributes(lib) + defn = result.value.definition.modules[IRPath((n("app"),))] + inputs = defn.value.values[n("main")].value.value.input_types # type: ignore[union-attr] + assert inputs == [(n("x"), TYPED, INT)] + assert [(w.code, w.construct, w.location) for w in result.warnings] == [ + (d.INPUT_ATTRIBUTES_DROPPED, "ValueDefinitionInput", (*INPUT_LOC, "x")) + ] + + @pytest.mark.parametrize( + "attrs", [v4.ValueAttributes(), TYPED], ids=["empty", "declared"] + ) + def test_input_that_loses_nothing_reports_nothing( + self, attrs: v4.ValueAttributes + ) -> None: + lib = library(inputs=[(n("x"), attrs, INT)]) + result = drop_input_attributes(lib) + assert result.warnings == () + + def test_let_bound_input_is_reset(self) -> None: + inner = v4.Definition( + [(n("y"), v4.ValueAttributes(inferred_type=STRING), INT)], + INT, + v4.Unit(TYPED), + ) + body = v4.LetDefinition(TYPED, n("f"), inner, v4.Unit(TYPED)) + result = drop_input_attributes(library(body=body)) + assert len(result.warnings) == 1 + assert result.warnings[0].location[-1] == "y" + assert "LetDefinition" in result.warnings[0].location + mod = result.value.definition.modules[IRPath((n("app"),))] + let = mod.value.values[n("main")].value.value.body # type: ignore[union-attr] + assert isinstance(let, v4.LetDefinition) + assert let.definition.input_types == [(n("y"), TYPED, INT)] # type: ignore[union-attr] + + def test_specs_have_no_inputs_to_drop(self) -> None: + specs = SpecsDistribution(PackageInfo(PKG), pkg4.Specification()) + assert drop_input_attributes(specs) == Ok(specs) + + def test_reference_model_upgrades_to_encodable_v4(self) -> None: + lib = load(REFERENCE_MODEL) + result = v3_to_v4(lib, drop_input_attributes=True) + assert len(result.warnings) == 10 + assert {w.code for w in result.warnings} == {d.INPUT_ATTRIBUTES_DROPPED} + document = v4_codec.encode_versioned_distribution(result.value) + assert document["formatVersion"] == 4 + + def test_default_upgrade_keeps_input_attributes(self) -> None: + lib = load(REFERENCE_MODEL) + result = v3_to_v4(lib) + assert result.warnings == () + back = v4_to_v3(result.value) + assert isinstance(back, Ok) + assert back.value == lib diff --git a/tests/unit/ir/test_v4_codec.py b/tests/unit/ir/test_v4_codec.py index af9481f9..fd720cad 100644 --- a/tests/unit/ir/test_v4_codec.py +++ b/tests/unit/ir/test_v4_codec.py @@ -12,6 +12,7 @@ from morphir.ir.diagnostics import DiagnosticError from morphir.ir.v4 import codec, module, package from morphir.ir.v4 import document as doc +from morphir.ir.v4 import incompleteness as inc from morphir.ir.v4 import literal as lit from morphir.ir.v4 import type as t from morphir.ir.v4 import type_def as td @@ -303,9 +304,9 @@ def test_attributes_and_hole(self) -> None: tpe = t.Unit(t.TypeAttributes(source=src_t, extensions=ext)) assert codec.decode_type(codec.encode_type(tpe)) == tpe attrs = v.ValueAttributes(source=src_v, inferred_type=tpe, extensions=ext) - hole = v.Hole(attrs, {"TypeMismatch": {"expected": "a", "found": "b"}}, INT) + hole = v.Hole(attrs, inc.TypeMismatch("a", "b"), INT) assert codec.decode_value(codec.encode_value(hole)) == hole - bare = v.Hole(v.ValueAttributes(), {"Draft": {}}, None) + bare = v.Hole(v.ValueAttributes(), inc.DeletedDuringRefactor("tx"), None) assert codec.decode_value(codec.encode_value(bare)) == bare def test_document_literal(self) -> None: @@ -357,9 +358,10 @@ def test_draft_notice_names_pin(self) -> None: class TestDecodeFailures: def test_unknown_node(self) -> None: - err = pytest.raises(DiagnosticError, codec.decode_type, {"Union": {}}).value + bad = {"Intersection": {}} + err = pytest.raises(DiagnosticError, codec.decode_type, bad).value assert err.codes == (d.UNKNOWN_NODE,) - assert err.diagnostics[0].location == ("Union",) + assert err.diagnostics[0].location == ("Intersection",) def test_compact_spelling_is_not_guessed(self) -> None: assert fail_codes(codec.decode_type, "morphir/SDK:basics#int") == ( @@ -402,9 +404,11 @@ def test_v3_is_refused(self) -> None: codec.decode_versioned_distribution, {"formatVersion": 3} ) == (d.UNSUPPORTED_FORMAT_VERSION_MAJOR,) - def test_other_definition_bodies_are_unknown(self) -> None: + def test_unknown_definition_body(self) -> None: + other = {"MacroBody": {"inputTypes": {}, "outputType": {"Unit": {}}}} + assert fail_codes(codec.decode_value_definition, other) == (d.UNKNOWN_NODE,) native = {"NativeBody": {"inputTypes": {}, "outputType": {"Unit": {}}}} - assert fail_codes(codec.decode_value_definition, native) == (d.UNKNOWN_NODE,) + assert fail_codes(codec.decode_value_definition, native) == (d.MISSING_MEMBER,) @pytest.mark.parametrize( "bad", diff --git a/tests/unit/ir/test_v4_upstream.py b/tests/unit/ir/test_v4_upstream.py new file mode 100644 index 00000000..b22fed15 --- /dev/null +++ b/tests/unit/ir/test_v4_upstream.py @@ -0,0 +1,719 @@ +"""Tests for the upstream v4 constructs (change align-ir-v4-with-upstream). + +Expected JSON is the expanded form of the finos/morphir kit cases at the pinned +commit; the case id is named in each test. +""" + +from __future__ import annotations + +import pytest + +from morphir.ir import diagnostics as d +from morphir.ir import v4 +from morphir.ir.diagnostics import DiagnosticError +from morphir.ir.migrate import Err, v4_to_v3 +from morphir.ir.v4 import annotation as ann +from morphir.ir.v4 import codec, module, package +from morphir.ir.v4 import incompleteness as inc +from morphir.ir.v4 import type as t +from morphir.ir.v4 import type_def as td +from morphir.ir.v4 import type_spec as ts +from morphir.ir.v4 import value as v +from morphir.ir.v4.access_controlled import Public +from morphir.ir.v4.distribution import ( + LibraryDistribution, + PackageInfo, + SpecsDistribution, +) +from morphir.ir.v4.documented import Documented +from morphir.ir.v4.fqname import FQName +from morphir.ir.v4.name import Name +from morphir.ir.v4.path import Path + + +def n(*words: str) -> Name: + return Name(words) + + +def p(*names: Name) -> Path: + return Path(names) + + +TA = t.TypeAttributes() +SDK = p(n("morphir"), n("s", "d", "k")) +INT = t.Reference(TA, FQName(SDK, p(n("basics")), n("int"))) +STRING = t.Reference(TA, FQName(SDK, p(n("string")), n("string"))) +INT_JSON = {"Reference": {"fqname": "morphir/SDK:basics#int"}} +STRING_JSON = {"Reference": {"fqname": "morphir/SDK:string#string"}} +MISSING = FQName(p(n("my", "org"), n("project")), p(n("module")), n("missing")) +TYPED = v.ValueAttributes(inferred_type=INT) + + +def fail_codes(fn, *args) -> tuple[str, ...]: # type: ignore[no-untyped-def] + with pytest.raises(DiagnosticError) as info: + fn(*args) + return info.value.codes + + +def library( + *, + types: dict[Name, td.Definition] | None = None, + values: dict[Name, v.ValueDefinitionBody] | None = None, +) -> LibraryDistribution: + mod = module.Definition( + types={k: Public(Documented("doc", x)) for k, x in (types or {}).items()}, + values={k: Public(Documented("doc", x)) for k, x in (values or {}).items()}, + ) + return LibraryDistribution( + PackageInfo(p(n("my"), n("pkg"))), + package.Definition({p(n("app")): Public(mod)}), + ) + + +def downgrade_errors(lib: LibraryDistribution) -> list[tuple[str, str, str]]: + result = v4_to_v3(lib) + assert isinstance(result, Err) + return [ + (x.code, x.construct, "/".join(map(str, x.location))) + for x in result.diagnostics + ] + + +class TestIncompleteness: + def test_exported_from_v4(self) -> None: + assert v4.Incompleteness is inc.Incompleteness + assert v4.HoleReason is inc.HoleReason + + def test_type_mismatch_hole_definitions_0014(self) -> None: + defn = td.IncompleteTypeDefinition( + [n("a")], + inc.HoleIncompleteness(inc.TypeMismatch("morphir/SDK:basics#int", "s")), + ) + expected = { + "IncompleteTypeDefinition": { + "typeParams": ["a"], + "incompleteness": { + "Hole": { + "reason": { + "TypeMismatch": { + "expected": "morphir/SDK:basics#int", + "found": "s", + } + } + } + }, + } + } + assert codec.encode_type_definition(defn) == expected + assert codec.decode_type_definition(expected) == defn + + def test_draft_with_partial_type_definitions_0015(self) -> None: + defn = td.IncompleteTypeDefinition([], inc.DraftIncompleteness(), INT) + expected = { + "IncompleteTypeDefinition": { + "typeParams": [], + "incompleteness": {"Draft": {}}, + "partialTypeExp": INT_JSON, + } + } + assert codec.encode_type_definition(defn) == expected + assert codec.decode_type_definition(expected) == defn + + def test_hole_keeps_partial_body_definitions_0024(self) -> None: + defn = td.IncompleteTypeDefinition( + [], inc.HoleIncompleteness(inc.UnresolvedReference(MISSING), INT) + ) + expected = { + "IncompleteTypeDefinition": { + "typeParams": [], + "incompleteness": { + "Hole": { + "reason": { + "UnresolvedReference": { + "target": "my-org/project:module#missing" + } + }, + "partialBody": INT_JSON, + } + }, + } + } + assert codec.encode_type_definition(defn) == expected + assert codec.decode_type_definition(expected) == defn + + def test_deleted_during_refactor_reason(self) -> None: + hole = v.Hole(TYPED, inc.DeletedDuringRefactor("tx-42"), INT) + encoded = codec.encode_value(hole) + assert encoded["Hole"]["reason"] == { + "DeletedDuringRefactor": {"tx-id": "tx-42"} + } + assert codec.decode_value(encoded) == hole + + def test_draft_is_not_a_hole_reason_definitions_0026(self) -> None: + bad = { + "IncompleteTypeDefinition": { + "typeParams": [], + "incompleteness": {"Hole": {"reason": {"Draft": {}}}}, + } + } + assert fail_codes(codec.decode_type_definition, bad) == (d.UNKNOWN_NODE,) + + def test_reason_must_be_an_object(self) -> None: + bad = {"Hole": {"reason": "Draft"}} + assert fail_codes(codec.decode_value, bad) == (d.INVALID_JSON,) + + def test_incomplete_type_definition_does_not_downgrade(self) -> None: + defn = td.IncompleteTypeDefinition([], inc.DraftIncompleteness()) + errors = downgrade_errors(library(types={n("t"): defn})) + assert errors == [ + ( + d.UNSUPPORTED_V4_DOWNGRADE, + "IncompleteTypeDefinition", + "distribution/modules/app/types/t/IncompleteTypeDefinition", + ) + ] + + +INPUTS_INT = [(n("a"), INT), (n("b"), INT)] +X_JSON = {"Variable": {"attributes": {"inferredType": INT_JSON}, "name": "x"}} + + +class TestDefinitionBodies: + def test_exported_from_v4(self) -> None: + assert v4.ValueDefinitionBody is v.ValueDefinitionBody + assert v4.NativeBody is v.NativeBody + + def test_native_body_definitions_0009(self) -> None: + body = v.NativeBody(INPUTS_INT, INT, v.NativeInfo(v.Arithmetic())) + expected = { + "NativeBody": { + "inputTypes": {"a": INT_JSON, "b": INT_JSON}, + "outputType": INT_JSON, + "nativeInfo": {"hint": {"Arithmetic": {}}}, + } + } + assert codec.encode_value_definition(body) == expected + assert codec.decode_value_definition(expected) == body + + @pytest.mark.parametrize( + "hint", + [ + v.Comparison(), + v.StringOp(), + v.CollectionOp(), + v.PlatformSpecific("wasm"), + ], + ids=lambda h: type(h).__name__, + ) + def test_native_hints_round_trip(self, hint: v.NativeHint) -> None: + body = v.NativeBody([], INT, v.NativeInfo(hint)) + assert ( + codec.decode_value_definition(codec.encode_value_definition(body)) == body + ) + + def test_platform_specific_names_its_platform_definitions_0030(self) -> None: + bad = { + "NativeBody": { + "inputTypes": {}, + "outputType": INT_JSON, + "nativeInfo": {"hint": {"PlatformSpecific": {}}}, + } + } + assert fail_codes(codec.decode_value_definition, bad) == (d.MISSING_MEMBER,) + + def test_external_body_definitions_0007(self) -> None: + body = v.ExternalBody( + [(n("msg"), STRING)], + INT, + [v.ExternalBinding("javascript", "console.log")], + ) + expected = { + "ExternalBody": { + "inputTypes": {"msg": STRING_JSON}, + "outputType": INT_JSON, + "externals": [ + {"targetPlatform": "javascript", "externalName": "console.log"} + ], + } + } + assert codec.encode_value_definition(body) == expected + assert codec.decode_value_definition(expected) == body + + def test_external_body_with_fallback_definitions_0008(self) -> None: + body = v.ExternalBody( + [(n("x"), INT)], + INT, + [ + v.ExternalBinding("erlang", "math:abs"), + v.ExternalBinding("javascript", "Math.abs"), + ], + v.Variable(TYPED, n("x")), + ) + encoded = codec.encode_value_definition(body) + assert encoded["ExternalBody"]["body"] == X_JSON + assert codec.decode_value_definition(encoded) == body + + def test_duplicate_platform_is_refused(self) -> None: + bindings = [v.ExternalBinding("js", "a"), v.ExternalBinding("js", "b")] + body = v.ExternalBody([], INT, bindings) + assert fail_codes(codec.encode_value_definition, body) == (d.DUPLICATE_MEMBER,) + bad = { + "ExternalBody": { + "inputTypes": {}, + "outputType": INT_JSON, + "externals": [ + {"targetPlatform": "js", "externalName": "a"}, + {"targetPlatform": "js", "externalName": "b"}, + ], + } + } + assert fail_codes(codec.decode_value_definition, bad) == (d.DUPLICATE_MEMBER,) + + def test_external_body_needs_a_binding(self) -> None: + body = v.ExternalBody([], INT, []) + assert fail_codes(codec.encode_value_definition, body) == (d.INVALID_JSON,) + bad = { + "ExternalBody": {"inputTypes": {}, "outputType": INT_JSON, "externals": []} + } + assert fail_codes(codec.decode_value_definition, bad) == (d.INVALID_JSON,) + + def test_incomplete_body_definitions_0016(self) -> None: + body = v.IncompleteBody( + [(n("x"), INT)], + inc.HoleIncompleteness(inc.DeletedDuringRefactor("tx-42")), + INT, + ) + expected = { + "IncompleteBody": { + "inputTypes": {"x": INT_JSON}, + "outputType": INT_JSON, + "incompleteness": { + "Hole": {"reason": {"DeletedDuringRefactor": {"tx-id": "tx-42"}}} + }, + } + } + assert codec.encode_value_definition(body) == expected + assert codec.decode_value_definition(expected) == body + + def test_incomplete_body_keeps_partial_value_definitions_0025(self) -> None: + body = v.IncompleteBody( + [], inc.DraftIncompleteness(), INT, v.Variable(TYPED, n("x")) + ) + encoded = codec.encode_value_definition(body) + assert encoded["IncompleteBody"]["partialBody"] == X_JSON + assert codec.decode_value_definition(encoded) == body + no_output = v.IncompleteBody([], inc.DraftIncompleteness()) + encoded = codec.encode_value_definition(no_output) + assert "outputType" not in encoded["IncompleteBody"] + assert codec.decode_value_definition(encoded) == no_output + + def test_let_binding_holds_any_body(self) -> None: + native = v.NativeBody([], INT, v.NativeInfo(v.Arithmetic())) + let = v.LetDefinition(TYPED, n("f"), native, v.Variable(TYPED, n("f"))) + assert codec.decode_value(codec.encode_value(let)) == let + + def test_library_mixing_bodies_round_trips(self) -> None: + lib = library( + values={ + n("native"): v.NativeBody([], INT, v.NativeInfo(v.Arithmetic())), + n("expr"): v.Definition([], INT, v.Unit(TYPED)), + } + ) + document = codec.encode_versioned_distribution(lib) + assert codec.decode_versioned_distribution(document) == lib + + def test_new_bodies_do_not_downgrade(self) -> None: + native = v.NativeBody([], INT, v.NativeInfo(v.Arithmetic())) + external = v.ExternalBody([], INT, [v.ExternalBinding("js", "f")]) + incomplete = v.IncompleteBody([], inc.DraftIncompleteness()) + let = v.Definition( + [], INT, v.LetDefinition(TYPED, n("g"), native, v.Unit(TYPED)) + ) + lib = library( + values={ + n("n"): native, + n("e"): external, + n("i"): incomplete, + n("l"): let, + } + ) + values = "distribution/modules/app/values" + assert downgrade_errors(lib) == [ + (d.UNSUPPORTED_V4_DOWNGRADE, "NativeBody", f"{values}/n/NativeBody"), + (d.UNSUPPORTED_V4_DOWNGRADE, "ExternalBody", f"{values}/e/ExternalBody"), + ( + d.UNSUPPORTED_V4_DOWNGRADE, + "IncompleteBody", + f"{values}/i/IncompleteBody", + ), + ( + d.UNSUPPORTED_V4_DOWNGRADE, + "NativeBody", + f"{values}/l/body/LetDefinition/definition/NativeBody", + ), + ] + + def test_input_drop_walks_external_fallback(self) -> None: + from morphir.ir.migrate import drop_input_attributes + + inner = v.Definition( + [(n("y"), v.ValueAttributes(inferred_type=STRING), INT)], + INT, + v.Unit(TYPED), + ) + fallback = v.LetDefinition(TYPED, n("g"), inner, v.Unit(TYPED)) + external = v.ExternalBody([], INT, [v.ExternalBinding("js", "f")], fallback) + result = drop_input_attributes(library(values={n("e"): external})) + assert [w.location[-1] for w in result.warnings] == ["y"] + codec.encode_versioned_distribution(result.value) + + +ANNOTATIONS = p(n("annotations")) +ORG = p(n("my", "org"), n("project")) + + +def anno(local: str) -> FQName: + return FQName(ORG, ANNOTATIONS, n(local)) + + +def lit_string(text: str) -> v.Value: + from morphir.ir.v4 import literal + + return v.LiteralValue(v.ValueAttributes(), literal.StringLiteral(text)) + + +class TestAnnotations: + def test_exported_from_v4(self) -> None: + assert v4.Annotation is ann.Annotation + + def test_type_specification_annotations_definitions_0020(self) -> None: + spec = ts.TypeAliasSpecification( + [], + STRING, + [ + ann.CompactAnnotation(anno("deprecated"), "Use user-v2 instead"), + ann.StructuredAnnotation( + anno("since"), + [ + ann.PositionalArgument(lit_string("1.2.0")), + ann.NamedArgument(n("reason"), lit_string("renamed")), + ], + ), + ann.StructuredAnnotation(anno("internal")), + ], + ) + string_literal = {"literal": {"StringLiteral": "1.2.0"}} + expected = { + "TypeAliasSpecification": { + "annotations": [ + "my-org/project:annotations#deprecated:Use user-v2 instead", + { + "name": "my-org/project:annotations#since", + "arguments": [ + {"Literal": string_literal}, + { + "name": "reason", + "value": { + "Literal": {"literal": {"StringLiteral": "renamed"}} + }, + }, + ], + }, + {"name": "my-org/project:annotations#internal"}, + ], + "typeParams": [], + "typeExp": STRING_JSON, + } + } + encoded = codec.encode_type_specification(spec) + assert encoded == expected + assert next(iter(encoded["TypeAliasSpecification"])) == "annotations" + assert codec.decode_type_specification(expected) == spec + + def test_empty_arguments_read_as_structured(self) -> None: + j = { + "OpaqueTypeSpecification": { + "annotations": [ + {"name": "my-org/project:annotations#internal", "arguments": []} + ] + } + } + spec = codec.decode_type_specification(j) + assert spec == ts.OpaqueTypeSpecification( + [], [ann.StructuredAnnotation(anno("internal"))] + ) + + def test_compact_text_starts_after_first_colon_after_hash(self) -> None: + j = { + "OpaqueTypeSpecification": { + "annotations": ["my-org/project:annotations#version:1.0.0:rc"] + } + } + spec = codec.decode_type_specification(j) + assert spec.annotations == [ann.CompactAnnotation(anno("version"), "1.0.0:rc")] + + def test_value_specification_definitions_0021(self) -> None: + spec = v.Specification( + [(n("x"), INT)], INT, [ann.CompactAnnotation(anno("pure"))] + ) + expected = { + "annotations": ["my-org/project:annotations#pure"], + "inputs": {"x": INT_JSON}, + "output": INT_JSON, + } + assert codec.encode_value_specification(spec) == expected + assert codec.decode_value_specification(expected) == spec + + def test_module_specification_definitions_0022(self) -> None: + spec = module.Specification( + doc="The domain module", + annotations=[ann.CompactAnnotation(anno("stable"))], + ) + expected = { + "annotations": ["my-org/project:annotations#stable"], + "types": {}, + "values": {}, + "doc": "The domain module", + } + encoded = codec.encode_module_specification(spec) + assert encoded == expected + assert list(encoded) == ["annotations", "types", "values", "doc"] + assert codec.decode_module_specification(expected) == spec + + def test_no_annotations_writes_no_member(self) -> None: + spec = ts.OpaqueTypeSpecification([]) + assert codec.encode_type_specification(spec) == { + "OpaqueTypeSpecification": {"typeParams": []} + } + + def test_definition_carries_no_annotations_definitions_0023(self) -> None: + bad = { + "TypeAliasDefinition": { + "annotations": [], + "typeParams": [], + "typeExp": STRING_JSON, + } + } + assert fail_codes(codec.decode_type_definition, bad) == (d.UNKNOWN_MEMBER,) + + def test_annotations_do_not_downgrade(self) -> None: + spec = module.Specification( + types={ + n("t"): Documented( + "doc", + ts.OpaqueTypeSpecification( + [], [ann.CompactAnnotation(anno("internal"))] + ), + ) + }, + values={ + n("f"): Documented( + "doc", + v.Specification([], INT, [ann.CompactAnnotation(anno("pure"))]), + ) + }, + annotations=[ann.CompactAnnotation(anno("stable"))], + ) + lib = library() + lib = LibraryDistribution( + lib.package, + lib.definition, + {p(n("dep")): package.Specification({p(n("m")): spec})}, + ) + deps = "distribution/dependencies/dep/m" + assert downgrade_errors(lib) == [ + (d.UNSUPPORTED_V4_DOWNGRADE, "Annotations", f"{deps}/annotations"), + (d.UNSUPPORTED_V4_DOWNGRADE, "Annotations", f"{deps}/types/t/annotations"), + ( + d.UNSUPPORTED_V4_DOWNGRADE, + "Annotations", + f"{deps}/values/f/annotations", + ), + ] + + def test_specs_distribution_with_annotations_round_trips(self) -> None: + spec = module.Specification(annotations=[ann.CompactAnnotation(anno("x"))]) + specs = SpecsDistribution( + PackageInfo(p(n("my"), n("pkg"))), + package.Specification({p(n("m")): spec}), + ) + document = codec.encode_versioned_distribution(specs) + assert codec.decode_versioned_distribution(document) == specs + + +class TestUnion: + def test_exported_from_v4(self) -> None: + assert t.Union in v4.Type.__args__ # type: ignore[attr-defined] + + def test_union_round_trips(self) -> None: + union = t.Union(TA, [STRING, INT]) + expected = {"Union": {"alternatives": [STRING_JSON, INT_JSON]}} + assert codec.encode_type(union) == expected + assert codec.decode_type(expected) == union + + def test_order_duplicates_nesting_and_attributes_are_kept(self) -> None: + attrs = t.TypeAttributes(source=t.SourceLocation(1, 1, 1, 9)) + union = t.Union(attrs, [INT, t.Union(TA, [STRING, INT]), INT]) + encoded = codec.encode_type(union) + assert "attributes" in encoded["Union"] + assert codec.decode_type(encoded) == union + + @pytest.mark.parametrize("alternatives", [[], [INT]], ids=["empty", "singleton"]) + def test_fewer_than_two_alternatives_are_refused( + self, alternatives: list[t.Type] + ) -> None: + union = t.Union(TA, alternatives) + assert fail_codes(codec.encode_type, union) == (d.INVALID_JSON,) + bad = {"Union": {"alternatives": [codec.encode_type(x) for x in alternatives]}} + assert fail_codes(codec.decode_type, bad) == (d.INVALID_JSON,) + + def test_map_attributes_keeps_order_and_duplicates(self) -> None: + union = t.Union(TA, [INT, STRING, INT]) + src = t.TypeAttributes(source=t.SourceLocation(1, 1, 1, 1)) + mapped = t.map_attributes(union, lambda _: src) + assert isinstance(mapped, t.Union) + assert mapped.attributes == src + assert [type(x).__name__ for x in mapped.alternatives] == ["Reference"] * 3 + assert all(x.attributes == src for x in mapped.alternatives) + + def test_union_does_not_downgrade(self) -> None: + defn = td.TypeAliasDefinition([], t.Union(TA, [STRING, INT])) + errors = downgrade_errors(library(types={n("t"): defn})) + assert errors == [ + ( + d.UNSUPPORTED_V4_DOWNGRADE, + "Union", + "distribution/modules/app/types/t/Union", + ) + ] + + +class TestDeprecated: + @pytest.mark.parametrize( + ("cls", "replacement"), + [(v.Native, "NativeBody"), (v.External, "ExternalBody")], + ) + def test_value_nodes_name_their_replacement( + self, cls: type, replacement: str + ) -> None: + doc = cls.__doc__ or "" + assert "Deprecated" in doc + assert replacement in doc + + def test_type_constraints_are_deprecated(self) -> None: + from morphir.ir.v4.type_constraints import TypeConstraints + + assert "Deprecated" in (TypeConstraints.__doc__ or "") + + +class TestInputDropCoverage: + def test_walks_incomplete_partial_body_and_keeps_native(self) -> None: + from morphir.ir.migrate import drop_input_attributes + + inner = v.Definition( + [(n("y"), v.ValueAttributes(inferred_type=STRING), INT)], + INT, + v.Unit(TYPED), + ) + partial = v.LetDefinition(TYPED, n("g"), inner, v.Unit(TYPED)) + incomplete = v.IncompleteBody([], inc.DraftIncompleteness(), None, partial) + native = v.NativeBody([], INT, v.NativeInfo(v.Arithmetic())) + lib = library(values={n("i"): incomplete, n("n"): native}) + result = drop_input_attributes(lib) + assert [w.location[-1] for w in result.warnings] == ["y"] + mod = result.value.definition.modules[p(n("app"))] + assert mod.value.values[n("n")].value.value == native + + def test_application_distribution(self) -> None: + from morphir.ir.migrate import drop_input_attributes + from morphir.ir.v4.distribution import ApplicationDistribution + + defn = v.Definition( + [(n("x"), v.ValueAttributes(inferred_type=STRING), INT)], + INT, + v.Unit(TYPED), + ) + lib = library(values={n("main"): defn}) + app = ApplicationDistribution(lib.package, lib.definition) + result = drop_input_attributes(app) + assert isinstance(result.value, ApplicationDistribution) + assert len(result.warnings) == 1 + + +def attributed_let() -> v.Value: + inner = v.Definition( + [(n("y"), v.ValueAttributes(inferred_type=STRING), INT)], INT, v.Unit(TYPED) + ) + return v.LetDefinition(TYPED, n("g"), inner, v.Unit(TYPED)) + + +def annotated_spec() -> module.Specification: + def structured(name: str) -> ann.StructuredAnnotation: + return ann.StructuredAnnotation( + anno(name), + [ + ann.PositionalArgument(attributed_let()), + ann.NamedArgument(n("k"), attributed_let()), + ], + ) + + return module.Specification( + types={ + n("t"): Documented( + "doc", ts.OpaqueTypeSpecification([], [structured("type")]) + ) + }, + values={ + n("f"): Documented("doc", v.Specification([], INT, [structured("value")])) + }, + annotations=[ann.CompactAnnotation(anno("stable")), structured("module")], + ) + + +class TestInputDropInSpecifications: + def test_specs_distribution_annotation_arguments(self) -> None: + from morphir.ir.migrate import drop_input_attributes + + specs = SpecsDistribution( + PackageInfo(p(n("my"), n("pkg"))), + package.Specification({p(n("m")): annotated_spec()}), + ) + with pytest.raises(DiagnosticError): + codec.encode_versioned_distribution(specs) + result = drop_input_attributes(specs) + assert len(result.warnings) == 6 + assert {w.location[:3] for w in result.warnings} == { + ("distribution", "modules", "m") + } + codec.encode_versioned_distribution(result.value) + + def test_library_dependency_specifications(self) -> None: + from morphir.ir.migrate import drop_input_attributes + + lib = library() + lib = LibraryDistribution( + lib.package, + lib.definition, + {p(n("dep")): package.Specification({p(n("m")): annotated_spec()})}, + ) + result = drop_input_attributes(lib) + assert len(result.warnings) == 6 + assert all( + w.location[:3] == ("distribution", "dependencies", "dep") + for w in result.warnings + ) + codec.encode_versioned_distribution(result.value) + + def test_application_dependency_definitions(self) -> None: + from morphir.ir.migrate import drop_input_attributes + from morphir.ir.v4.distribution import ApplicationDistribution + + dep = library(values={n("f"): v.Definition([], INT, attributed_let())}) + app = ApplicationDistribution( + dep.package, package.Definition(), {p(n("dep")): dep.definition} + ) + result = drop_input_attributes(app) + assert [w.location[:3] for w in result.warnings] == [ + ("distribution", "dependencies", "dep") + ] + codec.encode_versioned_distribution(result.value)