feat(ir): write the compact canonical v4 spelling by default

openspec/changes/write-compact-ir-v4/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: html-design
+created: 2026-10-27

openspec/changes/write-compact-ir-v4/design.html

@@ -0,0 +1,67 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>write-compact-ir-v4: design</title>
+<!--
+  Agent document template (docs/templates/agent-document.html). Specs, plans and reports that a person reviews
+  start from this file: copy it, keep the <head> as it is, and write the content inside <main>
+<h1>write-compact-ir-v4</h1>
+<p class="meta">OpenSpec change write-compact-ir-v4 · Bead morphir-ukvn.8.16 · 2026-10-27 · Proposed</p>
+
+<nav class="toc" aria-label="Contents">
+<ul>
+<li><a href="#context">Context</a></li>
+<li><a href="#goals">Goals and non-goals</a></li>
+<li><a href="#decisions">Decisions</a></li>
+<li><a href="#risks">Risks and trade-offs</a></li>
+<li><a href="#migration">Migration and rollback</a></li>
+</ul>
+</nav>
+
+<h2 id="context">Context</h2>
+<p>See proposal.md, Why. The v4 reader already accepts every spelling the pinned schema allows. The writer builds each node in <code>packages/morphir/src/morphir/ir/v4/codec.py</code> with <code>_wrap(tag, _with_attributes(attrs, members))</code>, recursing through about 50 call sites that thread only a location. morphir-core 0.2.0 is the reference writer at the same profile: it chooses compact or expanded per node from a thread-local encoding and writes <code>doc</code> flattened and first.</p>
+<p>Encoding the morphir-core output of the compiler fixtures with the current Python writer differs only in the node shapes listed in the proposal: type references, tuples, literals, variables and references.</p>
+
+<h2 id="goals">Goals and non-goals</h2>
+<p>The default output of a model must equal morphir-core's output for the same model, byte for byte, including member order. Nothing in the decoded model may change. Not covered: field order in records or maps beyond what the model holds, number lexemes that Python's <code>json</code> does not keep, and the deferred re-sync items listed in the proposal.</p>
+
+<h2 id="decisions">Decisions</h2>
+<h3>CDC-1. Select the spelling per public call with a context variable</h3>
+<p>A <code>Spelling</code> enum (<code>COMPACT</code>, <code>EXPANDED</code>) is held in a <code>contextvars.ContextVar</code> that defaults to <code>COMPACT</code>. Public encoders take a keyword-only <code>spelling: Spelling | None = None</code>. They set the variable for the duration of the call only when given one, so nested public encoders inherit the caller's choice. Each node encoder reads it once.</p>
+<p>Alternative: a parameter on every internal encoder. Rejected because about 50 recursive call sites and the <code>_keyed</code> helpers would each have to forward it, and one missed forward would silently mix spellings. The context variable is task- and thread-local, matches morphir-core's design, and tests check both spellings over every node kind.</p>
+<h3>CDC-2. Compact only nodes without attributes</h3>
+<p>Each node first builds its expanded members as today, then returns the compact form when its attributes are empty. The table follows morphir-core's <code>serialize_compact_type</code>, <code>serialize_value</code>, <code>serialize_pattern</code> and <code>serialize_literal</code>.</p>
+<table>
+<thead><tr><th>Node</th><th>Compact form</th></tr></thead>
+<tbody>
+<tr><td>Type Variable / argument-free Reference</td><td><code>"a"</code> / <code>"pkg:mod#name"</code></td></tr>
+<tr><td>Type Reference with arguments</td><td><code>{"Reference": [fqname, arg, ...]}</code></td></tr>
+<tr><td>Type Tuple</td><td><code>{"Tuple": [type, ...]}</code></td></tr>
+<tr><td>Record, ExtensibleRecord, Function, Unit, Union types</td><td>Wrapper without <code>attributes</code>, as today</td></tr>
+<tr><td>Literal</td><td><code>{"IntegerLiteral": 42}</code></td></tr>
+<tr><td>Literal, Constructor, Reference, Variable, FieldFunction, Tuple, List values</td><td>Payload directly under the tag</td></tr>
+<tr><td>TuplePattern / LiteralPattern</td><td>Pattern list / literal directly under the tag</td></tr>
+</tbody>
+</table>
+<h3>CDC-3. Flatten documentation in both spellings</h3>
+<p><code>_encode_documented</code> writes <code>{"doc": text, **members}</code> with <code>doc</code> first. Decision 0010 makes this the canonical form, and the nested wrapper is only a reader compatibility spelling, so the expanded spelling uses it as well. A documented node whose encoding is not an object, such as a compact string type, cannot occur at the documented positions: type and value definitions and specifications are always objects. The encoder fails with <code>no_v4_spelling</code> if it ever does.</p>
+<h3>CLI-1. Expose the choice in <code>ir convert</code> only</h3>
+<p><code>ir convert --to v4</code> gains <code>--v4-spelling compact|expanded</code>, default <code>compact</code>. <code>compile --ir-version v4</code> writes the default compact spelling, which is also what the native compiler emits.</p>
+
+<h2 id="risks">Risks and trade-offs</h2>
+<table>
+<thead><tr><th>Risk</th><th>Mitigation</th></tr></thead>
+<tbody>
+<tr><td>A downstream reader accepts only the expanded spelling</td><td><code>--v4-spelling expanded</code> and <code>Spelling.EXPANDED</code>; the proposal marks the change BREAKING.</td></tr>
+<tr><td>Compact and expanded spellings mix in one document</td><td>A test walks the expanded output of every node kind and finds no compact form. The byte-for-byte fixture test covers the compact side.</td></tr>
+<tr><td>Ambiguous strings</td><td>The reader already separates FQName strings (containing <code>:</code> and <code>#</code>) from names, as the schema patterns do. Round-trip tests cover both.</td></tr>
+</tbody>
+</table>
+
+<h2 id="migration">Migration and rollback</h2>
+<p>This is a minor release of <code>morphir</code> and, for the CLI option, of <code>morphir-python</code>. Readers are unaffected. To roll back, select the expanded spelling or revert the change; no stored data needs migration, because both spellings decode to the same model.</p>
+</main>
+</body>
+</html>

openspec/changes/write-compact-ir-v4/proposal.md

@@ -0,0 +1,31 @@
+# Proposal
+
+## Why
+
+The finos/morphir v4 JSON profile at the codec's pin (`efa36060`) says writers SHOULD emit the compact canonical vocabulary unless expanded output is explicitly selected. The Python v4 writer emits only the expanded spelling and the pre-decision nested `{"doc", "value"}` wrapper. Its output is therefore larger than other Morphir tools' output and spelled differently: it is 30–35% larger than morphir-core's output for the same compiled models (benchmark run 20261027T194412Z). The reader half landed with the Python binding migration; this change does the writer half of `morphir-ukvn.8.16`.
+
+## What Changes
+
+- The v4 encoders write the compact canonical spelling by default. A node without attributes uses its shortest schema-valid form: a type variable as its name string; a type reference without arguments as its FQName string, or `{"Reference": [fqname, args...]}` with arguments; a tuple type as `{"Tuple": [...]}`; a literal as `{"Kind": payload}`; `Literal`, `Constructor`, `Reference`, `Variable`, `FieldFunction`, `Tuple` and `List` values, and `TuplePattern` and `LiteralPattern` patterns, with the payload directly under the tag. A node that carries attributes keeps the expanded spelling.
+- Documentation is written flattened, with `doc` first beside the documented node's members (decision 0010), instead of the nested `{"doc", "value"}` wrapper.
+- Callers can still select the expanded spelling: `Spelling.EXPANDED` in the library, `--v4-spelling expanded` in `morphir-python ir convert`. **BREAKING** for consumers that parse v4 JSON written by this library and accept only the expanded spelling; every schema-valid reader, including this library and morphir-core, reads both.
+- Out of scope, still waiting for upstream v4 to settle: legacy-spelling warnings, kit diagnostic renames, `Union` and 4.1.0 linked annotations, and the morphir-diwy downgrade code names.
+
+## Capabilities
+
+### New Capabilities
+
+None.
+
+### Modified Capabilities
+
+- `ir/v4`: the writer's default spelling, flattened documentation, and spelling selection.
+- `tools/ir-interchange`: `ir convert --to v4` writes the compact spelling by default and offers `--v4-spelling`.
+
+## Beads
+
+- **Tracked by**: `morphir-ukvn.8.16` (writer half; its other scope stays deferred on upstream).
+
+## Impact
+
+`packages/morphir/src/morphir/ir/v4/codec.py`, its tests, the `ir convert` command and documentation. Encoded v4 output changes shape, but the decoded model does not. Generation in `morphir_tools.python` sends compact v4 to the native compiler, which reads both spellings.

openspec/changes/write-compact-ir-v4/specs/ir/v4/spec.md

@@ -0,0 +1,32 @@
+# Spec Delta
+
+## ADDED Requirements
+
+### Requirement: Write the compact canonical v4 spelling
+By default, the v4 encoders SHALL write the compact canonical spelling of the pinned finos/morphir v4 JSON profile, as morphir-core writes it at the same profile. A node without attributes SHALL use its shortest schema-valid form, and a node that carries attributes SHALL keep the expanded spelling with `attributes` first. (bead: morphir-ukvn.8.16)
+
+#### Scenario: Compiler output re-encodes byte for byte
+- **WHEN** a v4 document written by morphir-core 0.2.0 for a compiled Python model is decoded and encoded with the default spelling
+- **THEN** the encoded JSON equals the original document, including member order
+
+#### Scenario: Compact node forms
+- **WHEN** a model with type variables, argument-free and applied type references, tuples, literals, variables, references, constructors, field functions, lists, tuple patterns and literal patterns is encoded
+- **THEN** they are written as name or FQName strings, `{"Reference": [fqname, args...]}`, `{"Tuple": [...]}`, `{"Kind": payload}` literals, and wrappers holding their payload directly
+
+#### Scenario: Attributes keep the expanded spelling
+- **WHEN** a type or value carrying a source location or an inferred type is encoded
+- **THEN** that node is written in the expanded spelling with `attributes` first, while nodes without attributes stay compact
+
+### Requirement: Write documentation flattened
+The v4 encoders SHALL write documentation flattened, with `doc` first beside the documented node's own members (decision 0010), and SHALL NOT write the nested `{"doc", "value"}` wrapper. (bead: morphir-ukvn.8.16)
+
+#### Scenario: Documented type definition
+- **WHEN** a documented type definition is encoded
+- **THEN** `doc` is its first member, followed by the definition's members, and decoding returns the same documentation and definition
+
+### Requirement: Select the expanded spelling
+Callers SHALL be able to select the expanded spelling, which writes every node in its expanded form. Both spellings SHALL decode to the model that was encoded. (bead: morphir-ukvn.8.16)
+
+#### Scenario: Expanded spelling on request
+- **WHEN** a caller selects the expanded spelling for a model
+- **THEN** every node is written expanded, and decoding returns the same model as decoding the default output

openspec/changes/write-compact-ir-v4/specs/tools/ir-interchange/spec.md

@@ -0,0 +1,14 @@
+# Spec Delta
+
+## ADDED Requirements
+
+### Requirement: Choose the v4 spelling on conversion
+`ir convert --to v4` SHALL write the compact canonical spelling by default and SHALL accept `--v4-spelling expanded` to write the expanded spelling. The option SHALL be rejected as a usage error with any other value, and SHALL have no effect on v3 output. (bead: morphir-ukvn.8.16)
+
+#### Scenario: Default conversion is compact
+- **WHEN** the user converts a format 3 file with `ir convert --to v4`
+- **THEN** the written document uses compact spellings, such as type references as FQName strings, and `ir inspect` reads it
+
+#### Scenario: Expanded conversion on request
+- **WHEN** the user adds `--v4-spelling expanded`
+- **THEN** the written document uses the expanded spellings and decodes to the same model as the compact output

openspec/changes/write-compact-ir-v4/tasks.md

@@ -0,0 +1,9 @@
+# Tasks
+
+Implement against `design.html` and the two delta specs. Follow TDD; claim the group bead before work and close it with test and commit evidence.
+
+## 1. Write compact canonical v4 and flattened documentation (morphir-ukvn.8.16.1)
+
+- [x] 1.1 Add `Spelling` and its per-call selection to the v4 encoders, write the compact forms of types, literals, values and patterns without attributes, and keep the expanded form for nodes with attributes; verify that decoding and re-encoding the checked-in morphir-core compiler fixtures gives byte-identical JSON, that a unit test covers each compact node form, and that a test over every node kind finds no compact form in expanded output.
+- [x] 1.2 Write documentation flattened with `doc` first in both spellings; verify round trips of documented type and value definitions and specifications, and refusal with `no_v4_spelling` of a documented node that is not an object.
+- [x] 1.3 Add `--v4-spelling compact|expanded` to `ir convert`, update the codec docstring, README and CLI docs; verify CLI tests for both values and a rejected value, `ir inspect` of compact output, the existing v3/v4 conversion and BDD tests, and `mise run check`.

packages/morphir-tools/README.md

@@ -39,6 +39,7 @@ morphir-python ir inspect morphir-ir.json
 morphir-python ir convert morphir-ir.json --to v4 -o out.json --package-version 1.2.0
 morphir-python ir convert v4.json --to v3 -o out.json --drop-source-locations
 morphir-python ir convert old-format-1.json --to v3 -o out.json
+morphir-python ir convert morphir-ir.json --to v4 --v4-spelling expanded -o out.json
 ```
 
 `inspect` prints the normalized format version, the distribution kind
@@ -47,6 +48,9 @@ draft and formats 1 and 2 as legacy. `convert` handles `Library` and `Specs` in
 both directions. It writes a v3 `Library` as format `3` and a v3 `Specs` as
 `"3.1.0"`, the lowest version that holds each. A legacy format 1 or 2 file
 converts to a format 3.0.0 `Library` or draft v4; nothing writes format 1 or 2.
+Draft v4 is written in the compact canonical spelling, as other Morphir tools
+write it; `--v4-spelling expanded` writes every node in the expanded form
+instead. Both spellings read back to the same model.
 `convert` writes through a temporary file and renames it only on success, so a
 failed conversion leaves no output file.
 Diagnostics go to standard error as `code at location: message`.

packages/morphir-tools/src/morphir_tools/cli.py

@@ -53,6 +53,13 @@
 app.add_typer(ir_app, name="ir")
 
 
+class V4Spelling(str, Enum):
+    """The draft v4 JSON spelling to write."""
+
+    COMPACT = "compact"
+    EXPANDED = "expanded"
+
+
 class Target(str, Enum):
     """The IR version to convert to."""
 
@@ -184,6 +191,14 @@ def convert(
             "writing v4, which spells an input as a bare type."
         ),
     ] = False,
+    v4_spelling: Annotated[
+        V4Spelling,
+        typer.Option(
+            "--v4-spelling",
+            help="The v4 JSON spelling to write: compact, the canonical form, or "
+            "expanded. Ignored when writing v3.",
+        ),
+    ] = V4Spelling.COMPACT,
 ) -> None:
     """Convert between format 3 (3.0.x and 3.1.x) and draft IR v4.
 
@@ -209,7 +224,9 @@ def convert(
                 upgraded = Ok(migrated.value, migrated.warnings)
             if drop_input_attributes:
                 upgraded = drop_inputs(upgraded.value)
-            document = v4_codec.encode_versioned_distribution(upgraded.value)
+            document = v4_codec.encode_versioned_distribution(
+                upgraded.value, spelling=v4_codec.Spelling(v4_spelling.value)
+            )
             warnings: tuple[Diagnostic, ...] = upgraded.warnings
         else:
             if not _is_v4(version):

packages/morphir/README.md

@@ -177,6 +177,13 @@ the distribution wrapped by its kind, never a format-3 shape. The module
 docstring lists the gaps between the Python v4 model and upstream v4, each of
 which fails with a diagnostic instead of guessing or dropping data.
 
+The reader accepts every spelling the pinned schema allows. The writer emits the
+compact canonical spelling that the profile prefers and Rust `morphir-core`
+writes, so `{"Reference": {"fqname": "morphir/SDK:basics#int"}}` is written as
+`"morphir/SDK:basics#int"`. Pass `spelling=v4_codec.Spelling.EXPANDED`, or use
+`with v4_codec.use_spelling(v4_codec.Spelling.EXPANDED):`, to write every node in
+the expanded form. Both spellings decode to the same model.
+
 The v4 model follows the upstream reference model at that commit:
 
 - A value definition is one of four bodies (`ValueDefinitionBody`): the