Skip to content

multi language packaging - #10

Merged
jd-lara merged 8 commits into
mainfrom
jd/multi-language-packaging
Sep 21, 2026
Merged

jd-lara merged 8 commits into
mainfrom
jd/multi-language-packaging

Conversation

@jd-lara

@jd-lara jd-lara commented Sep 13, 2026

Copy link
Copy Markdown
Member

This PR is pretty large becuase it adds the capability to add TypeScript autogenerated with validation using zod.

WORK IN PROGRESS — the TypeScript postprocess has not completed. The six
domains are generated but NOT deduped, and the *Default consts are still
exported (6304 in operations alone). Do not publish from this state.

Layout: Python moves under python/, TypeScript joins under typescript/, and
rust/ is a prepared slot. Shared at the root: the .schema-version pin, the
fixtures both languages round-trip, codegen/, and the cross-language check.

Python: every path rewritten for the new depth. Regeneration verified
byte-identical to the pre-move tree; 52 tests pass.

TypeScript: orval with client: 'zod', giving runtime validation at parity with
pydantic rather than compile-time-only types. The orval config is generated
(orval rejects glob $ref allowlists) and committed.

Release: lockstep versioning. One v* tag fans out to PyPI and npm, both gated
on scripts/check_version_sync.py, which fails when manifests disagree, when the
tag does not match them, and — with --require — when a manifest a release
claims to publish is absent.

Cross-language check: it had been passing vacuously. It matched
`Base.@kwdef mutable struct` while the Julia generator emits
`Base.@kwdef struct`, so it parsed zero structs, compared zero types, and
printed "Surfaces agree" with exit 0. Fixed, plus a guard that refuses to
report agreement when nothing was compared, and a fix for comparing Julia
struct identifiers instead of the JSON keys the codec actually uses — which
had manufactured 352 false positives.

What it now reports (323 real divergences, Python vs Julia): 290 cases where
Julia leaves an omitted defaulted field ABSENT and Python materializes the
schema default. Verified by running both. That is a pre-existing bug in another
repo; the Julia arm is therefore report-only while Python<->TypeScript will be
the blocking gate. Issue drafted, not filed, at
.claude/plans/DRAFT-issue-poweropenapimodels-defaults.md

Field-name casing is NOT a divergence: Julia snake_cases the struct field but
_decode/_encode use the schema key, so documents round-trip correctly.
make generate-typescript now succeeds and is idempotent (a second run leaves
the tree unchanged).

The postprocess had been failing, correctly. 5 of the 106 duplicates were not
byte-identical: for the same $ref, orval renders a single-valued discriminator
as zod.enum(['STORAGE']) in one domain and zod.literal("STORAGE") in another,
depending on whether that schema also appears as a discriminated-union member
in the importing project's own spec. It refused to guess, which is the
contract.

The two forms are interchangeable, verified rather than assumed:
  - runtime: both accept only "X" and reject everything else, returning the
    same value ('STORAGE', 'storage', 'THERMAL', '', 1, null, undefined, {}
    all agree)
  - types: zod.input of each is mutually assignable, i.e. the same type

So the comparator now canonicalizes a ONE-element enum to its literal form.
Deliberately narrow: a two-element enum is a real difference and still fails
loudly.

Verified independently of the generating agent's report:
  - 106 re-exports written, matching the measured table exactly
    (core 12, operations 52, investments 38, dynamics 3, timeseries 1)
  - 0 cross-domain duplicate schema exports remain
  - 0 exported *Default consts; their .default() call sites survive
    (core 1767, operations 5188, investments 1072, ...)
  - CostCurve resolves from operations to ../core/models

KNOWN NOT DONE — do not publish from this state:
  - 7 tsc --strict errors, all pre-existing in orval's raw output: 6 empty
    array-literal defaults inferring any[], and 1 ImportExportCost composite
    default missing power_units (the TS analogue of the Python side's
    fix_missing_composite_defaults workaround). None introduced by the dedup.
  - typescript/package.json, exports map, tsup build, hand-written document.ts
  - the Python<->TypeScript arm of the cross-language gate
Completes the 0.1.0 multi-language release. Both packages now build, test,
and pack clean, and a CI gate proves they agree.

TypeScript package surface
  @sienna-platform/power-openapi-models 0.1.0: zod schemas for all six
  domains with inferred types, eight subpath exports, dual ESM/CJS with
  .d.ts, zod as a peer dep (^4), engines node >=22. Hand-written document.ts
  mirrors document.py. Verified by installing the packed tarball and
  importing every subpath in both module systems; export counts match the
  schema exactly (20/67/126/60/12/50).

The parity gate earned its keep
  It began the day green while comparing ZERO types. Once fixed it found 156
  real disagreements, every one confirmed against the schema. Both generators
  drop schema defaults, in opposite places:

  - datamodel-code-generator honours `required` but drops the `default` when
    a property is both. CostCurve(...) omitting vom_cost raised
    ValidationError in Python while the same document loaded in TypeScript.
    Fixed generally; subsumes the one-off fix_costcurve_power_units_default.
  - orval drops a property's default outright. An omitted
    EmissionsData.mass_unit was 'KG' in Python and undefined in TypeScript.
  - orval PascalCases all-caps titles, so the packages exported different
    identifiers for the same schema (AGC vs Agc). TypeScript now uses the
    schema title verbatim.
  - orval also INVENTED a default: FuelCurve.startup_fuel_offtake got a full
    curve the schema never declares. Stripped.

  Python<->TypeScript is now a hard CI gate at 0 divergences, and it still
  fails on a perturbed default (verified, given this check's history).
  Python<->Julia runs report-only: its 301 divergences are one pre-existing
  bug in another repo, printed in full every run so the count cannot quietly
  grow.

document.py did not round-trip
  model_dump() materialized name/description/frequency as explicit nulls,
  adding three keys the input never had; keys were emitted in field-
  declaration order; no trailing newline. exclude_unset (NOT exclude_none --
  time_series_storage_file is required and legitimately null), sorted keys,
  trailing newline. Both fixtures now round-trip byte-identically.

Known, bounded, documented
  TypeScript re-orders $.ext keys to ascending numeric order: ECMAScript
  enumerates integer-like own properties that way in every engine. Exactly
  one object in each fixture is affected; content is untouched and JSON key
  order is not significant. Tests assert this is the ONLY difference, so it
  cannot grow.

Follow-ups, not done here
  - fix_missing_composite_defaults is now dead code (its hardcoded pairs stop
    matching now that the generated type is LossCurve)
  - the MarketBidCost.no_load_cost exemption is stale (renamed to
    minimum_energy_offer)
  - PowerOpenAPIModels defaults bug: issue drafted, not filed, at
    .claude/plans/DRAFT-issue-poweropenapimodels-defaults.md

Before releasing: branch protection still names the old `test` job (now
python, typescript, version-sync, cross-language); npm needs one manual
publish to attach a trusted publisher; PyPI needs a pending publisher.
A /simplify pass over the branch's hand-written surface. Generated models are
untouched; regeneration output is byte-identical to before, which is the proof
none of this changed behaviour.

EXEMPTIONS could only rot, never self-correct
  Entries were looked up ONLY from detected divergences, so an entry whose
  divergence stopped occurring went permanently silent -- waiving nothing
  while still reading as deliberate coverage. The same shape as this script's
  own bug earlier on this branch, where it reported "surfaces agree" after
  comparing zero types.

  Now an exemption matching nothing fails the run. It immediately found three
  stale entries: MarketBidCost.no_load_cost (renamed to minimum_energy_offer
  in the schema), and MarketBidCost1 / StorageCostStartUpOneOf, which describe
  artifacts of the Java openapi-generator that PowerOpenAPIModels replaced
  with OpenAPI.jl's native generator. 7 entries -> 4.

Dead code
  fix_missing_composite_defaults had been matching nothing since a schema
  refactor changed its target type to LossCurve -- silently contributing
  nothing while looking like coverage. Deleted, with its now-orphaned
  constant. Three comments also named fix_costcurve_power_units_default, a
  function that no longer exists (it was folded into
  fix_required_fields_with_schema_defaults).

Duplication
  - test.yml resolved the same SiennaSchemas sha in three copy-pasted blocks;
    across the 7-way python matrix that is nine API calls for one commit. Now
    one schema-sha job with an output.
  - The six domain specs were hand-listed in two TypeScript files. Now one
    shared codegen/typescript/domains.ts.
  - postprocess.ts re-read and re-parsed schema JSON on every $ref hop, while
    the Python side already memoized. Now cached, matching _schema_json_cache.
  - check_version_sync.py had the same tomllib-with-regex-fallback twice.

Verbosity
  A Makefile comment restating postprocess.py's docstring verbatim, a 12-line
  ext explanation in document.ts duplicating the README, the first-publish
  rationale stated in three places, and a docstring promising "a one-line
  reason" for entries running five to ten lines.

Also adds typescript/tests/readme.test.ts, the counterpart of
python/tests/test_readme.py: it executes every ts block in the README, so the
examples cannot rot silently. Verified it actually runs them -- injecting a
throw fails exactly the blocks containing it. Suite 19 -> 27.

Not done, deliberately: the TS bracket scanner duplicated across
postprocess.ts and check_cross_language.py is genuinely cross-language;
folding fixMissingCostCurvePowerUnits into the general default machinery is
nontrivial against working code; SystemDocument being hand-mirrored per
language wants a fix in SiennaSchemas, not here.

Verified: gate exit 0, both generators idempotent with byte-identical output,
52 python + 27 typescript tests, both fixtures still round-tripping
byte-identically, lint clean, all five workflows valid.
They are local working notes, not repo content: a design doc for work now
described by the code itself, and an unfiled issue draft for another
repository. .claude/plans/ is gitignored so they stay local.

Nothing tracked referenced either file. Note that c9e5e63's message points at
.claude/plans/DRAFT-issue-poweropenapimodels-defaults.md, which is now local
only -- the PowerOpenAPIModels defaults divergence it documents is still
unfiled.
@jd-lara

jd-lara commented Sep 13, 2026

Copy link
Copy Markdown
Member Author

Reviewer's guide

This PR looks enormous — 151,790 insertions — but ~147,500 of those are generated model code and ~4,700 are lockfiles. The actual hand-written surface is ~4,800 lines added / ~600 removed.

Nothing under python/src/power_openapi_models/*/models.py, typescript/src/*/models.ts, typescript/src/*/index.ts, or typescript/orval.config.ts is hand-written. All of it is regenerated from SiennaSchemas, and CI fails if it drifts. Skip those files.


What this does

Ships power-openapi-models 0.1.0 to PyPI and @sienna-platform/power-openapi-models 0.1.0 to npm from one repo, one schema pin, one tag — with a prepared slot for Rust.

Decision Choice
Layout Symmetric monorepo: python/, typescript/, rust/ slot
TS output orval client: 'zod' — runtime validation at parity with pydantic, not types only
npm shape One package, six subpath exports, mirroring the Python distribution
Versioning Lockstep — one version, one v* tag, fanned out to every registry
Parity Hand-written document.ts and a blocking Python↔TypeScript CI gate

Structure

├── .schema-version          ONE pin, every language
├── fixtures/                shared — both languages round-trip the same bytes
├── codegen/
│   ├── python/postprocess.py
│   └── typescript/{gen-orval-config,postprocess,domains}.ts
├── scripts/
│   ├── check_cross_language.py    the equivalence gate
│   └── check_version_sync.py      lockstep enforcement
├── python/          pyproject.toml, src/, tests/, README
├── typescript/      package.json, src/, tests/, README
└── rust/            README only — upcoming

Suggested review order

1. Start here — the gate. scripts/check_cross_language.py

It compares the generated surfaces field by field: names, requiredness, enum values, scalar kinds, defaults. Python↔TypeScript is blocking; Python↔Julia is report-only (see below).

It was passing vacuously before this PR. It matched Base.@kwdef mutable struct while the Julia generator emits Base.@kwdef struct, so it parsed zero structs, compared zero types, and printed "Surfaces agree" with exit 0. It now refuses to report agreement when it compared nothing.

2. What the fixed gate found. Both generators silently drop schema defaults, in opposite places — 156 real disagreements, each confirmed against the schema:

  • CostCurve(...) omitting vom_cost raised ValidationError in Python while the identical document loaded fine in TypeScript. datamodel-code-generator honours required but drops the default when a property is both.
  • An omitted EmissionsData.mass_unit was 'KG' in Python and undefined in TypeScript. orval drops property-level defaults.
  • The packages exported different identifiers for the same schema (AGC vs Agc) — orval PascalCases all-caps titles.
  • orval also invented a default: FuelCurve.startup_fuel_offtake got a full curve the schema never declares.

All fixed against the schema, generally rather than case-by-case. Now 0 divergences, and the gate still fails on a perturbed default — verified, given its history.

3. The generation pipelines. codegen/typescript/postprocess.ts, codegen/python/postprocess.py

orval has no ref→module mapping (unlike datamodel-codegen's --external-ref-mapping), so each domain re-emits every shared type — 106 duplicates. The postprocess rewrites each into a re-export only if the bodies are identical, and otherwise aborts with a diff rather than guessing which copy wins.

4. The package surfaces. typescript/package.json, typescript/src/document.ts, python/src/power_openapi_models/document.py

document.ts is hand-written, mirroring Core/SystemDocument.json, because that schema is deliberately excluded from the six generated specs. It's the counterpart of document.py.

5. Release machinery. .github/workflows/, scripts/check_version_sync.py

One v* tag fires both release workflows; both gate on version sync, which fails when manifests disagree, when the tag doesn't match, and — with --require — when a manifest a release claims to publish is absent.


Things worth a reviewer's attention

write_document did not round-trip. It materialized name/description/frequency as explicit nulls (adding three keys the input never had), emitted keys in field-declaration order, and omitted the trailing newline. Both fixtures now round-trip byte-identically. Note exclude_unset, not exclude_none — time_series_storage_file is required and legitimately null, so dropping every None strips a required key.

One documented asymmetry. TypeScript re-orders $.ext keys to ascending numeric order: ECMAScript enumerates integer-like own properties that way in every engine. Exactly one object per fixture is affected, content is untouched, and key order isn't significant per RFC 8259. The tests assert this is the only difference, so it can't grow.

Python↔Julia is report-only, deliberately. It reports ~300 divergences from one confirmed pre-existing bug in PowerOpenAPIModels: an omitted defaulted field decodes to ABSENT there where Python materializes the schema default (verified by running both — SEXS without V_ref is ABSENT in Julia, 1.0 in Python). Not fixable by editing generated code here. It prints in full on every run, so the count can't quietly grow. To be filed separately against that repo.

Exemptions could only rot. They were looked up only from detected divergences, so an entry whose divergence stopped occurring went permanently silent while still reading as coverage. A rot detector now fails the run on an exemption matching nothing — it immediately found three stale entries, two describing a Java generator that repo no longer uses.


Before this can release

Three steps outside the diff:

  1. Branch protection — the required check test no longer exists. Jobs are now schema-sha, python, typescript, version-sync, cross-language.
  2. npm first publish must be manual — npm has no pending-publisher flow, so a trusted publisher can only attach to a package that already exists: cd typescript && npm publish --access public once, then wire the trusted publisher. The workflow is written for OIDC and fails until then rather than falling back to a stored token.
  3. PyPI pending publisher — configure before tagging, since the project is unclaimed.

Then git tag v0.1.0 && git push --tags releases both.


Verification

Gate exit 0 · both generators idempotent, byte-identical output · 52 Python + 27 TypeScript tests · both fixtures round-tripping byte-identically · tsc --noEmit 0 errors · subpath exports verified by installing the packed tarball (ESM + CJS) · npm publish --dry-run 2.2 MB / 112 files with no src/, orval, or codegen leakage.

…nfig

CI's "Generated output is up to date" check was failing. Not a test failure --
all 27 TypeScript tests passed; the drift guard caught two real problems.

The pin named a pre-release commit
  .schema-version held 0e233ef, which has diverged from the SiennaSchemas
  v0.1.0 release (9de1405). For a release PR the pin has to be the released
  version, and update-schema.yml already writes a tag rather than a sha.

  Regenerating against the release produces ZERO model changes -- the
  committed models were always built from v0.1.0, only the pin was stale. So
  this is a one-line correction, not a regeneration. __schema_version__ now
  reports "v0.1.0", which is far more useful in a bug report than a bare sha,
  and README's claim that the file "pins the exact SiennaSchemas release" is
  finally true.

orval.config.ts baked in a machine-specific path
  Every `target:` and a header comment carried the SCHEMA_DIR the generator
  happened to be run with, so a committed config could not be reproduced
  anywhere else and the drift check failed on every CI run.

  Relativizing is NOT enough, and testing caught that: a local checkout is a
  sibling of the repo (../../SiennaSchemas from typescript/) while CI checks
  SiennaSchemas out INSIDE the workspace (../SiennaSchemas). Any baked path
  differs. The config now resolves SCHEMA_DIR at orval runtime and bakes
  nothing, so it is byte-identical however it is invoked -- verified with both
  relative and absolute inputs. The Makefile passes SCHEMA_DIR through to
  orval for this.

Verified: the CI drift check now passes (regeneration is a no-op for both
generators), tsc 0 errors, 27 typescript + 52 python tests, cross-language
gate exit 0, version sync accepts v0.1.0, both fixtures still round-tripping
byte-identically, lint clean.
@jd-lara
jd-lara requested a review from daniel-thom September 14, 2026 00:22
check_json_compat.py had no caller: no Makefile target, no workflow, no
test. It pointed at PowerFlowFileParser.jl's inspection_output, whose emit
layer its own docstring called unimplemented, and it hand-maintained a copy
of SystemDocument's required-key list that document.SystemDocument already
encodes. test_serde_fixture.py covers the round-trip comparison it claimed.

bun.lock tracked 296 lines for a tool nothing runs; CI installs with
npm ci against package-lock.json.

Also drops test_serde_fixture.py's pointer to the deleted script.
`Investments/PortfolioDocument.json` is the document an investment portfolio belongs in:
its own required keys (`aggregation`, `requirements_associations`, `base_system_file`),
which a SystemDocument has nowhere to put. Julia has carried a hand-written container for
it since the schema landed; Python and TypeScript had none, so neither language could read
or write a portfolio at all.

Like SystemDocument, both are hand-written rather than generated, for the reason the schema
itself records: `components` is a map from type name to an array of heterogeneous objects,
which neither generator can express as typed buckets, so both skip the document type.

Python gets `PortfolioDocument`, `read_portfolio_document`, and `write_portfolio_document`;
TypeScript the same four names. Rather than copy the Node-22 raw-number-preservation
machinery into a second pair of functions, the TypeScript reader and writer are now generic
over the zod schema and the four exports are thin wrappers -- one implementation, so the two
containers cannot drift on number formatting or key sorting. Python's `write_document` is
split the same way around a shared `_write`.

Tests check the field set and the required list against the schema file in both languages,
so neither container can drift from what it mirrors: 14 properties, 8 required, matching
Julia. Verified end to end -- a portfolio written by TypeScript is read by Python and
rewritten byte-identically, and Julia reads the same file and resolves every field.
@jd-lara
jd-lara merged commit 5316592 into main Sep 21, 2026
10 of 11 checks passed
jd-lara added a commit that referenced this pull request Sep 30, 2026
portfolio-document.test.ts reads Investments/PortfolioDocument.json and falls
back to a sibling of the repo when SIENNA_SCHEMAS_DIR is unset. CI checks the
schemas out inside the workspace, so the two portfolio schema tests failed with
ENOENT on main since #10. The Python job already sets the variable; the
TypeScript job now does too.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant