Repository navigation
multi language packaging - #10
Conversation
WORK IN PROGRESS — the TypeScript postprocess has not completed. The six domains are generated but NOT deduped, and the *Default consts are still exported (6304 in operations alone). Do not publish from this state. Layout: Python moves under python/, TypeScript joins under typescript/, and rust/ is a prepared slot. Shared at the root: the .schema-version pin, the fixtures both languages round-trip, codegen/, and the cross-language check. Python: every path rewritten for the new depth. Regeneration verified byte-identical to the pre-move tree; 52 tests pass. TypeScript: orval with client: 'zod', giving runtime validation at parity with pydantic rather than compile-time-only types. The orval config is generated (orval rejects glob $ref allowlists) and committed. Release: lockstep versioning. One v* tag fans out to PyPI and npm, both gated on scripts/check_version_sync.py, which fails when manifests disagree, when the tag does not match them, and — with --require — when a manifest a release claims to publish is absent. Cross-language check: it had been passing vacuously. It matched `Base.@kwdef mutable struct` while the Julia generator emits `Base.@kwdef struct`, so it parsed zero structs, compared zero types, and printed "Surfaces agree" with exit 0. Fixed, plus a guard that refuses to report agreement when nothing was compared, and a fix for comparing Julia struct identifiers instead of the JSON keys the codec actually uses — which had manufactured 352 false positives. What it now reports (323 real divergences, Python vs Julia): 290 cases where Julia leaves an omitted defaulted field ABSENT and Python materializes the schema default. Verified by running both. That is a pre-existing bug in another repo; the Julia arm is therefore report-only while Python<->TypeScript will be the blocking gate. Issue drafted, not filed, at .claude/plans/DRAFT-issue-poweropenapimodels-defaults.md Field-name casing is NOT a divergence: Julia snake_cases the struct field but _decode/_encode use the schema key, so documents round-trip correctly.
make generate-typescript now succeeds and is idempotent (a second run leaves
the tree unchanged).
The postprocess had been failing, correctly. 5 of the 106 duplicates were not
byte-identical: for the same $ref, orval renders a single-valued discriminator
as zod.enum(['STORAGE']) in one domain and zod.literal("STORAGE") in another,
depending on whether that schema also appears as a discriminated-union member
in the importing project's own spec. It refused to guess, which is the
contract.
The two forms are interchangeable, verified rather than assumed:
- runtime: both accept only "X" and reject everything else, returning the
same value ('STORAGE', 'storage', 'THERMAL', '', 1, null, undefined, {}
all agree)
- types: zod.input of each is mutually assignable, i.e. the same type
So the comparator now canonicalizes a ONE-element enum to its literal form.
Deliberately narrow: a two-element enum is a real difference and still fails
loudly.
Verified independently of the generating agent's report:
- 106 re-exports written, matching the measured table exactly
(core 12, operations 52, investments 38, dynamics 3, timeseries 1)
- 0 cross-domain duplicate schema exports remain
- 0 exported *Default consts; their .default() call sites survive
(core 1767, operations 5188, investments 1072, ...)
- CostCurve resolves from operations to ../core/models
KNOWN NOT DONE — do not publish from this state:
- 7 tsc --strict errors, all pre-existing in orval's raw output: 6 empty
array-literal defaults inferring any[], and 1 ImportExportCost composite
default missing power_units (the TS analogue of the Python side's
fix_missing_composite_defaults workaround). None introduced by the dedup.
- typescript/package.json, exports map, tsup build, hand-written document.ts
- the Python<->TypeScript arm of the cross-language gate
Completes the 0.1.0 multi-language release. Both packages now build, test,
and pack clean, and a CI gate proves they agree.
TypeScript package surface
@sienna-platform/power-openapi-models 0.1.0: zod schemas for all six
domains with inferred types, eight subpath exports, dual ESM/CJS with
.d.ts, zod as a peer dep (^4), engines node >=22. Hand-written document.ts
mirrors document.py. Verified by installing the packed tarball and
importing every subpath in both module systems; export counts match the
schema exactly (20/67/126/60/12/50).
The parity gate earned its keep
It began the day green while comparing ZERO types. Once fixed it found 156
real disagreements, every one confirmed against the schema. Both generators
drop schema defaults, in opposite places:
- datamodel-code-generator honours `required` but drops the `default` when
a property is both. CostCurve(...) omitting vom_cost raised
ValidationError in Python while the same document loaded in TypeScript.
Fixed generally; subsumes the one-off fix_costcurve_power_units_default.
- orval drops a property's default outright. An omitted
EmissionsData.mass_unit was 'KG' in Python and undefined in TypeScript.
- orval PascalCases all-caps titles, so the packages exported different
identifiers for the same schema (AGC vs Agc). TypeScript now uses the
schema title verbatim.
- orval also INVENTED a default: FuelCurve.startup_fuel_offtake got a full
curve the schema never declares. Stripped.
Python<->TypeScript is now a hard CI gate at 0 divergences, and it still
fails on a perturbed default (verified, given this check's history).
Python<->Julia runs report-only: its 301 divergences are one pre-existing
bug in another repo, printed in full every run so the count cannot quietly
grow.
document.py did not round-trip
model_dump() materialized name/description/frequency as explicit nulls,
adding three keys the input never had; keys were emitted in field-
declaration order; no trailing newline. exclude_unset (NOT exclude_none --
time_series_storage_file is required and legitimately null), sorted keys,
trailing newline. Both fixtures now round-trip byte-identically.
Known, bounded, documented
TypeScript re-orders $.ext keys to ascending numeric order: ECMAScript
enumerates integer-like own properties that way in every engine. Exactly
one object in each fixture is affected; content is untouched and JSON key
order is not significant. Tests assert this is the ONLY difference, so it
cannot grow.
Follow-ups, not done here
- fix_missing_composite_defaults is now dead code (its hardcoded pairs stop
matching now that the generated type is LossCurve)
- the MarketBidCost.no_load_cost exemption is stale (renamed to
minimum_energy_offer)
- PowerOpenAPIModels defaults bug: issue drafted, not filed, at
.claude/plans/DRAFT-issue-poweropenapimodels-defaults.md
Before releasing: branch protection still names the old `test` job (now
python, typescript, version-sync, cross-language); npm needs one manual
publish to attach a trusted publisher; PyPI needs a pending publisher.
A /simplify pass over the branch's hand-written surface. Generated models are
untouched; regeneration output is byte-identical to before, which is the proof
none of this changed behaviour.
EXEMPTIONS could only rot, never self-correct
Entries were looked up ONLY from detected divergences, so an entry whose
divergence stopped occurring went permanently silent -- waiving nothing
while still reading as deliberate coverage. The same shape as this script's
own bug earlier on this branch, where it reported "surfaces agree" after
comparing zero types.
Now an exemption matching nothing fails the run. It immediately found three
stale entries: MarketBidCost.no_load_cost (renamed to minimum_energy_offer
in the schema), and MarketBidCost1 / StorageCostStartUpOneOf, which describe
artifacts of the Java openapi-generator that PowerOpenAPIModels replaced
with OpenAPI.jl's native generator. 7 entries -> 4.
Dead code
fix_missing_composite_defaults had been matching nothing since a schema
refactor changed its target type to LossCurve -- silently contributing
nothing while looking like coverage. Deleted, with its now-orphaned
constant. Three comments also named fix_costcurve_power_units_default, a
function that no longer exists (it was folded into
fix_required_fields_with_schema_defaults).
Duplication
- test.yml resolved the same SiennaSchemas sha in three copy-pasted blocks;
across the 7-way python matrix that is nine API calls for one commit. Now
one schema-sha job with an output.
- The six domain specs were hand-listed in two TypeScript files. Now one
shared codegen/typescript/domains.ts.
- postprocess.ts re-read and re-parsed schema JSON on every $ref hop, while
the Python side already memoized. Now cached, matching _schema_json_cache.
- check_version_sync.py had the same tomllib-with-regex-fallback twice.
Verbosity
A Makefile comment restating postprocess.py's docstring verbatim, a 12-line
ext explanation in document.ts duplicating the README, the first-publish
rationale stated in three places, and a docstring promising "a one-line
reason" for entries running five to ten lines.
Also adds typescript/tests/readme.test.ts, the counterpart of
python/tests/test_readme.py: it executes every ts block in the README, so the
examples cannot rot silently. Verified it actually runs them -- injecting a
throw fails exactly the blocks containing it. Suite 19 -> 27.
Not done, deliberately: the TS bracket scanner duplicated across
postprocess.ts and check_cross_language.py is genuinely cross-language;
folding fixMissingCostCurvePowerUnits into the general default machinery is
nontrivial against working code; SystemDocument being hand-mirrored per
language wants a fix in SiennaSchemas, not here.
Verified: gate exit 0, both generators idempotent with byte-identical output,
52 python + 27 typescript tests, both fixtures still round-tripping
byte-identically, lint clean, all five workflows valid.
They are local working notes, not repo content: a design doc for work now described by the code itself, and an unfiled issue draft for another repository. .claude/plans/ is gitignored so they stay local. Nothing tracked referenced either file. Note that c9e5e63's message points at .claude/plans/DRAFT-issue-poweropenapimodels-defaults.md, which is now local only -- the PowerOpenAPIModels defaults divergence it documents is still unfiled.
Reviewer's guideThis PR looks enormous — 151,790 insertions — but ~147,500 of those are generated model code and ~4,700 are lockfiles. The actual hand-written surface is ~4,800 lines added / ~600 removed. Nothing under What this doesShips
StructureSuggested review order1. Start here — the gate. It compares the generated surfaces field by field: names, requiredness, enum values, scalar kinds, defaults. Python↔TypeScript is blocking; Python↔Julia is report-only (see below). It was passing vacuously before this PR. It matched 2. What the fixed gate found. Both generators silently drop schema defaults, in opposite places — 156 real disagreements, each confirmed against the schema:
All fixed against the schema, generally rather than case-by-case. Now 0 divergences, and the gate still fails on a perturbed default — verified, given its history. 3. The generation pipelines. orval has no ref→module mapping (unlike datamodel-codegen's 4. The package surfaces.
5. Release machinery. One Things worth a reviewer's attention
One documented asymmetry. TypeScript re-orders Python↔Julia is report-only, deliberately. It reports ~300 divergences from one confirmed pre-existing bug in Exemptions could only rot. They were looked up only from detected divergences, so an entry whose divergence stopped occurring went permanently silent while still reading as coverage. A rot detector now fails the run on an exemption matching nothing — it immediately found three stale entries, two describing a Java generator that repo no longer uses. Before this can releaseThree steps outside the diff:
Then VerificationGate exit 0 · both generators idempotent, byte-identical output · 52 Python + 27 TypeScript tests · both fixtures round-tripping byte-identically · |
…nfig CI's "Generated output is up to date" check was failing. Not a test failure -- all 27 TypeScript tests passed; the drift guard caught two real problems. The pin named a pre-release commit .schema-version held 0e233ef, which has diverged from the SiennaSchemas v0.1.0 release (9de1405). For a release PR the pin has to be the released version, and update-schema.yml already writes a tag rather than a sha. Regenerating against the release produces ZERO model changes -- the committed models were always built from v0.1.0, only the pin was stale. So this is a one-line correction, not a regeneration. __schema_version__ now reports "v0.1.0", which is far more useful in a bug report than a bare sha, and README's claim that the file "pins the exact SiennaSchemas release" is finally true. orval.config.ts baked in a machine-specific path Every `target:` and a header comment carried the SCHEMA_DIR the generator happened to be run with, so a committed config could not be reproduced anywhere else and the drift check failed on every CI run. Relativizing is NOT enough, and testing caught that: a local checkout is a sibling of the repo (../../SiennaSchemas from typescript/) while CI checks SiennaSchemas out INSIDE the workspace (../SiennaSchemas). Any baked path differs. The config now resolves SCHEMA_DIR at orval runtime and bakes nothing, so it is byte-identical however it is invoked -- verified with both relative and absolute inputs. The Makefile passes SCHEMA_DIR through to orval for this. Verified: the CI drift check now passes (regeneration is a no-op for both generators), tsc 0 errors, 27 typescript + 52 python tests, cross-language gate exit 0, version sync accepts v0.1.0, both fixtures still round-tripping byte-identically, lint clean.
check_json_compat.py had no caller: no Makefile target, no workflow, no test. It pointed at PowerFlowFileParser.jl's inspection_output, whose emit layer its own docstring called unimplemented, and it hand-maintained a copy of SystemDocument's required-key list that document.SystemDocument already encodes. test_serde_fixture.py covers the round-trip comparison it claimed. bun.lock tracked 296 lines for a tool nothing runs; CI installs with npm ci against package-lock.json. Also drops test_serde_fixture.py's pointer to the deleted script.
`Investments/PortfolioDocument.json` is the document an investment portfolio belongs in: its own required keys (`aggregation`, `requirements_associations`, `base_system_file`), which a SystemDocument has nowhere to put. Julia has carried a hand-written container for it since the schema landed; Python and TypeScript had none, so neither language could read or write a portfolio at all. Like SystemDocument, both are hand-written rather than generated, for the reason the schema itself records: `components` is a map from type name to an array of heterogeneous objects, which neither generator can express as typed buckets, so both skip the document type. Python gets `PortfolioDocument`, `read_portfolio_document`, and `write_portfolio_document`; TypeScript the same four names. Rather than copy the Node-22 raw-number-preservation machinery into a second pair of functions, the TypeScript reader and writer are now generic over the zod schema and the four exports are thin wrappers -- one implementation, so the two containers cannot drift on number formatting or key sorting. Python's `write_document` is split the same way around a shared `_write`. Tests check the field set and the required list against the schema file in both languages, so neither container can drift from what it mirrors: 14 properties, 8 required, matching Julia. Verified end to end -- a portfolio written by TypeScript is read by Python and rewritten byte-identically, and Julia reads the same file and resolves every field.
portfolio-document.test.ts reads Investments/PortfolioDocument.json and falls back to a sibling of the repo when SIENNA_SCHEMAS_DIR is unset. CI checks the schemas out inside the workspace, so the two portfolio schema tests failed with ENOENT on main since #10. The Python job already sets the variable; the TypeScript job now does too.
This PR is pretty large becuase it adds the capability to add TypeScript autogenerated with validation using zod.