Skip to content

Remove PSY dependency from parser, and parse to OpenAPI dicts and a database - #42

Draft
hannahchubin wants to merge 2 commits into
psy6from
hc/fix_removePSY_breaks
Draft

Remove PSY dependency from parser, and parse to OpenAPI dicts and a database#42
hannahchubin wants to merge 2 commits into
psy6from
hc/fix_removePSY_breaks

Conversation

@hannahchubin

Copy link
Copy Markdown
Collaborator

Summary

  • Introduce ParsedOpenAPIObjects, a typed intermediate representation of
    PowerModelsData dicts targeting PowerOpenAPIModels (POM) component types
  • Decompose PSS/E transformers into the POM shape: one TransformerCircuit
    per winding, referenced by id from TwoWindingTransformer and
    ThreeWindingTransformer holders. Replaces the older Transformer2W /
    TapTransformer / PhaseShiftingTransformer
  • Add make_database: writes the typed collections into a SQLite database
    via a new src/dbinterface/ layer (schema.sql, triggers.sql,
    TABLE_SCHEMAS, OPENAPI_FIELDS_TO_DB, db_write.jl), ordered so
  • Subtype-inheritance DB layout: TransformerCircuit,
    TwoWindingTransformer, ThreeWindingTransformer,
    DiscreteControlledACBranch, TwoTerminalLCCLine, TwoTerminalVSCLine,
    FACTSControlDevice, SwitchedAdmittance, and
    InterruptibleStandardLoad share their parent's table, with entity_type
    preserving the concrete class
  • Adopt POM.ImpedanceCorrectionData (drops the local shim)
  • src/pm_io/psse.jl: emit Dict (not NamedTuple) for the nested MinMax
    limit fields on LCC / VSC / DiscreteControlledACBranch so
    OpenAPI.from_json can construct them
  • Consolidate PM/OpenAPI helpers, IDGenerator, and
    _MAKE_DATABASE_TYPE_ORDER into src/definitions.jl

Tests

New coverage for the POM/DB pipeline (previously untested):

  • test/test_parse_to_openapi_objects.jl (37 assertions) — typed
    collection counts against case14.raw and case16_all_components.raw,
    transformer decomposition invariants (each 2W → 1 circuit, each 3W → 3
    circuits), holder → circuit id references, cross-component id
    uniqueness.
  • test/test_make_database.jl (36 assertions) — DB row counts by
    entity_type match parse_to_openapi_objects output, subtype-inheritance
    coverage for all 7 subtypes on case16_all_components.raw, dedicated
    transformer table counts, arcs and 3W FK integrity, and the on-disk
    path kwarg.

Rebased on top of psse-parser-consolidation (single PowerModels parse
path, no PowerFlowData dep). Introduces the POM-target intermediate
layer and the SQLite writer:

- ParsedOpenAPIObjects: typed containers for each POM component built
  from PowerModelsData dicts.
- Transformer decomposition to POM shape: TransformerCircuit rows
  shared by TwoWindingTransformer / ThreeWindingTransformer holders,
  replacing the old Transformer2W / TapTransformer / PhaseShiftingTransformer
  triplet.
- Adopt POM.ImpedanceCorrectionData (dropped the local shim it duplicated).
- src/dbinterface: SQLite schema (schema.sql, triggers.sql), TABLE_SCHEMAS
  / OPENAPI_FIELDS_TO_DB routing, and db_write.jl to send OpenAPI-typed
  rows to the DB, ordered so topology/arcs land before FK-dependent rows.
- Subtype inheritance in the DB write: the 7 previously-skipped POM types
  (TransformerCircuit, TwoWindingTransformer, ThreeWindingTransformer,
  DiscreteControlledACBranch, TwoTerminalLCCLine, TwoTerminalVSCLine,
  FACTSControlDevice, SwitchedAdmittance, SynchronousCondenser,
  InterruptibleStandardLoad) share their parent's table with entity_type
  preserving the concrete class.
- src/pm_io/psse.jl: emit Dicts (not NamedTuples) for the nested MinMax
  limit fields on LCC / VSC / DiscreteControlledACBranch so
  OpenAPI.from_json can build them.
- Consolidate constants into definitions.jl (matching the rebase base's
  convention; merged our PM/OpenAPI helpers, IDGenerator, and
  _MAKE_DATABASE_TYPE_ORDER into it.

Full state documented in .claude/HANDOFF.md.
EOF
)
New test files exercise the POM/DB pipeline that landed in the rebase.
Nothing tested it before this commit; only the shared PowerModelsData
dict path had coverage.

- test_parse_to_openapi_objects.jl (37 assertions): typed collection
  counts against case14.raw and case16_all_components.raw, transformer
  decomposition invariants (each 2W → 1 circuit, each 3W → 3 circuits),
  circuit-id references from holders, and cross-component id uniqueness.
- test_make_database.jl (36 assertions): DB row counts by entity_type
  match parse_to_openapi_objects results, subtype-inheritance coverage
  (all 7 POM subtypes we route via shared parent tables appear as
  entity_type rows on case16), dedicated transformer table row counts,
  arcs and 3W FK integrity, and the on-disk path kwarg.
- test/Project.toml: add DBInterface and SQLite as test-only deps so
  the DB tests can query the returned SQLite handle directly.

case16_all_components.raw is the one PSB file that exercises every
subtype simultaneously (TransformerCircuit, TwoWinding, ThreeWinding,
DiscreteControlledACBranch, TwoTerminalLCCLine, TwoTerminalVSCLine,
FACTSControlDevice, SwitchedAdmittance, InterruptibleStandardLoad).

Full suite: 574 pass, 0 fail (73 new).

@jd-lara jd-lara left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd remove the DB writter in this PR for now since we need to still make the SiennaGridDB compliant with the new schemas.

@luke-kiernan luke-kiernan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're using boatloads of strings and symbols here. Those aren't compiler-friendly as a type hierarchy or a Val([scoped enum]).

Now, we don't build systems in hotloops, so we don't need to optimize the heck out of this code. But imo someone should measure how much the type instability and runtime dispatch is costing us. [This could be deferred to post "Sienna 1.0."]

Fetch column `col_name` from the OpenAPI struct `c`, honoring the
(table, db_column) → openapi_field renames in `DB_TO_OPENAPI_FIELDS` and
JSON-serializing any column listed in `JSON_COLUMNS` (except the
`thermal_generators.fuel` FK, which is a plain string).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FK?

`thermal_generators.fuel` FK, which is a plain string).
Returns `nothing` for missing properties.
"""
function get_row_field(c::OpenAPI.APIModel, table_name::AbstractString, col_name::Symbol)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ick. Not very compiler-friendly. Ideas:

  • Symbol and AbstractString instances aren't good for multiple dispatch. I'd consider making table_name and/or col_name Val-types instead.
  • Does hasproperty here really need the instance c, or just the type? I suspect the 2nd, in which case we should be using a type parameter: c::T and then write !hasproperty(T, k).
  • There's only 2 runtime calls: val = getproperty(c,k) and JSON.json(val). Rest are determined by table_name and col_name. Makes me think about @generated.

Many of these comments apply to the below functions in this same file.

We're encoding "here's a dozen different possibilities" here. When that's the case, imo there's 2 performant options:

  1. Create a scoped_enum and pass Val([enum instance]).
  2. Create a type hierarchy and pass types.

The downside of (1) is that constructing Val instances at runtime is slow; the downside of (2) is that it scales poorly and puts strain on the compiler's function lookup table.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants