Remove PSY dependency from parser, and parse to OpenAPI dicts and a database - #42
Remove PSY dependency from parser, and parse to OpenAPI dicts and a database#42hannahchubin wants to merge 2 commits into
Conversation
Rebased on top of psse-parser-consolidation (single PowerModels parse path, no PowerFlowData dep). Introduces the POM-target intermediate layer and the SQLite writer: - ParsedOpenAPIObjects: typed containers for each POM component built from PowerModelsData dicts. - Transformer decomposition to POM shape: TransformerCircuit rows shared by TwoWindingTransformer / ThreeWindingTransformer holders, replacing the old Transformer2W / TapTransformer / PhaseShiftingTransformer triplet. - Adopt POM.ImpedanceCorrectionData (dropped the local shim it duplicated). - src/dbinterface: SQLite schema (schema.sql, triggers.sql), TABLE_SCHEMAS / OPENAPI_FIELDS_TO_DB routing, and db_write.jl to send OpenAPI-typed rows to the DB, ordered so topology/arcs land before FK-dependent rows. - Subtype inheritance in the DB write: the 7 previously-skipped POM types (TransformerCircuit, TwoWindingTransformer, ThreeWindingTransformer, DiscreteControlledACBranch, TwoTerminalLCCLine, TwoTerminalVSCLine, FACTSControlDevice, SwitchedAdmittance, SynchronousCondenser, InterruptibleStandardLoad) share their parent's table with entity_type preserving the concrete class. - src/pm_io/psse.jl: emit Dicts (not NamedTuples) for the nested MinMax limit fields on LCC / VSC / DiscreteControlledACBranch so OpenAPI.from_json can build them. - Consolidate constants into definitions.jl (matching the rebase base's convention; merged our PM/OpenAPI helpers, IDGenerator, and _MAKE_DATABASE_TYPE_ORDER into it. Full state documented in .claude/HANDOFF.md. EOF )
New test files exercise the POM/DB pipeline that landed in the rebase. Nothing tested it before this commit; only the shared PowerModelsData dict path had coverage. - test_parse_to_openapi_objects.jl (37 assertions): typed collection counts against case14.raw and case16_all_components.raw, transformer decomposition invariants (each 2W → 1 circuit, each 3W → 3 circuits), circuit-id references from holders, and cross-component id uniqueness. - test_make_database.jl (36 assertions): DB row counts by entity_type match parse_to_openapi_objects results, subtype-inheritance coverage (all 7 POM subtypes we route via shared parent tables appear as entity_type rows on case16), dedicated transformer table row counts, arcs and 3W FK integrity, and the on-disk path kwarg. - test/Project.toml: add DBInterface and SQLite as test-only deps so the DB tests can query the returned SQLite handle directly. case16_all_components.raw is the one PSB file that exercises every subtype simultaneously (TransformerCircuit, TwoWinding, ThreeWinding, DiscreteControlledACBranch, TwoTerminalLCCLine, TwoTerminalVSCLine, FACTSControlDevice, SwitchedAdmittance, InterruptibleStandardLoad). Full suite: 574 pass, 0 fail (73 new).
jd-lara
left a comment
There was a problem hiding this comment.
I'd remove the DB writter in this PR for now since we need to still make the SiennaGridDB compliant with the new schemas.
luke-kiernan
left a comment
There was a problem hiding this comment.
We're using boatloads of strings and symbols here. Those aren't compiler-friendly as a type hierarchy or a Val([scoped enum]).
Now, we don't build systems in hotloops, so we don't need to optimize the heck out of this code. But imo someone should measure how much the type instability and runtime dispatch is costing us. [This could be deferred to post "Sienna 1.0."]
| Fetch column `col_name` from the OpenAPI struct `c`, honoring the | ||
| (table, db_column) → openapi_field renames in `DB_TO_OPENAPI_FIELDS` and | ||
| JSON-serializing any column listed in `JSON_COLUMNS` (except the | ||
| `thermal_generators.fuel` FK, which is a plain string). |
| `thermal_generators.fuel` FK, which is a plain string). | ||
| Returns `nothing` for missing properties. | ||
| """ | ||
| function get_row_field(c::OpenAPI.APIModel, table_name::AbstractString, col_name::Symbol) |
There was a problem hiding this comment.
Ick. Not very compiler-friendly. Ideas:
SymbolandAbstractStringinstances aren't good for multiple dispatch. I'd consider makingtable_nameand/orcol_nameVal-types instead.- Does
haspropertyhere really need the instancec, or just the type? I suspect the 2nd, in which case we should be using a type parameter:c::Tand then write!hasproperty(T, k). - There's only 2 runtime calls:
val = getproperty(c,k)andJSON.json(val). Rest are determined bytable_nameandcol_name. Makes me think about@generated.
Many of these comments apply to the below functions in this same file.
We're encoding "here's a dozen different possibilities" here. When that's the case, imo there's 2 performant options:
- Create a
scoped_enumand passVal([enum instance]). - Create a type hierarchy and pass types.
The downside of (1) is that constructing Val instances at runtime is slow; the downside of (2) is that it scales poorly and puts strain on the compiler's function lookup table.
Summary
ParsedOpenAPIObjects, a typed intermediate representation ofPowerModelsDatadicts targeting PowerOpenAPIModels (POM) component typesTransformerCircuitper winding, referenced by id from
TwoWindingTransformerandThreeWindingTransformerholders. Replaces the olderTransformer2W/TapTransformer/PhaseShiftingTransformermake_database: writes the typed collections into a SQLite databasevia a new
src/dbinterface/layer (schema.sql,triggers.sql,TABLE_SCHEMAS,OPENAPI_FIELDS_TO_DB,db_write.jl), ordered soTransformerCircuit,TwoWindingTransformer,ThreeWindingTransformer,DiscreteControlledACBranch,TwoTerminalLCCLine,TwoTerminalVSCLine,FACTSControlDevice,SwitchedAdmittance, andInterruptibleStandardLoadshare their parent's table, withentity_typepreserving the concrete class
POM.ImpedanceCorrectionData(drops the local shim)src/pm_io/psse.jl: emitDict(notNamedTuple) for the nested MinMaxlimit fields on LCC / VSC / DiscreteControlledACBranch so
OpenAPI.from_jsoncan construct themIDGenerator, and_MAKE_DATABASE_TYPE_ORDERintosrc/definitions.jlTests
New coverage for the POM/DB pipeline (previously untested):
test/test_parse_to_openapi_objects.jl(37 assertions) — typedcollection counts against
case14.rawandcase16_all_components.raw,transformer decomposition invariants (each 2W → 1 circuit, each 3W → 3
circuits), holder → circuit id references, cross-component id
uniqueness.
test/test_make_database.jl(36 assertions) — DB row counts byentity_typematchparse_to_openapi_objectsoutput, subtype-inheritancecoverage for all 7 subtypes on
case16_all_components.raw, dedicatedtransformer table counts, arcs and 3W FK integrity, and the on-disk
pathkwarg.