Skip to content

Parse PSS/E v34 RAW files and tolerate two record shapes that aborted the build - #61

Merged
jd-lara merged 3 commits into
psy6from
ac/v34_updates
Sep 25, 2026
Merged

jd-lara merged 3 commits into
psy6from
ac/v34_updates

Conversation

@annacasavant

Copy link
Copy Markdown

v34 parsing

  • Revision is read from the REV header field. The @! column marker no longer implies v35; v34 exports carry it too.
  • One (sections, dtypes, defaults) registry per layout family. v34 reuses the v35 layout with v33 field order in seven records, new fields appended.
  • Stage 2 branches on version families instead of literal revision strings.
  • Fixes found along the way: double-quoted strings kept their quotes, and the v35 VSC subline table was one field short.

Duplicate two-winding transformers

  • Records on the same bus pair with the same circuit id collided in the id registry. The first record is kept, later duplicates are dropped with a warning naming the PSS/E record.

Out-of-service VSC lines

  • The reader errored when no converter was TYPE 1, the shape PSS/E writes for MDC = 0. The check is now gated on the in-service flag; such lines are kept as unavailable with placeholder setpoints and a warning.

Verification

  • Full suite passes, 2,563 assertions, including new fixtures for each change.

@codecov

codecov Bot commented Sep 22, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.26549% with 11 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/pm_io/pti.jl 89.06% 7 Missing ⚠️
src/pm_io/psse.jl 91.83% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@jd-lara
jd-lara marked this pull request as ready for review September 22, 2026 18:57
@m-bossart
m-bossart self-requested a review September 23, 2026 16:56

@m-bossart m-bossart left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Proposal: reject duplicate TRANSFORMER records in the pti → PowerModels translation

Move the duplicate-transformer rejection out of the openapi layer and into
_psse2pm_transformer! (src/pm_io/psse.jl).

Why there

  • pti stays a faithful record reader.
  • _psse2pm_transformer! iterates pti_data["TRANSFORMER"], a Vector in file order, so
    "keep the first record" is guaranteed by construction rather than by sorting on pm
    index downstream.
  • Every consumer of parse_file / PowerModelsData gets the PSS/E behavior, not just the
    openapi build.
  • The rule is keyed on the PSS/E identity, not on a display name that
    branch_name_formatter can change.

Changes in _psse2pm_transformer!

  1. Before the loop, create a Set of seen identities.
  2. For each record, build the key (I, J, K, uppercase(strip(CKT))).
    This one key covers two-winding (K == 0) and three-winding records.
    It uses the RAW bus numbers, not the routed node-bus endpoints.
  3. If the key was already seen, @warn with the buses and CKT, then continue.
    Otherwise record it and build the entry as today.
  4. Write the stripped CKT into the two-winding source_id (currently the raw
    pop!(transformer, "CKT")).
    Three-winding already strips it for "circuit".

Changes in openapi

  • Remove _is_duplicate_branch and its call in read_branches!
    (src/openapi/branch.jl).
  • register! goes back to being the only guard.
    A duplicate name reaching it then means a parser bug, and it fails loudly.

Tests

  • Add a parse-level test in test/test_parse_psse.jl using
    synthetic_v35_duplicate_transformer_names.raw.
    It checks the warning, a single 201–202 transformer entry holding the first record's
    values (x = 0.05, rating 40), and that the 203–204 transformer is untouched.
  • Add a record whose CKT differs only in whitespace (e.g. ' 1' vs '1 ') to the
    fixture, or to a new fixture.
    Both current duplicate records use '1 ', so the fixture doesn't cover the real case.
  • Keep the openapi test in test/test_openapi_branch.jl as a build test, but drop the
    warning expectation.
    It then checks that the build succeeds and that one circuit survives.

v34 shares v35's file structure but keeps the v33 field order in seven
records and appends its new fields at the end, where v35 inserts them
mid-record. The parser treated any file with @! column comments as v35,
so a real rawd34 export failed on its first generator record.

Stage 1
- Detect the revision from the REV header field for every file; @!
  comments only mark columns. Accept 29, 30, 32, 33, 34 and 35.
- Replace the three-way table selector with one (sections, dtypes,
  defaults) registry per layout family, so parsing and defaulting can
  no longer disagree about the version.
- Add v34 tables for LOAD, GENERATOR, TWO-TERMINAL DC, VSC sublines,
  FACTS, SWITCHED SHUNT and the transformer winding lines, where a real
  v34 export writes NODn last, after CNXAn, unlike the manual's summary.
- Accept the three-field v34 substation 'S' terminal, defaulting its id.
- Wire the v35 default variants into the v35 registry; they were defined
  but never used.

Stage 2
- Branch on two version-family constants instead of literal revision
  strings, with dedicated v34 arms for generator and switched shunt.

Fixes exposed by the v34 file
- Strip double as well as single quotes from string fields; DC-line and
  FACTS names carried literal quote characters.
- Give the v35 VSC subline table its sixteenth field. NREG had been read
  as RMPCT and RMPCT dropped; the fixture only had fifteen fields.
…g the build

Two-winding transformers on the same bus pair with the same circuit id
collided in the id registry. Keep the record read first and drop later
duplicates with a warning naming the PSS/E record. How PSS/E resolves
the repeat is still to be confirmed.

The VSC reader errored when neither converter was TYPE 1, the shape
PSS/E writes for an out-of-service line. Gate that check on the
in-service flag and keep such lines as unavailable with placeholder
setpoints and a warning.
Move the duplicate check from read_branches! into _psse2pm_transformer!,
where the RAW identity is still available. Records are keyed by their
RAW bus numbers and the stripped, case-folded circuit id, covering two-
and three-winding transformers alike. The record read first is kept;
later duplicates are dropped with a warning naming the buses and
circuit id. The two-winding source_id now stores the stripped circuit
id, as the three-winding path already did.

register! is again the only guard in the OpenAPI layer, so a duplicate
name reaching it is a parser bug and fails loudly.
The fixture's duplicate now differs only in circuit-id whitespace,
'1 ' against ' 1', which is the case that occurs in practice.
@jd-lara
jd-lara merged commit 8e8e59d into psy6 Sep 25, 2026
6 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants