Export DOIs and PMCIDs as an identifiers field

[gaurav]

Adds DOIs and PMCIDs to the JSON export as an identifiers array of CURIEs, and
fixes an upstream parser bug that surfaced while doing it. Closes #4

{
  "id": "PMID:16954148",
  "identifiers": ["PMID:16954148", "PMC:PMC1904490", "doi:10.1242/jcs.03153"],
  "journal_name": "Journal of cell science",
  "...": "..."
}

What changed

identifiers is derived at export, not stored. The DOIs and PMCIDs were
already being loaded into the article_id table, so this adds a CTE joined on
(pmid, source_file) — the same key the abstract CTE uses, which is what keeps a
superseded version's DOI out of the export. No schema change and no reload needed
to backfill.

Prefixes match Babel's src/prefixes.py exactly: PMID, lowercase doi, and
PMC. PubMed's PMCID values already begin with PMC, hence the doubled
PMC:PMC6423490. DOI: or a de-doubled PMC:6423490 would silently fail to join
against the Babel publication compendium. Values keep PubMed's own case, so
consumers must match case-insensitively — DOIs are case-insensitive per spec
and PubMed is not internally consistent.

Only DOIs and PMCIDs are promoted. Other ArticleId types (pii, mid, …) stay
in the article_id table and the Parquet export.

validate gained the matching cross-check. efetch_documents rebuilds the
CURIEs from the same ArticleIdList using the exporter's own prefix map, and
check_fields compares them as a set — it is the one list-valued field, so the
string path cannot handle it.

Upstream bug fixed along the way

pubmed_downloader's _extract_article builds Article.xrefs from
.//ArticleIdList/ArticleId under PubmedData. .// matches at any depth and
PubmedData/ReferenceList/Reference carries an ArticleIdList of its own, so
every cited reference's DOI and PMCID was being stored as an identifier of the
citing article. One real record (PMID:41136637) contributed 426 foreign DOIs
alongside its own. parse._xrefs now re-extracts from the direct child — the path
Babel reads.

This has silently corrupted article_id since the first load. The JSON export
dodged it only because it never read that table; the Parquet export did not.

Important

Existing databases need pubmed2db load --force over the corpus to clean
article_id. Nothing detects this automatically, because the source files
themselves have not changed.

Verification

61 tests pass, offline. Each new test was mutation-checked — reverting the
behaviour it covers makes it fail.

Verified end-to-end against a real update file (14,199 records): 96.7% carry a
DOI, 57% a PMCID, and the maximum identifiers on any one record is 3 (it was 427
before the parser fix). validate against live Entrez reports only pre-existing
ahead-of-print staleness — two sampled records have since been assigned both a
journal issue and a PMCID upstream, which shows up in volume/issue/pub_month
as well as in identifiers.

Follow-ups

Noted while doing this work; none are started.

• reference_citation is always empty. Same wrong-scope bug class as the
  one fixed here: cites_pubmed_ids searches MedlineCitation for
  .//ReferenceList/Reference, but ReferenceList is a child of PubmedData, a
  sibling — so it never matches. 0 rows loaded from 14,201 real articles whose
  records carry hundreds of references each. The path fix is one line; the real
  question is whether the citation graph is wanted at full scale (~444 rows for a
  single article). Documented in FUTURE.md.
• Add an identifier-coverage stat to validate. A DOI rate that collapsed
  between two exports would pass silently in the offline structure check. Roughly
  three lines (pct_with_doi / pct_with_pmc); small enough to fold in here.
• Report the xrefs bug upstream to cthoyt/pubmed-downloader. The fix
  there is anchoring the XPath to the direct child. Tracked in FUTURE.md.
• Re-load the production database with load --force so its article_id
  table drops the cited-reference pollution, then re-run the export.
• ELocationID DOIs are still unread. A DOI can also appear as
  Article/ELocationID[@EIdType="doi"], normally duplicating ArticleIdList.
  Babel ignores it too and ArticleIdList is authoritative, so this is a
  fallback worth adding only if records turn up with one and not the other.
• Confirm the identifiers field with Node Annotator. The
  DocumentMetadataAPI spec documents nine string fields and carries the
  identifier only as the results map key — it has no id field and no
  identifier field. id was already an extension; identifiers is a second one,
  and the first non-string value in the record.

🤖 Generated with Claude Code