Add a verbatim pub_date field for dates the parsed fields can't hold

[gaurav]

Stacked on #15 — review that first; this PR's base is support-approximate-dates.

Addresses the cross-year case raised on #14.

The problem

pub_year/pub_month/pub_day are parsed conveniences, and no arrangement of three fields represents a cross-year range. PMID:10188493 is <MedlineDate>1998 Dec-1999 Jan</MedlineDate>, which we split into:

"pub_year": "1998", "pub_month": "Dec-1999 Jan", "pub_day": ""

Lossless — the two concatenate back to the source — but a field named pub_month holding a year is a wart, and every alternative split just moves the ambiguity somewhere else.

NCBI already solved this

esummary ships a verbatim pubdate on every record, alongside a normalized sortpubdate:

PMID	pubdate	sortpubdate
30690000	"2019 Mar 15"	2019/03/15
8000234	"1994 Sep-Dec"	1994/09/01
10188493	"1998 Dec-1999 Jan"	1998/01/01

This PR takes the first half. pub_date carries PubMed's own string verbatim, on every record. The three parsed fields stay exactly as they are. Consumers rendering a citation read pub_date; consumers sorting or filtering read pub_year; neither parses the other's output.

It fixes every lossy shape at once — seasons, bare year ranges, "n.d." — not just the one that prompted it.

The rule

pub_date = MedlineDate verbatim, if present
         | otherwise: Year + normalize_month(Month or Season) + Day, blanks skipped

"1998 Dec-1999 Jan"  (MedlineDate)   -> "1998 Dec-1999 Jan"
1994 + <Season>Sep-Dec               -> "1994 Sep-Dec"     <- converges with the line above
2019 + Mar + 15                      -> "2019 Mar 15"
2019 (year only)                     -> "2019"
nothing                              -> ""

The convergence is the load-bearing property. PubMed serves the same record as <Year>+<Season> from efetch and as a bare <MedlineDate> in the baseline; both must produce one string, or validate reads every such record as a mismatch. That's asserted directly rather than left implied.

Why pub_year keeps the leading year

Left unchanged at 1998, not 1999, and the reasoning is now written into _year_from_medline_date's docstring where a reader will look. The trailing year is arguably the better semantic answer — the issue mostly reached readers in Jan 1999 — but:

1. NCBI's own sortpubdate uses the leading year on every cross-year shape sampled ("1997 Dec-1998 Jan" → 1997, "1987-1988" → 1987). A consumer joining our pub_year against anything Entrez-derived would disagree otherwise.
2. Every other shape already uses it; a trailing-year rule would fire only when two years appear, making one rare shape behave unlike the rest.
3. It's ~0.07% of PubMed — 4 of 5,773 uniformly sampled records, and three of those were bare "1987-1988" ranges we already handle. The problem shape proper is ~1 in 5,773 (~7k records of 40.9M).

Anyone who needs 1999 can read it from pub_date.

Verification

• test_pub_date_sql_matches_python — 4-way cross product of year × month × day × MedlineDate (~1,080 cases), the same twin-pinning pattern as pub_month. The 4-way product matters: a disagreement can hide in any pairing, notably an absent month leaving a double space in one implementation but not the other.
• test_pub_date_matches_ncbi_pubdate — offline table pinned to esummary's real output for the PMIDs above, including both renderings of 8000234.
• End-to-end: PMIDs 30690000, 8000234 and 10188493 through parse → load → export, then diffed against live esummary — pub_date byte-identical for all three, and 8000234's two renderings converge.
• validate now compares pub_date as a core field against Entrez.
• 135 tests pass, no new lint findings.

Notes

• No schema change, no reload — medline_date and the three components are already stored; this is entirely export-layer. A re-export is required to ship it.
• Size: one string field × 40.9M records, ~800MB on a 16GB export (~5%).
• Field count is now twelve, locked by test_expected_fields_matches_spec. Nine are DocumentMetadataAPI's; id, identifiers and now pub_date are ours. Worth raising on Replace DocumentMetadataAPI by incorporating it into Node Annotator NCATSTranslator/Core-Components-Working-Group#15, where the field set is already under discussion (URLs, authors).
• This defuses Decide whether to normalize month names inside pub_month ranges #16 — with the raw string preserved, normalizing "September-December" → "Sep-Dec" inside pub_month becomes cosmetic.
• Not in scope: sortpubdate, the other half of NCBI's design. Nothing downstream range-filters yet; recorded in FUTURE.md.

TODO

Before merge:

• Measure <Season> prevalence in the archival XML. zgrep -c "<Season>" <a baseline file>. Every case seen so far is <MedlineDate> in the baseline with <Season> appearing only from efetch, so the <Season> half of Keep approximate months in pub_month (issue #14) #15 may be pure insurance. FUTURE.md currently records this as unmeasured — the answer either confirms that claim or lets it be deleted.
• Settle the field name: pub_date or pubdate. Raised on CCWG#15. pub_date is consistent with the neighbouring pub_year/pub_month/pub_day; pubdate matches NCBI exactly. One line in _JSON_FIELDS while this PR is open, a re-export after it lands.

After merge / separate:

• Run Decide whether to normalize month names inside pub_month ranges #16's query against a full load. With the raw string now preserved in pub_date, normalizing "September-December" → "Sep-Dec" inside pub_month is cosmetic rather than a fidelity question; expect this to close as won't-fix unless the counts surprise. Needs a full corpus load.
• Consider a sortpubdate equivalent. The other half of NCBI's design — a normalized YYYY/MM/DD sort key — deliberately not taken here, since nothing downstream range-filters yet. Recorded in FUTURE.md. Needs a consumer that wants ordering, plus its own call on what to emit for an unparseable range (NCBI drops precision to January rather than guessing).
• Re-baseline export size and timing with the twelfth field. ~800MB estimated on a 16GB export, unmeasured. Folds into the Slurm re-baselining FUTURE.md already tracks for the COPY-based export.

🤖 Generated with Claude Code