Initial implementation: download, store, and export PubMed abstracts

[gaurav]

Initial implementation of pubmed2db: a uv/click tool that downloads all PubMed abstracts, loads them into a full-version-history DuckDB database, and exports the latest version of every abstract to JSON (for Node Annotator / ElasticSearch) and Parquet (for downloadable queries).

Pipeline download → load → export:

• download — reuses cthoyt/pubmed-downloader for bulk baseline/update fetching; adds .md5 sidecar tracking so new/changed checksums drive incremental reloads.
• parse — drives the XML iteration itself, calling @cthoyt's _extract_article for the rich record while additionally capturing raw PubDate components (full date fidelity) and <DeleteCitation> PMIDs.
• load — full history, every version tagged with source_file provenance; latest_article view selects the newest non-deleted version per PMID. Journal names come from the NLM Catalog.
• export — one configurable command: sharded NDJSON with DocumentMetadataAPI field names (empty string, never null; pub_month as a 3-letter abbreviation) and per-table Parquet (latest or full history).

The database uses PubMed's own field names; DocumentMetadataAPI names are applied only at JSON export.

Step ordering is enforced from DB state rather than a run flag: load errors if nothing is downloaded, export errors if nothing is loaded and warns about unloaded files or an empty journal table, and the read-only status command reports the same signals.

Design decisions

• Reuse pubmed-downloader as-is (no upstream changes) for download + parse; DuckDB is our store (we don't use its JSONL cache).
• Keep full version history; fully normalized tables; DuckDB for native Parquet + scale.
• MD5 is low-priority (HTTP downloads are reliable, PubMed files immutable): store the checksum and reload only on new/changed files.
• Columnar bulk load: each file's rows go in as an Arrow table via INSERT ... SELECT, ~25–90× faster than the original row-by-row executemany (~20 min/file → ~5–6 s/file).

Operations

• Everything runs from the repo with uv run pubmed2db …; nothing is installed.
• Group-level --threads / --temp-dir (env: PUBMED2DB_THREADS, PUBMED2DB_DUCKDB_TEMP_DIR) cap DuckDB's thread pool and set its spill directory, since DuckDB otherwise sizes its pool from the node's cores rather than the Slurm allocation.
• slurm/README.md documents sizing for both jobs: load needs ~16 GB (memory is bounded by the largest single file), export needs ~256 GB (whole-corpus snapshot + sort).
• The README documents why re-running download/load after a gap can't duplicate data, and the one case that costs real time: a new baseline year is a full re-parse that stores a second version of every PMID.

Known issue worked around

pubmed_downloader.catalog.process_journal_overview() (≤ 0.0.14) raises on the real J_Entrez.txt (its Journal model requires start_year/end_year the file omits), so we parse the overview file ourselves. Tracked in FUTURE.md to revert once fixed upstream via PR cthoyt/pubmed-downloader#16

Verification

• 43 tests pass (no network): parse fidelity, latest-version/delete logic, idempotent + MD5-change reloads, journal parsing, JSON spec fields, Parquet filtering, DuckDB tuning options, CLI end-to-end.
• Full-scale run on RENCI's cluster: the complete corpus exported as 40,901,984 documents across 16 NDJSON shards in 23m13s (≈30k documents/s), peak RSS 201.1 GiB under --mem=256G --cpus-per-task 8.
• Earlier single-file live run: MD5-verified download → 16,664 articles + 56 deletions → 41,923 journals → 16,664 docs across even NDJSON shards (zero nulls, journal names/abbrevs resolved, months normalized) and 15 consistent Parquet files.

TODO (undecided: fix here or file)

• Confirm the pub_year backfill at corpus scale after the rebuild. The
  fix is verified end-to-end on one real baseline file (pubmed26n0005.xml.gz:
  3,625 of 30,000 records, 127 distinct MedlineDate shapes, all recovering a
  year). The corpus-wide check is that validate's core-fields drops from 20
  sampled mismatches to ~2 on the next export.
• Investigate the two residual mismatches that the backfill does not
  explain: issue exported blank vs Entrez "Suppl" (PMID 10137601), and
  article_title blank vs "[Not Available]." (PMID 28972331). Both are
  "exported blank where Entrez has a value", so neither is corrupt data — but
  they may be a third parse gap. Read the archival XML before assuming a cause;
  efetch alone points at the wrong layer (see CLAUDE.md).
• Decide load --force vs a fresh rebuild, and write it into the README's
  "Re-running after a gap". schema.sql only ever adds tables, so a forced
  reload refreshes article_id's rows but leaves the dropped
  reference_citation table sitting there. Rebuilding is probably the simpler
  answer; it is currently only noted in FUTURE.md.
• Test download --limit (newest-N) and the --verify gating. Two
  one-line behaviour changes with no coverage, because _sync_kind would need
  _ensure_urls, requests.get and ensure_module.ensure all mocked.
• Already filed: Consider dropping the global sort in the JSON export #8 — drop the JSON export's global ORDER BY la.pmid in
  favour of pmid % shards. Measure first, since restricting the abstract
  aggregation to the latest snapshot already cut into the same peak RSS.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Initial implementation of pubmed2db, a Python CLI tool that downloads PubMed baseline/update XMLs, loads them into a full-version-history DuckDB schema, and exports the latest non-deleted article set to NDJSON (DocumentMetadataAPI fields) and Parquet.

Changes:

• Added end-to-end pipeline modules (download → parse → load → export → status) plus a Click CLI wrapper.
• Introduced a normalized DuckDB schema with a latest_article view for newest-version selection and deletion handling.
• Added comprehensive pytest coverage with XML/journal fixtures, plus docs and Slurm run guidance.

Reviewed changes

Copilot reviewed 26 out of 30 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/pubmed2db/__init__.py	Defines package version and module docstring.
src/pubmed2db/cli.py	Click CLI implementing download, journals, load, export, update, status.
src/pubmed2db/db.py	DuckDB connection + schema init + source-file registry helpers.
src/pubmed2db/download.py	PubMed sync via pubmed-downloader with .md5 sidecar tracking and optional verification.
src/pubmed2db/export.py	JSON (DocumentMetadataAPI fields, empty-string-not-null) and Parquet exports.
src/pubmed2db/load.py	Normalized loading with full history, idempotent reloads, and journal overview parsing/loading.
src/pubmed2db/parse.py	Self-driven XML iteration using _extract_article, plus raw PubDate + DeleteCitation capture.
src/pubmed2db/schema.sql	Full normalized schema + latest_article view implementing newest-version selection with deletions.
src/pubmed2db/status.py	Read-only pipeline readiness/state summarization derived from DB state.
src/pubmed2db/util.py	Shared helpers for peak RSS and duration formatting.
tests/conftest.py	Shared fixtures (gzipping XML fixtures, DuckDB connections, staged download layout).
tests/test_cli.py	CLI end-to-end tests (export, status, error messaging, directory scan, gzip export).
tests/test_db_download.py	Tests for filename ordering, registry upsert behavior, and MD5 parsing/helpers.
tests/test_export.py	Tests for month normalization, JSON field contract, sharding, and Parquet latest/all behavior.
tests/test_journals.py	Tests for journal overview parsing and journal load behavior.
tests/test_load.py	Tests for full history retention, latest selection, deletion behavior, and MD5-triggered reloads.
tests/test_parse.py	Tests for raw-date fidelity, rich extraction fields, and DeleteCitation capture.
tests/__init__.py	Marks tests as a package.
tests/fixtures/pubmed25n0001.xml	Baseline fixture XML used for parsing/load/export tests.
tests/fixtures/pubmed25n0002.xml	Update fixture XML including a revised article and a DeleteCitation.
tests/fixtures/J_Entrez_sample.txt	Sample NLM journal overview fixture for journal parser tests.
scripts/benchmark_load.py	Script to benchmark parsing vs insertion throughput and peak RSS.
slurm/README.md	Operational notes for running loads on Slurm (memory/time sizing, monitoring).
README.md	Project documentation: purpose, pipeline, usage examples, and development notes.
CLAUDE.md	Architecture/design orientation document for the repository.
FUTURE.md	Deferred work items and known limitations.
pyproject.toml	Project metadata, dependencies, and pytest configuration.
.gitignore	Ignores downloaded data while keeping directory skeleton.
data/.gitkeep	Keeps data/ directory present in the repository.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 30 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (4)

src/pubmed2db/download.py:57

• file_md5() uses hashlib.file_digest(), which is only available in Python 3.11+. Since pyproject.toml declares requires-python = ">=3.10", this will raise AttributeError on Python 3.10 and break download --verify (and test_file_md5). Implement MD5 hashing in a way that works on 3.10+ (e.g., chunked hashlib.md5() updates), or bump the project’s minimum Python to 3.11.

def file_md5(path: Path) -> str:
    """Compute the MD5 hex digest of a local file."""
    with path.open("rb") as fh:
        return hashlib.file_digest(fh, "md5").hexdigest()

src/pubmed2db/download.py:73

• Type annotation for registry is incorrect: sync() builds it as {file_name: published_md5} (a str|None), but _sync_kind() declares registry: dict[str, dict]. This makes prior's type misleading and can hide real type errors.

    ensure_module,
    registry: dict[str, dict],
    limit: int | None,
    verify: bool,

pyproject.toml:17

• pydantic is declared as a direct dependency but does not appear to be used anywhere in src/ or tests/. Keeping it increases install size and dependency surface area unnecessarily; consider removing it unless it’s needed for upcoming work.

    "pyarrow>=14",
    "click>=8.1",
    "pydantic>=2",
    "requests>=2.31",
    "tqdm>=4.66",
    "lxml>=5",

pyproject.toml:10

• The code imports and uses pubmed_downloader private APIs (pubmed_downloader.api._extract_article in src/pubmed2db/parse.py and pubmed_downloader.api._ensure_urls in src/pubmed2db/download.py). With an unconstrained pubmed-downloader>=0.0.14 requirement, a future upstream release can break pubmed2db without any changes here. Consider pinning pubmed-downloader to the exact tested version (or at least adding an upper bound) until these private API usages are replaced with stable/public ones.

dependencies = [
    "pubmed-downloader>=0.0.14",
    "duckdb>=1.1",

[Copilot]

Pull request overview

Copilot reviewed 26 out of 30 changed files in this pull request and generated no new comments.

Suppressed comments (2)

src/pubmed2db/download.py:73

• Type hint for registry is incorrect: sync() builds it as dict[str, str | None] (file_name -> published_md5), but _sync_kind() annotates it as dict[str, dict]. This makes type checking misleading and can hide real issues around the MD5 comparison logic.

    list_cache: Path,
    ensure_module,
    registry: dict[str, dict],
    limit: int | None,
    verify: bool,

src/pubmed2db/download.py:57

• file_md5() uses hashlib.file_digest, which is not available in Python 3.10. Since pyproject.toml declares requires-python = ">=3.10", this will raise AttributeError on the minimum supported version and break download --verify (and its tests). Use a 3.10-compatible MD5 implementation (e.g., incremental hashlib.md5().update() loop), or bump the minimum Python version.

def file_md5(path: Path) -> str:
    """Compute the MD5 hex digest of a local file."""
    with path.open("rb") as fh:
        return hashlib.file_digest(fh, "md5").hexdigest()