Skip to content

Add CLI for querying Babel cross-references via DuckDB and NodeNorm - #1

Open
gaurav wants to merge 102 commits into
mainfrom
basic-implementation-in-uv
Open

Add CLI for querying Babel cross-references via DuckDB and NodeNorm#1
gaurav wants to merge 102 commits into
mainfrom
basic-implementation-in-uv

Conversation

@gaurav

@gaurav gaurav commented Dec 3, 2025

Copy link
Copy Markdown
Collaborator

Introduces babel-explorer, a CLI tool to query Babel intermediate files (Parquet) via DuckDB and NodeNorm. BabelDownloader handles caching and freshness, BabelXRefs handles querying, NodeNorm handles label enrichment, and cli.py wires them together with Click. Three commands: xrefs, ids and test-concord.

Endpoint configuration

Babel and NodeNorm endpoints are read from .env rather than hardcoded, so the repository ships only public URLs. BABEL_URL, BABEL_LOCAL_DIR, BABEL_CHECK_DOWNLOAD, NODENORM_URL and BABEL_ALLOW_VERSION_MISMATCH each have a matching command-line option, with precedence running flag > environment variable > .env > built-in default.

The committed .env.example carries the public Babel URL only, with a note telling Translator team members to contact the Babel developers for the Translator-specific URL.

Babel version handling

The release behind BABEL_URL is resolved from VERSION.txt, falling back to the final URL path segment for older trees that predate it, so latest/ resolves to whichever release it currently points at.

BABEL_LOCAL_DIR holds one Babel release at a time. When the release changes, the .meta sidecars under <local_dir>/duckdb/ are deleted so the existing ETag path re-checks each cached file and re-downloads only what changed — the Parquet files themselves are never deleted outright. This keeps Concord.parquet and Identifiers.parquet from being read together across two different Babel releases, which is the failure ETag alone does not prevent.

xrefs fails when NodeNorm's status endpoint reports a different babel_version than the Babel being queried, since labels and cliques would not match the cross-references. --allow-version-mismatch overrides it. The check runs only where NodeNorm is actually consulted.

WIP:

Blocked on Babel/NodeNorm deployments

The shipped default (BABEL_URL=https://stars.renci.org/var/babel/latest/) does not work end to end yet, because public Babel releases do not publish the DuckDB Parquet files. babel-explorer reports this explicitly rather than failing mid-download, but these still need doing:

  • Build a new public Babel that includes the DuckDB files (duckdb/Concord.parquet, duckdb/Identifiers.parquet), then confirm the shipped BABEL_URL default works end to end.
  • Publish the current Babel to its public endpoints.
  • Update NodeNorm Dev (https://nodenormalization-sri.renci.org/) to the latest Babel. Its status endpoint currently reports 2025sep1, so xrefs --labels fails the version check against any current Babel unless --allow-version-mismatch is passed.
  • Add a BABEL_URL repository secret so CI integration tests run against a Babel that publishes the Parquet files. Without it the 24 Parquet-dependent integration tests skip.

Linting

The repository had no [tool.ruff] section, so ruff ran with its default rule set and never checked import ordering. Rules are now E, F, I (import sorting) and UP (pyupgrade), with E501 left to the formatter and *.md excluded (ruff 0.16+ reformats Python inside Markdown code blocks). Line length stays at ruff's default of 88 rather than Babel's 120, which would have reflowed 12 of 15 files for no correctness gain.

CI passes --output-format github so lint failures annotate the diff inline, and keeps using uv run ruff rather than astral-sh/ruff-action — uv already resolves the ruff pinned in uv.lock, so CI and local runs share a version without extra plumbing. CLAUDE.md and README.md now say explicitly to run ruff check and ruff format before committing.

Smaller fixes folded in

  • ids gains --labels, so identifier records can carry NodeNorm labels instead of only raw Parquet columns.
  • --paths with --format json/tsv/csv is now rejected. It previously ignored the flag and emitted the full recursive cross-reference list, which looks like a successful --paths run but is not one.
  • The test data directory is now removed once all xdist workers finish. addopts = "-n auto" made every run parallel, and the old teardown was guarded on a "master" worker that never exists under xdist, so data/test/ survived every run.
  • .idea/ is gitignored.

Testing

192 unit tests pass. Integration tests run against whatever BABEL_URL points at and skip when that release does not publish the Parquet files (currently 13 pass, 24 skip against the public default).

Verified against the live servers: the public default produces the missing-Parquet error; the internal latest/ resolves to 2026jul22 via VERSION.txt; a 2025nov192026jul22 flip removes the .meta sidecars while leaving Concord.parquet in place; the NodeNorm skew check fires before any download; and a full xrefs query returns cross-references end to end.

gaurav and others added 19 commits December 2, 2025 15:38
- Add IdentifierRecord dataclass to babel_xrefs.py (resolves TODO)
- Add 89 tests across 3 files: test_downloader (26), test_babel_xrefs (31), test_nodenorm (23)
- Unit tests (71) use mocks and run without network; integration tests (18) use real downloads/APIs
- Add session-scoped fixtures in conftest.py for shared Parquet file downloads
- Parametrize integration tests over tests/data/valid_curies.txt for easy expansion
- Add integration and slow pytest markers to pyproject.toml
- Update CLAUDE.md and README.md with testing documentation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request implements a basic version of babel-explorer in Python using the uv package manager. It's a tool for querying Babel intermediate files to understand why biological/chemical identifiers are considered equivalent. The implementation includes a downloader for large Parquet files with MD5 validation and resume support, NodeNorm API integration for label enrichment, DuckDB-based cross-reference querying, and a Click-based CLI.

Changes:

  • Initial project structure with uv-based package management (pyproject.toml, Python 3.11+)
  • Core functionality: BabelDownloader with streaming downloads and MD5 validation, NodeNorm API client with LRU caching, BabelXRefs for DuckDB-based Parquet queries
  • CLI with three commands: xrefs, ids, and test-concord
  • Comprehensive test suite with 80 tests split between unit tests (mocked) and integration tests (real network calls)

Reviewed changes

Copilot reviewed 15 out of 19 changed files in this pull request and generated 9 comments.

Show a summary per file
File Description
pyproject.toml Project configuration with dependencies (click, duckdb, requests, tqdm) and pytest markers
.python-version Specifies Python 3.11 requirement
.gitignore Excludes /data directory for downloaded files
README.md User documentation with setup, usage examples, and testing instructions
CLAUDE.md AI assistant guidance documentation (contains outdated wget reference)
src/babel_explorer/cli.py Click-based CLI with xrefs, ids, and test-concord commands
src/babel_explorer/core/downloader.py Streaming file downloader with MD5 validation and resume capability
src/babel_explorer/core/nodenorm.py NodeNorm API client for identifier normalization
src/babel_explorer/core/babel_xrefs.py DuckDB-based cross-reference query engine (has frozen dataclass bug)
tests/conftest.py Session-scoped pytest fixtures for shared test resources
tests/constants.py Shared test constants and CURIE loader utility
tests/data/valid_curies.txt Parametrized test data (one CURIE)
tests/test_downloader.py 26 tests for BabelDownloader (22 unit, 3 integration, 1 slow)
tests/test_nodenorm.py 23 tests for NodeNorm (18 unit, 5 integration)
tests/test_babel_xrefs.py 31 tests for BabelXRefs (22 unit, 8 integration, 1 slow)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/babel_explorer/core/nodenorm.py Outdated
Comment thread src/babel_explorer/core/downloader.py Outdated
Comment thread src/babel_explorer/cli.py Outdated
Comment thread pyproject.toml Outdated
Comment thread src/babel_explorer/core/downloader.py Outdated
Comment thread src/babel_explorer/core/babel_xrefs.py Outdated
Comment thread src/babel_explorer/core/babel_xrefs.py Outdated
Comment thread src/babel_explorer/core/nodenorm.py Outdated
Comment thread CLAUDE.md Outdated
gaurav and others added 6 commits March 2, 2026 17:35
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
- Remove _calculate_md5/_fetch_remote_md5 (too slow on 2.5-3.9 GB files)
- Add sidecar .meta JSON files (ETag, Last-Modified, Content-Length, last_checked)
- Three-tier logic: freshness window → HEAD/ETag check → full re-download
- Add freshness_seconds param to BabelDownloader (default 3h)
- Add --check-download CLI option to xrefs and ids commands (e.g. 3h, never)
- Update tests: replace MD5 test classes with meta/ETag/tier coverage

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 24 changed files in this pull request and generated 3 comments.

Comment thread tests/test_babel_xrefs.py
@pytest.mark.parametrize("curie", VALID_CURIES)
def test_get_curie_xref(babel_xrefs, curie):
"""get_curie_xref returns non-empty CrossReferences with the queried CURIE."""
babel_xrefs.get_curie_xref.cache_clear()
Comment thread src/babel_explorer/cli.py Outdated
Comment thread src/babel_explorer/core/babel_xrefs.py Outdated
gaurav and others added 2 commits May 17, 2026 23:14
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
gaurav and others added 7 commits May 17, 2026 23:52
…ution

Replaces the implicit DuckDB pattern (assign a relation to a Python
variable, then reference that name in a SQL string) with explicit
read_parquet($N) parameterised calls. The recursive query wraps the
scan in a MATERIALIZED CTE so the parquet is read once regardless of
how many CTEs reference it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lru_cache on get_downloaded_file caused a latent bug: once it cached a
local path, subsequent calls bypassed the os.path.exists() guard —
returning a stale path if the file was deleted mid-run. The three-tier
freshness logic (meta + ETag) already prevents redundant network calls,
so the cache adds no benefit and only introduces this risk.

Also corrects the misleading "connection only" comment on the requests
timeout: it is a per-read idle timeout, not a total-transfer limit.

Tests updated to remove cache_clear() calls and rename the caching test
to reflect that the freshness window is now the deduplication mechanism.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes #14. lru_cache on instance methods holds a reference to self in
the cache key, preventing garbage collection of NodeNorm instances for
the lifetime of the process.

Each of the three methods now checks and populates a dedicated dict
(_normalize_cache, _identifier_cache, _clique_cache) on the instance.
This makes caching scope explicit: the cache lives and dies with the
object, and callers who need fresh results simply instantiate a new
NodeNorm. HTTP errors in normalize_curie are intentionally not cached
so a transient failure does not permanently suppress retries.

Tests updated to remove cache_clear() calls — unit tests already
construct a fresh NodeNorm per test case via _make_nn(), and integration
tests are parametrized per-CURIE so cached results do not interfere.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--paths finds and displays the shortest path between each pair of query
CURIEs (implies --recurse), making indirect clique connections immediately
readable. --recurse output now colors each CURIE by BFS distance from the
nearest query term (bold cyan → bold yellow → yellow → green → dim).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the previous parenthesis format so labels are easy to parse
by downstream tools. Embedded backslashes and double quotes are escaped
(\\ and \"). Documents the convention in CLAUDE.md and README.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Consistent with xrefs --labels behaviour: a missing label produces no
output, not "" or "-". Documents the rule in CLAUDE.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@gaurav gaurav changed the title Basic CLI Add CLI for querying Babel cross-references via DuckDB and NodeNorm Aug 14, 2026
gaurav and others added 17 commits August 14, 2026 18:19
CI runs `ruff format --check src/ tests/`, but these two files had drifted
out of ruff-formatted shape. No behaviour change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolve the Babel version from VERSION.txt, which every full release
publishes as `Babel <version>`, falling back to the final URL path segment
for older trees that predate it (such as the 2025nov19 development
directory). `latest/` therefore resolves to whichever release it currently
points at rather than being treated as a version in its own right.

The local cache holds one Babel release at a time, recorded in a
.babel-version marker. When the release changes, sync_cache_version()
deletes the .meta sidecars under <local_dir>/duckdb/ so the existing
ETag path re-checks every cached file immediately, re-downloading only
what actually changed.

Deleting sidecars rather than the Parquet files themselves means nothing
large is destroyed if the version cannot be trusted, an interrupted
refresh self-heals (a .meta is only written after a successful download),
and a directory the user pointed us at is never cleared wholesale. The
hazard this closes is not cache invalidation, which ETag already covers,
but cross-release mixing: Concord.parquet and Identifiers.parquet refresh
independently, so without a pin a query can read two files from different
Babel releases.

Public Babel releases do not publish the DuckDB Parquet files this tool
queries, so a 404 under duckdb/ now raises MissingBabelFileError naming
the release and pointing at BABEL_URL, instead of being retried ten times
with backoff before failing opaquely.

Also add NodeNorm.get_babel_version(), reading `babel_version` from the
status endpoint, so callers can tell which Babel a NodeNorm was built
from. It stays silent in offline mode, where every lookup is
short-circuited already.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The default Babel URL was a Translator-internal server that should not
ship in a public repository. BABEL_URL, BABEL_LOCAL_DIR,
BABEL_CHECK_DOWNLOAD, NODENORM_URL and BABEL_ALLOW_VERSION_MISMATCH are
now read from .env via python-dotenv, wired through Click's envvar= so
precedence runs flag > environment > .env > built-in default. The
committed .env.example carries the public URL only, with a note telling
Translator team members to ask the Babel developers for the internal one.

The public default does not work end to end yet, because public releases
do not publish the DuckDB Parquet files; that now surfaces as a plain
error naming BABEL_URL rather than an opaque failure part-way through a
multi-gigabyte download.

Enriching cross-references with a NodeNorm built from a different Babel
release yields labels and cliques that do not match the cross-references,
so `xrefs` now fails on that mismatch, overridable with
--allow-version-mismatch. The check runs only where NodeNorm is actually
consulted: plain `xrefs` constructs one but never queries it, `ids` has
none, and `test-concord` takes no --babel-url, since comparing NodeNorm
against a rebuild is the whole point of that command.

Integration tests now run against whatever BABEL_URL points at and skip
when that release does not publish the Parquet files, so the suite stays
usable for both Translator developers and public contributors.

Also fold the duplicated --nodenorm-url declaration into a shared
decorator, show defaults for --babel-url and --local-dir in --help, and
drop a dead local and a placeholder-free f-string that were failing
`ruff check`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replace the Translator-internal URL and pinned 2025nov19 version
throughout the docs with the public URL and the .env workflow, and
describe how the single-release cache and the NodeNorm version check
behave.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repository had no [tool.ruff] section, so ruff ran with its default
rule set (E4, E7, E9, F) and never checked import ordering. Select E, F,
I and UP, matching NCATSTranslator/Babel, with E501 left to the formatter
since it owns wrapping.

Line length stays at ruff's default of 88 rather than Babel's 120:
adopting 120 would reflow 12 of 15 files for no correctness gain.

Exclude *.md, because ruff 0.16 began formatting Python inside Markdown
code blocks and this repository's snippets are illustrative fragments
rather than runnable modules. ruff is currently 0.15.2 here, but the
dependency is declared as >=0.11.0, so a lock refresh would hit this.

The 30 resulting violations are all mechanical and auto-fixed: unsorted
imports, datetime.timezone.utc to datetime.UTC, IOError to OSError, a
redundant open() mode, and lru_cache(maxsize=None) to functools.cache.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing told contributors or coding agents to run ruff before pushing, so
formatting drift reached main and only surfaced as a red PR later. Add an
explicit "run before committing or pushing" instruction to CLAUDE.md and
README.md, including what to do when ruff reports files you did not touch:
commit that reformatting separately rather than reverting it.

In CI, pass --output-format github so failures appear as inline
annotations on the diff instead of buried in the log, and drop the
hardcoded `src/ tests/` paths now that [tool.ruff] defines the scope.

CI keeps using `uv run ruff` rather than astral-sh/ruff-action: uv already
resolves the ruff pinned in uv.lock, so CI and local runs share a version
without the action's version-file plumbing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing in src/ called it: BabelXRefs builds DuckDB paths itself and now
queries Parquet through inline read_parquet(), so the helper only existed
to be tested. Its @functools.cache decorator also kept a strong reference
to self for the lifetime of the process, the same leak that motivated
replacing lru_cache with instance dicts in NodeNorm.

Drops its three tests, and a vestigial patch.object() in
test_get_curie_xref_calls_downloader that stubbed the method without ever
asserting on it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ping

Three behaviours added this session had no tests:

NodeNorm.get_babel_version() — reads the status endpoint, stays silent in
offline mode, returns None rather than raising when NodeNorm is
unreachable or reports no version, and caches both outcomes so a failed
lookup is not retried on every call.

The group-level conversion of MissingBabelFileError into a Click error,
so a Babel release that does not publish the Parquet files reads as a
message rather than a traceback, for every command rather than just
xrefs.

That a cache refresh clears only <local_dir>/duckdb/*.meta. The glob is
deliberately not recursive: local_path may hold other Babel releases in
nested directories, and sweeping those up would force needless re-checks
of gigabyte files. Verified the test fails when the glob is made
recursive again.

Also drop the per-file test-count table from CLAUDE.md. It had drifted
badly (test_downloader listed 41 unit tests against an actual 49,
test_formatting was missing entirely), so replace it with the collect-only
commands, and note that integration tests skipping in bulk is the expected
result without a Translator BABEL_URL.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The GitHub Python template ships this line commented out. JetBrains project
files are local editor state, so uncomment it rather than have .idea/ show up
as untracked in every git status.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`addopts = "-n auto"` means every run is parallel, and the session fixture's
teardown was guarded on being the "master" worker -- which never happens under
xdist. data/test/ therefore survived every run, contrary to the comment saying
it was removed so the next run starts fresh.

Move the cleanup to pytest_sessionfinish, which the xdist controller runs after
all workers exit. That removes the race the guard existed to avoid (gw0 deleting
Concord.parquet while gw5 still reads it) without disabling cleanup, and still
fires on a non-parallel run, where there is no worker either.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--paths has a renderer only for the console format. With --format json, tsv or
csv the flag was silently ignored and the full recursive cross-reference list
was emitted instead, which looks like a successful --paths run but is not one.

Fail with a usage error naming the alternative, checked before anything is
downloaded so the mistake costs nothing. Emitting paths as structured records
would be a feature rather than a fix; nothing asks for it yet.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`ids` had no NodeNorm integration at all, so IdentifierRecord output carried
only the raw Identifiers.parquet columns and there was no way to see what a
CURIE actually refers to without a second xrefs or test-concord call.

IdentifierRecord grows a label field, populated from NodeNorm when
--labels is passed, and rendered in double quotes immediately after the CURIE
per the console output convention. As with xrefs, the Babel version check runs
only when labels are requested, since that is the only time NodeNorm is
consulted.

An absent label is omitted from serialized output rather than emitted as an
empty string, matching the console convention and keeping TSV/CSV columns
stable for runs that did not ask for labels.

Also escape ids console output as Rich markup: Parquet values are arbitrary
text and a stray bracket would otherwise be swallowed as a style tag.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The `CURIE "label"` console convention had four independent implementations:
_fmt_label and _curie_str in cli.py, an inline copy in each of the xrefs and
test-concord console loops, and a hand-rolled escape in IdentifierRecord.__str__
that had already drifted (it escaped quotes but never rich markup).

formatting.py now owns it via escape_label(), curie_with_label() and
format_identifier_record(), so the convention and its escaping rules are
defined once. IdentifierRecord loses the console __str__ it should never have
carried in core/, and hl_curie/hl_curie_at_depth collapse into one depth-based
function — the boolean variant was just depth 0 or None, and every call site
was branching between the two.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
write_records took its CSV/TSV field names from the first row alone, so any run
where one record carried a label and another did not raised a ValueError inside
DictWriter on the first record with an extra key. This was already reachable
via `ids --labels` whenever NodeNorm knew some CURIEs but not others.

Field names are now the union of keys across all rows, with restval="" filling
the gaps. The omit-an-absent-label rule also moves off the literal field name
"label" and onto any field ending in it, so LabeledCrossReference's
subj_label/obj_label follow the same convention that ids already did — they
were being emitted as "" in JSON and TSV.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three N+1 patterns dominated runtime on anything larger than a toy query:

- Every labelled CURIE cost its own get_normalized_nodes round-trip, so
  `xrefs --labels --recurse` over a 500-CURIE clique issued ~500 sequential
  HTTPS requests. NodeNorm.normalize_curies() now prefetches a whole batch
  (100 CURIEs per request) and the per-CURIE accessors serve from cache.
- Multi-CURIE `xrefs` ran one full scan of the multi-gigabyte Concord.parquet
  per CURIE. One scan now matches every CURIE, with results bucketed back into
  the per-CURIE cache; a CURIE with no cross-references caches an empty list so
  it is not rescanned.
- _print_paths rebuilt the undirected neighbour map for each of the C(n,2)
  pairs, and build_depth_map built the same structure a third time. All three
  share one build_adjacency().

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
parse_duration spent 39 lines and four separately-worded error messages on
what one regex rejects in a single branch: empty, negative, and non-integer
values now share one message, and the bare-seconds path stops duplicating the
unit-suffix path.

BabelDownloader and NodeNorm each hand-rolled the same lazy-once cache as a
value field plus a _resolved flag, the flag existing only because the resolved
value may legitimately be None. functools.cached_property caches None too, so
both collapse to a single property.

Also extracts _write_meta(), which the tier-2 ETag refresh had been inlining
alongside _save_meta, and drops two parameters no caller ever passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scoped to node_modules/ rather than /web so that frontend source added under
web/ is still tracked — .gitignore already anticipates web/src/lib/.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants