Add CLI for querying Babel cross-references via DuckDB and NodeNorm - #1
Open
gaurav wants to merge 102 commits into
Open
Add CLI for querying Babel cross-references via DuckDB and NodeNorm#1gaurav wants to merge 102 commits into
gaurav wants to merge 102 commits into
Conversation
- Add IdentifierRecord dataclass to babel_xrefs.py (resolves TODO) - Add 89 tests across 3 files: test_downloader (26), test_babel_xrefs (31), test_nodenorm (23) - Unit tests (71) use mocks and run without network; integration tests (18) use real downloads/APIs - Add session-scoped fixtures in conftest.py for shared Parquet file downloads - Parametrize integration tests over tests/data/valid_curies.txt for easy expansion - Add integration and slow pytest markers to pyproject.toml - Update CLAUDE.md and README.md with testing documentation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This pull request implements a basic version of babel-explorer in Python using the uv package manager. It's a tool for querying Babel intermediate files to understand why biological/chemical identifiers are considered equivalent. The implementation includes a downloader for large Parquet files with MD5 validation and resume support, NodeNorm API integration for label enrichment, DuckDB-based cross-reference querying, and a Click-based CLI.
Changes:
- Initial project structure with uv-based package management (pyproject.toml, Python 3.11+)
- Core functionality: BabelDownloader with streaming downloads and MD5 validation, NodeNorm API client with LRU caching, BabelXRefs for DuckDB-based Parquet queries
- CLI with three commands: xrefs, ids, and test-concord
- Comprehensive test suite with 80 tests split between unit tests (mocked) and integration tests (real network calls)
Reviewed changes
Copilot reviewed 15 out of 19 changed files in this pull request and generated 9 comments.
Show a summary per file
| File | Description |
|---|---|
| pyproject.toml | Project configuration with dependencies (click, duckdb, requests, tqdm) and pytest markers |
| .python-version | Specifies Python 3.11 requirement |
| .gitignore | Excludes /data directory for downloaded files |
| README.md | User documentation with setup, usage examples, and testing instructions |
| CLAUDE.md | AI assistant guidance documentation (contains outdated wget reference) |
| src/babel_explorer/cli.py | Click-based CLI with xrefs, ids, and test-concord commands |
| src/babel_explorer/core/downloader.py | Streaming file downloader with MD5 validation and resume capability |
| src/babel_explorer/core/nodenorm.py | NodeNorm API client for identifier normalization |
| src/babel_explorer/core/babel_xrefs.py | DuckDB-based cross-reference query engine (has frozen dataclass bug) |
| tests/conftest.py | Session-scoped pytest fixtures for shared test resources |
| tests/constants.py | Shared test constants and CURIE loader utility |
| tests/data/valid_curies.txt | Parametrized test data (one CURIE) |
| tests/test_downloader.py | 26 tests for BabelDownloader (22 unit, 3 integration, 1 slow) |
| tests/test_nodenorm.py | 23 tests for NodeNorm (18 unit, 5 integration) |
| tests/test_babel_xrefs.py | 31 tests for BabelXRefs (22 unit, 8 integration, 1 slow) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This was referenced Feb 25, 2026
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
- Remove _calculate_md5/_fetch_remote_md5 (too slow on 2.5-3.9 GB files) - Add sidecar .meta JSON files (ETag, Last-Modified, Content-Length, last_checked) - Three-tier logic: freshness window → HEAD/ETag check → full re-download - Add freshness_seconds param to BabelDownloader (default 3h) - Add --check-download CLI option to xrefs and ids commands (e.g. 3h, never) - Update tests: replace MD5 test classes with meta/ETag/tier coverage Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| @pytest.mark.parametrize("curie", VALID_CURIES) | ||
| def test_get_curie_xref(babel_xrefs, curie): | ||
| """get_curie_xref returns non-empty CrossReferences with the queried CURIE.""" | ||
| babel_xrefs.get_curie_xref.cache_clear() |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…ution Replaces the implicit DuckDB pattern (assign a relation to a Python variable, then reference that name in a SQL string) with explicit read_parquet($N) parameterised calls. The recursive query wraps the scan in a MATERIALIZED CTE so the parquet is read once regardless of how many CTEs reference it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lru_cache on get_downloaded_file caused a latent bug: once it cached a local path, subsequent calls bypassed the os.path.exists() guard — returning a stale path if the file was deleted mid-run. The three-tier freshness logic (meta + ETag) already prevents redundant network calls, so the cache adds no benefit and only introduces this risk. Also corrects the misleading "connection only" comment on the requests timeout: it is a per-read idle timeout, not a total-transfer limit. Tests updated to remove cache_clear() calls and rename the caching test to reflect that the freshness window is now the deduplication mechanism. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fixes #14. lru_cache on instance methods holds a reference to self in the cache key, preventing garbage collection of NodeNorm instances for the lifetime of the process. Each of the three methods now checks and populates a dedicated dict (_normalize_cache, _identifier_cache, _clique_cache) on the instance. This makes caching scope explicit: the cache lives and dies with the object, and callers who need fresh results simply instantiate a new NodeNorm. HTTP errors in normalize_curie are intentionally not cached so a transient failure does not permanently suppress retries. Tests updated to remove cache_clear() calls — unit tests already construct a fresh NodeNorm per test case via _make_nn(), and integration tests are parametrized per-CURIE so cached results do not interfere. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--paths finds and displays the shortest path between each pair of query CURIEs (implies --recurse), making indirect clique connections immediately readable. --recurse output now colors each CURIE by BFS distance from the nearest query term (bold cyan → bold yellow → yellow → green → dim). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the previous parenthesis format so labels are easy to parse by downstream tools. Embedded backslashes and double quotes are escaped (\\ and \"). Documents the convention in CLAUDE.md and README.md. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Consistent with xrefs --labels behaviour: a missing label produces no output, not "" or "-". Documents the rule in CLAUDE.md. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CI runs `ruff format --check src/ tests/`, but these two files had drifted out of ruff-formatted shape. No behaviour change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolve the Babel version from VERSION.txt, which every full release publishes as `Babel <version>`, falling back to the final URL path segment for older trees that predate it (such as the 2025nov19 development directory). `latest/` therefore resolves to whichever release it currently points at rather than being treated as a version in its own right. The local cache holds one Babel release at a time, recorded in a .babel-version marker. When the release changes, sync_cache_version() deletes the .meta sidecars under <local_dir>/duckdb/ so the existing ETag path re-checks every cached file immediately, re-downloading only what actually changed. Deleting sidecars rather than the Parquet files themselves means nothing large is destroyed if the version cannot be trusted, an interrupted refresh self-heals (a .meta is only written after a successful download), and a directory the user pointed us at is never cleared wholesale. The hazard this closes is not cache invalidation, which ETag already covers, but cross-release mixing: Concord.parquet and Identifiers.parquet refresh independently, so without a pin a query can read two files from different Babel releases. Public Babel releases do not publish the DuckDB Parquet files this tool queries, so a 404 under duckdb/ now raises MissingBabelFileError naming the release and pointing at BABEL_URL, instead of being retried ten times with backoff before failing opaquely. Also add NodeNorm.get_babel_version(), reading `babel_version` from the status endpoint, so callers can tell which Babel a NodeNorm was built from. It stays silent in offline mode, where every lookup is short-circuited already. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The default Babel URL was a Translator-internal server that should not ship in a public repository. BABEL_URL, BABEL_LOCAL_DIR, BABEL_CHECK_DOWNLOAD, NODENORM_URL and BABEL_ALLOW_VERSION_MISMATCH are now read from .env via python-dotenv, wired through Click's envvar= so precedence runs flag > environment > .env > built-in default. The committed .env.example carries the public URL only, with a note telling Translator team members to ask the Babel developers for the internal one. The public default does not work end to end yet, because public releases do not publish the DuckDB Parquet files; that now surfaces as a plain error naming BABEL_URL rather than an opaque failure part-way through a multi-gigabyte download. Enriching cross-references with a NodeNorm built from a different Babel release yields labels and cliques that do not match the cross-references, so `xrefs` now fails on that mismatch, overridable with --allow-version-mismatch. The check runs only where NodeNorm is actually consulted: plain `xrefs` constructs one but never queries it, `ids` has none, and `test-concord` takes no --babel-url, since comparing NodeNorm against a rebuild is the whole point of that command. Integration tests now run against whatever BABEL_URL points at and skip when that release does not publish the Parquet files, so the suite stays usable for both Translator developers and public contributors. Also fold the duplicated --nodenorm-url declaration into a shared decorator, show defaults for --babel-url and --local-dir in --help, and drop a dead local and a placeholder-free f-string that were failing `ruff check`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Replace the Translator-internal URL and pinned 2025nov19 version throughout the docs with the public URL and the .env workflow, and describe how the single-release cache and the NodeNorm version check behave. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repository had no [tool.ruff] section, so ruff ran with its default rule set (E4, E7, E9, F) and never checked import ordering. Select E, F, I and UP, matching NCATSTranslator/Babel, with E501 left to the formatter since it owns wrapping. Line length stays at ruff's default of 88 rather than Babel's 120: adopting 120 would reflow 12 of 15 files for no correctness gain. Exclude *.md, because ruff 0.16 began formatting Python inside Markdown code blocks and this repository's snippets are illustrative fragments rather than runnable modules. ruff is currently 0.15.2 here, but the dependency is declared as >=0.11.0, so a lock refresh would hit this. The 30 resulting violations are all mechanical and auto-fixed: unsorted imports, datetime.timezone.utc to datetime.UTC, IOError to OSError, a redundant open() mode, and lru_cache(maxsize=None) to functools.cache. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing told contributors or coding agents to run ruff before pushing, so formatting drift reached main and only surfaced as a red PR later. Add an explicit "run before committing or pushing" instruction to CLAUDE.md and README.md, including what to do when ruff reports files you did not touch: commit that reformatting separately rather than reverting it. In CI, pass --output-format github so failures appear as inline annotations on the diff instead of buried in the log, and drop the hardcoded `src/ tests/` paths now that [tool.ruff] defines the scope. CI keeps using `uv run ruff` rather than astral-sh/ruff-action: uv already resolves the ruff pinned in uv.lock, so CI and local runs share a version without the action's version-file plumbing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nothing in src/ called it: BabelXRefs builds DuckDB paths itself and now queries Parquet through inline read_parquet(), so the helper only existed to be tested. Its @functools.cache decorator also kept a strong reference to self for the lifetime of the process, the same leak that motivated replacing lru_cache with instance dicts in NodeNorm. Drops its three tests, and a vestigial patch.object() in test_get_curie_xref_calls_downloader that stubbed the method without ever asserting on it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ping Three behaviours added this session had no tests: NodeNorm.get_babel_version() — reads the status endpoint, stays silent in offline mode, returns None rather than raising when NodeNorm is unreachable or reports no version, and caches both outcomes so a failed lookup is not retried on every call. The group-level conversion of MissingBabelFileError into a Click error, so a Babel release that does not publish the Parquet files reads as a message rather than a traceback, for every command rather than just xrefs. That a cache refresh clears only <local_dir>/duckdb/*.meta. The glob is deliberately not recursive: local_path may hold other Babel releases in nested directories, and sweeping those up would force needless re-checks of gigabyte files. Verified the test fails when the glob is made recursive again. Also drop the per-file test-count table from CLAUDE.md. It had drifted badly (test_downloader listed 41 unit tests against an actual 49, test_formatting was missing entirely), so replace it with the collect-only commands, and note that integration tests skipping in bulk is the expected result without a Translator BABEL_URL. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The GitHub Python template ships this line commented out. JetBrains project files are local editor state, so uncomment it rather than have .idea/ show up as untracked in every git status. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`addopts = "-n auto"` means every run is parallel, and the session fixture's teardown was guarded on being the "master" worker -- which never happens under xdist. data/test/ therefore survived every run, contrary to the comment saying it was removed so the next run starts fresh. Move the cleanup to pytest_sessionfinish, which the xdist controller runs after all workers exit. That removes the race the guard existed to avoid (gw0 deleting Concord.parquet while gw5 still reads it) without disabling cleanup, and still fires on a non-parallel run, where there is no worker either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
--paths has a renderer only for the console format. With --format json, tsv or csv the flag was silently ignored and the full recursive cross-reference list was emitted instead, which looks like a successful --paths run but is not one. Fail with a usage error naming the alternative, checked before anything is downloaded so the mistake costs nothing. Emitting paths as structured records would be a feature rather than a fix; nothing asks for it yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`ids` had no NodeNorm integration at all, so IdentifierRecord output carried only the raw Identifiers.parquet columns and there was no way to see what a CURIE actually refers to without a second xrefs or test-concord call. IdentifierRecord grows a label field, populated from NodeNorm when --labels is passed, and rendered in double quotes immediately after the CURIE per the console output convention. As with xrefs, the Babel version check runs only when labels are requested, since that is the only time NodeNorm is consulted. An absent label is omitted from serialized output rather than emitted as an empty string, matching the console convention and keeping TSV/CSV columns stable for runs that did not ask for labels. Also escape ids console output as Rich markup: Parquet values are arbitrary text and a stray bracket would otherwise be swallowed as a style tag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The `CURIE "label"` console convention had four independent implementations: _fmt_label and _curie_str in cli.py, an inline copy in each of the xrefs and test-concord console loops, and a hand-rolled escape in IdentifierRecord.__str__ that had already drifted (it escaped quotes but never rich markup). formatting.py now owns it via escape_label(), curie_with_label() and format_identifier_record(), so the convention and its escaping rules are defined once. IdentifierRecord loses the console __str__ it should never have carried in core/, and hl_curie/hl_curie_at_depth collapse into one depth-based function — the boolean variant was just depth 0 or None, and every call site was branching between the two. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
write_records took its CSV/TSV field names from the first row alone, so any run where one record carried a label and another did not raised a ValueError inside DictWriter on the first record with an extra key. This was already reachable via `ids --labels` whenever NodeNorm knew some CURIEs but not others. Field names are now the union of keys across all rows, with restval="" filling the gaps. The omit-an-absent-label rule also moves off the literal field name "label" and onto any field ending in it, so LabeledCrossReference's subj_label/obj_label follow the same convention that ids already did — they were being emitted as "" in JSON and TSV. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three N+1 patterns dominated runtime on anything larger than a toy query: - Every labelled CURIE cost its own get_normalized_nodes round-trip, so `xrefs --labels --recurse` over a 500-CURIE clique issued ~500 sequential HTTPS requests. NodeNorm.normalize_curies() now prefetches a whole batch (100 CURIEs per request) and the per-CURIE accessors serve from cache. - Multi-CURIE `xrefs` ran one full scan of the multi-gigabyte Concord.parquet per CURIE. One scan now matches every CURIE, with results bucketed back into the per-CURIE cache; a CURIE with no cross-references caches an empty list so it is not rescanned. - _print_paths rebuilt the undirected neighbour map for each of the C(n,2) pairs, and build_depth_map built the same structure a third time. All three share one build_adjacency(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
parse_duration spent 39 lines and four separately-worded error messages on what one regex rejects in a single branch: empty, negative, and non-integer values now share one message, and the bare-seconds path stops duplicating the unit-suffix path. BabelDownloader and NodeNorm each hand-rolled the same lazy-once cache as a value field plus a _resolved flag, the flag existing only because the resolved value may legitimately be None. functools.cached_property caches None too, so both collapse to a single property. Also extracts _write_meta(), which the tier-2 ETag refresh had been inlining alongside _save_meta, and drops two parameters no caller ever passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scoped to node_modules/ rather than /web so that frontend source added under web/ is still tracked — .gitignore already anticipates web/src/lib/. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces babel-explorer, a CLI tool to query Babel intermediate files (Parquet) via DuckDB and NodeNorm.
BabelDownloaderhandles caching and freshness,BabelXRefshandles querying,NodeNormhandles label enrichment, andcli.pywires them together with Click. Three commands:xrefs,idsandtest-concord.Endpoint configuration
Babel and NodeNorm endpoints are read from
.envrather than hardcoded, so the repository ships only public URLs.BABEL_URL,BABEL_LOCAL_DIR,BABEL_CHECK_DOWNLOAD,NODENORM_URLandBABEL_ALLOW_VERSION_MISMATCHeach have a matching command-line option, with precedence running flag > environment variable >.env> built-in default.The committed
.env.examplecarries the public Babel URL only, with a note telling Translator team members to contact the Babel developers for the Translator-specific URL.Babel version handling
The release behind
BABEL_URLis resolved fromVERSION.txt, falling back to the final URL path segment for older trees that predate it, solatest/resolves to whichever release it currently points at.BABEL_LOCAL_DIRholds one Babel release at a time. When the release changes, the.metasidecars under<local_dir>/duckdb/are deleted so the existing ETag path re-checks each cached file and re-downloads only what changed — the Parquet files themselves are never deleted outright. This keepsConcord.parquetandIdentifiers.parquetfrom being read together across two different Babel releases, which is the failure ETag alone does not prevent.xrefsfails when NodeNorm'sstatusendpoint reports a differentbabel_versionthan the Babel being queried, since labels and cliques would not match the cross-references.--allow-version-mismatchoverrides it. The check runs only where NodeNorm is actually consulted.WIP:
test-concordtakes no--babel-url, so it cannot report which Babel release its NodeNorm is behind. Giving it one would make the before/after clique comparison self-describing, but needs a decision on what it should compare against — deferred rather than folded in here.Blocked on Babel/NodeNorm deployments
The shipped default (
BABEL_URL=https://stars.renci.org/var/babel/latest/) does not work end to end yet, because public Babel releases do not publish the DuckDB Parquet files. babel-explorer reports this explicitly rather than failing mid-download, but these still need doing:duckdb/Concord.parquet,duckdb/Identifiers.parquet), then confirm the shippedBABEL_URLdefault works end to end.statusendpoint currently reports2025sep1, soxrefs --labelsfails the version check against any current Babel unless--allow-version-mismatchis passed.BABEL_URLrepository secret so CI integration tests run against a Babel that publishes the Parquet files. Without it the 24 Parquet-dependent integration tests skip.Linting
The repository had no
[tool.ruff]section, so ruff ran with its default rule set and never checked import ordering. Rules are nowE,F,I(import sorting) andUP(pyupgrade), withE501left to the formatter and*.mdexcluded (ruff 0.16+ reformats Python inside Markdown code blocks). Line length stays at ruff's default of 88 rather than Babel's 120, which would have reflowed 12 of 15 files for no correctness gain.CI passes
--output-format githubso lint failures annotate the diff inline, and keeps usinguv run ruffrather thanastral-sh/ruff-action— uv already resolves the ruff pinned inuv.lock, so CI and local runs share a version without extra plumbing.CLAUDE.mdandREADME.mdnow say explicitly to runruff checkandruff formatbefore committing.Smaller fixes folded in
idsgains--labels, so identifier records can carry NodeNorm labels instead of only raw Parquet columns.--pathswith--format json/tsv/csvis now rejected. It previously ignored the flag and emitted the full recursive cross-reference list, which looks like a successful--pathsrun but is not one.addopts = "-n auto"made every run parallel, and the old teardown was guarded on a "master" worker that never exists under xdist, sodata/test/survived every run..idea/is gitignored.Testing
192 unit tests pass. Integration tests run against whatever
BABEL_URLpoints at and skip when that release does not publish the Parquet files (currently 13 pass, 24 skip against the public default).Verified against the live servers: the public default produces the missing-Parquet error; the internal
latest/resolves to2026jul22viaVERSION.txt; a2025nov19→2026jul22flip removes the.metasidecars while leavingConcord.parquetin place; the NodeNorm skew check fires before any download; and a fullxrefsquery returns cross-references end to end.