Rank normalization quality by source and CURIE prefix, over a local KGX Storage mirror - #1
Draft
gaurav wants to merge 27 commits into
Draft
Rank normalization quality by source and CURIE prefix, over a local KGX Storage mirror#1gaurav wants to merge 27 commits into
gaurav wants to merge 27 commits into
Conversation
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Python >=3.12, no dependencies yet. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Mirrors only normalization-metadata.json, normalization_failures.txt and normalization_map.json (~970 MB across 267 files) into data/kgx-storage.ci.transltr.io/, preserving the remote layout and pruning files that disappear upstream. The multi-gigabyte normalized_nodes.jsonl / normalized_edges.jsonl files are never fetched: the --include filters apply to the listing walk, so only directory HTML is requested. rclone's http backend is used because the S3 bucket behind KGX Storage is not anonymously listable -- the site's HTML listings are the only public index. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Records what had to be discovered by probing the live server: no anonymous S3 listing, HTML-only index, presigned-URL redirects, the 502 on large JSON files, the path layout, what each normalization file contains, and how big they all are. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
31 files of ~600 B, and it makes "which build is current for this source" authoritative instead of inferred from directory mtimes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
load_rows() returns one plain dict per (build, prefix) from the local mirror; summarize() pools by (source, case-insensitive prefix). No Dash imports, so a notebook or a static-JSON exporter can reuse it. Prefixes are pooled case-insensitively because NodeNorm resolves CURIE prefixes case-insensitively -- ENSEMBL:, Ensembl: and ensembl: all return NCBIGene:675. Individual rows keep the case observed in the files and the summary reports the spellings it merged, so a fold is never silent. success_rate is recomputed from the counts rather than read from the file: the pipeline truncates it (99.556 -> 99.55) in 128 of 587 rows, which would not agree with rates derived from pooled counts. tests/test_loader.py checks these invariants against the real mirror. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
One sortable, filterable table of every data source x CURIE prefix, sorted worst normalization rate first, defaulting to each source's latest build (241 rows; 242 with all 90 builds shown). Failed counts are a column so the ranking can be flipped to "how many CURIEs are at stake", which is the better prioritization signal at 0%. Running locally keeps individual CURIEs off the public web and leaves the deployment question open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…_rate Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gaurav
changed the title
normalized_nodes.jsonl carries the label, full Biolink category list and clique
members for every normalized node; the pre-normalization transform output
({source}_nodes.jsonl) carries what the upstream source knew about CURIEs we
could not normalize -- but only for some sources. pathbank, ncbi_gene and goa
have names and descriptions there; ubergraph has only id + biolink:NamedThing.
Recorded rather than downloaded: these run to hundreds of MB per source.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
5.7 GB over all 90 builds, so mirroring these would mean restricting to the latest build per source rather than adding them to the sync filters. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It is gitignored, survives reboots and sits next to the code, so a one-off job can be picked up or re-run later instead of being redone from scratch. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Expands a CURIE through the Biolink prefix map (vendored, 266 prefixes, so the app works offline), falling back to bioregistry.io for prefixes the map lacks. The map covers 75.5% of the 898,042 distinct unnormalized CURIEs; the fallback covers the rest, including PathBank, which is absent from the Biolink map and is by itself 215,953 of them. Malformed CURIEs get the reason instead of a link, because the malformation is often the whole explanation for the normalization failure. Real species in the data: rhea:RHEA:13065 (double prefix), CL:0000089 ∩ UBERON:0000473 (post-composed class expression), UniProtKB:B3DHD6 Q6XCC7 (two accessions), UniProtKB: (empty). The curies package is not used: the Biolink file is a flat prefix -> URI-stem dict, so expansion is a dict lookup and a string concat. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
build_url points at the normalization output directory in KGX Storage that a row's numbers came from; the summary renders one markdown link per pooled build. failures_paths lets the UI find the CURIEs behind a row without re-deriving paths from the mirror layout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Clicking a row lists the CURIEs that failed to normalize for that source and prefix, each linked out so it can be checked, with malformed ones flagged. Failures files are read on demand -- the biggest is under 7 MB, and preloading all 87 would cost 3.2M lines for nothing. Table changes: CURIE Prefix moves to the leftmost column and is styled as the row header it effectively is; versions link to their KGX Storage directory; prefixes that fully normalize are hidden by default, with the hidden count in the checkbox label; 75 rows per page, enough to show all 67 rows below 80% at once. The drill-down reads derived_viewport_data rather than data, since active_cell indexes the sorted and filtered page. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The mirror path was relative to the working directory, so starting the app from anywhere but the repository root -- an IDE run configuration, uv run --project -- loaded zero rows in silence. Fall back to the repo-relative path, and raise a FileNotFoundError naming the sync script when no mirror is found at all, instead of rendering an empty table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
DataTable cells may only hold a string, number or boolean, so the list of failures paths carried on each summary row made the browser reject the whole table with "Invalid argument `data[0].failures_paths`". The check is client-side, so no server-side test or HTTP request saw it. The drill-down now looks its paths up from the loaded rows by (source, prefix) instead, which also stops sending local file paths to the browser at all. test_loader.py asserts every summary value is a scalar, so this class of error fails a self-check rather than only the UI. Also honours PORT, so a second instance can run alongside one already on 8050. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Up to five real unnormalized CURIEs per row, one per column, monospaced and linked. They keep the case the source file spells them with, which is much of the value: PathBank's failures are PathBank:Reaction_13124 and PathBank:Compound_102409 -- not pathway identifiers at all -- and goa's are RNAcentral:URS0002337E65_9606, taxon-suffixed. Collected in one pass per failures file rather than per row: 87 files, 3.2M lines, 0.8s at startup. Latest builds are scanned first so examples come from current data. Also bracket the counts in "Normalized to" (CHEBI (5), not CHEBI 5), which read as a possibly-malformed CURIE, and let the 14-column table scroll rather than crushing every cell. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The first five CURIEs in a failures file are often all alike, since a prefix's failures are grouped in file order. Spreading the sample across the whole list shows the variety instead: pathbank now reads Reaction_13124, Compound_46510, Reaction_8770 rather than three consecutive Reaction ids. Still one pass per file and 0.8s, holding only one file's CURIEs at a time, and a source's examples now all come from its latest build rather than whichever build filled the bucket first. Observed as leaves the table: it equals the CURIE Prefix in every row today, because both case collisions are cross-source, and the Example columns show the original casing anyway. summarize() still carries the field. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gaurav
changed the title
The page container capped itself at 1400px, and a DataTable can only fill the width its parent gives it -- so roughly four columns sat off-screen behind a scrollbar. This looked like a Dash limitation and nearly cost us a rewrite into hand-built HTML/JS; it was two lines of our own CSS. Also drops the "Normalized to" column, which at 200+ characters for semmeddb was the one value wide enough to force a scrollbar by itself. Where a prefix normalizes to is a renormalization-diff question rather than a today question; summarize() still carries normalized_to_str for when that work starts. Prose keeps a 70ch cap -- wide tables are good, wide paragraphs are not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Clicking a cell used to highlight just that cell, which says nothing -- which column you happened to hit is not information. The whole row now highlights, matched on (prefix, source) identity rather than row index, so it follows the row when the table is re-sorted or filtered, and the active cell gets no colour of its own. The unnormalized CURIE listing is now a <ul> grouped by CURIE shape -- everything up to the first digit -- with each group's members spread across the whole group rather than taken from its head. pathbank's 215,953 failures read as 175,039 Reaction_, 31,182 Compound_, 8,886 ProteinComplex_, 416 Bound_; bgee's ENSEMBL failures split into one group per species. Grouping declines above 12 groups and falls back to a flat spread list, because InChIKeys have no digits to split on and would otherwise produce 87 groups of one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The listing grouped by CURIE shape, which described the identifiers without diagnosing them -- rhea:RHEA: was reported as just another shape rather than as a double prefix. It now groups by problem, most actionable first: malformed CURIEs, then prefixes the Biolink model has never heard of, then the ordinary "no Babel clique" remainder. Shape grouping survives one level down, where it was doing real work: pathbank's 215,953 unknown-prefix CURIEs still break into Reaction_, Compound_, ProteinComplex_, Bound_, ElementCollection_, NucleicAcid_. The prefix check is labelled as a signal rather than a cause -- Babel decides coverage for itself, not from the Biolink prefix map -- but a prefix the model has never heard of is rarely a coincidence when all of its CURIEs fail. When no shape is shared, as for the 87 digit-free InChIKeys, the fallback list is capped at 30 spread across the set rather than printing every one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The intro paragraph kept a 70ch cap while the table went full width, which just looked broken. Nothing caps its width now. The summary line reported only row and source counts. It now carries the totals worth knowing at a glance: Showing 144 of 241 rows — 31 sources, 84 CURIE prefixes, 2,793,224 of 3,652,613 CURIEs normalized (76.5%), 859,389 failed Every figure but the row count describes the whole build selection rather than the visible rows, so hiding the fully-normalizing prefixes cannot make the overall score look worse than it is. CURIEs are counted once per source, which the intro now says: the all-builds total of 3,225,939 failures matches the line count across the mirrored failures files exactly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The intro was one paragraph of prose and the totals sat in a greyed-out line above the table, which is the least readable place to put the numbers people actually want. Both become one <ul>: three bold figure bullets -- CURIEs normalized and the percentage, CURIEs failed, and the source/prefix/pair counts -- followed by the fixed notes. The line above the table now reports only what it can honestly claim, the number of rows currently displayed, and is no longer greyed out. The figures still describe the whole build selection, so hiding the fully-normalizing prefixes cannot move the headline score. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A full runbook from a fresh checkout to a running dashboard: prerequisites, the sync and its variants, what lands where, how to keep it current, and troubleshooting for the failure modes we actually hit (the ubergraph 502, a port already in use, a missing mirror, an empty table). README keeps a quick start and links to it. Also states plainly that there is no static export yet and what hosting would take, rather than leaving the deployment question implied. Drops the README's reference to the "Observed as" column, which no longer exists. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It still described Dash as a possible future port and claimed the project had no dependencies. Now records the current shape, which goals are met, and the gotchas that cost real time to find: - DataTable cells hold only scalars, and the check is client-side, so no server-side test or curl of the callback endpoint can catch a violation - active_cell indexes the viewport, not the data prop - row identity beats row index for conditional styling - the mirror resolves relative to the repo, not the working directory - table width was never a Dash limit, just a maxWidth on our own container Plus how the click-through listing groups CURIEs by problem and by shape, and why the prefix check is labelled a signal rather than a cause. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gets the normalization data onto disk, then makes it answerable: which prefixes, in which sources,
are failing to normalize — and what the failing identifiers actually look like.
Mirroring KGX Storage
scripts/sync-kgx-normalization.shis onerclone synccall. It mirrors onlynormalization-metadata.json,normalization_failures.txt,normalization_map.jsonandlatest-build.jsonintodata/kgx-storage.ci.transltr.io/, preserving the remote layout andpruning files that disappear upstream. The multi-gigabyte
normalized_nodes.jsonl/normalized_edges.jsonlfiles are never fetched — the--includefilters apply to the listingwalk, so only directory HTML is requested. A full sync is ~970 MB across 267 files.
rclone's http backend is used because the S3 bucket is not anonymously listable; the site's HTML
listings are the only public index.
The dashboard
A local Dash app (
uv run normalization-dashboard, orPORT=8051 …for a second instance): everydata source × CURIE prefix, sorted worst normalization rate first, defaulting to each source's
latest build. Prefixes that fully normalize are hidden by default (n=97 of 241).
Failedis a sortable column, because at 0% the percentage says nothing about how many CURIEsare at stake — FOODON at 0% of 28,920 and a prefix at 0% of 3 rank identically by rate.
the source spells them, each linked out. This is usually the fastest explanation of a 0% row.
Running locally keeps individual CURIEs off the public web and leaves deployment (Pages export, k8s,
folding into another Translator dashboard) open.
loader.pyhas no Dash imports, so a notebook or astatic-JSON exporter can reuse it.
What it already shows
PathBank:Reaction_13124andPathBank:Compound_46510— internal reaction and compound identifiers, not pathway identifiers."Babel lacks PathBank" is the wrong diagnosis.
RNAcentral:URS0002337E65_9606.rhea:RHEA:13065— double-prefixed, fullyexplained by malformation, no Babel work needed.
Things found in the data
ENSEMBL:,Ensembl:,ensembl:allreturn
NCBIGene:675), so summaries pool on the uppercased prefix while records keep the observedcase. Merges 86 observed prefixes into 84.
success_raterather than rounding it (99.556 → 99.55, in 128 of 587rows), so the loader recomputes the rate from the counts.
unnormalized CURIEs; those fall back to
bioregistry.iofor linking.Scale, for the "do we need a database" question
normalization-metadata.jsonnormalization_failures.txtnormalization_map.jsonNo database needed for the prefix summary, or for a drill-down that reads one build's failures file
on demand. Only the renormalization-diff work will want DuckDB or similar.
Known upstream issue
Ubergraph's 247 MB
normalization_map.jsonreturns 502 — the origin appears to buffer JSONresponses in memory, and a 48 MB map from the same source downloads fine. A full sync lands 266/267
files and exits non-zero. Worth reporting to DINGO.
Verification
uv run python tests/test_loader.pyandtests/test_curie.py— invariants against the realmirror: counts reconcile, rates match the counts,
normalized_tosums tosucceeded, exactly onelatest build per source,
summarize()conserves totals and leaves no case-duplicate prefixes, andevery summary value is a scalar (a list in a row makes the browser reject the whole DataTable, and
no server-side check sees that).
🤖 Generated with Claude Code