Skip to content

Give HPRC_R2's gaps and T2T chromosome tables a data type (#765) - #783

Merged
NoopDog merged 3 commits into
mainfrom
noopdog/765-hprc-chrom-assignment-annotations
Nov 10, 2026
Merged

NoopDog merged 3 commits into
mainfrom
noopdog/765-hprc-chrom-assignment-annotations

Conversation

@NoopDog

@NoopDog NoopDog commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • src/meta_disco/rules/value_map.yaml: the rows data_type.gaps and data_type.sequences were seeded and declared nothing. Both now declare data_type: annotations, apply to AnVIL_HPRC_R2 only, and give a reason citing HPRC's evidence.
  • src/meta_disco/rules/activities.yaml: the genome annotation step's reason now names the gaps and telomere-to-telomere sequence files among the kinds it covers. Wording only.
  • tests/test_value_map.py: the two rows are no longer in the list of rows left unfilled, and the HPRC fixture's review queue drops from 7 values to 5.
  • docs/: regenerated from run 20261109_191616, classified and reconciled at 42c09af.

Why

AnVIL_HPRC_R2 has two submitter tables, gaps and t2t_sequences, that hold 924 files with no data type: 462 *.gaps.bed and 462 *.t2t_chromosomes.tsv. They were the only assembly annotation tables without one. Each already has a genome annotation step from its assembly (#682). That step expects an annotations file, but a file with no data type is never checked against it.

The evidence for annotations:

  • HPRC stores both kinds of file under each assembly's annotation/chrom_assignment/ folder.
  • The HPRC Data Explorer's annotations catalog (catalog/build/intermediate/annotations.csv) lists 462 files each under the annotation types "ChromAlias Gaps" and "ChromAlias T2T".
  • A gaps BED holds intervals on the assembly's own sequences. For example, HG03453_pat_hprc_r2_v1.0.1.gaps.bed has 19 intervals on contigs such as HG03453#1#CM088011.1.
  • A .t2t_chromosomes.tsv is a table of the assembly's telomere-to-telomere sequences. Its columns are sequence_id, chr_name, level, sequence_length, num_gaps and total_gap_length; I read one file from AnVIL's public mirror.

Assumptions I made

  • Both tables get annotations, not a new term. Dave decided this. A T2T list could be read as a quality report on the assembly, but it describes the assembly's own sequences the way censat and segdups do. annotations is also the kind the genome annotation step expects, so the step needs no change.
  • Both rows apply to AnVIL_HPRC_R2 only. "gaps" and "sequences" are the words the slot map takes from the table names, and they are generic. In another dataset, a sequences table could hold the sequences themselves. The IGVF rows are limited to their dataset for the same reason. Neither value appears in any other dataset's evidence today.
  • The slot map's span for t2t_sequences stays sequences. Renaming it to t2t_sequences would make the raw value say what the table holds, but it would mean re-importing the evidence. Limiting the row to HPRC_R2 was the smaller change.
  • No test pins the rows' HPRC_R2 scope. Dave declined one, following the rule against restating the bundled table's rows in tests.

How to verify

Definition of done, from #765:

  • The 924 files have a data_type, and HPRC_R2's data_type gap falls by 924 (steps 1 and 2).
  • The reconcile report's lineage misfits show no output kind misfit for GenomeAnnotationActivity (step 3).
  1. Run make reconcile RUN_DIR=<a run>, or use run 20261109_191616. Among AnVIL_HPRC_R2's records, the 462 *.gaps.bed files (in bed_classifications) and the 462 *.t2t_chromosomes.tsv files (in remaining_classifications) are all classified as annotations.
  2. Open docs/reconcile-report.md and find its comparison with run 20261104_183742. Its data_type row shows +924 filled and -924 not classified. The per-dataset table lists only AnVIL_HPRC_R2's data type: annotations goes from 2,772 to 3,696, and not_classified from 3,709 to 2,785. No other dataset's count of any value moved.
  3. In output/anvil/20261109_191616/reconciled/reconcile_report.json, lineage.misfits has no entry whose activity is GenomeAnnotationActivity.
  4. Open docs/review-queue-report.md: the 462 'gaps' and 462 'sequences' rows are gone. In docs/rules-report.md, data_type.gaps and data_type.sequences show their reasons and fire 462 times each.
  5. The other reports change only their run id and commit. docs/anvil-coverage-report.md counts inference, which this change doesn't touch.
  6. make test-all passes (3,732 root tests and the schema's 98), as do make lint-all (which includes the typecheck) and make format-check. make classify runs validate-metadata first, and it passed.

Closes #765

🤖 Generated with Claude Code

https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
🐦 Built with cc-claude-tools

NoopDog and others added 2 commits November 9, 2026 19:16
AnVIL_HPRC_R2's gaps BEDs and t2t_chromosomes.tsv files (462 each) had no
data_type: their value-map rows were seeded and declared nothing. Both now
declare `annotations`, scoped to AnVIL_HPRC_R2 because `gaps` and `sequences`
are generic words. HPRC files both under each assembly's
annotation/chrom_assignment/ folder, and the HPRC Data Explorer's catalog
lists them as "ChromAlias Gaps" and "ChromAlias T2T".

The GenomeAnnotationActivity reason now names them among the kinds it covers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
Co-authored-by: cc-claude-tools <cc-claude-tools@clevercanary.com>
Classified and reconciled at 42c09af. Against run 20261104_183742, only
AnVIL_HPRC_R2's data_type moved: annotations 2,772 -> 3,696 and
not_classified 3,709 -> 2,785 (924 files). The review queue loses the
gaps and sequences rows; the rules report shows each firing 462 times.
The other reports change only their run id and commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
Co-authored-by: cc-claude-tools <cc-claude-tools@clevercanary.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 03:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the dataset-scoped annotation mapping changes.

No open issues 🎉

What changed in this PR

This PR fills the missing data types for 924 HPRC_R2 gap and telomere-to-telomere chromosome files through the existing reconciliation translation table, aligning them with their genome annotation lineage steps (#765).

Changes:

  • Maps gaps and sequences to annotations, scoped to AnVIL_HPRC_R2 and supported by documented evidence.
  • Expands the genome annotation rationale without changing activity behavior and updates review-queue test expectations.
  • Refreshes reports to show 924 additional classified files and two fewer queued values; inference coverage remains unchanged.
File Description
tests/​test_value_map.py Removes the two authored mappings from seeded-row expectations.
src/​meta_disco/​rules/​value_map.yaml Defines dataset-scoped annotation mappings and their rationale.
src/​meta_disco/​rules/​activities.yaml Updates rationale wording to include gaps and T2T tables.
docs/​unprocessable-report.md Refreshes the referenced run.
docs/​rules-report.md Documents both mappings and 462 applications each.
docs/​review-queue.html Removes resolved values and displays their authored mappings.
docs/​review-queue-report.md Updates queue totals and mapping documentation.
docs/​reconcile-report.md Records the 924-file improvement and updated provenance.
docs/​coverage-dashboard.html Refreshes the run timestamp; inference counts stay unchanged.
docs/​consistency-report.md Refreshes the run reference, retaining zero reported violations.
docs/​anvil-coverage-report.md Refreshes the classification timestamp.

Review effort: Balanced

The genome annotation step's reason said every file it covers is written in
the sequence's own coordinates; a .t2t_chromosomes.tsv holds one row per
sequence instead. Found by the no-context review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VF7tbFa5ZTmYWZ6JWKiBBS
Co-authored-by: cc-claude-tools <cc-claude-tools@clevercanary.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 03:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the HPRC_R2 annotation mapping changes.

No open issues 🎉

Review effort: Balanced

@NoopDog
NoopDog merged commit 295c09f into main Nov 10, 2026
2 checks passed
@NoopDog
NoopDog deleted the noopdog/765-hprc-chrom-assignment-annotations branch November 10, 2026 04:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

HPRC_R2: give the gaps and T2T chromosome tables a data type

2 participants