perf: materialize sections once in resolve_batch and trim before resolution - #110
Merged
Conversation
…lution resolve_batch re-executed the entire upstream lazy plan (source scan, encoding/regex ops, NLP normalization) once per resolved node column: distinct().collect() per column, join_matches' collect per column, and log_unmatched's anti-join collect with --log. A section resolving subject + object + two qualifiers ran that pipeline ~5x (~9x logging). ### Implementation - resolve_batch collects the frame exactly once; term extraction, unmatched logging, and per-column join-backs run on the in-memory frame. - join_matches keeps its LazyFrame contract over a new eager core (_join_matches_eager) that resolve_batch calls directly. - trim moves from the finalize ops to just before resolve_batch, so the materialized frame and downstream joins drop the spent raw column_<n> columns early; its phase label falls back to transform. ### Testing - uv run pytest -q: 928 passed (1 failure + 1 collection error are pre-existing: this venv lacks the [qc] extra; both fail on the clean tree too). - ruff check / ruff format --check / pyright: clean. - Synthetic 50k-row, 4-column benchmark: resolve phase 0.14s -> 0.08s (~1.75x), identical output rows.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
SkyeAv
added a commit
that referenced
this pull request
Aug 24, 2026
Cut 13.0.0 and bump the package version in pyproject.toml, uv.lock, and CITATION.cff. Major: three breaking changes since 12.1.0. The inlined supporting study now carries current Biolink Study metadata with disjoint ids and names and no `#` composition; biolink-model 4.4.4 types the statistical edge slots, so `effect_size` ships as a real JSON number and `statistical_significance_qualifier` rides the edge as a bare enum token (both 99099a1); and an unpaired `effect_size`/`effect_type` half is now DROPPED with an `UnpairedEffectAnnotationWarning` instead of failing the section, retiring the `annotation-effect-size-without-type` / `annotation-effect-type-without-size` codes (#105). Also ships the two final-graph QC assertion sets (#106, #107), the resolve_batch single-materialization win (#110), and the agent's task pre-rendering, planning-off, improve-round cap, and build memoization (#113). The legacy TableConfigs importer (#105) is deliberately absent from the changelog: it landed and was removed (58787f4) inside this window, so it never appeared in a released version and is a net no-op for users. main was red at 7140c37; fixed here so the release is cuttable. Both failures are #112 fixtures/expectations written against biolink-model 4.4.3 and merged after the 4.4.4 bump landed: - The vendored DAKP configs listed `AffinityMeasurement` in `avoid:`, a class 4.4.4 renamed to `ProteinLigandAssayResult`. DAKP generates `avoid` as the sorted complement of each side's prioritize tuple, so the old name is a 4.4.3 generation artifact rather than an intentional deviation; rewritten in place and recorded in the fixture README. - test_copysign_transformation_in_pipeline asserted `effect_size == "-0.85"`, the pre-4.4.4 `{:.4g}` string form. It is a real JSON number now. Deliberate Biolink departures are untouched: p-value columns keep their controlled scientific notation despite the model typing them `float`, and `approval_ids` remains a curated pending pass-through. docs/cli.md's validate-kgx section still printed `biolink-model 4.4.3` and claimed `effect_size`/`effect_type` were pending; rewritten around the fields that are actually pending today (`approval_ids` plus the KGX denormalized carryovers), noting the pair graduated when 4.4.4 shipped #1774. Testing: - uv run pytest -q -> 1047 passed, 15 skipped (94% coverage) - uv run ruff check . && uv run ruff format --check . && uv run pyright -> clean / 0 errors Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HZQ8rLfhvtyErcq9S4Ao6b
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
resolve_batchre-executed the entire upstream lazy plan — source scan, encoding/regex ops, NLP normalization — once per resolved node column:distinct(...).collect()per column,join_matches' collect per column, andlog_unmatched's anti-join collect with--log. A section resolving subject + object + two qualifiers ran that pipeline ~5× (~9× with logging).Changes
resolve_batch(fullmap.py) collects the frame exactly once; per-column term extraction, unmatched logging, and join-backs all run against the in-memory frame. Output rows are unchanged.join_matcheskeeps its LazyFrame-in/LazyFrame-out contract over a new eager core (_join_matches_eager) thatresolve_batchcalls directly.trimmoves from the finalize ops (_provenance_ops) to just beforeresolve_batch(_node_ops), so the materialized frame and every downstream join drop the spent rawcolumn_<n>columns early. All readers of those columns run earlier, so this is safe;prune_to_classskips them regardless via itsnot any(accepts)branch.trim'sPHASE_OFentry is removed so progress falls back totransforminstead of flashingfinalizebeforeresolve.No reordering of filters was needed — every row-reducing op already runs before resolution (pinned by
tests/test_lib.py).Testing
uv run pytest -q→ 928 passed; the 1 failure + 1 collection error are pre-existing environment issues (this venv lacks the[qc]extra — both fail on the clean tree too).ruff check,ruff format --check,pyright→ clean.--log): resolve phase 0.14s → 0.08s (~1.75×), identical output rows. Real tables with heavier encoding chains should gain more, since the eliminated cost is per-column re-execution of the whole upstream plan.Changelog entry added under Unreleased → Performance.