feat: ranking and filtering for biolink:statistical_significance_qualifier

[SkyeAv]

Adds ranking boosts and an edge-level significance filter for biolink:statistical_significance_qualifier (biolink/biolink-model#1766) across all Shepherd ARAs, modeled on the equivalent ARAX implementation (RTXteam/RTX#2859). Closes BioPack-team#134.

Shared module

• shepherd_utils/statistical_significance_qualifier.py (NEW): single source of truth for SIGNIFICANCE_BAND_SCORES (0.70…0.0), SIGNIFICANCE_ORDINAL (4…0), SIGNIFICANCE_SOURCE_WEIGHT (0.5), and get_statistical_significance(edge) — a qualifiers-only lookup that strips any biolink: prefix. Reused by all three consumers below.

Ranking — aragorn_score (Aragorn + BTE)

• workers/aragorn_score/worker.py: extracts the qualifier in get_edge_values() via the shared helper (the attribute loop only walks edge["attributes"]; the qualifier lives in edge["qualifiers"]). Adds it as a scored property feeding the graph-Laplacian: band_score × 0.5 → admittance -1/log(weight). Max effective weight 0.35 → bounded boost.
• Covers both ARAs: aragorn_score is the shared Aragorn + BTE ranker (both workflows run aragorn.score); one change updates both. There is no separate BTE ranker.
• No penalty: not_significant and qualifier-less edges are unchanged (property_w > 0 guard, also avoids log(0)).

Ranking — arax_rank (ARAX dict port)

• workers/arax_rank/ranker.py: categorical-bypass injection in _calculate_edge_confidence() — the qualifier is looked up separately from the numeric attribute normalizer (the enum string would otherwise hit float() → ValueError). Appends band_score × SIGNIFICANCE_SOURCE_WEIGHT to edge_attribute_score_list. Works even for attribute-less edges (injected at method level, not nested inside if edge.get("attributes")).
• Mirrors RTX post-refactor: trust weight is a standalone constant, not in known_attributes_to_trust (RTX moved it out in the same refactor).

Filtering — filter_edges_by_statistical_significance

• shepherd_utils/shared.py: filter_edges_by_statistical_significance(message, minimum_significance, logger) — ordinal edge removal (edges explicitly below threshold removed; qualifier-less kept), with analysis/result pruning to keep TRAPI valid. Downstream filter_kgraph_orphans removes orphaned nodes.
• workers/filter_edges_by_statistical_significance/ (NEW): super-thin async worker (inline, no process pool — mirrors filter_kgraph_orphans). Reads minimum_significance from the nested parameters dict (the older filter_*_top_n workers read flat keys and silently use defaults; this one honors the parameter).
• compose.yml: service registered, mirroring filter_kgraph_orphans.

Workflow wiring

• workers/aragorn/worker.py + workers/bte/worker.py: filter inserted after aragorn.score, before sort_results_score, in all four lookup branches (two per ARA). Pathfinder branches untouched.
• Default minimum_significance=suggestive: removes only edges explicitly labeled not_significant; qualifier-less edges kept.

Design

• Qualifier-only placement: reads edge["qualifiers"] exclusively — the qualifier is_a: statement_qualifier, so BMT/Retriever route it there. Matches ARAX (qualifier-only as of RTX#2859; its defensive edge.attributes lookup was removed). A one-line fallback in the shared helper covers a KP ever sending it as an attribute.
• No penalty for qualifier-less edges: rollout is incremental across KGs; penalizing non-adopters would be unfair. Ranking is boost-only; filtering removes only explicitly-labeled edges.
• Conservative heuristics: band scores (0.70…0.0) and trust (0.5) match RTX exactly. These are starting points, not empirically validated. Deferred: revisit post-rollout.
• Deferred: significance-aware path scoring in score_paths (currently purely structural); edge.attributes fallback; per-ARA threshold tuning.
• Inert until retriever#197 ships: no edges carry the qualifier yet, so the filter removes 0 edges today and auto-activates when qualifier data starts flowing.

Testing

• pytest tests/unit/ -q → 511 passed (24 new tests: 7 shared-helper, 4 aragorn_score, 5 arax_rank, 8 filter).
• 3 existing workflow-assertion tests updated to include the new filter op.

[coderabbitai]

Warning

Review limit reached

@SkyeAv, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 2 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: cd6d101e-5597-4877-9fea-cb5a67632ca2

📥 Commits

Reviewing files that changed from the base of the PR and between 0450fde and 2a1e242.

📒 Files selected for processing (19)

• .gitignore
• compose.yml
• shepherd_utils/shared.py
• shepherd_utils/statistical_significance_qualifier.py
• tests/unit/aragorn/test_aragorn.py
• tests/unit/aragorn/test_aragorn_examine_query.py
• tests/unit/aragorn/test_aragorn_score_ranker.py
• tests/unit/test_arax_rank_ranker.py
• tests/unit/test_bte.py
• tests/unit/test_filter_edges_by_statistical_significance.py
• tests/unit/test_statistical_significance_qualifier.py
• workers/aragorn/worker.py
• workers/aragorn_score/worker.py
• workers/arax_rank/ranker.py
• workers/bte/worker.py
• workers/filter_edges_by_statistical_significance/Dockerfile
• workers/filter_edges_by_statistical_significance/__init__.py
• workers/filter_edges_by_statistical_significance/requirements.txt
• workers/filter_edges_by_statistical_significance/worker.py

---

_{Comment @coderabbitai help to get the list of available commands.}

[SkyeAv]

Superseded by the upstream cross-repo PR BioPack-team#135 — closing so the change is proposed to BioPack-team/shepherd (the original repo) rather than the fork.