fix: normalize supporting study provenance

CHANGELOG.md

@@ -4,6 +4,15 @@ All notable changes to this project are documented in this file.
 
 ## Unreleased
 
+### Changed
+- **The inlined supporting study now carries the current Biolink Study metadata, with disjoint ids and names and no `#` composition.** Supporting studies were verbose and self-duplicating: a published Excel section keyed its `Study` by `"PMC11708054#all correlations"`, repeated the sheet as the `name`, and fabricated a per-edge `StudyResult` (`"…#row42"` / `"all correlations row 42"`) even when it carried no evidence; an unpublished section keyed the study by its own config filename (`my_table.yaml`). The contract is now: `Study.id` is a real identifier — the section's publication CURIE, or the config stem (YAML filename minus its extension) as the rescue fallback — and `Study.name` a *disjoint* human label (worksheet for spreadsheet sources, source filename otherwise; omitted when it would equal the id). Study-level metadata declared as annotations (`study_size`, `study_cohort`, `study_context`, `study_date_range`, `study_method_description`, `study_method_types` — the `Study` node properties that [biolink-model#1770](https://github.com/biolink/biolink-model/pull/1770) replaced the deprecated `supporting_study_*` association slots with) lands as real typed fields **on the Study**, never on the edge and never inside a description string. The `StudyResult` is identified only by the scoped CURIE `row:<N>` — its source row — and carries no name; its `description` is reserved for values with no structured home (routed unsatisfiable columns and class-pruned qualifiers). No `#` delimiter appears anywhere in the output. A published section always keeps the `{id, name}` wrapper; an unpublished one emits the struct only when it has metadata, routed unsatisfiable values, or pruned values to carry (the contentless-skip behavior is unchanged).
+
+  **Migration:** readers keying `has_supporting_studies` by the composed `publication#sheet` string should key by the publication CURIE; consumers scraping `"row N"` result names or `supporting_study_size=…` description entries should read `StudyResult.id` (`row:<N>`) and the typed `Study.study_*` fields instead. Declaring the deprecated `supporting_study_size` / `sample_size` / `supporting_study_*` annotation names still builds — the coercion phase renames them onto the canonical `study_*` properties — but emits a `BiolinkRelocationWarning` naming the destination. If multiple study-size synonyms coexist, the canonical/winning value is retained and the losing synonyms are dropped as duplicate representations of the same Study property.
+
+- **biolink-model 4.4.4: the statistical edge slots the model now declares are emitted as real model-typed values.** The dependency moved from 4.4.3 to 4.4.4 and every model-derived set follows it with no code change: `effect_size` (`float`) and `effect_type` (`EffectTypeEnum`) are real `Association` slots ([#1774](https://github.com/biolink/biolink-model/pull/1774)) and leave the pending-exemption set, so `effect_size` ships as a real JSON number instead of a `{:.4g}` string; `statistical_significance_qualifier` is attached to every association class and rides the edge as a bare `StatisticalSignificanceQualifierEnum` token (`"strongly_significant"`, no `biolink:` prefix) instead of being relocated into the study description; `EffectTypes` is now derived from the installed model's enum rather than a local tuple. `relationship_strength` is deliberately no longer listed among unsatisfiable study-routed names — it is a legacy alias the coercion renames to the edge slot `effect_size`.
+
+- **`openpyxl` is an explicit test dependency.** It was only ever transitive via `linkml` 1.10; biolink-model 4.4.4's `linkml` 1.11 bump dropped it, and `tests/test_cover_lib.py` imports it directly.
+
 ### Added
 - **The final graph QC now asserts no field in the emitted NDJSON is null or empty, and that every node carries a name.** `build-kg --qc`'s stage-7 study pass (in the spirit of `studyKGtsvs.pl`) gained two assertions. `empty-or-null-values`: any field in the nodes or edges file whose value is JSON `null`, a string that strips to empty, or an empty container — checked recursively, so a null or blank nested inside an `attributes` list counts — fails the build. The check is deliberately stricter than the NDJSON writer's `strip_nulls` (`rust/src/json.rs`), which scrubs dict entries at every depth but passes array scalars (`["x", ""]`) and emptied nested objects (`[{}]`) through verbatim; the study stage now asserts the stronger contract — no null or empty value anywhere — so the first such shape to reach an emitted file fails the build loudly instead of shipping silently. Null-like *strings* (`NA`/`NaN`/`null`/`none`) are also dropped by the writer but are neither null nor empty, and are deliberately not flagged; the `original_*` whitespace exemption does not extend to emptiness. `unnamed-nodes`: a node record whose `name` key is missing, `null`, or strips to empty fails the build — the missing-key case is what pipeline output surfaces, since `strip_nulls` deletes empty and null-like names before the file is written. Both report offenders per field or per node id, capped at 10 examples like the other assertions.
 - **The final graph QC now asserts every node has an `id` and every edge has `subject`, `predicate`, and `object`.** Two more stage-7 study assertions join the `unnamed-nodes` check (which already requires a non-empty node `name`). `unidentified-nodes`: a node record whose `id` key is missing, `null`, or strips to empty fails the build; since the id is exactly what is absent, examples key on the node's `name` (or `<no name>` when it has none). `incomplete-edges`: an edge record missing any of the three core slots — missing key, `null`, or strips-to-empty — fails, counted per slot (e.g. `predicate (2)`). The writer's `strip_nulls` deletes a null slot outright rather than emitting it, so on pipeline output a hit means the slot was null upstream and the record shipped broken — exactly the condition these assertions exist to catch loudly. Non-string, non-null ids and slots pass, mirroring the name convention (no writer emits them).

README.md

@@ -41,7 +41,7 @@ template:
   provenance: { repo: PMID, publication: "12345678" }
   annotations:
     - { annotation: p_value, method: column, encoding: C }
-    - { annotation: supporting_study_size, method: column, encoding: D }
+    - { annotation: study_size, method: column, encoding: D }
 ```
 
 Wrap it in a graph config (`graph.yaml`) pointing at your fullmap entity-resolution database
@@ -85,7 +85,7 @@ Output is one JSON object per line: nodes with Biolink categories, edges with an
 ```
 
 ```json
-{"subject":"HGNC:11998","predicate":"biolink:associated_with","object":"MONDO:0008903","p_value":"1.0000e-03","supporting_study_size":"450"}
+{"subject":"HGNC:11998","predicate":"biolink:associated_with","object":"MONDO:0008903","p_value":"1.0000e-03","has_supporting_studies":{"PMID:12345678":{"id":"PMID:12345678","name":"gene-disease.csv","study_size":450,"has_study_results":[{"id":"row:2"}]}}}
 ```
 
 See the [Tutorial](https://skyeav.github.io/Tablassert/tutorial/) for the full walkthrough.

docs/agent.md

@@ -215,24 +215,29 @@ cannot drift from the model the build validates against:
 - any predicate is safe for: Gene~Gene, Gene~Pathway, ChemicalEntity~Disease, …
 ```
 
-!!! note "`effect_size` / `effect_type` / `approval_ids` are exempt"
-    Tablassert emits `effect_size` / `effect_type` deliberately, pending
-    [biolink-model#1774](https://github.com/biolink/biolink-model/pull/1774), and emits
-    `approval_ids` as a translator-ingest pass-through. The installed model declares none of
-    these on `Association`, so a strict check rejects edges carrying them. `biolink_valid_pct`
-    exempts them (and the other curated KGX carryovers) so the agent is scored on **its own**
-    decisions.
-    The exempt set is *derived* (`TABLASERT_EDGE_EXTRAS - <fields any association declares>`), so it
-    empties itself when the model catches up, with no code change.
+!!! note "`approval_ids` is exempt"
+    Tablassert emits `approval_ids` as a translator-ingest pass-through; the installed model
+    declares no such slot on `Association`, so a strict check rejects edges carrying it.
+    `biolink_valid_pct` exempts it (and the other curated KGX carryovers) so the agent is
+    scored on **its own** decisions.
+    The exempt set is *derived* from `TABLASERT_EDGE_EXTRAS` minus the fields any association
+    declares, so it empties itself when the model catches up. That is exactly what happened to
+    `effect_size` and `effect_type`: they were exempt pending
+    [biolink-model#1774](https://github.com/biolink/biolink-model/pull/1774), and dropped out
+    of the set when biolink-model 4.4.4 shipped them as real `Association` slots.
 
 Two related silent behaviours the agent's prompt now names, since neither raises:
 
-- An annotation like `supporting_study_size` or `sample_size` is declared in the LinkML schema but
-  attached to **no** Pydantic class, so its value is routed onto the inlined `StudyResult` rather than
-  emitted on the edge. Names that are not association slots at all (`q_value`, `fold_change`, …) are
-  folded into `supporting_text`. Authoring either now emits a `BiolinkRelocationWarning` naming where
-  the value actually went, a warning, not an error: nothing is lost, and every existing config
-  keeps building.
+- An annotation like `supporting_study_size` or `sample_size` names study-level metadata.
+  biolink-model 4.4.4 ([PR #1770](https://github.com/biolink/biolink-model/pull/1770))
+  deprecated the old `supporting_study_*` association slots and replaced them with `Study`
+  node properties, so the value is carried on the edge's inlined supporting `Study` (as
+  `study_size`, `study_cohort`, and related fields) rather than emitted on the edge.
+  `relationship_strength` is not one of these. It is a legacy alias coerced to the real edge
+  slot `effect_size`. Names that are not association slots at all (`fold_change` alone,
+  `z_score`, and similar names) are folded into `supporting_text`. Authoring any relocated
+  name emits a `BiolinkRelocationWarning` naming where the value actually went, a warning,
+  not an error: nothing is lost, and every existing config keeps building.
 - Enum-ranged qualifiers take a literal token (`object_direction_qualifier: increased`), never a
   CURIE, and are deliberately **not** entity-resolved. `map_coverage` skips them for the same reason
   the build does, so they no longer depress a config's coverage score for working correctly.

docs/cli.md

@@ -144,7 +144,7 @@ The positional `GRAPH-CONFIGURATION-FILE` (also `--configuration-file`, `-f`) is
 | Option | Type | Required | Default | Description |
 | --- | --- | --- | --- | --- |
 | `GRAPH-CONFIGURATION-FILE` (`--configuration-file`, `-f`) | Path | Yes | n/a | Graph YAML |
-| `--release`, `-r` | Flag | No | `False` | Emit a slim, significant-only graph (drops `biolink:not_significant` edges before resolution) |
+| `--release`, `-r` | Flag | No | `False` | Emit a slim, significant-only graph (drops `not_significant` edges before resolution) |
 | `--qc`, `-q` | Flag | No | `False` | Audit resolved mappings (exact → fuzzy → abbreviation → SapBERT) so low-confidence edges are flagged; requires the `[qc]` extra, checked before the build starts. Also runs a final study stage that asserts over the emitted NDJSON: no duplicate node ids, every node has a non-empty `id` and `name`, every edge has a non-empty `subject`, `predicate`, and `object`, no undeclared or isolated nodes, no malformed lines, no null or empty values in any field (checked recursively), and no stray whitespace (verbatim `original_*` fields excepted from the whitespace check, since they are faithful source copies) and fails the build (non-zero exit) on any violation |
 | `--log`, `-l` | Flag | No | `False` | Enable verbose per-section logging |
 | `--head`, `-hd` | Flag | No | `False` | Fast output-shape preview: ≤5 random rows/section, cached to `.head.parquet`, never clobbers a full build |

docs/configuration/advanced-example.md

@@ -53,7 +53,7 @@ template:
   # Statistical metadata as edge annotations (method: value = constant,
   # method: column = per-row)
   annotations:
-    - {annotation: supporting_study_size, method: value, encoding: 9}
+    - {annotation: study_size, method: value, encoding: 9}      # Study metadata -> inlined supporting Study
     - {annotation: p_value, method: column, encoding: C}
     - {annotation: multiple_testing_correction_method, method: value, encoding: Benjamini Hochberg}
     - {annotation: effect_size, method: column, encoding: B}   # Spearman rho value
@@ -99,26 +99,34 @@ whole table or `method: column` to pull per-row notes from the source (see
 {"id":"CHEBI:41774","name":"13C-tamoxifen","category":["biolink:ChemicalEntity"]}
 ```
 
-**Edges:** Allow-listed annotation columns (`supporting_study_size`, `p_value`, `effect_size`,
-`effect_type`) stay as
-top-level edge fields (numeric annotations as controlled-notation strings). Any non-Biolink-slot name
-(here `assertion_method`, `multiple_testing_correction_method`, `miscellaneous_notes`) folds into the
-edge's `supporting_text` list as `"name: value"` entries (sorted alphabetically), alongside the built-in
-`extracted_from_row_number`:
+**Edges:** Allow-listed annotation columns (`p_value`, `effect_size`, `effect_type`) stay as
+top-level edge fields (`p_value` in controlled scientific-notation string form, `effect_size` as a real
+JSON number since biolink-model 4.4.4). Study metadata (`study_size`, here declared as a constant)
+lands on the edge's inlined supporting `Study`; the `Study.id` is the publication CURIE, `Study.name`
+the worksheet, and the single `StudyResult` anchors the source row as its `row:<N>` id. Any
+non-Biolink-slot name such as `assertion_method`, `multiple_testing_correction_method`, and
+`miscellaneous_notes` folds into the edge's `supporting_text` list as `"name: value"` entries
+(sorted alphabetically):
 
 ```json
 {
   "id": "2cfea591-0f8f-33af-a7df-03da531d3359",
   "subject": "NCBITaxon:47715",
   "predicate": "biolink:correlated_with",
   "object": "CHEBI:41774",
-  "supporting_study_size": "9",
+  "has_supporting_studies": {
+    "PMCID:PMC11708054": {
+      "id": "PMCID:PMC11708054",
+      "name": "all correlations",
+      "study_size": 9,
+      "has_study_results": [{"id": "row:3"}]
+    }
+  },
   "p_value": "1.0000e-03",
-  "effect_size": "0.85",
+  "effect_size": 0.85,
   "effect_type": "spearmans_rho",
   "supporting_text": [
     "assertion_method: Spearman correlation",
-    "extracted_from_row_number: 3",
     "miscellaneous_notes: Correlation analysis between microbial composition and 13C-tamoxifen abundance after FDR correction",
     "multiple_testing_correction_method: Benjamini Hochberg"
   ]