RO-Crate static-map arm: retarget the interface-table rows that no longer resolve, and test every row against the schema (#2915)

[realmarcin]

Summary

d4d rocrate map builds the registered deterministic_ours arm (rocrate_static_map) by applying data/ro-crate_mapping/d4d_rocrate_interface_mapping.tsv. 54 of its 136 rows placed nowhere in a Dataset record at schema 3.0.0. 52 named a slot or class the schema does not have; the other two name FormatDialect, a class that exists but that no Dataset slot ranges over. The arm dropped every value those rows read, including exactMatch/none rows. Each report row gave the schema's reason (for example "'vulnerable_populations' is not a slot on Dataset"), but no test failed on a table row the schema contradicts.

This PR does four things:

• Retargets the 11 rows that have an obvious successor. For each one, the new D4D path was checked against the merged schema and the crate path against the CHORUS, CM4AI and VOICE crates.
• Declares why the other 43 rows place nowhere. Two new columns at the end of each row hold this: Unplaced (10 out_of_scope, 33 owner_question) and Unplaced_Reason. map_crate adds the declaration to the row's detail, and the Outcome legend counts the declared rows.
• Adds TestTableAgainstSchema. It checks every row's D4D path with the same placement logic map_crate uses. It fails on a row that places nowhere and has no declaration, and on a row that places but still has one. A corpus-lane test also requires every committed report to match the current table's declarations: the same rows, each ending in the declared kind and reason, and the legend's per-kind counts.
• Regenerates data/ro-crate_packages/{CHORUS,CM4AI,VOICE}/processed/*_crate_mapped_d4d.yaml and *_crate_mapping_provenance.md. All three still validate (PASS, schema 3.0.0).

Refs #2915. The issue stays open: 33 rows need an owner's ruling (listed below). #2915's Part 1 criterion "no row declaring exactMatch + none is unplaceable" is not yet met. 24 such rows remain: 4 are out of scope and 20 are owner questions.

Retargets (11)

Line	Was	Now	Crate side, checked
81	Dataset.vulnerable_populations	Dataset.at_risk_populations (renamed in #129)	rai:atRiskPopulations, which no crate carries, corrected to d4d:atRiskPopulations. That is the slot's own slot_uri, it is what CM4AI carries, and the FAIRSCAPE converter already reads it. This is the prefix correction notes/D4D_GENERATION_ARMS.md deferred until the row was retargeted.
28	Dataset.external_resource	Dataset.external_resources	relatedLink: no crate carries it (the row is now empty)
77	Dataset.machine_annotation_analyses	Dataset.machine_annotation_tools	rai:machineAnnotationTools: VOICE
78	MachineAnnotation.tool_name	MachineAnnotationTools.tools	same property; subsumed by line 77 in VOICE
96	Maintenance.frequency	UpdatePlan.frequency	rai:dataReleaseMaintenancePlan; subsumed by Dataset.updates in all three
129, 130	Subset.is_data_split, Subset.is_sub_population	DataSubset.is_data_split, DataSubset.is_subpopulation	N/A (unresolvable)
131, 132	Variable.name, Variable.type	VariableMetadata.variable_name, VariableMetadata.data_type	N/A (unresolvable)
133, 134	SamplingStrategy.strategy_type, SamplingStrategy.details	SamplingStrategy.strategies, SamplingStrategy.description	d4d:samplingStrategy, which map_crate treats as not a crate path (unresolvable)

On each retargeted row, the d4d: slot token in Exchange_Layer_URI is renamed to match (line 81 also gets its crate token), and line 77's D4D_Type class name is updated. The SKOS relation and information loss are unchanged on every row. A note in Transformation_Notes records each retarget.

Before / after

Project	filled	subsumed	empty	unresolvable	unplaceable	distinct slots
CHORUS	32 → 32	0 → 1	47 → 51	4 → 10	54 → 43	32 → 32
CM4AI	42 → 43	0 → 1	37 → 40	4 → 10	54 → 43	42 → 43
VOICE	44 → 45	4 → 6	31 → 33	4 → 10	54 → 43	44 → 45

All 43 remaining unplaceable rows per project carry the table's declaration ("the mapping table says why for 43 of them: 10 out of scope, 33 awaiting an owner's decision"). AI_READI is still refused, because its crate is not UTF-8 (#3357).

Before any change, the run reproduced every committed output exactly, apart from the verdict date.

Values the arm now writes:

• CHORUS: none. Its mapped record is byte-identical to main's; only its report changes.
• CM4AI: at_risk_populations ({name: "None — no human subjects involved; commercially sourced de-identified cell lines only."}).
• VOICE: machine_annotation_tools (seven tool objects: OpenSMILE, Praat/parselmouth, TorchAudio pipelines, sparc, phonetic posteriorgram models, Whisper Large, b2aiprep).

Declared out of scope (10)

These rows map nowhere by design.

• File-level slots (8). bytes (the contentSize row), encoding, path, md5, sha256, hash, dialect and media_type are slots on File, which sits two levels below Dataset. A row places only one level deep.
  • contentSize is a rounded human string ('1.2 tb'), not a byte count, and is not parsed into one. total_size_bytes reads the exact evi:totalContentSizeBytes, but only CM4AI's crate carries it, so CHORUS's and VOICE's records carry no size, as on main. The row's reason says so.
  • contentUrl already fills download_url.
• FormatDialect.delimiter and FormatDialect.header. These describe one file's CSV dialect, and their source is prose, not a crate path.

Owner questions (33)

Each row's Unplaced_Reason gives the specifics.

• Pending generate_interface_mapping.py (pinned) writes rai:prohibitedUses / rai:ethicalReview / rai:imputationProtocol, and the committed interface table disagrees with it #4043 (mapping columns unchanged): line 87 EthicalReview.irb_id.
• Line 89 DatasetCollection.data_governance_committee (mapping columns unchanged). The successor is Dataset.data_governance (DataGovernance.committee_name, No governance or stewardship class exists; the one DAC contact slot is nested under ExportControlRegulatoryRestrictions #503). But committee_name is "the body responsible for access decisions", and in each crate the arm maps (CHORUS, CM4AI, VOICE) dataGovernanceCommittee names a person (CHORUS's adds an email address). AI_READI's names a consortium, but the arm refuses that crate for its encoding (Decide whether the AI_READI crate is decoded as its declared windows-1252 or stays refused #3357). Whether to place it, in committee_name or as a contact, is the owner's decision (Interface table line 89: dataGovernanceCommittee names a person in every crate; where, if anywhere, should the static-map arm place it? #4386).
• Line 90 DatasetCollection.principal_investigator (mapping columns unchanged). Its one slot is Creator.principal_investigator (range Person), under Dataset.creators, which the author row fills. Retargeted there, the row is refused in all three crates, because merging two crate properties into one object is not decided. Making a Creator or Person of the free-text name (CHORUS's carries an email address) is the decision RO-Crate mapper fidelity: 54 of 136 interface-table rows cannot be placed, and nested *.description rows overwrite filled host slots #2915's proposal asks for, and RO-Crate mapper fidelity: 54 of 136 interface-table rows cannot be placed, and nested *.description rows overwrite filled host slots #2915 tracks it.
• Successor exists, but placing it is a judgement:
  • line 26 DatasetCollection.parent_datasets → Dataset.parent_datasets. CHORUS's and CM4AI's isPartOf name an organization and a project, which is FAIRSCAPE isPartOf names an organization and a project; the reverse converter now places them in parent_datasets #4047's open question.
  • line 27 related_datasets. DatasetRelationship requires a relationship_type that relatedLink does not state.
  • line 97 Maintenance.versioning_strategy. Both candidate homes, UpdatePlan.update_details and VersionAccess.version_details, would be a judgement.
• Duplicates. Each obvious successor is already another row's target from the same crate property: line 105 EvidenceMetadata.total_content_size_bytes (Dataset.total_size_bytes), line 106 EvidenceMetadata.formats (Dataset.distribution_formats) and line 110 DatasetCollection.funding_and_acknowledgements (Dataset.funders). For each: remove it, or keep it as a documented duplicate.
• No slot at schema 3.0.0, and the schema never had one (git log -S over src/data_sheets_schema/schema/): lines 61, 62, 65, 66, 73 and 74, the step_type, pipeline_step, annotator_type and evidence_type rows.
• No slot at schema 3.0.0 (adding one is a schema decision):
  • HumanSubjectResearch.exemption (88)
  • contact_email (93), data_sharing_agreement (94)
  • seven EvidenceMetadata counts (98–104)
  • completeness (107), summary_statistics (108), quality_control (109), provenance_and_lineage (111)
  • ValidationMetrics.validation_method (112)
  • QualityControl.accuracy (113), QualityControl.data_quality_report (114), QualityControl.fda_compliant (115)

Boundaries kept

• Line 67 (Dataset.imputation_protocols, rai:imputationProtocol) and line 87 (EthicalReview.irb_id, rai:ethicalReview) wait for the owner's ruling on generate_interface_mapping.py (pinned) writes rai:prohibitedUses / rai:ethicalReview / rai:imputationProtocol, and the committed interface table disagrees with it #4043. Their mapping columns are byte-identical. Line 67 places, so it gains only two empty fields. Line 87 gains its owner_question declaration.
• The pinned generator .claude/agents/scripts/generate_interface_mapping.py is untouched. Regenerating from it would undo this PR and the earlier hand corrections, as generate_interface_mapping.py (pinned) writes rai:prohibitedUses / rai:ethicalReview / rai:imputationProtocol, and the committed interface table disagrees with it #4043 notes.
• No data/d4d_concatenated/** file changes, and the published arm labels stay as they are. Deterministic crate mappers write resolver-URL DOIs: both deterministic arms fail the anchored doi pattern (#646) while crate-mapping provenance still reports PASS #2916 mints new deterministic labels.
• The FAIRSCAPE converter does not read the table at run time. Its output is identical between origin/main and this branch on seven crates: the four bundled examples and the CHORUS, VOICE and CM4AI release crates.
  • Its cross-arm tests do read the table. at_risk_populations is now a slot both arms fill, and the arms agree on CM4AI's value. It joins updates and human_subject_research as a single-valued class-range slot where the arms differ on a list mixing values with lists (FAIRSCAPE converter: a single-valued class-range slot places nothing for text beside a nested list, which rocrate_map keeps #4194's class). _coerce's comment and test_documented_exceptions_across_all_shared_pairs_and_value_kinds now name it.
  • The arms also agree on VOICE's machine_annotation_tools.
• dataGovernanceCommittee is placed by neither arm. This one declares line 89 an owner question; the converter neither reads the property nor records it as dropped (Reverse FAIRSCAPE converter maps four RAI properties to slots other than those the schema declares, and leaves FAIRSCAPE properties with declared homes unread #4046).

Follow-ups filed

• Interface table: Dataset.human_subject_research reads d4d:humanSubject, which no crate carries (crates carry humanSubjectResearch / humanSubjects / humanSubjectExemption) #4383: Dataset.human_subject_research reads d4d:humanSubject, which no crate carries; the crates carry humanSubjectResearch, humanSubjects and humanSubjectExemption.
• Interface table: remove or document the three duplicate rows declared owner_question in #2915 #4384: remove or document the three duplicate rows (lines 105, 106, 110).
• Regenerate set_2303 fig09 crate-vs-generation data from the #2915 static-map outputs #4385: regenerate set_2303 fig09 from these outputs (RO-Crate mapper fidelity: 54 of 136 interface-table rows cannot be placed, and nested *.description rows overwrite filled host slots #2915's last acceptance item).
• Interface table line 89: dataGovernanceCommittee names a person in every crate; where, if anywhere, should the static-map arm place it? #4386: line 89, dataGovernanceCommittee, names a person in each crate the arm maps. Where, if anywhere, should the arm place it?
• Interface table line 90 (principalInvestigator): where, if anywhere, should the static-map arm place it? #4449 (from review round 1): Interface table line 90 (principalInvestigator): where, if anywhere, should the static-map arm place it?

Reviewing

• The table diff. Use git diff --ignore-space-at-eol on the TSV: 55 lines change (the header, 11 retargets, 43 declarations). Every other line gains only two empty trailing fields (the file keeps CRLF and its quoting; unchanged rows round-trip byte for byte through csv).
• Regenerating CM4AI. It needs data/ro-crate_packages/CM4AI/crate/ re-extracted from the tracked zip (see that directory's README).
• Six commits. The first retargeted line 89 to DataGovernance.committee_name. The second reverts that before the PR opened: in each crate the arm maps the value names a person, and CHORUS's carries an email address, so the arm was writing a person, and for CHORUS an address, into a slot that names a committee. Line 89 keeps main's mapping columns and is declared an owner question. The last four are review round 1 (below). Review them together; the counts above are the final ones.
• Commit messages that the round-1 corrections supersede. The message of 1a3f13817 says the report gave a bare "unplaceable" (it gave the schema's reason) and that every one of the 54 rows named a missing slot or class (52 did). The message of 424ba2478 says every crate's dataGovernanceCommittee names a person (true of the three crates the arm maps, not of AI_READI's). History is not rewritten; the table, the reports, the notes, the docstrings and this description carry the corrected statements.

Tests

• Round 0, at 424ba2478. tests/test_rocrate, tests/test_fairscape_integration, tests/test_cli/test_rocrate_cli.py, tests/test_cli/test_render_gate.py, tests/test_method_split.py, tests/test_deterministic_arm_validity.py and tests/test_rocrate_file_collection.py, corpus tests included: 647 passed, 6 skipped. The skips are environmental and existed before.
• Round 1, at 0782e61ca. The final head, 70c66ec2b, changes only one sentence of notes/D4D_GENERATION_ARMS.md, which no test reads. The round-0 files, plus four more that read the table, the reports or the package directory (tests/test_review_pack.py, tests/test_generation_specificity_skill.py, tests/test_release_inventory.py and tests/test_download/test_audit_bundles.py), corpus tests included: 942 passed, 6 skipped. The six skips are environmental: linkml-map and the FAIRSCAPE integration are not installed on the development machine.
• The full -m "not corpus" lane was not run on the development machine, because it was under heavy load (load average about 80 in round 0 and above 100 in round 1). CI runs it.
• Mutation checks, round 0, run on a git archive copy: 13 of 13 caught.
  • Table mutations: the renamed slot restored; a declaration's kind cleared; a declaration on a row that places; line 81's crate side reverted; line 89 retargeted to DataGovernance.committee_name again (the first commit's version; caught by test_the_retargeted_rows_place_the_crate_values_they_read, which finds data_governance in the record).
  • Mapper mutations: the declaration not appended; the kind not recorded; the legend clause dropped.
  • Committed-report mutations: a declared reason removed; a declared row missing.
  • Guard-helper mutations: each of its three checks disabled.
  • The first twelve were run at the first commit; the line-89 mutation at 424ba2478.
• Mutation checks, round 1, run on a git archive copy of 0782e61ca: 8 of 8 caught.
  • Committed-report mutations (Committed-report guard checks only the reason text, so stale reports pass after a change to a row's declared kind #4413), each with the reports left as committed: a declared kind flipped with its reason kept (line 116); a committed legend's per-kind counts left stale (CHORUS); a table reason shortened (line 13). Each is caught by the corpus guard.
  • The guard's old substring filter for undecided merges restored. It drops line 90's row from the comparison, so the corpus guard fails, and so does TestTheGuardReadsWhatTheMapperWrites.
  • Mapper mutations: the undecided-merge note reworded (caught by TestTheGuardReadsWhatTheMapperWrites); the legend's declared count replaced by the unplaceable total (Legend test cannot tell 'declared' from 'unplaceable': counting all unplaceable rows as declared survives every test #4419; caught by both legend tests and by TestTheGuardReadsWhatTheMapperWrites); a row that places given its declared kind, or its declared reason in its detail (Guard docstrings say a placed row keeping a declaration "would be reported unplaced"; the mapper ignores it #4417; each caught by test_the_guard_fails_a_declaration_on_a_row_that_places).
  • An A/B on the pre-round data at 424ba2478: unmutated, the old and the new guard both pass. Under each of the three committed-report mutations, the old guard passes and the new one fails.

Review round 1

Eight issues were filed from the round-1 review of 424ba2478. All eight are fixed here.

• Notes give the first commit's filled-row counts (33, 44, 46); the final reports say 32, 43, 45 #4412 (minor) — fixed. notes/D4D_GENERATION_ARMS.md gave the first commit's filled-row counts after the retargets (33, 44, 46). It now gives CM4AI 43 and VOICE 45, and says CHORUS's record is unchanged.
• Committed-report guard checks only the reason text, so stale reports pass after a change to a row's declared kind #4413 (minor) — fixed. The committed-report guard compared only the set of rows and looked for the reason anywhere in a row, so a kind flipped with its reason kept, a stale legend, or a shortened reason all passed. It now requires each declared row's cell to end with the declared kind and the whole reason, and the legend to end with the table's per-kind counts. It recognises an undecided-merge row by the ending map_crate writes, not by a phrase anywhere in the row: line 90's new reason uses the same words, and the old filter would have dropped that row from the comparison. TestTheGuardReadsWhatTheMapperWrites pins all three readings to what the mapper writes, and runs in the pull-request lane.
• principal_investigator is declared out_of_scope, but its own reason describes a pending owner decision #4414 (minor) — fixed. Line 90 (principal_investigator) was declared out_of_scope, but its own reason described a decision still to be made. It is now owner_question, and its reason says why. The legend moves from 11 / 32 to 10 / 33, and the exactMatch + none split from 5 / 19 to 4 / 20. The three reports are regenerated; the three mapped records are byte-identical.
• Dataset.bytes reason says total_size_bytes carries the size, but CHORUS and VOICE carry no evi:totalContentSizeBytes #4415 (minor) — fixed. Line 13's reason (and this description) said the record's size is total_size_bytes, read from the exact evi:totalContentSizeBytes. Only CM4AI's crate carries that property. The reason now says CHORUS's and VOICE's records carry no size; the row stays out of scope.
• The governance reason says every crate names a person, but the tracked AI_READI crate names a consortium #4416 (nit) — fixed. "Every crate's dataGovernanceCommittee names a person" was false for the tracked AI_READI crate, which names a consortium. The table reason, the reports, the notes, the test docstring and this description now scope the claim to the crates the arm maps. The 424ba2478 commit message keeps the old wording.
• Guard docstrings say a placed row keeping a declaration "would be reported unplaced"; the mapper ignores it #4417 (nit) — fixed. The guard's docstrings said a row that places but keeps its declaration "would be reported unplaced". map_crate ignores such a declaration. The docstrings now say so, and test_the_guard_fails_a_declaration_on_a_row_that_places asserts it: the row is filled, carries no declaration, and its value is in the record.
• "54 rows named a slot or class the schema does not have" is false for the two FormatDialect rows, and main's reports did name each mismatch #4418 (nit) — fixed. "54 rows named a slot or class the schema does not have" was false for the two FormatDialect rows, and the reports did give a reason for each row. The test docstring, the notes and this description now say 52 plus two, and that what was missing was a test. The 1a3f13817 commit message keeps the old wording.
• Legend test cannot tell 'declared' from 'unplaceable': counting all unplaceable rows as declared survives every test #4419 (nit) — fixed. The legend test ran on a graph where every unplaceable row was declared, so writing the unplaceable total in place of the declared count passed every test. The real-table legend test now maps a crate with one undecided-merge row (44 unplaceable, 43 declared), and the small-fixture legend test adds a declared row beside an undecided one.

🤖 Generated with Claude Code