You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Let class derivations declare an identity key for required identifier slots #342
When a target class declares an identifier slot, linkml-map warns if the spec derives no value for it (Required target slot 'id' has no derivation) and emits the record anyway. The output then fails validation against the target schema. In our BDCHM transforms that is roughly 6,400 of 6,641 class derivation blocks.
The workaround is to hand-write a uuid5() expression per block. That works — it is what our Visit, Participant, and Person derivations already do — but it puts identical derivation logic in thousands of places, and the thing being hashed is usually a key the spec has already spelled out in its reference slots. A MeasurementObservation is identified in practice by its participant, its visit, and its observation type; all three are already derived a few lines above.
Proposal: let a class derivation declare which of its derived slots constitute identity, and have linkml-map hash them into the identifier slot.
uuid5 already exists as an injected eval function, so the mechanism is in place; what's missing is somewhere to declare the key. The values are hashed after derivation, so the result is stable across runs, independent of row order, and unaffected by chunking or parallel execution.
The reason this should be declared rather than inferred is that identity is not recoverable from the data. Whether a changed cell means "same entity, corrected" or "a different entity" is a modeling judgement, and no rule over the input can settle it. Hashing the whole record would be deterministic but would change the identifier whenever any value changed, which is the wrong semantics for a stable key. Declaring the key keeps that decision with the spec author while keeping the derivation in one place.
Two things worth settling in the design:
Collisions are possible and meaningful. Two measurements of the same type for the same participant at the same visit would hash alike; sometimes that is correct, sometimes it needs a discriminator. Being explicit about the key at least makes the question visible per class.
Nested derivations have the same problem. Our value_quantity blocks derive a Quantity, which also inherits a required identifier, so whatever shape this takes should be usable on nested class derivations too.
Happy to implement this if the approach seems right.
I tested the identity_from proposal against the full BDC trans-spec corpus (6,225 class derivation blocks with no id derivation), and it doesn't hold up. Keying on participant, visit, and observation type collides in 2,486 blocks (40%). Those blocks are not mistakes. ARIC HDL at exam 1 is four separate measurements (longitudinal file, lipid assay, two recalibrations) that share all three slots and differ only in method_type and source. A declared key will always be one forgotten slot away from merging distinct records.
Revised proposal: when the target class has a required identifier and the spec derives none, synthesize it from the derived record itself.
id = uuid5(namespace, class + canonical serialization of the derived record, minus the identifier)
Same harmonized record, same id; different record, different id. The id is computed after derivation from what the block already produced, so it needs no new spec syntax, no knowledge of source tables or row shape, and gives the same result regardless of row order, chunking, or parallel execution.
The thing to get right is the canonicalization, which becomes a contract:
sorted keys, nulls dropped
one number format (5 vs 5.0, float noise from unit conversions)
a defined rule for multivalued slot order
nested inlined objects included
Consequences worth stating up front:
These ids are content hashes, not stable identifiers. Any change to a derived value or to the spec's output changes the id. That's fine for leaf records nobody references; entities that are referenced (Participant, Visit) should keep deriving a natural-key id explicitly, and an explicit id derivation always wins.
Identical records get identical ids. Duplicates become detectable and safe to drop, since a shared id guarantees identical content. Removing them is the caller's job, not linkml-map's.
Future hook for stability: if stable ids are ever needed, an optional per-class exclude list (slots to leave out of the hash) is the way to add it. Exclude rather than include, because a missed exclusion only costs stability, while a missed inclusion merges records.
Activity
amc-corey-cox commentedon Oct 9, 2026
I tested the
identity_fromproposal against the full BDC trans-spec corpus (6,225 class derivation blocks with noidderivation), and it doesn't hold up. Keying on participant, visit, and observation type collides in 2,486 blocks (40%). Those blocks are not mistakes. ARIC HDL at exam 1 is four separate measurements (longitudinal file, lipid assay, two recalibrations) that share all three slots and differ only inmethod_typeand source. A declared key will always be one forgotten slot away from merging distinct records.Revised proposal: when the target class has a required identifier and the spec derives none, synthesize it from the derived record itself.
Same harmonized record, same id; different record, different id. The id is computed after derivation from what the block already produced, so it needs no new spec syntax, no knowledge of source tables or row shape, and gives the same result regardless of row order, chunking, or parallel execution.
The thing to get right is the canonicalization, which becomes a contract:
5vs5.0, float noise from unit conversions)Consequences worth stating up front:
idderivation always wins.I'll implement this.