Skip to content

Let class derivations declare an identity key for required identifier slots #342

Closed

Description

@amc-corey-cox

When a target class declares an identifier slot, linkml-map warns if the spec derives no value for it (Required target slot 'id' has no derivation) and emits the record anyway. The output then fails validation against the target schema. In our BDCHM transforms that is roughly 6,400 of 6,641 class derivation blocks.

The workaround is to hand-write a uuid5() expression per block. That works — it is what our Visit, Participant, and Person derivations already do — but it puts identical derivation logic in thousands of places, and the thing being hashed is usually a key the spec has already spelled out in its reference slots. A MeasurementObservation is identified in practice by its participant, its visit, and its observation type; all three are already derived a few lines above.

Proposal: let a class derivation declare which of its derived slots constitute identity, and have linkml-map hash them into the identifier slot.

MeasurementObservation:
  populated_from: pht012853
  identity_from: [associated_participant, associated_visit, observation_type]

uuid5 already exists as an injected eval function, so the mechanism is in place; what's missing is somewhere to declare the key. The values are hashed after derivation, so the result is stable across runs, independent of row order, and unaffected by chunking or parallel execution.

The reason this should be declared rather than inferred is that identity is not recoverable from the data. Whether a changed cell means "same entity, corrected" or "a different entity" is a modeling judgement, and no rule over the input can settle it. Hashing the whole record would be deterministic but would change the identifier whenever any value changed, which is the wrong semantics for a stable key. Declaring the key keeps that decision with the spec author while keeping the derivation in one place.

Two things worth settling in the design:

  • Collisions are possible and meaningful. Two measurements of the same type for the same participant at the same visit would hash alike; sometimes that is correct, sometimes it needs a discriminator. Being explicit about the key at least makes the question visible per class.
  • Nested derivations have the same problem. Our value_quantity blocks derive a Quantity, which also inherits a required identifier, so whatever shape this takes should be usable on nested class derivations too.

Happy to implement this if the approach seems right.

Activity

  1. amc-corey-cox commented on Oct 9, 2026

    @amc-corey-cox
    ContributorAuthor

    I tested the identity_from proposal against the full BDC trans-spec corpus (6,225 class derivation blocks with no id derivation), and it doesn't hold up. Keying on participant, visit, and observation type collides in 2,486 blocks (40%). Those blocks are not mistakes. ARIC HDL at exam 1 is four separate measurements (longitudinal file, lipid assay, two recalibrations) that share all three slots and differ only in method_type and source. A declared key will always be one forgotten slot away from merging distinct records.

    Revised proposal: when the target class has a required identifier and the spec derives none, synthesize it from the derived record itself.

    id = uuid5(namespace, class + canonical serialization of the derived record, minus the identifier)
    

    Same harmonized record, same id; different record, different id. The id is computed after derivation from what the block already produced, so it needs no new spec syntax, no knowledge of source tables or row shape, and gives the same result regardless of row order, chunking, or parallel execution.

    The thing to get right is the canonicalization, which becomes a contract:

    • sorted keys, nulls dropped
    • one number format (5 vs 5.0, float noise from unit conversions)
    • a defined rule for multivalued slot order
    • nested inlined objects included

    Consequences worth stating up front:

    • These ids are content hashes, not stable identifiers. Any change to a derived value or to the spec's output changes the id. That's fine for leaf records nobody references; entities that are referenced (Participant, Visit) should keep deriving a natural-key id explicitly, and an explicit id derivation always wins.
    • Identical records get identical ids. Duplicates become detectable and safe to drop, since a shared id guarantees identical content. Removing them is the caller's job, not linkml-map's.
    • Future hook for stability: if stable ids are ever needed, an optional per-class exclude list (slots to leave out of the hash) is the way to add it. Exclude rather than include, because a missed exclusion only costs stability, while a missed inclusion merges records.

    I'll implement this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions