Carry the terms a 51Did was created under, so they travel with it

[jwrosewell]

Draft. The specification this implements has merged as 51Degrees/specifications#27, and this stays in draft until the cloud has been updated to write both fields. pipeline-dotnet#409 goes first, because the cloud is .NET and needs that package published before it can write anything.

What this is

Two fields of the 51Did payload, read by the fodid crate.

The terms is one byte after the match key and before the creator context, saying which terms document the identifier was created under. A 51Did created for marketing may only be used by a receiver that has accepted the terms it was created under, so the identifier has to say which terms those are. In OpenRTB there is somewhere to put the answer, being the Terms Document Locator on the eids entry, and everywhere else there is not. An identifier passed as a query string parameter arrives on its own, and any hop can drop an answer carried beside it without the identifier looking any different, so the terms travel inside the identifier.

The payload version is bits 4 and 5 of the flags byte, saying which layout the payload follows.

The terms

The byte is an index into a table the specification publishes and is not a version number. An index rather than a version number, so a later document can live at any address rather than only at one the specification could compose from a number. An index rather than the address itself, because an address is long and because a receiver has to know the exact document that was in force when the identifier was made, which an index that maps to one immutable document can answer years later and an address whose contents can be edited cannot.

FodId::terms answers with the address of the document, as Option<&'static str>. The crate turns the index into the address, so a caller never handles the byte.

Index	Document	FodId::terms
0	Not stated in the identifier	None
1	Model Terms for Marketing, version 2	Some("https://m4ow.uk/mtm/2.txt")
any other	One this crate cannot name	None

That one member is the whole of the addition to the public surface. The crate answers with the address and never fetches it, and the address is never an empty string and never built from the index. The Terms enumeration keeps the names the specification gives for each index and is pub(crate), so it is no longer re-exported from the crate root.

The payload version

This crate reads version 0 and refuses every other version with Error::UnsupportedPayloadVersion { version }, which carries the version it found so the message names it.

No field is read under the layout this crate knows once the version says otherwise. A later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent, which is worse than refusing. A version that nothing checks protects nothing.

The version is not exposed. Either this crate read the layout, in which case the accessors are the answer, or it did not, in which case there is no identifier to read fields from.

The parts that are easy to get wrong

No address is ever built from an index this crate cannot name. That would name a document nobody wrote, and a receiver would record having accepted terms that do not exist. An index of zero and an index added after this release therefore give a caller the same answer, which is deliberate, since both say the identifier does not give the terms and the answer has to come from somewhere else.

The terms byte is read at the offset the match key ends at. Where it sits moves with the match key length the identifier type requires, being offset 37 for a probabilistic or hashed email identifier and 21 for a random one. Reading it at a fixed offset would take a match key byte for a random identifier, so there is a test for both lengths.

A payload with no byte after the match key reads as index 0. The reader takes the byte with payload.get(...).unwrap_or(0), so absence and zero need no telling apart, no presence flag exists and no format version is needed.

IdType::Reserved answers with no address. That type has no assigned match key length, and this crate reads every byte after the header as its value best effort, so there is no byte a reserved identifier could have its terms taken from. That is right rather than a fault, a comment at the read says so, a test pins it, and it stops being a special case as soon as a reserved type is assigned a length.

No address does not mean the identifier is unrestricted. It says only that the identifier does not carry the answer, so the answer has to come from the surrounding protocol. Where an identifier may go is a separate question FodId::usage answers.

The terms sits before the creator context section. An issuer writing a context section writes the terms byte before it, and this crate holds no upper bound on a payload, so a longer one is accepted and left in place.

The version is read on its own bits. A reader masking the wrong bits would refuse a version 0 identifier of some usages, or let a later version through, so there is a test over every combination of the usage and type bits.

Files changed

• fodid/src/fodid.rs, the Terms enum (now pub(crate)), the terms_index field on FodId, the version read and the terms read in from_owid, and the terms accessor.
• fodid/src/error.rs, Error::UnsupportedPayloadVersion { version } with its Display line.
• fodid/src/lib.rs, the crate documentation, the payload layout table, the worked example and the re-exports.
• fodid/README.md, the same documentation, whose code blocks are compiled and run as documentation tests.
• fodid/tests/fodid_tests.rs, the fixtures and the tests below.

No offsets or lengths were added to the public surface, because the specification says a package must not expose them, and the version constant is pub(crate). The offset of the terms byte is computed where it is needed. The constants already public on main are left exactly as they are, since taking them off the surface is #38 and not this change.

The test fixtures write what an issuer writes

canonical_payload() is the creating side, so it now carries the payload version 0 in the flags byte and the terms byte of the document a personalized marketing identifier is created under. payload_ending_at_match_key() is the fixture for a payload that carries no terms byte, which a reader takes as index 0 and which no issuer would write.

Tests

All in fodid/tests/fodid_tests.rs, each building its payload byte by byte the way the tests around them do, and signing a real envelope through the existing Fixture.

1. a_payload_ending_at_the_match_key_has_no_terms_address
2. an_explicit_zero_reads_the_same_as_a_missing_terms_byte
3. index_one_is_the_model_terms_for_marketing_and_carries_its_address
4. an_index_this_crate_does_not_know_has_no_address, covering 2, 127, 200 and 255
5. no_terms_stated_and_an_unknown_index_both_have_no_address
6. every_terms_index_decodes_as_the_specification_publishes_it, covering 0, 1, 2, 200 and 255
7. the_terms_byte_is_read_after_the_match_key_for_both_match_key_lengths
8. a_creator_context_after_the_terms_leaves_both_the_match_key_and_terms_read, with 1, 40 and 300 bytes of context after the byte
9. a_reserved_identifier_states_no_terms
10. the_terms_survive_a_base64_round_trip
11. payload_version_zero_reads_every_field
12. an_unassigned_payload_version_is_refused_and_names_the_version
13. a_refused_payload_version_names_itself_in_the_message
14. the_payload_version_is_read_apart_from_the_usage_and_type_bits

The address is written out in the test file rather than taken from the crate, so the test checks the crate against the specification rather than against itself, and it is pinned character for character as https://m4ow.uk/mtm/2.txt.

The terms table

The address, the index and the name are one table, so adding a terms document is one row and not a search for every place a number was written down. That is the point of the byte being an index rather than a version number, and the first version of this change did not honour it, spreading the same fact over three, being the variant, an arm of from_index and an arm of url places that could disagree.

TERMS_TABLE now holds one row per document carrying the index, the name and the address together, and both lookups read it. The bare 0 the absent byte fell back to is now NOT_STATED_INDEX, named where it is explained.

Four unit tests check the table itself: every row round trips through both lookups and carries an https address, no row claims the not stated index, no index appears twice, and every one of the 256 indexes the table does not carry is Unknown with no address.

What was run, and what it reported

On Windows, from the repository root, after git submodule update --init and pwsh ./ci/copy-owid-source.ps1.

Command	Result
cargo fmt -p fodid -- --check	clean, no diff
cargo test -p fodid	28 passed (unit); 0 passed, 1 ignored (the live cloud test, which needs a resource key); 52 passed (integration); 19 passed (documentation tests). 0 failed.
cargo clippy -p fodid --all-targets -- -D warnings	finished, no warnings
cargo build -p fiftyone-fodid-cloud --all-targets	finished, no errors

Note for review

• The specification is not merged, so the documentation links in fodid/src/fodid.rs and fodid/README.md point at did-specification/identifier-layout.md on main, which will show the terms and version sections once 51Degrees/specifications#27 has merged, so this waits only on the cloud.
• #38 touches the same part of fodid/src/fodid.rs, adding Usage and taking the raw flags byte and the offset constants off the public surface. Whichever merges second will need a small rebase, and nothing here works against what that one is doing.
• The five other 51Did packages carry the same two fields. All six were run over thirteen identifiers built by hand outside every package, at each terms index that matters and at each version, and gave the same terms and match key on every one. The fixtures are not in this pull request.