Carry the terms in the identifier, so they travel with it - #27
Merged
Merged
Conversation
A 51Did created for marketing is only usable by a receiver that has accepted the terms it was created under, and until now the identifier did not say which terms those were. The answer had to travel beside it, which works in OpenRTB where there is somewhere to put it and fails wherever an identifier is passed on its own, a query string parameter being the ordinary case. An identifier that arrives without its terms cannot be acted on safely. A single byte after the Match Key carries an index into a table in this specification. Zero says the terms are not stated in the identifier and the receiver must consult the data accompanying it. One is the Model Terms for Marketing version 2, at https://m4ow.uk/mtm/2.txt. A package answers with the address for an index it knows and never fetches it. An index rather than a version number is carried so that a later document can live at any address, rather than only at an address this specification could compose from a number. An index rather than the address itself is carried because an address is long, and because a receiver has to know the exact document in force when the identifier was made, which an address whose contents can be edited cannot prove years later. An index is therefore never reused or repointed once published. A package will meet an index added after it was released, so it reports the index, answers with no address, and must not read it as zero. Zero says no terms are stated whilst an unknown index says terms are stated that this package cannot name, and a receiver confusing the two would read an identifier created under terms as one created under none. This is why the surface exposes the raw index alongside the named value, which is the one place the specification asks for a raw value. Nothing about existing identifiers changes. One issued before this has a payload that ends at the Match Key, and a reader treats a missing byte as zero, which is the right answer for it, so absence and zero mean the same thing and neither has to be told apart from the other. The byte sits before the creator context, so the context section moves by one and that section is not in production use. This does not remove the need for a Terms Document Locator where a protocol carries one, and the page says so, because the byte answers for the identifier and the locator answers for the transaction. Zero does not mean unrestricted. It means this identifier does not carry the answer. A non-marketing identifier carries zero, since the Model Terms govern marketing and it was not created under them, and it remains barred from a demand source by its Usage. A marketing identifier with a zero index must not be issued.
jwrosewell
force-pushed
the
feature/terms-in-the-identifier
branch
from
September 9, 2026 13:34
d9d7d11 to
5a41bd8
Compare
jwrosewell
changed the title
Six packages are implementing this at once and the table gave indexes and addresses but no names, so each was about to invent its own and a customer reading two of them would think they were different features. The names are now stated once, with the casing left to each language. The Reserved type has no defined Match Key length, so a reader takes every byte after the header as the Match Key and no byte is left for the Terms. Such an identifier reads as index 0, which is correct under the rule above rather than a gap in a package, and the page now says so.
Implementing this in Python showed the consequence plainly. A reader takes the first byte after the Match Key as the Terms and nothing in the payload says which field that byte belongs to, so a pre-change identifier carrying a creator context would have the first byte of that section read as an index and would report whatever it happened to be. That is accepted rather than defended against, because the creator context was never in production use, and it is the reason the byte can go here at all. The page now says so, and says that anything issuing an identifier with a context section writes the Terms byte before it.
This was referenced Sep 9, 2026
Merged
The page said a non-marketing identifier carries zero and that a marketing one must not, which leaves the reader to work out the positive rule for themselves. It now states it as a table, being zero for non-marketing and one for both standard and personalized marketing, one being the Model Terms for Marketing version 2. It also says what happens when a later document is published, which is that it gains an index and the issuer writes that instead whilst identifiers already issued keep saying what they were created under.
Nothing here has reached a customer, so a reader implementing this has never seen an identifier without a Terms byte and does not need the account of how the byte came to sit where it does. The rules stay and the narrative goes, being what a payload with no byte after the Match Key means, and that an issuer writing a creator context writes the Terms before it. The same sentence in the Type section, explaining that identifiers predating the type bits read as Probabilistic, goes for the same reason. The rule it wrapped, that a reader must not refuse the reserved type, stays.
The envelope has a version and the payload has none, so the terms byte could only be added by leaning on the payload length and on the creator context not being in production. That trick does not work twice, because a second field added after the terms cannot be told apart from a context section, and by then identifiers carrying the terms and no version would be in circulation. Bits 4 and 5 become the payload version, which was the only field in the byte with no name. Zero is the layout on this page, so nothing on the wire changes and no identifier already issued is affected. Three values are left unassigned, and whoever assigns the last of them has to say how the flags are extended beyond it. A reader must refuse a version it does not know rather than read the fields as though the version were zero, because a later version exists precisely because a field moved, so reading it under the old layout gives wrong values rather than absent ones. That also makes reading the field compulsory, since a version nothing checks protects nothing, the first identifier carrying a new layout being misread by every package that ignored it.
The surface asked for three members, being the named value, the raw index and the address. It is now one, Terms, answering with the address of the document the identifier was created under, and the package turns the index into the address so a caller never handles the byte. Nothing already on the page changes, so a caller written against an earlier release keeps working untouched. A caller can no longer tell an index the package does not know from an index of zero, since both answer with no address. That is accepted, because the two lead a caller to the same place, which is that the identifier does not tell them the terms and they must look elsewhere. What a package must still never do is compose an address for an index it does not know, since that names a document it cannot know exists and a receiver would record having accepted terms nobody wrote.
The payload table now states the Terms offset as 37 or 21, following the Match Key length, with a rule that a reader takes the offset from the length it read and never from a constant, since a constant is right for one Type and silently wrong for the other. "where the index is one it does not know" could be read as index 1, which is the one index that carries an address, so it now says "not in the table it knows". The note about a caller written against an earlier release is removed, because nothing has been released. The version rule now says the version belongs to the refusal and must not appear as a member of the identifier, so a refusal that carries it in a message or in a field on the error reads as the compliance it is. The Terms index and the Payload version join the table of things a package does not make public.
Automation51D
pushed a commit
to 51Degrees/pipeline-java
that referenced
this pull request
Sep 11, 2026
…y the terms a 51Did was created under in the identifier' * FEAT: Carry the terms a 51Did was created under in the identifier A 51Did created for marketing may only be used by a receiver that has accepted the terms it was created under, and until now the identifier did not say which terms those are, so the answer had to travel beside it. In OpenRTB there is somewhere to put it, being the Terms Document Locator on the eids entry, and everywhere else there is not. An identifier passed as a query string parameter arrives on its own and any hop can drop a sidecar without the identifier looking any different, so a receiver that gets one without its terms cannot act on it safely. One byte after the match key now answers that, and it is an index into a table in the specification rather than a version number, so a later document can live at any address rather than only at one a number could compose. Terms names the values, being NOT_STATED for index 0, MODEL_TERMS_FOR_MARKETING_2 for index 1 and UNKNOWN for an index this package does not know. FodId.getTerms() answers the named value, getTermsIndex() the raw byte and getTermsUrl() the address, which is null for both NOT_STATED and UNKNOWN and is never an empty string and never built from the index. The package never fetches the address, it returns it. UNKNOWN is deliberately not NOT_STATED. Zero says no terms are stated whilst an unknown index says terms are stated that this package cannot name, and a receiver confusing the two would read an identifier created under terms as one created under none. The raw index is exposed alongside the named value so that a caller meeting an index added after this release can look the document up by hand and can report which index it could not read. Existing identifiers are unaffected. One issued before the byte existed has a payload that ends at the match key, and a missing byte reads as index 0, which says the terms are not stated in the identifier, so absence and zero mean the same thing, no reader has to tell them apart and no presence flag is needed. The byte adds nothing to the minimum payload lengths this package enforces, so no identifier that read before fails to read now, and every existing test passes untouched. Depends on 51Degrees/specifications#27, which must merge first. pipeline.did tests: 137 run, 0 failures, 2 skipped (the live cloud tests), five new. The fodid developer example prints the three members and its tests run 9, 0 failures. * Keep the change inside the 51Did package The example module is outside pipeline.did and printing the new members there is a separate change, so it is taken back out and the diff is the package alone. * DOC: Describe the Terms byte as it is, rather than as a change No reader has ever seen a 51Did without the Terms byte, so describing the field as a change from a previous state gives a reader history they cannot use. Every rule the history was wrapped around is kept and reworded to describe the payload instead. A payload that ends at the match key has no Terms byte, and a missing byte reads as index zero, which says the terms are not stated in the identifier, so absence and zero mean the same thing and no presence flag is needed. The Terms still adds nothing to the minimum payload lengths the package enforces. * FEAT: Refuse a payload version this package cannot read, and answer the Terms with its address Bits 4 and 5 of the Flags byte are the payload version. This package reads version 0 and refuses any other with FodIdParseStatus.UNSUPPORTED_PAYLOAD_VERSION, with the throwing readers naming the version they found. No field is read under the layout this package knows once the version says otherwise, because a later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent. The version is not exposed, because either the package read the layout or there is no identifier to read fields from. FodIdParseResult carries the version it found so the message can name it, and that field is not public. The Terms is one member rather than three. getTerms() answers with the address of the document the identifier was created under, and the package turns the index into the address so a caller never handles the byte. The raw index and the separate address member are gone and the Terms enumeration is package-private. An index of zero and an index this package cannot name both answer with no address, which a caller cannot tell apart, and that is deliberate because both say the identifier does not give the terms and the answer has to come from somewhere else. No address is ever built from an index, since that would name a document nobody wrote. The test factory is the creating side, so it writes both new fields. The canonical Flags byte carries version 0, the canonical payload carries the Terms of a personalized marketing identifier and the canonical Random payload carries the zero a non-marketing identifier carries. A payload that ends at the match key is now a fixture of its own, since a reader takes it as a Terms of zero and no issuer would write one. pipeline.did tests: 142 run, 0 failures, 0 errors, 2 skipped, being the live cloud tests. Javadoc builds with no warnings. * TEST: A Reserved identifier states no terms The Reserved type takes every byte after the header as the match key, so no byte is left to read as the Terms and getTerms() answers null. The other five packages each test this and Java did not. * REFACTOR: One terms table, so a new document is one member Adding a terms document meant editing the enumeration and a second switch that repeated the index, so the two could disagree. Each member now carries its index alongside its address, and the index to member map is built from the members in a static block rather than written out again, so fromIndex is one lookup and a new document is one new member and nothing else. A test walks every member and fails if one does not read back from its own index, or names a document with no address, which is what would happen if a member and its address were added apart. mvn test -pl pipeline.did: 34 in FodIdParseTests, 40 in FodIdTests, 0 failures.
Automation51D
pushed a commit
to 51Degrees/pipeline-python
that referenced
this pull request
Sep 11, 2026
… the terms in the identifier, so they travel with it' * FEAT: Read the Terms a 51Did was created under The 51Did payload gains one byte after the match key, the Terms, which says which terms document the identifier was created under so that the terms travel with the identifier instead of alongside it. The byte is an index into a table in the specification and not a version number, so a later document can live at any address, and an index is never reused or repointed once published because an identifier issued under it has to stay readable years later. Three members are added to FodId, being terms as a named value, the raw terms_index, and terms_url which is the address or None. A new Terms enumeration in terms.py carries the vocabulary and the addresses, the same shape as the existing Usage and IdType. Terms.NOT_STATED and Terms.UNKNOWN are different answers. NOT_STATED is index 0 and says the identifier does not carry the answer, so it has to come from the surrounding protocol, whilst UNKNOWN says the identifier does state terms this package cannot name because the index was added after the package was released. Reading UNKNOWN as NOT_STATED would read an identifier created under terms as one created under none, so the two are separate members and terms_index is exposed to let a caller say which index it could not read. Neither has an address, and the package answers with an address and never fetches it. Existing identifiers are unaffected. A payload ending at the match key has no byte to read and answers index 0, which is the same answer as a zero byte, so absence and zero mean the same thing and no presence flag exists. No field before the Terms moves and no existing test changed. A Reserved type has no defined match key length and keeps its documented best-effort reading, taking every byte after the header as its match key, so it leaves no byte to read as the Terms and answers index 0. Depends on 51Degrees/specifications#27 and must not merge before it. * REFACTOR: Name the Terms members the same in every language The specification gives the table of terms documents but no names for the enumeration members, and six language packages were about to invent six different sets. The names are now fixed across all of them, being NotStated for index 0, ModelTermsForMarketing2 for index 1 and Unknown for an index the package does not know, cased the way each language cases its own enumeration members. Python therefore uses NOT_STATED, MODEL_TERMS_FOR_MARKETING_2 and UNKNOWN. Only the middle one changes here, from MODEL_TERMS_FOR_MARKETING_VERSION_2, along with the string behind it. The comment on reading the Terms of a Reserved identifier now says plainly that answering with terms not stated is the right answer and not a defect, because no match key length is defined for that type, so every byte after the header is its match key and none is left to read. The next reader should not spend time looking for a bug there. * DOC: Describe the Terms byte as it is, rather than as a change No reader has ever seen a 51Did without the Terms byte, so describing the field as a change from a previous state gives a reader history they cannot use. Every rule the history was wrapped around is kept and reworded to describe the payload instead. A payload that ends at the match key reads as index 0, which says the terms are not stated in the identifier, so absence and zero mean the same thing and no presence flag is needed. A Reserved identifier still reads as index 0, because no match key length is defined for that type so every byte after the header is its match key. * FEAT: Refuse a payload version this package cannot read, and answer the Terms with its address Bits 4 and 5 of the flags byte are the payload version. This package reads version 0 and refuses any other with FodIdParseStatus.UNSUPPORTED_PAYLOAD_VERSION, and the raising readers name the version they found. No field is read under the layout this package knows once the version says otherwise, because a later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent. The version is not exposed, because either the package read the layout or there is no identifier to read fields from. The Terms is one member rather than three. fod_id.terms answers with the address of the document the identifier was created under, and the package turns the index into the address so a caller never handles the byte. The raw index and the separate address member are gone, and terms.py is now _terms.py, private to the package as the layout module is, and is no longer exported. An index of zero and an index this package cannot name both answer with None, which a caller cannot tell apart, and that is deliberate because both say the identifier does not give the terms and the answer has to come from somewhere else. No address is ever built from an index, since that would name a document nobody wrote. The test payload builders are the creating side, so they write both new fields. The canonical flags byte carries version 0, the canonical payload carries the Terms of a personalized marketing identifier and the canonical Random payload carries the zero a non-marketing identifier carries. A payload that ends at the match key is now a fixture of its own, since a reader takes it as a Terms of zero and no issuer would write one. Tests: 171 passed, 2 skipped, 29 subtests passed. The offline example prints the address. * REFACTOR: One terms table, so a new document is one member Adding a terms document meant editing the member, an index to member dict and an address dict, three places that could disagree. The address lookup also subscripted the dict directly, so a member missing from it would raise KeyError at the caller rather than answer with no address. Each member now carries its index and its address, and the index to member map is built from the members rather than written out again, so a new document is one new member and nothing else. The address comes off the member, so there is nothing left to raise. A test that claimed a member without an address entry would raise there described the old design and could not fail under the new one. It is replaced by two that check what now holds, being that every member reads back from its own index and that a member naming a document has an https address, and that no two members share an index. pytest: 89 passed, 29 subtests, 0 failures. * DOC: Correct what the package says about the version refusal The readme told a caller to catch OwidError for a payload version this package cannot read. _unpack_or_raise raises ValueError for every status _read_payload returns, so the one new failure this branch introduces was the one a caller following the readme would not catch. The status joins the ValueError list, and the table of statuses, which described the version refusal in prose without ever listing it. Five counts still said two where the version check made them three, being the members added to the OWID status vocabulary, the payload rules _read_payload applies and the ones _from_read applies in turn. The test pinning the vocabulary was renamed to _plus_three; the prose around it was not. The Terms docstring told a caller to treat UNKNOWN as terms it cannot read and to refuse the identifier. No caller can meet UNKNOWN. It is a private member and terms answers None for it exactly as it does for NOT_STATED, which the package surface specification requires, so the paragraph described a surface removed earlier on this branch. It now says why the two reach a caller as one answer and why UNKNOWN is still a member of its own, being that nothing may compose an address from an index the table does not carry. The claim that members carry a name and not the index went with it, every member having carried its index since the terms table was made one place. _payload_length_valid kept the old wording about the bytes beyond the match key, which was reworded in fod_id.py, the readme and the envelope builder when the Terms byte took that position. The readme now also says that no identifier already issued is refused by the version check, the two bits having been written as zero by every issuer before the field was defined, which is the question a reader of that section asks. Also tidied whilst here: two PEP 8 blank-line slips in test_fodid.py, a docstring line in fod_id.py left at 92 characters by a reflow, and the terms constants in test_fodid.py moved to the top block with with_terms above its first caller. * REORG: One place records the index that states no terms ABSENT_TERMS_INDEX in _layout.py and Terms.NOT_STATED in _terms.py both recorded that index 0 says the terms are not stated in the identifier, so the fact the terms table was made the single place for was still written down twice and could disagree. _layout.py holds offsets and lengths, and an index value is neither. _read_terms_index now reads Terms.NOT_STATED.index, and the tests and the envelope builder that wrote the constant name the same member. TERMS_LENGTH stays in _layout.py, being a length. The assertion that the layout gives the constant the value zero is dropped, because test_every_member_agrees_with_the_table already pins NOT_STATED to index 0 and there is no second place left to disagree with it. --------- Co-authored-by: Oleksandr Lazarenko <oleksandr.lazarenko@postindustria.com>
Automation51D
pushed a commit
to 51Degrees/pipeline-node
that referenced
this pull request
Sep 11, 2026
…y the terms in the identifier, so a receiver can read what a 51Did was created under' * FEAT: Read the terms a 51Did was created under from the identifier A 51Did created for marketing may only be used by a receiver that has accepted the terms it was created under, and until now the identifier did not say which terms those are, so the answer had to travel beside it. In OpenRTB there is somewhere to put it, being the Terms Document Locator on the eids entry, and everywhere else there is not. An identifier passed as a query string parameter arrives on its own, and any hop can drop what was sent alongside it without the identifier looking any different. The specification adds one byte after the match key and before the creator context, and this package reads it. The byte is an index into a table in the specification and is not a version number, so that a later document can live at any address rather than only at an address a number could be turned into. Index 0 says the terms are not stated in the identifier and index 1 is the Model Terms for Marketing version 2, at https://m4ow.uk/mtm/2.txt. An index is never reused or repointed once published, because repointing one would rewrite what a past identifier says it agreed to. Three members, named as the package surface page states for Node. 1. fodId.terms, a named value like the existing usage and type, being one of Terms.NOT_STATED, Terms.MODEL_TERMS_FOR_MARKETING_2 and Terms.UNKNOWN. 2. fodId.termsIndex, the byte itself. It is the one place the specification asks for a raw value, and it is asked for because a caller meeting an index added after this package was released otherwise has a value meaning unknown and no way to say which index it could not read. 3. fodId.termsUrl, the address, which is null for NOT_STATED and for UNKNOWN, never an empty string and never an address built from the index. Nothing here fetches the address, because what to do with the document is the caller's decision. An index this package does not know is not zero. NOT_STATED says no terms are stated, whilst UNKNOWN says terms are stated that this package cannot name, and a caller confusing the two would read an identifier created under terms as one created under none, so the two values are distinct and a test holds them apart. Existing identifiers are unaffected. One issued before the byte existed has a payload that ends at the match key, and a payload with no byte to read answers with an index of zero, which is the right answer for it, so absence and zero mean the same thing and no presence flag or format version is needed. The least payload lengths are unchanged, so nothing that parsed before stops parsing. An identifier of the Reserved type cannot carry a byte this reader can find, because the match key length for that type is not defined and every byte after the header is the match key, so it reads as NOT_STATED too, which is correct and is noted in a comment so the next reader does not take it for a missing case. The byte follows the match key, whose length depends on the identifier type, so the offset is worked out from the type rather than being a constant. internal/layout.js gains TERMS_LENGTH and no offset. The readme gains a section on the terms, the payload layout paragraph now names the byte, and the offline example builds a sample carrying index 1 and prints all three members. The generated type declarations were rebuilt for the three files this change touches. Depends on 51Degrees/specifications#27 and must not merge before it. fiftyone.pipeline.did tests: 157 passed, 13 new, 144 before. Whole repository unit run: 287 passed in 23 suites. Lint: 0 errors. * DOC: Describe the Terms byte as it is, rather than as a change No reader has ever seen a 51Did without the Terms byte, so describing the field as a change from a previous state gives a reader history they cannot use. Every rule the history was wrapped around is kept and reworded to describe the payload instead. A payload that ends at the match key carries no terms byte and reads as index zero, which says the terms are not stated, so absence and zero are the same answer. The byte is read at the offset the match key ends at, so where it sits follows the match key length the type selects, and a Reserved identifier leaves no byte for it. * FEAT: Refuse a payload version this package cannot read, and answer the terms with their address Bits 4 and 5 of the flags byte are the payload version. This package reads version 0 and refuses any other with ParseStatus.UNSUPPORTED_PAYLOAD_VERSION, and the error the throwing surfaces raise names the version it found. No field is read under the layout this package knows once the version says otherwise, because a later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent. The version is not exposed, because either the package read the layout or there is no identifier to read fields from. The terms are one member rather than three. fodId.terms answers with the address of the document the identifier was created under, and the package turns the index into the address so a caller never handles the byte. The raw index and the separate address member are gone, and terms.js is no longer exported from the package entry point. An index of zero and an index this package cannot name both answer with no address, which a caller cannot tell apart, and that is deliberate because both say the identifier does not give the terms and the answer has to come from somewhere else. No address is ever built from an index, since that would name a document nobody wrote. The test envelope builders are the creating side, so they write both new fields. The canonical flags byte carries version 0, the canonical payload carries the terms of a personalized marketing identifier and the canonical Random payload carries the zero a non-marketing identifier carries. A payload that ends at the match key is now a fixture of its own, since a reader takes it as a terms index of zero and no issuer would write one. fiftyone.pipeline.did tests: 162 passed, 2 skipped, being the live cloud tests. Lint: 0 errors. The offline example prints the address. * REFACTOR: One terms table, so a new document is one row The name and the address lived in two lists side by side, indexed by position, so adding a terms document meant editing both and a named value, and a row added to one list and not the other would answer with undefined rather than no address. One TABLE now holds a row per document carrying the name and the address together, and fromIndex, name and url all read it. A new document is one row and one named value. A test walks all 256 indexes and fails if a row does not answer its own index, if a row that names a document has no https address, or if index 0 answers with anything but no address. npx jest: 82 passed. eslint: 0 errors, with the same 113 pre-existing warnings. * DOC: Say shipped code, since the tests carry the address on purpose The table comment claimed to be the only place in the package carrying the address. A reader who greps finds it in the tests too, where it is written out deliberately so that a test never compares the reader with itself, so the claim now says shipped code and explains the test. * DOC: Say that a refused payload version throws RangeError The throwing surface table named RangeError for two payload statuses and sent every other status to FodIdParseError, so a caller reading it would catch the wrong class for the one new failure this branch introduces. errorFor throws RangeError for UNSUPPORTED_PAYLOAD_VERSION as well, and the table now says so. The status table gained the row it never had, having been described in prose further up and nowhere a caller enumerates the vocabulary from. The test that pins the throwing contract checked two of the three and said "the two 51Did payload statuses" over the top of them. It now builds a payload of version 1 and asserts RangeError carrying the status, so the class a caller is told to catch is the class the suite holds the reader to. Nothing pinned that before, which is how the table came to disagree with the reader. Four places still said two 51Did payload statuses where the version check made them three, being the errorFor comment, the readme's account of the vocabulary, the test comment above the case that now covers all three, and the class comment on FodIdParseError. That last file is not otherwise touched by this branch, so nothing in the change set led to it; it was found by searching for the claim rather than by walking the diff. unpack returns payloadVersion on the refusal path and errorFor reads it to name the version, but only termsIndex reached the documented return type when the two were added together. * REORG: Put the terms table on the internal path, and bound its lookups terms.js sat at the package root beside fodId.js, idType.js and usage.js, which are the exported modules, whilst the module that is not public in this package is internal/layout.js. Nothing leaked, because index.js does not export it and the exports map in package.json offers only the entry point, but it read as public by where it was. It now sits beside layout, and its generated type sits beside layout's. name and url subscripted the table with whatever they were given, so a value that was neither UNKNOWN nor a row raised a TypeError at the caller rather than answering with no address. The one call in the reader passes the result of fromIndex and was always safe, so nothing was broken, but the shape that raises had no reason to stay: the table is the thing a new terms document changes, and a lookup into it should answer. rowFor decides the bounds once. fromIndex, name and url all go through it, where fromIndex carried a second copy of the same bounds test. A test covers a value outside the table, and it fails against the old lookup with the TypeError it used to raise. The index a payload with no terms byte reads as is Terms.NOT_STATED rather than a bare zero, since that is the value it means and the comment above it was already having to say so. --------- Co-authored-by: Oleksandr Lazarenko <oleksandr.lazarenko@postindustria.com>
jwrosewell
added a commit
to 51Degrees/pipeline-php-did
that referenced
this pull request
Sep 13, 2026
* FEAT: Carry the terms a 51Did was created under, so they travel with it The payload now holds one byte after the match key, an index into the terms table in the specification rather than a version number. Index 0 says the terms are not stated in the identifier, and index 1 is the Model Terms for Marketing version 2 at https://m4ow.uk/mtm/2.txt. FodId gains getTerms(), getTermsIndex() and getTermsUrl(), named as the package surface page requires, and Terms is a named value alongside the existing IdType. An index this release does not know reads as Terms::Unknown and never as Terms::NotStated, because zero says no terms are stated whilst an unknown index says terms are stated that this package cannot name, and reading the two as one would take an identifier created under terms for one created under none. The index is reported so that a caller can say which one it could not read, and the address is returned and never fetched. Existing identifiers are unaffected. A payload ending at the match key reads as index 0, so absence and a byte holding zero mean the same thing and no presence flag is needed. No length rule changed. Depends on 51Degrees/specifications#27. * DOC: Describe the Terms byte as it is, rather than as a change No reader has ever seen a 51Did without the Terms byte, so describing the field as a change from a previous state gives a reader history they cannot use. Every rule the history was wrapped around is kept and reworded to describe the payload instead. A payload that ends at the match key carries no Terms byte and reads as index 0, so absence and a byte holding zero mean the same thing and no presence flag exists. The byte sits at the offset the match key ends at, so where it sits follows the match key length the type selects, and a Reserved identifier leaves none for it. * FEAT: Refuse a payload version this package cannot read, and answer the terms with their address Bits 4 and 5 of the flags byte are the payload version. This package reads version 0 and refuses any other with FodIdParseStatus::UnsupportedPayloadVersion, and the raising readers name the version they found. No field is read under the layout this package knows once the version says otherwise, because a later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent. The version is not exposed, because either the package read the layout or there is no identifier to read fields from. The terms are one member rather than three. getTerms() answers with the address of the document the identifier was created under, and the package turns the index into the address so a caller never handles the byte. The raw index and the separate address member are gone and the Terms enum is marked internal, so it is not part of the published surface. An index of zero and an index this package cannot name both answer with null, which a caller cannot tell apart, and that is deliberate because both say the identifier does not give the terms and the answer has to come from somewhere else. No address is ever built from an index, since that would name a document nobody wrote. The test payload builders are the creating side, so they write both new fields. The canonical flags byte carries version 0, the canonical payload carries the terms of a personalized marketing identifier and the canonical Random payload carries the zero a non-marketing identifier carries. A payload that ends at the match key is now a fixture of its own, since a reader takes it as an index of zero and no issuer would write one. Tests: 137, assertions 1333, 2 skipped, being the live client tests. The offline example prints the address and php -l is clean. * DOC: Drop the reference to a Terms index accessor that does not exist The Unknown case pointed a reader at FodId::getTermsIndex(), which is not a member of this package, since the index is deliberately not published. It now says what Unknown answers with, being no address, and why. * REFACTOR: One terms table, so a new document is one row Adding a terms document meant editing the case, a constant for its address, an arm of fromIndex and an arm of url, four places carrying the same fact. The enum is now backed by the Terms index, so fromIndex is tryFrom with Unknown as the fallback, and one ADDRESSES table keyed by the case's own value carries the addresses, so the index number is written once. A new document is one new case and one new row. Unknown is backed by -1. A Terms index read from a payload is one byte, so it is 0 to 255 and can never be negative, which is what makes -1 safe as the value that is not in the table. Two tests check the table: every case reads back from its own index and a case naming a document has an https address, and all 256 indexes the table does not carry are Unknown with no address. phpunit: 74 tests, 1655 assertions, 0 failures. * DOC: Say shipped code, since the tests carry the address on purpose The table comment claimed to be the only place in the package carrying the address. A reader who greps finds it in the tests too, where it is written out deliberately so that a test never compares the reader with itself, so the claim now says shipped code and explains the test. * FIX: Key the address table with a literal, so PHP 8.1 can read it The table was keyed by self::ModelTermsForMarketing2->value, which reads an enum property inside a constant expression. That needs PHP 8.3, and this package supports 8.1, so it was a fatal parse error there before a single test ran. CI said so on both 8.1 jobs and passed on both 8.3 ones. It went unnoticed because it was written and checked on 8.5. The key is now the literal 1, which costs the single source the enum value gave, so a test holds the two together and fails if the literal ever stops matching the case. phpunit: 75 tests, 1657 assertions, 0 failures, on 8.5 locally. The 8.1 job is what this is for. * TEST: Read the terms, the flags and the consent bit from the live cloud The live test created one non-marketing identifier and checked that it verified. It never read the terms, the usage, the identifier type or the bit that says where the usage came from, so it passed just as happily against a service that wrote none of them. Two tests are added. The first asks for each id.usage the service offers and asserts the usage this package answers with, the terms address that usage must produce, that the usage was recorded as stated by the caller, and that the type is probabilistic. Read through the accessors rather than by masking, because the usage values are cumulative, being 001, 011 and 111, so a caller masking the byte for the non-marketing bit reads every marketing identifier as non-marketing. The second covers the half a caller cannot state for itself. A consent management platform sends an IAB TCF consent string and no usage of its own, the service decodes the purposes, decides the usage, and records in the identifier that it decided rather than was told. Two strings are sent, one granting all twelve purposes and one granting the Appendix 1 standard set, and each identifier must read as the matching usage with the consent bit set. No id.usage goes with them, because a stated usage wins over a consent string and sending one would prove the opposite. The address and the consent strings are written out here rather than taken from the package or shared with the service's own tests, because a test that asked either what it expects would agree with it whatever it said. These sit on this branch rather than a branch of their own, because getTerms() only exists here. main has the usage and type accessors but not the terms. Both tests write NOTHING PROVEN to stderr where they read no marketing identifier, so a key with no marketing entitlement cannot produce a pass that checked nothing. Verified: php -l clean, and vendor/bin/phpunit runs 142 tests where it ran 140, with 4 skipped rather than 2, those two extra being the new live tests with no resource key set. That is the state off CI, and the assertions need the live service. * TEST: Report a skip, not stderr, when the run proved nothing A runner prints the test counts and not the captured output, so a key that returned no marketing identifier would report a pass identical to one that proved the byte. The two summary reports are now skips, which show in the visible counts. The per-usage notes inside the loops stay on stderr, because they explain a skip rather than being one. Verified: php -l clean and phpunit runs 142 tests with 4 skipped.
jwrosewell
added a commit
to 51Degrees/rust
that referenced
this pull request
Sep 13, 2026
* FEAT: Read the terms a 51Did was created under A 51Did created for marketing may only be used by a receiver that has accepted the terms it was created under, and until now the identifier did not say which terms those are. The answer had to travel beside it, which works in OpenRTB, where the Terms Document Locator has somewhere to go, and nowhere else, because an identifier passed as a query string parameter arrives on its own and any hop can drop a sidecar without the identifier looking any different. The specification adds one byte after the match key, before the creator context, holding an index into a table it publishes. Index 0 says the terms are not stated in the identifier and index 1 is the Model Terms for Marketing, version 2, at https://m4ow.uk/mtm/2.txt. It is an index rather than a version number so that a later document can live at any address, and an index rather than the address itself because a receiver has to know the exact document in force when the identifier was made. This adds three members to FodId, matching the package surface page. terms() answers a named Terms value, terms_index() answers the raw byte, and terms_url() answers the address as an Option, being None for index 0 and for an index this crate does not know. An index this crate does not know is Terms::Unknown and never Terms::NotStated. Zero says no terms are stated, whilst an unknown index says terms are stated that this crate cannot name, and a receiver confusing the two would read an identifier created under terms as one created under none. The raw index stays available so a caller meeting a newer index can say which one it could not read. The address is answered and never fetched. Existing identifiers are unaffected. A payload issued before the terms existed ends at the match key, and a missing byte reads as index 0, which is exactly what such an identifier means, so absence and zero need no telling apart and no presence flag exists. Every test that passed before passes unchanged. The byte follows the match key, so where it sits moves with the match key length the identifier type requires, being 37 for a probabilistic or hashed email identifier and 21 for a random one. A reserved type has no defined match key length and takes every remaining byte as its value, so a reserved identifier states no terms until that length is assigned. Depends on 51Degrees/specifications#27, which must merge first. * FEAT: Use the agreed Terms member name across the packages The specification does not name the values, and the six packages were about to invent six different names for the same three concepts, so the names are now fixed across all of them. The concepts are NotStated for index 0, ModelTermsForMarketing2 for index 1 and Unknown for an index the package does not know, each cased the way its own language cases an enumeration member, which is NotStated, ModelTermsForMarketing2 and Unknown in Rust. Only the index 1 member changes here, from ModelTermsForMarketingVersion2. The read also says plainly why a reserved identifier states no terms, since a reserved type has no assigned match key length and takes every byte after the header as its value, so there is no byte left for the terms to be taken from. Index 0 is the right answer there and not a fault, and the next reader should not have to work that out. * DOC: Describe the terms byte as it is, rather than as a change No reader has ever seen a 51Did without the terms byte, so describing the field as a change from a previous state gives a reader history they cannot use. Every rule the history was wrapped around is kept and reworded to describe the payload instead. A payload that ends at the match key carries no terms byte and a missing byte is index 0, so absence and zero say the same thing. The byte sits after the match key, so where it sits follows the match key length the type selects, and a reserved identifier has no byte left for it and reads as index 0. * FEAT: Refuse a payload version this crate cannot read, and answer the terms with their address Bits 4 and 5 of the flags byte are the payload version. This crate reads version 0 and refuses any other with Error::UnsupportedPayloadVersion, which carries the version it found so the message names it. No field is read under the layout this crate knows once the version says otherwise, because a later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent. The version is not exposed, because either the crate read the layout or there is no identifier to read fields from. The terms are one member rather than three. FodId::terms answers with the address of the document the identifier was created under, and the crate turns the index into the address so a caller never handles the byte. The raw index and the separate address member are gone and the Terms enumeration is now pub(crate). An index of zero and an index this crate cannot name both answer with None, which a caller cannot tell apart, and that is deliberate because both say the identifier does not give the terms and the answer has to come from somewhere else. No address is ever built from an index, since that would name a document nobody wrote. The test fixtures are the creating side, so they write both new fields. The canonical flags byte carries version 0 and the canonical payload carries the terms of a personalized marketing identifier. A payload that ends at the match key is now a fixture of its own, since a reader takes it as an index of zero and no issuer would write one. fodid tests: 24 unit, 52 integration and 19 documentation tests pass, with 1 ignored, being the live cloud test. cargo fmt is clean and cargo clippy with -D warnings finds nothing. * REFACTOR: One terms table, so a new document is one row Adding a terms document meant editing the variant, an arm of from_index and an arm of url, so the index, the name and the address were written in three places that could disagree. TERMS_TABLE now holds one row per document carrying all three together, and both lookups read it, so a new document is one row and one variant. The bare 0 the absent byte fell back to is now NOT_STATED_INDEX, named where it is explained. Four unit tests check the table itself: every row round trips through both lookups and carries an https address, no row claims the not stated index, no index appears twice, and every one of the 256 indexes the table does not carry is Unknown with no address. cargo test -p fodid: 28 unit, 52 integration, 19 doc, 0 failures. Clippy clean over all targets. * DOC: Say shipped code, since the tests carry the address on purpose The table comment claimed to be the only place in the package carrying the address. A reader who greps finds it in the tests too, where it is written out deliberately so that a test never compares the reader with itself, so the claim now says shipped code and explains the test. * STYLE: Format the terms table and its tests the way rustfmt wants The only red job on this branch was `cargo fmt --all -- --check`, which reported three differences in fodid/src/fodid.rs, being the TERMS_TABLE constant and two of its tests. Every other job was green, including the wasm32-wasip1 build and the examples check. This is the output of `cargo fmt --all` and nothing else. No behaviour changes. Verified: `cargo fmt --all -- --check` is now silent, and `cargo test -p fodid` runs 99 tests with 0 failures and 1 ignored. * DOC: Point the terms note at the flags, which is the surface this crate has `cargo doc` failed the build with an unresolved intra-doc link, because the note on `terms` pointed at `FodId::usage` and this crate has no such item. It exposes the raw `flags` byte and documents the usage bits within it, so the link had nowhere to go and `-D warnings` turned that into an error. The sentence now points at `flags` and says what the usage bits do, which is bar a non-marketing identifier from a demand source. That is the wording the .NET package already carries for the same note. Verified: `cargo doc -p fodid --no-deps` with `RUSTDOCFLAGS=-D warnings` is clean, `cargo fmt --all -- --check` is silent, `cargo clippy -p fodid --all-targets -- -D warnings` is clean, and `cargo test -p fodid` runs 99 tests with 0 failures and 1 ignored. * TEST: Check the terms, the flags and the consent bit against the live cloud The live cloud test asked the service for a 51Did at each id.usage and then checked the envelope around it, being the match key length, a minimum payload length, a non-empty domain and a base64 round trip. It never read the terms and never read the flags, so it passed just as happily against a service that wrote neither. Every identifier it reads now asserts the terms address the usage must produce, which is nothing for non-marketing and the Model Terms for Marketing for the two marketing usages, the usage the crate answers with, that the usage was recorded as stated by the caller, and that the type is probabilistic. Read through the accessors rather than by masking, because the usage values are cumulative, being 001, 011 and 111, so a caller masking the byte for the non-marketing bit reads every marketing identifier as non-marketing. The new test covers the half a caller cannot state for itself. A consent management platform sends an IAB TCF consent string and no usage of its own, the service decodes the purposes, decides the usage, and records in the identifier that it decided rather than was told. Two strings are sent, one granting all twelve purposes and one granting the Appendix 1 standard set, and each identifier must read as the matching usage with the consent bit set. No id.usage goes with them, because a stated usage wins over a consent string and sending one would prove the opposite. The address and the consent strings are written out here rather than taken from the crate or shared with the service's own tests, because a test that asked either what it expects would agree with it whatever it said. The local case struct was called Usage and shadowed the crate's enum, which the assertions now need, so it is UsageCase. Rust has no inconclusive result, so where a run reads no marketing identifier the consent test says NOTHING PROVEN rather than leaving a pass to be read as evidence. Verified: cargo fmt --all --check silent, cargo clippy -p fodid --all-targets -D warnings clean, cargo doc clean under RUSTDOCFLAGS=-D warnings, and cargo test -p fodid at 103 passed, 0 failed and 2 ignored, those two being the live tests. The assertions need the live service, so the Cloud tests step on this pull request is what proves them. * BUILD: Move fodid to 4.5.4, so merging this actually publishes it crates.io already serves fodid 4.5.3, which is the version this branch carried, and ci/publish-crates.sh skips a crate whose version is already published. Merging as it stood would have published nothing and reported no error, so the terms reader would have reached no consumer. Rust 38 merged that way this morning and is in exactly that state. fodid-cloud moves with it, being unpublished and pinned to fodid by version, so the two stay in step as they were at 4.5.3. Verified: cargo build for both crates is clean, cargo fmt --all --check is silent, and cargo test -p fodid runs 103 tests with 0 failures. * REVERT: Drop the version bump, the release branch already carries it I bumped fodid to 4.5.4 here this morning, not having looked for an existing release pull request. There is one, 31, opened on 2 September, which moves the same line for the same reason and explains the same mechanism, being that ci/publish-crates.sh skips a version already on crates.io so a merge without a bump publishes nothing. This branch now targets that release branch rather than main, so the bump arrives from the base and duplicating it here would only conflict. The fodid-cloud change goes with it. Pull request 31 reasoned that fodid-cloud needs none, because its dependency reads a bare version which Cargo treats as a caret requirement and so accepts 4.5.4, and that is the author's decision to make rather than mine to override in passing.
jwrosewell
added a commit
to 51Degrees/rust
that referenced
this pull request
Sep 13, 2026
…live cloud test (#31) * CHORE: Release fodid 4.5.4 so the match key constants reach crates.io. The match key rename landed in 30 but the crate version did not move, and ci/publish-crates.sh skips any version already on crates.io, so the publish run on that merge finished green having published nothing. crates.io still serves 4.5.3, which carries the old constant names only. This bumps the fodid crate so the next publish has something to release. fiftyone-fodid-cloud needs no change, because its dependency on fodid is written as 4.5.3, which Cargo reads as caret 4.5.3 and which 4.5.4 satisfies. * Carry the terms a 51Did was created under, so they travel with it (#41) * FEAT: Read the terms a 51Did was created under A 51Did created for marketing may only be used by a receiver that has accepted the terms it was created under, and until now the identifier did not say which terms those are. The answer had to travel beside it, which works in OpenRTB, where the Terms Document Locator has somewhere to go, and nowhere else, because an identifier passed as a query string parameter arrives on its own and any hop can drop a sidecar without the identifier looking any different. The specification adds one byte after the match key, before the creator context, holding an index into a table it publishes. Index 0 says the terms are not stated in the identifier and index 1 is the Model Terms for Marketing, version 2, at https://m4ow.uk/mtm/2.txt. It is an index rather than a version number so that a later document can live at any address, and an index rather than the address itself because a receiver has to know the exact document in force when the identifier was made. This adds three members to FodId, matching the package surface page. terms() answers a named Terms value, terms_index() answers the raw byte, and terms_url() answers the address as an Option, being None for index 0 and for an index this crate does not know. An index this crate does not know is Terms::Unknown and never Terms::NotStated. Zero says no terms are stated, whilst an unknown index says terms are stated that this crate cannot name, and a receiver confusing the two would read an identifier created under terms as one created under none. The raw index stays available so a caller meeting a newer index can say which one it could not read. The address is answered and never fetched. Existing identifiers are unaffected. A payload issued before the terms existed ends at the match key, and a missing byte reads as index 0, which is exactly what such an identifier means, so absence and zero need no telling apart and no presence flag exists. Every test that passed before passes unchanged. The byte follows the match key, so where it sits moves with the match key length the identifier type requires, being 37 for a probabilistic or hashed email identifier and 21 for a random one. A reserved type has no defined match key length and takes every remaining byte as its value, so a reserved identifier states no terms until that length is assigned. Depends on 51Degrees/specifications#27, which must merge first. * FEAT: Use the agreed Terms member name across the packages The specification does not name the values, and the six packages were about to invent six different names for the same three concepts, so the names are now fixed across all of them. The concepts are NotStated for index 0, ModelTermsForMarketing2 for index 1 and Unknown for an index the package does not know, each cased the way its own language cases an enumeration member, which is NotStated, ModelTermsForMarketing2 and Unknown in Rust. Only the index 1 member changes here, from ModelTermsForMarketingVersion2. The read also says plainly why a reserved identifier states no terms, since a reserved type has no assigned match key length and takes every byte after the header as its value, so there is no byte left for the terms to be taken from. Index 0 is the right answer there and not a fault, and the next reader should not have to work that out. * DOC: Describe the terms byte as it is, rather than as a change No reader has ever seen a 51Did without the terms byte, so describing the field as a change from a previous state gives a reader history they cannot use. Every rule the history was wrapped around is kept and reworded to describe the payload instead. A payload that ends at the match key carries no terms byte and a missing byte is index 0, so absence and zero say the same thing. The byte sits after the match key, so where it sits follows the match key length the type selects, and a reserved identifier has no byte left for it and reads as index 0. * FEAT: Refuse a payload version this crate cannot read, and answer the terms with their address Bits 4 and 5 of the flags byte are the payload version. This crate reads version 0 and refuses any other with Error::UnsupportedPayloadVersion, which carries the version it found so the message names it. No field is read under the layout this crate knows once the version says otherwise, because a later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent. The version is not exposed, because either the crate read the layout or there is no identifier to read fields from. The terms are one member rather than three. FodId::terms answers with the address of the document the identifier was created under, and the crate turns the index into the address so a caller never handles the byte. The raw index and the separate address member are gone and the Terms enumeration is now pub(crate). An index of zero and an index this crate cannot name both answer with None, which a caller cannot tell apart, and that is deliberate because both say the identifier does not give the terms and the answer has to come from somewhere else. No address is ever built from an index, since that would name a document nobody wrote. The test fixtures are the creating side, so they write both new fields. The canonical flags byte carries version 0 and the canonical payload carries the terms of a personalized marketing identifier. A payload that ends at the match key is now a fixture of its own, since a reader takes it as an index of zero and no issuer would write one. fodid tests: 24 unit, 52 integration and 19 documentation tests pass, with 1 ignored, being the live cloud test. cargo fmt is clean and cargo clippy with -D warnings finds nothing. * REFACTOR: One terms table, so a new document is one row Adding a terms document meant editing the variant, an arm of from_index and an arm of url, so the index, the name and the address were written in three places that could disagree. TERMS_TABLE now holds one row per document carrying all three together, and both lookups read it, so a new document is one row and one variant. The bare 0 the absent byte fell back to is now NOT_STATED_INDEX, named where it is explained. Four unit tests check the table itself: every row round trips through both lookups and carries an https address, no row claims the not stated index, no index appears twice, and every one of the 256 indexes the table does not carry is Unknown with no address. cargo test -p fodid: 28 unit, 52 integration, 19 doc, 0 failures. Clippy clean over all targets. * DOC: Say shipped code, since the tests carry the address on purpose The table comment claimed to be the only place in the package carrying the address. A reader who greps finds it in the tests too, where it is written out deliberately so that a test never compares the reader with itself, so the claim now says shipped code and explains the test. * STYLE: Format the terms table and its tests the way rustfmt wants The only red job on this branch was `cargo fmt --all -- --check`, which reported three differences in fodid/src/fodid.rs, being the TERMS_TABLE constant and two of its tests. Every other job was green, including the wasm32-wasip1 build and the examples check. This is the output of `cargo fmt --all` and nothing else. No behaviour changes. Verified: `cargo fmt --all -- --check` is now silent, and `cargo test -p fodid` runs 99 tests with 0 failures and 1 ignored. * DOC: Point the terms note at the flags, which is the surface this crate has `cargo doc` failed the build with an unresolved intra-doc link, because the note on `terms` pointed at `FodId::usage` and this crate has no such item. It exposes the raw `flags` byte and documents the usage bits within it, so the link had nowhere to go and `-D warnings` turned that into an error. The sentence now points at `flags` and says what the usage bits do, which is bar a non-marketing identifier from a demand source. That is the wording the .NET package already carries for the same note. Verified: `cargo doc -p fodid --no-deps` with `RUSTDOCFLAGS=-D warnings` is clean, `cargo fmt --all -- --check` is silent, `cargo clippy -p fodid --all-targets -- -D warnings` is clean, and `cargo test -p fodid` runs 99 tests with 0 failures and 1 ignored. * TEST: Check the terms, the flags and the consent bit against the live cloud The live cloud test asked the service for a 51Did at each id.usage and then checked the envelope around it, being the match key length, a minimum payload length, a non-empty domain and a base64 round trip. It never read the terms and never read the flags, so it passed just as happily against a service that wrote neither. Every identifier it reads now asserts the terms address the usage must produce, which is nothing for non-marketing and the Model Terms for Marketing for the two marketing usages, the usage the crate answers with, that the usage was recorded as stated by the caller, and that the type is probabilistic. Read through the accessors rather than by masking, because the usage values are cumulative, being 001, 011 and 111, so a caller masking the byte for the non-marketing bit reads every marketing identifier as non-marketing. The new test covers the half a caller cannot state for itself. A consent management platform sends an IAB TCF consent string and no usage of its own, the service decodes the purposes, decides the usage, and records in the identifier that it decided rather than was told. Two strings are sent, one granting all twelve purposes and one granting the Appendix 1 standard set, and each identifier must read as the matching usage with the consent bit set. No id.usage goes with them, because a stated usage wins over a consent string and sending one would prove the opposite. The address and the consent strings are written out here rather than taken from the crate or shared with the service's own tests, because a test that asked either what it expects would agree with it whatever it said. The local case struct was called Usage and shadowed the crate's enum, which the assertions now need, so it is UsageCase. Rust has no inconclusive result, so where a run reads no marketing identifier the consent test says NOTHING PROVEN rather than leaving a pass to be read as evidence. Verified: cargo fmt --all --check silent, cargo clippy -p fodid --all-targets -D warnings clean, cargo doc clean under RUSTDOCFLAGS=-D warnings, and cargo test -p fodid at 103 passed, 0 failed and 2 ignored, those two being the live tests. The assertions need the live service, so the Cloud tests step on this pull request is what proves them. * BUILD: Move fodid to 4.5.4, so merging this actually publishes it crates.io already serves fodid 4.5.3, which is the version this branch carried, and ci/publish-crates.sh skips a crate whose version is already published. Merging as it stood would have published nothing and reported no error, so the terms reader would have reached no consumer. Rust 38 merged that way this morning and is in exactly that state. fodid-cloud moves with it, being unpublished and pinned to fodid by version, so the two stay in step as they were at 4.5.3. Verified: cargo build for both crates is clean, cargo fmt --all --check is silent, and cargo test -p fodid runs 103 tests with 0 failures. * REVERT: Drop the version bump, the release branch already carries it I bumped fodid to 4.5.4 here this morning, not having looked for an existing release pull request. There is one, 31, opened on 2 September, which moves the same line for the same reason and explains the same mechanism, being that ci/publish-crates.sh skips a version already on crates.io so a merge without a bump publishes nothing. This branch now targets that release branch rather than main, so the bump arrives from the base and duplicating it here would only conflict. The fodid-cloud change goes with it. Pull request 31 reasoned that fodid-cloud needs none, because its dependency reads a bare version which Cargo treats as a caret requirement and so accepts 4.5.4, and that is the author's decision to make rather than mine to override in passing. * Run the live cloud 51Did test in CI (#44) * Add the 51Did client: two-step verification, redeem and the outcome types (#33) * FEAT: Add the 51Did client, two-step verification, redeem and the outcome types Every other 51Did language package carries a client for the two-step verification, and Rust had none, so a Rust server could create a 51Did and check its signature but could not take part in the verification at all. This adds the fodid-client crate as a port of the .NET DidClient, which is the model the Java, Node, Python and PHP ports follow. The crate fetches and caches the published signing keys, verifies a signature offline against the key in force when the identifier was created, verifies a signature through the cloud, and redeems the sealed creator context result a browser relays, reading the outcomes the other packages report, misconfigured and invaliddate included. A factor of misconfigured is read on its own and never falls through to a mismatch, because it says the checking service could not determine that factor and the identifier says nothing about it either way. Every request goes through a DidHttpClient trait, and the crate builds without a network stack by default so it compiles for wasm32-wasip1 and an edge runtime can supply its own transport. The reqwest-client feature turns on the built-in blocking reqwest transport, following the cloud request engine. Credentials never travel in a URL, and the licence key is sent only in the redeem form body. Closes #32. * FEAT: Make the 51Did client asynchronous with a non-Send transport Every method of DidClient that may reach the network (public_keys, public_key_for, verify_signature, verify_signature_detailed, verify, verify_encoded, redeem, redeem_encoded) is now `pub async fn` with the same name and return type, and the synchronous versions are gone. The methods that never touch the network (resource_key, endpoint, has_licence_key and the key selection helpers) are unchanged. The DidHttpClient trait's one method now returns a LocalBoxFuture, a boxed future that borrows the request and transport and is deliberately not required to be Send. The crate carries no async runtime and no async-trait dependency, so it builds for wasm32-wasip1 and a single-threaded host such as a Trusted Server appliance can implement the transport and await the client, which is the same shape the cloud request engine's awaitable transport takes. The built-in transport behind the reqwest-client feature is now the asynchronous reqwest client with rustls and the form feature, keeping the thirty second default timeout, zero meaning none, and the default redirect handling the blocking client had. The key cache keeps every rule the tests pin (fetch on first use, again after a day, when no key covers the date, or when the date is past the newest start) and now shares one in-flight fetch between concurrent callers. A caller that finds a fetch in flight waits for it and answers from the keys it landed, fetching for itself only when that fetch failed, and a fetch dropped before it lands clears the in-flight mark so no waiter is stranded. The lock is never held across an await. The tests run on a tokio current-thread runtime as a dev-dependency, with new cases for the shared fetch, a failed shared fetch, a dropped fetch, and a transport whose future holds an Rc across an await to prove the future need not be Send. The README and crate documentation show the awaited calls and the new trait shape, and their examples remain documentation tests. * FIX: Publish the new crate, and stop its test reading internal constants Two things this branch needed before it could go into the release. ci/publish-crates.sh publishes from an explicit CRATES list in dependency order, and fodid-client was not in it, so merging this would have added the crate to the workspace and published nothing. It now sits directly after fodid, which it depends on. The fixture built a payload from fodid::HEADER_LENGTH and fodid::MATCH_KEY_LENGTH. Pull request 38 took the offsets and lengths off the public surface, because the only use a caller has for an offset is to read a field out of the payload by hand and that is how the usage comes out wrong. The fixture carries its own copy taken from the specification instead, which is what the reader's own tests already do and is what a test of a byte format should do anyway, since a payload built from the reader's constants agrees with the reader whatever either of them says. Verified: cargo fmt --all --check silent, cargo clippy on both crates with -D warnings clean, cargo doc clean under RUSTDOCFLAGS=-D warnings, and cargo test on both crates at 156 passed, 0 failed and 2 ignored, those two being the live cloud tests. * MERGE: Take the asynchronous client rewrite The branch had moved on while this was in progress. The fixture constants and the publish list entry both survive the merge unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Defines the Terms, one byte in the 51Did payload saying which terms document the identifier was created under, so the terms travel with the identifier instead of alongside it.
Why the identifier has to carry it
A 51Did created for marketing may only be used by a receiver that has accepted the terms it was created under. If the identifier does not say which terms those are, the answer has to travel beside it. In OpenRTB there is somewhere to put it, being the Terms Document Locator on the
eidsentry, and everywhere else there is not. An identifier passed as a query string parameter arrives on its own, and any hop can drop a sidecar without the identifier looking any different.The field
One byte after the Match Key, before the creator context. It is an index into a table in this specification, not a version number.
01https://m4ow.uk/mtm/2.txtAn index rather than a version number, so a later document can live at any address rather than only at one the specification could compose from a number. An index rather than the address itself, because an address is long and because a receiver has to know the document that was in force when the identifier was made, which an address whose contents can be edited cannot prove years later. An index is never reused or repointed once published.
What the issuer writes is stated as a table too. Non-marketing writes
0, standard and personalized marketing both write1.The parts that are easy to get wrong
An index a package does not know is not zero. A package released before an index existed reports the index, answers with no address, and must not read it as zero. Zero says no terms are stated, whilst an unknown index says terms are stated that this package cannot name, and a receiver confusing the two would read an identifier created under terms as one created under none.
Zero does not mean unrestricted. It means the identifier does not carry the answer, so the answer comes from elsewhere, being the Terms Document Locator or whatever the surrounding protocol provides. Carrying the Terms does not remove the need for a locator where a protocol has one, because the byte answers for the identifier and the locator answers for the transaction.
The Usage and the Terms answer different questions. Where an identifier may go, and under which document it was created. A non-marketing identifier carries zero and is still barred from a demand source by its Usage.
The surface
Three members in every language, being the named value, the raw index and the address. The raw index is the one place this specification asks for a raw value, and it is asked for because a package meeting an index newer than itself otherwise has a value meaning unknown and no way to say which index it could not read.
The names are fixed here too, being Not Stated, Model Terms For Marketing 2 and Unknown, cased the way each language cases its own enumeration members, so two packages describe one thing the same way.
Implementations
Six draft pull requests read the byte against this page and none may merge before it.
Each covers the absent byte, index 1, an unknown index, both Match Key lengths, and the byte read before a creator context section. Writing them turned up three things this page had left to the implementer, all now decided here rather than six times over, being the value names, that the Reserved type leaves no byte for the Terms and so reads as index 0, and the exact address form.
All six were then run over thirteen identifiers built by hand outside every package, covering the absent byte, an explicit zero, index 1 on both Match Key lengths and on the Hashed Email type, indexes 200 and 255, the Terms before a creator context section on both lengths, a non-marketing zero, and versions 1, 2 and 3. Every package gave the same terms and match key on every case, agreed with this page, and refused the three versions.
The remote server writing the byte is the remaining piece and follows this page rather than preceding it.