Carry the terms in the identifier, so a receiver can read what a 51Did was created under by jwrosewell · Pull Request #194 · 51Degrees/pipeline-node

jwrosewell · 2026-09-09T14:43:00Z

Draft. The specification this implements has merged as 51Degrees/specifications#27, and this stays in draft until the cloud has been updated to write both fields. pipeline-dotnet#409 goes first, because the cloud is .NET and needs that package published before it can write anything.

What this is

Two fields of the 51Did payload, read by fiftyone.pipeline.did.

The terms is one byte after the match key and before the creator context, saying which terms document the identifier was created under. A 51Did created for marketing may only be used by a receiver that has accepted the terms it was created under, so the identifier has to say which terms those are. In OpenRTB there is somewhere to put the answer, being the Terms Document Locator on the eids entry, and everywhere else there is not. An identifier passed as a query string parameter arrives on its own, and any hop can drop an answer sent alongside it without the identifier looking any different, so the terms travel inside the identifier.

The payload version is bits 4 and 5 of the flags byte, saying which layout the payload follows.

The terms

The byte is an index into a table in the specification and is not a version number, so that a later document can live at any address rather than only at an address a number could be turned into. An index is never reused or repointed once published, because repointing one would rewrite what a past identifier says it agreed to.

fodId.terms answers with the address of the document. The package turns the index into the address, so a caller never handles the byte.

Index	Document	`fodId.terms`
`0`	Not stated in the identifier	`null`
`1`	Model Terms for Marketing, version 2	`https://m4ow.uk/mtm/2.txt`
anything else	One this package cannot name	`null`

That one member is the whole of the addition to the public surface. Nothing here fetches the address, because what to do with the document is the caller's decision, and a test holds globalThis.fetch and asserts it was never called. terms.js keeps the names the specification gives for each index, and it is internal, so it is no longer exported from the package entry point.

The payload version

This package reads version 0 and refuses every other version with ParseStatus.UNSUPPORTED_PAYLOAD_VERSION, and the error the throwing surfaces raise names the version it found.

No field is read under the layout this package knows once the version says otherwise. A later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent, which is worse than refusing. A version that nothing checks protects nothing.

The version is not exposed. Either this package read the layout, in which case the accessors are the answer, or it did not, in which case there is no identifier to read fields from.

The parts that are easy to get wrong

No address is ever built from an index this package cannot name. That would name a document nobody wrote, and a receiver would record having accepted terms that do not exist. An index of zero and an index added after this release therefore give a caller the same answer, which is deliberate, since both say the identifier does not give the terms and the answer has to come from somewhere else.

The terms byte is read at the offset the match key ends at. The match key length depends on the identifier type, so the offset is worked out from the type rather than being a constant, and internal/layout.js gains TERMS_LENGTH and no offset. A reader with one fixed offset reads a match key byte on a 16 byte identifier.

A payload with no byte after the match key reads as index 0. That is the same answer a byte holding zero gives, so absence and zero mean the same thing and no presence flag or format version is needed. The least payload lengths stay 37 bytes and 21 bytes, because the byte is not counted in either.

An identifier of the Reserved type answers with no address. The match key length for that type is not defined and every byte after the header is therefore the match key, so no byte is left for the reader to find. That is correct rather than a missing case, and a comment in unpack says so.

No address does not mean the identifier is unrestricted. It says only that the identifier does not carry the answer, so the answer has to come from the surrounding protocol. Where an identifier may go is a separate question fodId.usage answers, which still bars a non-marketing identifier from a demand source.

The terms sits before the creator context section. An issuer writing a context section writes the terms byte before it, and this package puts no upper bound on a payload, so a longer one is accepted and the same fields are read.

The version is read on its own bits. A reader masking the wrong bits would refuse a version 0 identifier of some usages, or let a later version through, so there is a test over every combination of the usage and type bits.

Files changed

fiftyone.pipeline.did/terms.js, the named value for each index. Internal to the package, so it is not exported from index.js.
fiftyone.pipeline.did/fodId.js, reads the version before any field, refuses a version it does not know, reads the terms byte in unpack and adds the terms accessor.
fiftyone.pipeline.did/internal/layout.js, TERMS_LENGTH and SUPPORTED_PAYLOAD_VERSION.
fiftyone.pipeline.did/tests/envelope.js, the creating side, so it writes both new fields, plus withTerms and withPayloadVersion builders.
fiftyone.pipeline.did/tests/fodId.test.js, the tests below.
fiftyone.pipeline.did/readme.md, a terms section, a payload version section and the payload layout paragraph.
fiftyone.pipeline.did/examples/fodIdExample.js, the sample carries index 1 and prints the address.
fiftyone.pipeline.did/package.json and remote_package.json, the description sentence.
fiftyone.pipeline.did/types/*.d.ts, rebuilt with tsc for the files this change touches. Regenerating the whole repository also rewrites declarations in packages this change does not touch, because the checked in ones were produced by an older TypeScript, and those were restored so the diff stays on this change.

The test fixtures write what an issuer writes

canonicalPayload() and canonicalRandomPayload() are the creating side, so they now carry the payload version 0 in the flags byte and the terms byte their usage calls for, being index 1 for the personalized marketing canonical payload and index 0 for the non-marketing Random one. payloadEndingAtMatchKey() and randomPayloadEndingAtMatchKey() are the fixtures for a payload that carries no terms byte, which a reader takes as index 0 and which no issuer would write.

Tests

In fiftyone.pipeline.did/tests/fodId.test.js, each building the payload byte by byte the way the existing tests build one.

a payload ending at the match key answers with no address
terms index one is the Model Terms for Marketing address
an index this package does not know has no address
not stated and an unknown index both answer with no address
a zero terms byte written out reads the same as none at all
the terms byte is read after a 32 byte match key
the terms byte is read after a 16 byte match key
a context section after the terms byte leaves it read, with a 32 byte match key
a context section after the terms byte leaves it read, with a 16 byte match key
a Reserved payload answers with no address
the terms survive both base64 alphabets and the byte round-trip
the Terms table maps every index, name and address
reading the address does not fetch it
a flags byte with the version bits clear reads every field
payload version 1, 2 and 3 are refused and name the version
the version is read apart from the usage and type bits

The two context cases read the same identifier through all five surfaces, being fromBase64, fromByteArray, fromOwid, tryParse and tryFromByteArray.

Two deliberate breakages were made to check the tests catch a wrong reader, and both were reverted. Making Terms.fromIndex answer zero for an index it does not know failed 5 tests. Reading the byte at a fixed offset of 37 rather than after the match key failed 2 tests, both of them the 16 byte match key cases.

The terms table

The address, the index and the name are one table, so adding a terms document is one row and not a search for every place a number was written down. That is the point of the byte being an index rather than a version number, and the first version of this change did not honour it, spreading the same fact over three, being a named value and two lists side by side indexed by position, so a row added to one and not the other answered with undefined rather than no address places that could disagree.

One TABLE now holds a row per document carrying the name and the address together, and fromIndex, name and url all read it.

A test walks all 256 indexes and fails if a row does not answer its own index, if a row that names a document has no https address, or if index 0 answers with anything but no address.

What was run

Node v24.13.0 on Windows.

npx jest              (in fiftyone.pipeline.did)
  Test Suites: 1 skipped, 3 passed, 3 of 4 total
  Tests:       2 skipped, 163 passed, 165 total

npx eslint fiftyone.pipeline.did --ext .js
  0 errors, 113 warnings

tsc -p fiftyone.pipeline.did/tsconfig.json
  exit 0, no diagnostics

node fiftyone.pipeline.did/examples/fodIdExample.js
  Terms     : https://m4ow.uk/mtm/2.txt

The two skipped tests are the live cloud ones, which need a resource key. The lint warnings are all jsdoc advisories of the same class the rest of the repository already produces, and terms.js adds three newline-after-description warnings, which is exactly what usage.js produces for its own three methods.

The submodule fiftyone.pipeline.core/javascript-templates has to be initialised before the whole repository suite will run, otherwise 14 suites fail to load on a missing mustache template.

Not done here

The offline example has no "Find out more" section, which is a gap in that file rather than one this change introduces, so it is left for a separate pull request rather than widening this diff.

A 51Did created for marketing may only be used by a receiver that has accepted the terms it was created under, and until now the identifier did not say which terms those are, so the answer had to travel beside it. In OpenRTB there is somewhere to put it, being the Terms Document Locator on the eids entry, and everywhere else there is not. An identifier passed as a query string parameter arrives on its own, and any hop can drop what was sent alongside it without the identifier looking any different. The specification adds one byte after the match key and before the creator context, and this package reads it. The byte is an index into a table in the specification and is not a version number, so that a later document can live at any address rather than only at an address a number could be turned into. Index 0 says the terms are not stated in the identifier and index 1 is the Model Terms for Marketing version 2, at https://m4ow.uk/mtm/2.txt. An index is never reused or repointed once published, because repointing one would rewrite what a past identifier says it agreed to. Three members, named as the package surface page states for Node. 1. fodId.terms, a named value like the existing usage and type, being one of Terms.NOT_STATED, Terms.MODEL_TERMS_FOR_MARKETING_2 and Terms.UNKNOWN. 2. fodId.termsIndex, the byte itself. It is the one place the specification asks for a raw value, and it is asked for because a caller meeting an index added after this package was released otherwise has a value meaning unknown and no way to say which index it could not read. 3. fodId.termsUrl, the address, which is null for NOT_STATED and for UNKNOWN, never an empty string and never an address built from the index. Nothing here fetches the address, because what to do with the document is the caller's decision. An index this package does not know is not zero. NOT_STATED says no terms are stated, whilst UNKNOWN says terms are stated that this package cannot name, and a caller confusing the two would read an identifier created under terms as one created under none, so the two values are distinct and a test holds them apart. Existing identifiers are unaffected. One issued before the byte existed has a payload that ends at the match key, and a payload with no byte to read answers with an index of zero, which is the right answer for it, so absence and zero mean the same thing and no presence flag or format version is needed. The least payload lengths are unchanged, so nothing that parsed before stops parsing. An identifier of the Reserved type cannot carry a byte this reader can find, because the match key length for that type is not defined and every byte after the header is the match key, so it reads as NOT_STATED too, which is correct and is noted in a comment so the next reader does not take it for a missing case. The byte follows the match key, whose length depends on the identifier type, so the offset is worked out from the type rather than being a constant. internal/layout.js gains TERMS_LENGTH and no offset. The readme gains a section on the terms, the payload layout paragraph now names the byte, and the offline example builds a sample carrying index 1 and prints all three members. The generated type declarations were rebuilt for the three files this change touches. Depends on 51Degrees/specifications#27 and must not merge before it. fiftyone.pipeline.did tests: 157 passed, 13 new, 144 before. Whole repository unit run: 287 passed in 23 suites. Lint: 0 errors.

No reader has ever seen a 51Did without the Terms byte, so describing the field as a change from a previous state gives a reader history they cannot use. Every rule the history was wrapped around is kept and reworded to describe the payload instead. A payload that ends at the match key carries no terms byte and reads as index zero, which says the terms are not stated, so absence and zero are the same answer. The byte is read at the offset the match key ends at, so where it sits follows the match key length the type selects, and a Reserved identifier leaves no byte for it.

…he terms with their address Bits 4 and 5 of the flags byte are the payload version. This package reads version 0 and refuses any other with ParseStatus.UNSUPPORTED_PAYLOAD_VERSION, and the error the throwing surfaces raise names the version it found. No field is read under the layout this package knows once the version says otherwise, because a later version exists precisely because a field moved, so reading such a payload here would answer with values that are wrong rather than absent. The version is not exposed, because either the package read the layout or there is no identifier to read fields from. The terms are one member rather than three. fodId.terms answers with the address of the document the identifier was created under, and the package turns the index into the address so a caller never handles the byte. The raw index and the separate address member are gone, and terms.js is no longer exported from the package entry point. An index of zero and an index this package cannot name both answer with no address, which a caller cannot tell apart, and that is deliberate because both say the identifier does not give the terms and the answer has to come from somewhere else. No address is ever built from an index, since that would name a document nobody wrote. The test envelope builders are the creating side, so they write both new fields. The canonical flags byte carries version 0, the canonical payload carries the terms of a personalized marketing identifier and the canonical Random payload carries the zero a non-marketing identifier carries. A payload that ends at the match key is now a fixture of its own, since a reader takes it as a terms index of zero and no issuer would write one. fiftyone.pipeline.did tests: 162 passed, 2 skipped, being the live cloud tests. Lint: 0 errors. The offline example prints the address.

The name and the address lived in two lists side by side, indexed by position, so adding a terms document meant editing both and a named value, and a row added to one list and not the other would answer with undefined rather than no address. One TABLE now holds a row per document carrying the name and the address together, and fromIndex, name and url all read it. A new document is one row and one named value. A test walks all 256 indexes and fails if a row does not answer its own index, if a row that names a document has no https address, or if index 0 answers with anything but no address. npx jest: 82 passed. eslint: 0 errors, with the same 113 pre-existing warnings.

The table comment claimed to be the only place in the package carrying the address. A reader who greps finds it in the tests too, where it is written out deliberately so that a test never compares the reader with itself, so the claim now says shipped code and explains the test.

The throwing surface table named RangeError for two payload statuses and sent every other status to FodIdParseError, so a caller reading it would catch the wrong class for the one new failure this branch introduces. errorFor throws RangeError for UNSUPPORTED_PAYLOAD_VERSION as well, and the table now says so. The status table gained the row it never had, having been described in prose further up and nowhere a caller enumerates the vocabulary from. The test that pins the throwing contract checked two of the three and said "the two 51Did payload statuses" over the top of them. It now builds a payload of version 1 and asserts RangeError carrying the status, so the class a caller is told to catch is the class the suite holds the reader to. Nothing pinned that before, which is how the table came to disagree with the reader. Four places still said two 51Did payload statuses where the version check made them three, being the errorFor comment, the readme's account of the vocabulary, the test comment above the case that now covers all three, and the class comment on FodIdParseError. That last file is not otherwise touched by this branch, so nothing in the change set led to it; it was found by searching for the claim rather than by walking the diff. unpack returns payloadVersion on the refusal path and errorFor reads it to name the version, but only termsIndex reached the documented return type when the two were added together.

terms.js sat at the package root beside fodId.js, idType.js and usage.js, which are the exported modules, whilst the module that is not public in this package is internal/layout.js. Nothing leaked, because index.js does not export it and the exports map in package.json offers only the entry point, but it read as public by where it was. It now sits beside layout, and its generated type sits beside layout's. name and url subscripted the table with whatever they were given, so a value that was neither UNKNOWN nor a row raised a TypeError at the caller rather than answering with no address. The one call in the reader passes the result of fromIndex and was always safe, so nothing was broken, but the shape that raises had no reason to stay: the table is the thing a new terms document changes, and a lookup into it should answer. rowFor decides the bounds once. fromIndex, name and url all go through it, where fromIndex carried a second copy of the same bounds test. A test covers a value outside the table, and it fails against the old lookup with the TypeError it used to raise. The index a payload with no terms byte reads as is Terms.NOT_STATED rather than a bare zero, since that is the value it means and the comment above it was already having to say so.

jwrosewell mentioned this pull request Sep 9, 2026

Carry the terms in the identifier, so they travel with it 51Degrees/specifications#27

Merged

jwrosewell added 4 commits September 9, 2026 18:28

justadreamer marked this pull request as ready for review September 11, 2026 07:44

justadreamer requested a review from oleksandrlazarenko-pi September 11, 2026 07:45

oleksandrlazarenko-pi added 2 commits September 11, 2026 09:29

oleksandrlazarenko-pi approved these changes Sep 11, 2026

View reviewed changes

Automation51D merged commit 70a187a into main Sep 11, 2026
1 check passed

Automation51D deleted the feature/terms-in-the-identifier branch September 11, 2026 13:35

Navigation Menu

Uh oh!

Carry the terms in the identifier, so a receiver can read what a 51Did was created under - #194

Carry the terms in the identifier, so a receiver can read what a 51Did was created under#194
Automation51D merged 7 commits into
mainfrom
feature/terms-in-the-identifier

jwrosewell commented Sep 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

jwrosewell commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this is

The terms

The payload version

The parts that are easy to get wrong

Files changed

The test fixtures write what an issuer writes

Tests

The terms table

What was run

Not done here

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jwrosewell commented Sep 9, 2026 •

edited

Loading