Harden 51Did parsing to answer with a reason instead of throwing

[jwrosewell]

What changed and why

The OWID libraries were hardened so that an OWID reaches a caller only from a successful non-throwing parse or from a creator that signs it. The Rust port of that work is SWAN-community/owid-rust#4, and it is a breaking change to the owid crate. The fodid and fiftyone-fodid-cloud crates in this workspace build on that crate, so this pull request adapts them to the hardened surface and hardens 51Did parsing in the same spirit, meaning a read answers with a named reason instead of a message, never checks a signature, and never rejects a longer identifier merely for being longer.

There is no DidClient and no 4096 character input guard in this repository, so those parts of the shared brief do not apply here. Nothing in this workspace fetches a key during a read, and the fodid crate does not enable the owid fetch feature, so no HTTP code is compiled into it.

The 51Did read contract (fodid)

FodId::from_base64, FodId::from_byte_array, FodId::from_owid, FromStr, TryFrom<&[u8]> and TryFrom<Owid> all run one walk, in this order:

1. The OWID crate parses the envelope through its non-throwing surface. If that fails, the OWID ParseError is carried unchanged in fodid::Error::Parse, and .status() on it names the OWID reason (MissingInput, InvalidBase64, UnexpectedEnd, ByteCountMismatch, and so on). No OWID status is mapped down to a generic one and no message text is inspected.
2. The payload must hold the 5 byte 51Did header (flags byte plus little endian licence id), otherwise Error::PayloadTooShort.
3. The payload must hold the value the identifier type requires after the header, being 16 GUID bytes for Random and 32 hash bytes for Probabilistic and HashedEmail, otherwise the new Error::InvalidTypePayloadLength { id_type, expected, actual }. Reserved keeps the existing best effort read. The minimums are the existing HEADER_LENGTH, RANDOM_PAYLOAD_LENGTH and PAYLOAD_LENGTH constants, no new lengths were introduced.
4. Anything longer is accepted and left in the payload. There is no upper bound in the crate.
5. The parsed FodId is returned without any signature check. The documentation now says in several places that a parsed 51Did is not necessarily cryptographically valid, and points at verify_status_with_public_key, whose SignatureStatus keeps a signature that does not match apart from a key that could not be read or obtained.

Rust already has a non-throwing Result surface, so no try_parse alias was added. The Error enum itself is the status, and a caller branches on the variant. Error::Owid(owid::Error) remains for the exceptional case, which is a caller using ? on an OWID operation of a parsed value (serialising it again, verifying with an unreadable key). A read never produces that variant, and the From<owid::Error> conversion unwraps an owid::Error::Parse back into Error::Parse so a failed read has one representation whichever route a caller's ? took.

Before and after for a caller

// Before (owid 1.0 fields and error)
let domain = &fod_id.domain;
match FodId::from_base64(input) {
    Err(fodid::Error::Owid(e)) => log(e.to_string()),
    ..
}

// After
let domain = fod_id.domain();
match FodId::from_base64(input) {
    Err(fodid::Error::Parse(e)) => log(e.status()),       // the OWID status, unchanged
    Err(fodid::Error::PayloadTooShort { .. }) => ..,
    Err(fodid::Error::InvalidTypePayloadLength { .. }) => ..,
    ..
}

// Building a signed envelope in a test
creator.sign_bytes(payload)?   // before
creator.create(payload)?       // after

The full migration notes are in the fodid crate documentation and README.

fiftyone-fodid-cloud

The engine already turned a failed read into a no-value rather than a failure. Its message now reads "The 51Did value could not be read: ..." followed by the reason from the reader, because a payload that is too short for its type is not an envelope decode failure. The tests and their comments moved from sign_bytes to create and from field access to accessors.

Public API changes

fodid

• Changed: a failed read is Error::Parse(owid::ParseError) where it was Error::Owid(owid::Error). Error::Owid remains for OWID operations other than reading.
• Added: Error::InvalidTypePayloadLength { id_type: IdType, expected: usize, actual: usize }.
• Added: From<owid::ParseError> for Error.
• Added: re-exports fodid::ParseError, fodid::ParseStatus, fodid::SignatureStatus alongside the existing fodid::Owid, so a caller can branch on statuses without a direct owid dependency.
• Unchanged: from_base64, from_byte_array, from_owid, FromStr, TryFrom, the accessors, the offset and length constants. The Deref<Target = Owid> now reaches accessors (domain(), date(), payload(), signature(), version()) rather than public fields, which is the owid 2.0 change.

fiftyone-fodid-cloud

• No signature changes. The no-value message text for an unreadable identifier changed as described above.

Where the OWID code comes from

No owid crate is taken from crates.io or from git any more. James Rosewell
decided that no OWID package is to be owned by a personal account or managed
under SWAN, and that OWID will move to Prebid, so the fodid crate must not
depend on an OWID package at all. The OWID source is compiled into fodid as
the private module fodid::owid, copied before every build by
ci/copy-owid-source.ps1 from the new owid-rust git submodule, which points
at the 51Degrees fork (https://github.com/51Degrees/owid-rust) at the merged
hardening commit bbdb7aa4 on its main branch. This is the same arrangement
as the .NET package, which compiles the owid-dotnet source into
FiftyOne.Did.dll, and the Python package, which copies owid-python in as a
private module.

The copy is git ignored, carries a NOTICE naming the exact submodule commit
and the library's Apache 2.0 LICENSE, and is packaged into the published
crate through an explicit include list in the manifest, proven with
cargo package --list -p fodid. The OWID types a caller needs (Owid,
ParseError, ParseStatus, SignatureStatus, Crypto and the rest) are
re-exported from fodid, and the two that create and sign a new envelope sit
behind a new creator feature, off by default, so the reading crate compiles
nothing that signs unless a test or tool asks for it. The publish workflow
runs the copy script before ci/publish-crates.sh, whose comment now records
the arrangement, and the fodid crates are publishable on their own with
nothing to release on crates.io first. Nothing has been published from this
work. The existing owid 1.0.0 release on crates.io is still required by the
already published fodid 4.5.2, and what to do with that release is a
separate decision.

Tests

Counts are fodid plus fiftyone-fodid-cloud, with the two live cloud tests ignored as before.

• Before: 49 passed (fodid 23 integration + 1 doctest, fodid-cloud 19 unit + 4 integration + 2 doctests), 2 ignored.
• After: 105 passed (fodid 36 integration + 24 unit in the vendored OWID module + 19 doctests, fodid-cloud 20 unit + 4 integration + 2 doctests), 2 ignored. The doctests grew because the OWID module's own documentation examples now compile and run as fodid doctests, README examples included.

The fodid README is now compiled and run as documentation tests through a #[cfg(doctest)] #[doc = include_str!("../README.md")] item, which is where three of the five doctests come from, so the README examples are checked against the code on every cargo test. The parse_and_verify example was run and prints signature : SignatureValid.

New or reworked cases in fodid/tests/fodid_tests.rs, each asserting the three facts (succeeded or not, value present only on success, status read off the Error variant and never off message text):

• a longer self hosted creator domain is accepted and verifies.
• a longer payload is accepted for arbitrary extra lengths (1, 7, 64, 300 bytes) with the value still read and the extra bytes left in the payload, and the same for every identifier type.
• too short Probabilistic, HashedEmail and Random payloads are InvalidTypePayloadLength with the type and the existing minimum reported.
• 0 to 4 byte payloads are PayloadTooShort whatever the flags byte says.
• invalid base64 is the OWID InvalidBase64, empty string and empty buffer are MissingInput, a truncated envelope is UnexpectedEnd, an unknown version byte is UnsupportedVersion.
• an OWID declaration mismatch (a byte after the signature) is ByteCountMismatch, carried with its detail, and the 51Did rules are never reached.
• a structurally valid 51Did with a bit flipped in its hash parses successfully and then verifies as SignatureStatus::Invalid, and so does one checked against another creator's key.
• a key that cannot be read is SignatureStatus::InvalidKey, never Invalid, and the Result form of that check is an error taken into Error::Owid rather than Ok(false).
• FromStr, TryFrom<&[u8]> and TryFrom<Owid> report the same statuses as the direct functions.
• Display names the status and source() keeps the OWID parse error.

fiftyone-fodid-cloud gains a test that a well formed OWID whose payload is shorter than the 51Did header becomes a no-value whose message names PayloadTooShort.

Neutralisation

Each new check was disabled in turn (if false && ...), the fodid suite run, and the check restored.

• Header check disabled: 5 of 36 failed (payload_shorter_than_the_header_is_payload_too_short, constructor_from_owid_short_payload_errors, constructor_from_bytes_short_payload_errors, the_other_reading_routes_report_the_same_statuses, error_display_names_the_status_and_keeps_the_owid_source).
• Type length check disabled: 8 of 36 failed (probabilistic_payload_one_byte_short_is_invalid_type_payload_length, hashed_email_payload_one_byte_short_is_invalid_type_payload_length, random_payload_shorter_than_guid_is_invalid_type_payload_length, random_payload_with_only_the_header_is_invalid_type_payload_length, constructor_from_owid_short_payload_errors, constructor_from_bytes_short_payload_errors, the_other_reading_routes_report_the_same_statuses, error_display_names_the_status_and_keeps_the_owid_source).
• Restored: 36 of 36 pass.

Whole workspace

Across the whole workspace, cargo fmt --all -- --check is clean, cargo clippy --all-targets --all-features -- -D warnings passes (exit 0, every test target compiled), RUSTDOCFLAGS=-D warnings cargo doc --all-features --no-deps passes (exit 0), and the CI wasm steps for fiftyone-fodid-cloud (cargo build and cargo clippy -D warnings with --no-default-features --target wasm32-wasip1) pass.

cargo test --all-features over the whole workspace could not be completed on the machine used, because the build drive ran out of space part way through linking the native on-premise crates (0 bytes free, error 112, unrelated to the code). Only fodid and fiftyone-fodid-cloud depend on owid or fodid, so the other crates' tests cannot be affected by this change, and the two changed crates were tested from a clean build after the disk was recovered (67 passed, 2 ignored). CI runs the full suite on this pull request.

Checked with no issue

• The repository was searched for every use of the removed owid 1.0 surface (sign_bytes, sign_string, Owid::new, public field access on Owid). The only uses were in fodid, fiftyone-fodid-cloud and the parse_and_verify example, all fixed. The examples/ workspace does not use fodid on main.
• No constant, variant, message or test names an envelope, encoded length or creator context size. The lengths that appear are the pre-existing type minimums.
• The base64 handling is unchanged. The owid crate decodes the standard alphabet with or without padding, exactly as 1.0.0 did.
• No em dashes in any touched file, cargo fmt --all -- --check clean.

What remains

• The open #19 renames FodId::hash to match_key; this pull request does not build on it, so whichever merges second will need the tests and docs reconciled. #21 was not touched.
• Once nothing published requires it any more, the owid 1.0.0 release on crates.io can be yanked, which is James Rosewell's call as its owner.

Produced with AI assistance under James Rosewell's direction and needs human review.