Fetch the creator's public key asynchronously only

[jwrosewell]

What changed

Verification that reaches the network is now asynchronous and only asynchronous. Owid::verify and Owid::verify_status take a transport the caller supplies and return futures, and the synchronous forms are removed outright. This is one of a set of changes making every OWID port and every 51Did client asynchronous where the network is involved, so the libraries behave the same way in every language.

verify_with_public_key and verify_status_with_public_key take a key already in hand, are pure cryptography and are unchanged, as is public_key_url.

Removed

• pub fn verify(&self, scheme: &str, others: &[&Owid]) -> Result<bool>
• pub fn verify_status(&self, scheme: &str, others: &[&Owid]) -> SignatureStatus
• the ureq dependency

New

pub type LocalBoxFuture<'a, T> = Pin<Box<dyn Future<Output = T> + 'a>>;
pub struct FetchResponse { /* status, body, location */ }
pub trait PublicKeyFetch: Send + Sync {
    fn fetch<'a>(&'a self, url: &'a str) -> LocalBoxFuture<'a, Result<FetchResponse>>;
}
impl Owid {
    pub async fn verify(&self, fetch: &dyn PublicKeyFetch, scheme: &str, others: &[&Owid]) -> Result<bool>;
    pub async fn verify_status(&self, fetch: &dyn PublicKeyFetch, scheme: &str, others: &[&Owid]) -> SignatureStatus;
}

The future is deliberately not required to be Send. This crate is consumed on wasm32-wasip1, where the host environment produces futures that cannot be Send, so requiring it would shut that consumer out. A test proves the bound is absent by holding an Rc across an await inside the transport stub.

Features

fetch now adds no dependency at all and builds for WebAssembly targets, because the transport comes from the caller. A ready made one is available behind the new reqwest-fetch feature, ReqwestFetch, over asynchronous reqwest with rustls and redirect::Policy::none().

Kept exactly

The redirect refusal and the date parameter on the key URL. The refusal now lives in the library rather than the transport, so any 3xx reads as the key being unavailable no matter which transport is supplied and no transport can follow one by accident.

Verification

All clean from the repository root.

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
cargo build --no-default-features --features fetch --target wasm32-wasip1
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features

Tests: 128 passing across the suites with every feature on, including the converted fetch and redirect tests and the not-Send proof.

Cache keyed on the key's span, added 7 September 2026

The cache was keyed by the whole key url, and the url carries the identifier's date in minutes, so two identifiers signed a minute apart never shared an entry and a hundred identifiers over a hundred minutes made a hundred requests for one key. Keys are now held by creator end point, which is the key url without its date, each against the span of minutes the creator has confirmed it for. A key is in force from the start of its period until the next key starts, so a key the creator answers with at two minutes was in force at every minute between them, and an identifier dated inside a confirmed span is verified without a request. The span is never widened past a minute the creator has answered for, nor across a minute the creator answered with another key for. A date later than now is held against now, because the creator reads it that way, and a request with no date is read as now.

The bound of 1024 keys and the empty and refill are unchanged, as are the shared fetch between callers arriving together and the refusal to hold a failure. The same change is on every port that caches. Addresses #11.

cargo fmt --all -- --check, cargo clippy --all-targets --all-features -- -D warnings, cargo build --no-default-features --features fetch --target wasm32-wasip1 and cargo doc with warnings denied are all clean, and cargo test --all-features passed 157 on three consecutive runs. Five tests added. Delete the self.held >= MAXIMUM_CACHED_KEYS block in Cache::hold and the_cache_is_bounded fails with held 1025 of at most 1024. Replace held.in_flight.get(url) with None and callers_asking_for_one_key_at_the_same_time_share_one_fetch fails. Both checked.

Clock drift, added later the same day

The first version of this section read a date later than now as now and held the answer against now. A creator whose clock runs ahead signs identifiers dated in this process's future, and just after a rotation such an identifier would have been served the old key from a span confirmed up to now, reading as not matching for as long as the two clocks differ. A minute within fifteen minutes of now, or later, is now neither served from the cache nor held in it, and a request with no date is treated the same way. Live identifiers therefore cost one request per minute per creator, which is what they always cost, and every identifier older than the allowance is served from the spans. The test that read a future date as now is replaced by one that shows a recent minute asked about twice, a future minute and an undated request asked about, and a minute beyond the allowance held after its first request. fmt, clippy, the wasm build and doc clean and 157 tests on five consecutive runs after the change. The span tests count keys under their own end point only, because the cache is shared and the harness runs tests in parallel.

The answer carries the span, added 7 September 2026

The public key end point now answers with a JSON object carrying the key as publicKeySPKI together with validFrom and validTo, the UTC moments the key came into force and the next key starts. validFrom is null for a creator with one key and no schedule, and validTo is null for the last key in a schedule. The PEM alone as text is not a valid answer, and a client that receives it reports the key as one it cannot read. The answer is checked before it is sent, by the same code a client checks it with, so a store or schedule that would produce an answer a client refuses is a server error at the creator. endpoints::public_key_response_at and endpoints::public_key_answer build the answer from the new PublicKeySchedule, and PublicKeyAnswer reads, writes and checks the body without a dependency, so the fetch feature still adds none.

The client holds the key for the whole span the creator stated, so an identifier dated anywhere in it is verified without a request whatever the clock drift, and live identifiers cost one request per key rather than one per minute. A creator that states no span has its key held against the minutes it confirms, with the fifteen minute clock drift allowance kept out of that cache. A signature that does not verify under the key selected, where the identifier is dated within the drift allowance of an edge of the key's span, is checked against the neighbouring key before it is reported as not matching, because a creator's signing machines may not agree with its schedule to the minute.

The stand in creator in the tests answers with what this port's own server side code builds, so every client test runs against the response a creator built on this port sends. A test with eight threads, each driving its own runtime, verifying one OWID at the same moment shows one request between them.

fmt, clippy with warnings denied, the wasm32-wasip1 build and doc all clean, 167 tests on three consecutive runs. FetchResponse is unchanged, so a transport of the caller's own still compiles.

Brought into line with the specification as merged, 7 September 2026

Four commits, each on its own. The neighbouring key is asked for by the minute just beyond the edge of the span the creator stated, the span the neighbour check works from is the one the creator stated so a key with a start and no end has no later edge, a creator that stated no span has no neighbour to try, and where the creator's own statement puts the identifier's date outside the key it answered with and nothing verifies the key is reported as unavailable rather than the signature as not matching. The creator end point is gone, because the specification no longer has one. The answer carries the key as publicKey and its encoding as format, the one format is spki, a request without it receives spki, and any other value is answered 400. A signature covers the OWID's own bytes and nothing else, so signing and verifying over other OWIDs is gone with the chained interop fixtures, because nothing in production signs that way.

fmt, clippy with warnings denied, the wasm32-wasip1 build and doc with warnings denied all clean, 166 tests.

Notes

Written with AI assistance and reviewed before merging. The 51Degrees fork, the fodid crate that copies this source in, and the 51Did client follow this merge.

[jwrosewell]

Added in ec2e95e. This port had no way to empty the key cache, and every other port that caches offers one: clearCache in Java and PHP, clear_cache in Python, ClearKeyCache in Go, and ClearPublicKeyCache in .NET. I confirmed each of those exists rather than taking it on trust. The cache here lives as long as the process and is only emptied by reaching its own limit, so a long running verifier that had learned it should no longer trust a key it already held, because the creator rotated after a compromise, could do nothing about it.

clear_cache drops the keys held and the record of the fetches under way. A fetch already running is not stopped and the callers waiting on it still get its answer, because they hold the shared state through an Arc rather than through the map, so only what is held is dropped.

One thing worth recording, because it would have become an intermittent CI failure rather than an obvious one. Adding a test that empties a process wide cache made it possible to disturb keys_are_held_per_request_and_not_per_domain, which fetches two keys and then asserts that asking for the first again makes no third request. The harness runs tests in parallel, so an empty landing between its second and third call would have failed it. It did not fire in twenty runs, which is exactly why it is worth fixing rather than trusting: both tests now take a mutex, so the ordering is decided rather than left to chance.

Verified on the committed state: cargo fmt --all -- --check clean, cargo clippy --all-targets --all-features -- -D warnings clean, 152 tests passing, cargo build --no-default-features --features fetch --target wasm32-wasip1 succeeds, and cargo doc clean with warnings denied. Fifteen consecutive full runs of the suite pass.

The readme records what clear_cache is for and the exported surface table lists it.

[jwrosewell]

Verification run book for this change and the four alongside it: SWAN-community/owid-dotnet#17

It gives the commands and expected results for every port, the traps that waste time (Rust threads its tests where the other six do not, a restored file can leave a build stale and passing against an old binary, Python needs the owid submodule on PYTHONPATH with a Windows separator), and two deliberate breaks that must make specific tests fail, so the tests are shown to be worth something rather than assumed to be.