Make the public key fetch asynchronous with no synchronous form - #6
Merged
Conversation
Every function in owid.public_key_fetch that reaches the network is now a coroutine, so a caller awaits it, and the synchronous public_key_pem, signature_status and verify are removed rather than kept beside an async twin. public_key_url and clear_cache make no request and stay ordinary functions. Importing owid still never loads the fetch. A synchronous fetch blocks whichever thread calls it for up to the ten second timeout, which in an async application is the event loop itself, and keeping a synchronous twin beside the coroutine would leave that path in place for the next caller to pick up. Removing it is the point of the change. The transport a caller may supply is now an async callable of the same shape, taking the URL and the timeout and returning the response code and the body. The default transport runs the existing urllib code on a worker thread through asyncio.to_thread, which is blocking I/O on a worker thread rather than a non-blocking request, and the module and the README say so and point at an aiohttp or httpx based transport for a fully non-blocking one. The redirect refusal and the date parameter on the URL are unchanged and stay in the default transport. No dependency is added. The key cache is kept, and two callers who await the same key at the same moment now share one in-flight request, held as one asyncio task per URL, instead of making two. A failed request is forgotten so the next caller tries again, and cancelling one waiter does not cancel the request another waiter is still on. The fetch tests move to the standard library IsolatedAsyncioTestCase, so no test dependency is added, and gain tests for the shared in-flight request, for a failure reaching every waiter, for cancellation and for the contract that every network reaching function is a coroutine. The README example uses asyncio.run and every mention of the removed synchronous form is updated.
Contributor
Author
|
Verification run book for this change and the four alongside it: SWAN-community/owid-dotnet#17 It gives the commands and expected results for every port, the traps that waste time (Rust threads its tests where the other six do not, a restored file can leave a build stale and passing against an old binary, Python needs the owid submodule on PYTHONPATH with a Windows separator), and two deliberate breaks that must make specific tests fail, so the tests are shown to be worth something rather than assumed to be. |
The cache was keyed by the whole key url, and the url carries the date of the identifier being verified in minutes. A creator's key changes on the order of a week, so two identifiers signed a minute apart never shared an entry and a hundred identifiers over a hundred minutes made a hundred requests for one key. The cache only ever served a repeat verification of one identifier. Keys are now held by end point, which is the key url without its date, and each key carries the span of minutes the creator has confirmed it for. A key is in force from the start of its period until the next key starts, so a key the creator answers with at two minutes was in force at every minute between them. An identifier dated inside a confirmed span is verified without a request. One dated outside every span is asked about, and the answer widens the span when the same key comes back or adds a key when it does not. The span is never widened across a minute the creator has answered with another key for. A date later than now is held against now, because that is how a creator reads it. Held against the future minute, the key would still be served for that minute after the creator had rotated. A request without a date is read as now for the same reason. The bound of 1024 keys and the empty and refill are unchanged, as is the sharing of one request between callers awaiting together and the refusal to hold a failure. The same change is made to every port that caches, so the ports behave the same way. Tests cover a minute between two confirmed minutes being served without a request, a hundred identifiers inside one confirmed period making none, a key never being served outside its span across a rotation, and a future date being read as now. The bound test and the shared request test were checked by breaking the code and watching them fail.
…g them The span cache read a date later than now as now and held the answer against now. A creator whose clock runs ahead of this one's signs identifiers dated in this process's future, and just after a rotation the cache would have served such an identifier the old key from a span confirmed up to now. It would have read as not matching until this clock caught up, for as long as the two clocks differ. The cache keyed by url did not have this fault, because it asked the creator for the exact minute. A minute within fifteen minutes of now, or later, is now neither served from the cache nor held in it. The creator may have read such a minute as its present rather than as the minute named, so its answer says nothing certain about the minute. A request with no date is treated the same way. Live identifiers therefore cost one request per minute per creator, which is what they always cost, and every identifier older than the allowance is served from the spans. The same allowance is applied on every port that caches. The test that read a future date as now is replaced by one that shows a recent minute asked about twice, a future minute and an undated request asked about, and a minute beyond the allowance held after its first request.
The public key end point now answers with a JSON object carrying the key as publicKeySPKI together with validFrom and validTo, the UTC moments the key came into force and the next key starts. validFrom is null for a creator with one key and no schedule, and validTo is null for the last key in a schedule. The PEM alone as text is not a valid answer, and a client that receives it reports the key as one it cannot read. The answer is checked before it is sent, by the same code a client checks it with. The key must be a public key this library can read, a key valid to a moment must be valid from an earlier one, and the key must have been in force at the moment asked about. A creator whose store or schedule fails that check answers with a server error rather than a bad answer, so a fault on the server side shows up in the server's own tests and never reaches a client. The client holds the key for the whole span the creator stated, so an identifier dated anywhere in it is verified without a request whatever the clock drift, and live identifiers cost one request per key rather than one per minute. A creator that states no span has its key held against the minutes it confirms, with the fifteen minute clock drift allowance kept out of that cache as before. A signature that does not verify under the key selected for the identifier's minute, where that minute is within the drift allowance of an edge of the key's span, is checked against the neighbouring key before it is reported as not matching, because a creator's signing machines may not agree with its schedule to the minute. Where the key tried was never in force at the identifier's minute the neighbours are not tried. Callers verifying the same identifier at the same moment make one request between them, and a test with many threads proves it. The stand in creator in the tests answers with what this library's own server side code builds, so every client test runs against the response a creator built on this library sends, and the loop between the two halves is closed.
…d the stated span The neighbouring key is asked for by the minute just beyond the edge of the span the creator stated, rather than by a minute a fixed distance from the identifier, so a key in force for less than the drift allowance is still the one tried. The span a caller sees is the one the creator stated. A key stated with a start and no end has no later edge, whatever the cache holds it for, so a live identifier dated just after a rotation is checked against the key before it. A creator that stated no span has one key and no neighbour to try. Where the creator's own statement puts the identifier's date outside the span of the key it answered with and nothing verifies, the key is reported as unavailable rather than the signature as not matching, because a key that was not in force proves nothing about the identifier. The key request asks for the JSON form by name in its Accept header, and the README describes the cache as it is, so a creator that answers the PEM alone is refused and a live identifier from a creator that states its spans costs one request per key.
The specification no longer has a creator end point. It repeated the key the public-key end point serves and added fields no verifier read. The end point helpers now build the path and the body for the public-key end point alone, and the tests that exercised both end points exercise the one that remains.
The public-key answer carries the key as publicKey and the encoding it is in as format. The one format served is spki, which a request without the parameter receives, and any other value is answered 400 rather than in an encoding the caller did not ask for. The client asks for spki by name, reads the key from publicKey and refuses an answer that states another format as a key it cannot read.
A signature covers the OWID's own bytes without the signature field and nothing else. Signing and verifying over other OWIDs is gone from Creator, from every verification surface and from the tests, along with the chained interop fixtures, because nothing in production signs that way and an undocumented signing input is a liability for anyone implementing from the specification.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
The creator's public key is now fetched asynchronously and only asynchronously.
public_key_pem,signature_statusandverifyinowid.public_key_fetchare coroutines, the injectable transport is an async callable, and the synchronous forms are removed outright. This is one of a set of changes making every OWID port and every 51Did client asynchronous where the network is involved, so the libraries are consistent across languages.Nothing else in the package touches the network.
PublicKeyScheduleandOwidverify with a key already in hand and are unchanged,public_key_urlandclear_cachestay synchronous because they make no request, and importingowidstill never loads the fetch module.Removed
def public_key_pem(owid, scheme, transport=None) -> strdef signature_status(owid, scheme, others=None, transport=None) -> SignatureStatusdef verify(owid, scheme, others=None, transport=None) -> boolTransport = Callable[[str, float], Tuple[int, bytes]]New
async def public_key_pem(owid, scheme, transport=None) -> strasync def signature_status(owid, scheme, others=None, transport=None) -> SignatureStatusasync def verify(owid, scheme, others=None, transport=None) -> boolTransport = Callable[[str, float], Awaitable[Tuple[int, bytes]]]The default transport wraps the existing urllib code, redirect refusal included, in
asyncio.to_thread, so it is blocking I/O on a worker thread rather than fully non-blocking. The README says so and points at aiohttp or httpx for a transport that is. No dependency was added andrequires-pythonis unchanged at 3.9.Kept exactly
The redirect refusal from #5 and the date parameter on the key URL. The cache is unchanged in what it holds and now also keeps one in-flight task per URL, so concurrent awaits for one key make one request, a failure in flight reaches every waiter, and cancelling one waiter leaves the request running for the others.
Verification
CI's own command,
python -m unittest discover, run with deprecation and runtime warnings promoted to errors: 168 tests OK (163 before). Coverage ofpublic_key_fetch.pyis 96 percent, the same five lines uncovered as before. New tests cover the coroutine contract for every network function, shared in-flight requests over a stub and over the real transport, failure propagation and cancellation.One thing found on the way:
asyncio.iscoroutinefunctionis deprecated on Python 3.14, so the contract test usesinspect.iscoroutinefunction.Cache keyed on the key's span, added 7 September 2026
The cache was keyed by the whole key url, and the url carries the identifier's date in minutes, so two identifiers signed a minute apart never shared an entry and a hundred identifiers over a hundred minutes made a hundred requests for one key. Keys are now held by creator end point, which is the key url without its date, each against the span of minutes the creator has confirmed it for. A key is in force from the start of its period until the next key starts, so a key the creator answers with at two minutes was in force at every minute between them, and an identifier dated inside a confirmed span is verified without a request. The span is never widened past a minute the creator has answered for, nor across a minute the creator answered with another key for. A date later than now is held against now, because the creator reads it that way, and a request with no date is read as now.
The bound of 1024 keys and the empty and refill are unchanged, as are the shared request between callers awaiting together and the refusal to hold a failure. The same change is on every port that caches. Addresses #7.
python -m unittest discover172 OK, with deprecation and runtime warnings promoted to errors. Four tests added. Delete the_held_keys >= MAXIMUM_CACHED_KEYSblock in_holdandtest_the_cache_is_boundedfails with4 != 3. Replace_in_flight.get(url)withNoneandtest_concurrent_awaits_for_one_key_share_one_requestfails reporting two requests. Both checked.Clock drift, added later the same day
The first version of this section read a date later than now as now and held the answer against now. A creator whose clock runs ahead signs identifiers dated in this process's future, and just after a rotation such an identifier would have been served the old key from a span confirmed up to now, reading as not matching for as long as the two clocks differ. A minute within fifteen minutes of now, or later, is now neither served from the cache nor held in it, and a request with no date is treated the same way. Live identifiers therefore cost one request per minute per creator, which is what they always cost, and every identifier older than the allowance is served from the spans. The test that read a future date as now is replaced by one that shows a recent minute asked about twice, a future minute and an undated request asked about, and a minute beyond the allowance held after its first request.
python -m unittest discover172 OK after the change.The answer carries the span, added 7 September 2026
The public key end point now answers with a JSON object carrying the key as
publicKeySPKItogether withvalidFromandvalidTo, the UTC moments the key came into force and the next key starts.validFromis null for a creator with one key and no schedule, andvalidTois null for the last key in a schedule. The PEM alone as text is not a valid answer, and a client that receives it reports the key as one it cannot read. The answer is checked before it is sent, by the same code a client checks it with, so a store or schedule that would produce an answer a client refuses is a server error at the creator.endpoints.public_key_response_atandendpoints.public_key_answerbuild the answer from aPublicKeySchedule, andendpoints.validate_public_key_answeris the check.The client holds the key for the whole span the creator stated, so an identifier dated anywhere in it is verified without a request whatever the clock drift, and live identifiers cost one request per key rather than one per minute. A creator that states no span has its key held against the minutes it confirms, with the fifteen minute clock drift allowance kept out of that cache. A signature that does not verify under the key selected, where the identifier is dated within the drift allowance of an edge of the key's span, is checked against the neighbouring key before it is reported as not matching, because a creator's signing machines may not agree with its schedule to the minute.
The stand in creator in the tests answers with what this port's own server side code builds, so every client test runs against the response a creator built on this port sends. Requests are now shared across event loops in different threads as well as within one loop, and a test with eight threads verifying one OWID at the same moment shows one request between them.
python -m unittest discover178 OK with deprecation and runtime warnings promoted to errors.Brought into line with the specification as merged, 7 September 2026
Four commits, each on its own. The neighbouring key is asked for by the minute just beyond the edge of the span the creator stated, the span the neighbour check works from is the one the creator stated so a key with a start and no end has no later edge, a creator that stated no span has no neighbour to try, and where the creator's own statement puts the identifier's date outside the key it answered with and nothing verifies the key is reported as unavailable rather than the signature as not matching. The creator end point is gone, because the specification no longer has one. The answer carries the key as
publicKeyand its encoding asformat, the one format isspki, a request without it receivesspki, and any other value is answered 400. A signature covers the OWID's own bytes and nothing else, so signing and verifying over other OWIDs is gone with the chained interop fixtures, because nothing in production signs that way.python -W error -m unittest discover177 OK with warnings promoted to errors.Notes
Written with AI assistance and reviewed before merging. The 51Degrees fork and the
fiftyone_pipeline_didpackage follow this merge.