Conversation
acalcutt
force-pushed
the
feature/torrent-tile-source
branch
from
September 4, 2026 13:04
0e089a2 to
032bb2d
Compare
First half of the torrent-aware client: the archive reader, which is what lets a device resolve a tile to a byte range itself. There is nothing tile-specific in a swarm — it holds one file — so both ends need to know how to read tiles out of it, and this is that knowledge on the device side. A separate package rather than part of MapLibreNative.Maui. MonoTorrent and its transitive dependencies would otherwise land in every app that draws a map, and embedding a BitTorrent client is a decision an app should make deliberately. Written against an IByteRangeSource rather than against a swarm, so the same reader works over a torrent, a local file or HTTP ranges, and is testable without a network. Two entry points the C header already declared were never bound: mbgl_http_provider_claim_prefix and _clear_claims. They are what let this serve a few URLs from the swarm while everything else keeps maplibre's own network stack, retry and backoff intact. The tests are cross-implementation on purpose. Tile ids are checked against 1421 cases generated by the reference JavaScript, and archives are read from fixtures that same library's format produced — plain, fully gzipped, and gzipped tiles with plain directories. A self-consistent test would happily agree with a wrong Hilbert curve and misread every archive in existence. One assertion is worth keeping: neighbouring tiles are a median of one apart in the file in *both* directions. Row-major would score one horizontally and a full row vertically. That isotropy is why a single torrent piece tends to carry a whole neighbourhood, which is the property this design rests on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The layer between the archive reader and the swarm: turns "give me bytes 4096 to 8192" into "fetch piece 217". Ported from pmtiles-torrent, whose TypeScript is the reference for the behaviour here. Three things it does that a naive implementation does not: Concurrent callers of the same piece share one fetch. A map asks for a screenful at once and neighbouring tiles land in the same piece, so without this the swarm sees one request per tile on screen. Cancellation is reference counted. An abandoned read stops waiting immediately, but the fetch is only cancelled once every waiter has gone. Forwarding a caller's token straight through would let one abandoned tile kill a piece another tile still needs — and a panning map abandons requests constantly, so that is the common case, not an edge one. Pieces covering a range are fetched together rather than in sequence. Serialising them was the largest latency cost in the original: a range across three pieces paid three round-trips. Caught a real bug while testing the multi-file case. The fetch clipped a piece to the file with Math.Max(0, start) but the reassembly used the unclipped start, so every byte came back shifted by the file offset — plausible-looking wrong data rather than a failure. Both now go through one PieceFileRange helper, which is the only way they cannot drift. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The contract between pmtiles-swarm and this plugin. It describes the archive, not tiles — there is nothing tile-specific in a swarm, which holds one file that both ends know how to read tiles out of. Degrades rather than fails. A TileJSON with no torrent member parses fine and reports null, because an ordinary tile server is not an error; a block with no infohash is treated as absent, since half a descriptor is worse than none; and unexpected types fall back to "no acceleration" rather than breaking the map. Fixtures are generated by pmtiles-swarm's own TileJSON builder, so a change to either side that breaks the contract fails here. A hand-written fixture would only prove this parser agrees with itself. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reads go through MonoTorrent's streaming provider, which promotes the pieces around the read position instead of waiting for the normal picker. That is the difference between a tile arriving in seconds and arriving whenever a sequential download reaches it. The provider's stream is not safe for concurrent use, so reads are serialised. That gives up the parallel piece fetches the layer above would otherwise get — an acceptable trade on a client, where the limit is one slow swarm rather than many fast pieces. Verified against a live 71.93 GiB OpenMapTiles torrent: metadata 0.8s (from a .torrent; a magnet costs minutes) header 4.5s tile 0/0/0 34.8s cold — a whole 16 MiB piece must arrive tile 1/0/0 0.0s already inside a fetched piece fetched 32 MiB to serve 204 KB That second tile is the design working: Hilbert ordering puts map neighbours next to each other in the file, so the read amplification is the prefetch rather than waste. The measurement also showed a default of mine was wrong. Sizing the cache as pieces x piece length gives 128 MiB against the 16 MiB pieces these torrents use, which is not a thing to allocate on a phone. Piece count now sizes the cache relative to the piece length under a 64 MiB ceiling. The swarm test lives here but skips unless PMTILES_TORRENT_TEST_ID names a torrent, so nothing large is committed and CI never depends on public peers being reachable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The last piece. Point TorrentTileProvider.AttachAsync at a pmtiles-swarm tiles.json: if it carries a torrent block the archive is joined and its tiles come from swarm pieces, and if it does not, nothing changes. The style needs no special syntax either way, which is the point — one URL works for both kinds of client. Only the archive's own tile URLs are claimed, so every other resource the map fetches keeps maplibre's network stack with its retry and rate-limit handling. HTTP is never abandoned: it answers while the swarm connects, and remains the fallback for anything the swarm cannot produce within SwarmTimeout. Six seconds by default, because a cold 16 MiB piece can take thirty and a map that stalls that long reads as broken, whereas the piece will be there by the next pan. A missing tile is answered as 204 rather than passed to HTTP: a sparse archive genuinely has no tile there, and asking HTTP the same question only adds latency to a known answer. Also fixes something the cross-platform provider work missed. Those commits took the native side to every platform, but NativeMethods.cs still had the whole provider block inside #if ANDROID, so nothing could reach it anywhere else — the C# half of the same guard bug fixed in the C header. The guard now covers only the ANativeWindow helpers, which really are Android-only. The URL parser lives in src/ rather than platform/ so it is testable: it runs for every resource the map requests on Android, where the provider sits beneath OnlineFileSource and sees all traffic. Its job is mostly to say no, and the tests are mostly things that must not be claimed — sprites, fonts, style documents, and tile URLs belonging to other servers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The same bug as pmtiles-swarm had, and mine in both places: every missing tile answered 204. MapLibre only falls back to a parent tile when the child 404s, so a sparse raster-dem rendered as holes wherever the data was never built. Now decided by the archive's own tile type — raster 404, vector 204 — with a Sparse option for anything that wants the opposite. PMTiles cannot say whether raster data is a DEM, so raster defaults to the answer a DEM needs. Verified against a 698 GiB sparse webp planet: header in 19.2s, first tile 3.3s, neighbour 0.0s. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
pmtiles-swarm publishes an archived source as one string with both halves of the story in it: https://host/latest/<category>/tiles.json#torrent=<url>&magnet=<magnet> A fragment is never sent in a request, so that is an ordinary TileJSON URL to MapLibre and to everything else that reads the style; only something looking for it sees more. AttachAsync now takes that form. The document still wins. Its torrent block carries the infohash, the size, the web seeds and the mutable identity, none of which fit in a URL, so it is asked for first and the fragment is what remains when it cannot be had -- a server that is down, or a source that never published a block. That is the case the fragment exists for, so an unreadable document falls back rather than failing. A fragment names handles, not an identity, so the infohash is recovered from the magnet, or from the metainfo the .torrent URL points at -- which is written into the cache as it is read, so the join does not ask for it twice. Unlike the browser this does not require torrent=. WebTorrent has no DHT, so a magnet alone leaves it nothing to ask for the metadata a magnet omits; MonoTorrent has both a DHT and BEP 9, so a magnet on its own is a route this client can take and refusing it would discard one. Also ports pmtiles-torrent 0.4.6's tail prefetch, which landed after this port was written: one piece of the archive's end, hinted at normal priority as soon as the geometry is known, to cover the gap before a header can point anywhere. Inert with the bundled engine -- MonoTorrentEngine.Hint is a deliberate no-op because the streaming provider owns piece priority -- and marked as such. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The torrent work predates the rename that landed in 5.0.0, so replaying it onto main brought back three names that no longer exist. They applied cleanly, which is the point: nothing here conflicts, so nothing flagged them. Two are P/Invoke entry points, mbgl_http_provider_claim_prefix and mbgl_http_provider_clear_claims. The native side has always exported these -- main carries them as mln_* -- but never declared them in C#, so this is their first binding rather than a duplicate of one. Wrong here would have been an EntryPointNotFoundException the first time a prefix was claimed, not a build error. The third is the managed enum: NativeMethods.MbglHttpError is MlnHttpError. That one the compiler would have caught. Torrent library builds for net10.0 and net10.0-windows against main's bindings, and its 1510 tests pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This branch builds against a fork of maplibre-native (terrain-3d-color-relief)
and carries the torrent tile source, so what it produces is not the stable line
at a newer version -- it is a different build of the same API. Shipping it as a
prerelease under the stable IDs would make a fork native installable by
accident, so the whole family is renamed instead:
MapLibreNative.Maui -> MapLibreNative.Maui.Experimental
MapLibreNative.Maui.WPF -> MapLibreNative.Maui.Experimental.WPF
MapLibreNative.Maui.Vulkan -> MapLibreNative.Maui.Experimental.Vulkan
MapLibreNative.Maui.Handlers -> MapLibreNative.Maui.Experimental.Handlers
MapLibreNative.Maui.Torrent -> MapLibreNative.Maui.Experimental.Torrent
Only the package identity moves. AssemblyName and RootNamespace stay, so
consuming code needs no changes, and the two lines are mutually exclusive by
construction since both carry an assembly called MapLibreNative.Maui.
The rename lives in Directory.Build.targets behind -p:Experimental=true, which
as a global property flows across ProjectReference -- so a dependent packs with
a dependency on MapLibreNative.Maui.Experimental rather than on the stable
package. Verified by packing WPF and reading the nuspec back.
Versioned from VERSION.experimental at 5.0.0-experimental.1: a prerelease of the
next major, so 4.x stays free for minors and patches, and the version string
carries its own prerelease segment so the tag cannot collide with a stable v4.x.
Rather than duplicating five hundred lines of release.yml, the existing workflow
takes a `line` dispatch input. A push to main is always stable; the input only
exists on a manual dispatch. The torrent package is packed only on the
experimental line, because the stable native carries neither the host HTTP
provider nor URL claiming.
Sample apps are skipped on the experimental line: they reference the packages by
their stable IDs from a local feed, which under the renamed line holds nothing
they ask for. Making them work means giving those csproj files a conditional
reference, which is not done here.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things standing between this branch and a release. Nothing ran the torrent library's tests. There are 1510 of them and they are the only automated check the feature has, so a CI job now runs them. They build the plain net10.0 target rather than the solution -- archive parsing, piece mapping, URL and TileJSON handling are pure logic, needing no device, no map and no native library -- which is why this fits on a plain ubuntu runner in seconds. And VERSION.experimental still read 5.0.0-experimental.1, which is already on nuget.org for both MapLibreNative.Maui.Experimental and .Experimental.Torrent. Releasing again at that version would push nothing and say it succeeded, since the push step skips duplicates. Now 5.0.0-experimental.2. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pushing this branch produced a release run with zero jobs and a failure, which reads as a broken workflow file rather than a broken step -- because that is what it was. The version check chose its package line with `inputs.line || 'stable'`. The inputs context only exists for workflow_dispatch and workflow_call, so referencing it on a push fails validation for the whole file. Now guarded on github.event_name, which is defined for every event. And the Publish step carried `prerelease:` twice: one from the experimental line, one from main. They never conflicted -- the lines are far enough apart that git took both -- so the duplicate arrived through a clean merge. Keeping main's is enough on its own: it asks whether the version contains a suffix, which 5.0.0-experimental.N does as much as 5.0.0-pre.3, so the experimental condition it replaces was never adding anything. Both workflows now load with no duplicate keys. Torrent tests still pass (1510), and the library still builds for net10.0-windows against main's bindings after the rebase onto v5.0.0-pre.3. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The experimental package line existed for one reason, which its own comment gave: this branch built against a fork of maplibre-native, so its natives were not the ones the stable line shipped even at the same version. That stopped being true when the branch moved onto upstream feature/terrain-3d -- the submodule pin is now identical to main's -- so the renaming machinery protects against a situation that no longer exists. Removes Directory.Build.targets and VERSION.experimental, the workflow_ dispatch input that chose a line, the branch of version-check that read the other VERSION file, every -p:Experimental= flag, the release-notes preamble and the title prefix. MapLibreNative.Maui.Torrent now packs unconditionally alongside the others, so it ships as itself at whatever the next prerelease is. No 'experimental' references remain, and version-check is back to two outputs. Also adds both torrent projects to maplibre-maui.sln. They were never in it, so the solution build skipped them and they did not appear in the IDE -- easy to miss for a project whose tests are the feature's only automated check. release.yml loads with no duplicate keys, the solution restores, and the 1510 torrent tests still pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
acalcutt
force-pushed
the
feature/torrent-tile-source
branch
from
September 4, 2026 16:41
032bb2d to
d10b316
Compare
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.