Repository navigation
Add network statistics and Ping() to the Unreal SDK - #6600
Draft
krisajenkins wants to merge 11 commits into
Draft
krisajenkins wants to merge 11 commits into
krisajenkins wants to merge 11 commits into
Conversation
krisajenkins
requested review from
JasonAtClockwork and
lisandroct
and removed request for
JasonAtClockwork
November 16, 2026 16:24
JasonAtClockwork
self-requested a review
November 16, 2026 22:06
Game developers want to know their connection's round-trip time, how much it jitters, and what time the server thinks it is. Those numbers feed lag compensation, interpolation and the "ping" display that nearly every multiplayer game has. You'd think the protocol already provides this. The server sends a WebSocket Ping frame every 15 seconds, and the client answers each one with a Pong. But browsers never expose control frames to JavaScript (#5525), and none of our SDKs surface them, so that round trip is invisible to the application. So people approximate it. BitCraft syncs its clock at login with 20 sequential requests to `GET /v1/database/:name/unstable/timestamp`, shows an ICMP ping to the region host, and reads reducer latency from the C# SDK's `conn.stats`. Each measures something slightly different from the real connection: the timestamp route uses a separate HTTP connection, ICMP bypasses the WebSocket entirely, and reducer timings include queueing and execution on the server. Pogly runs its own ping/pong system on top of ours (#5357), and others time reducer calls, which runs a transaction for every sample. A ping sent as an ordinary data message would mislead too. The server handles a client's messages one at a time, and runs `Subscribe` and `OneOffQuery` to completion before moving on: while let Some((data, timer)) = recv_handler.next().await { let result = message_handler(data, timer).await; A ping sent just after an expensive subscription would wait for that query to finish, and the client would report the query's evaluation time as network latency. Nothing errors; the figure is just quietly wrong whenever the server is busy with that client, which is exactly when a game cares. This adds `ClientMessage::Ping` and `ServerMessage::Pong` to the v2 schema, and so to v3, which shares it. A `Ping` carries an opaque `client_send_time: u64`, which the `Pong` echoes along with the server's `server_receive_time` and `server_hold_duration`. Only clients that send a Ping ever receive a Pong, so existing clients are unaffected. A new client talking to an older server is disconnected by its first Ping, which we accept rather than bumping the protocol version. The websocket actor answers pings before they reach the incoming queue. A new stream stage, `ws_answer_pings`, sits between the socket and `ws_recv_queue`. It decodes any payload small enough to be a lone Ping and replies through the unordered channel, which the send loop services ahead of ordered messages. The hold duration is measured when the encoder builds the Pong, as late as possible before serialisation. Pings batched with other messages in one v3 payload fall through to the normal handler, which answers them in order. - `decode_lone_ping` only decodes payloads of up to 32 bytes, so only small messages are ever decoded twice. - The Rust SDK gets an `unreachable!` arm for `Pong`, as it never sends pings. - The TypeScript, C# and Unreal SDKs are unchanged. Deriving the RTT, jitter and server-clock metrics there is follow-up work on #5926.
…tion to the TypeScript SDK. An app built on the TypeScript SDK has no way to tell how far away the server is or what time it is there. Games want the round-trip time to tune input prediction and show a latency readout, and they want the server's clock to schedule things against reducer timestamps. Today the only option is to guess from the local wall clock, which can be minutes off and gives no latency information at all. The server now answers ClientMessage::Ping with a Pong carrying its receive time and how long it held the Ping, which is everything needed to measure both, but the SDK's generated protocol types predate those messages, so it can neither send a Ping nor decode a Pong. There is a second, related gap. Picture a client with automatic reconnect whose network goes away without the socket being told: Wi-Fi drops, a NAT forgets the mapping, or (to reproduce it) Chrome DevTools switches the tab to "Offline". Surely automatic reconnect kicks in? It doesn't. The websocket stays OPEN, and reconnect waits for an onclose that won't arrive until the OS gives up on TCP, possibly minutes later. Nothing errors and nothing closes, so it presents as "the connection looks fine but nothing is happening". Once the SDK is sending a Ping every second, those Pings are a liveness probe as well as a measurement, and a run of unanswered ones is exactly the signal that is missing. Regenerated src/sdk/client_api with pnpm generate:client-api to add the Ping and Pong types and their ClientMessage/ServerMessage variants. On top of that, the connection now pings once on InitialConnection and every second after that, and exposes the results as conn.networkStats. It is undefined until the first Pong, then a fresh frozen snapshot on each read: rtt and jitter (RFC 6298 SRTT and RTTVAR), rttLatest, rttMin over the last 10 samples, serverNow, a Timestamp on the same clock as ctx.timestamp, serverTimeLatest, the most recent Pong's unadjusted server_receive_time, so a debug display can show what was measured alongside what was inferred, and pongWait, how long the oldest unanswered Ping has been waiting, which stays below rtt on a healthy link and keeps rising on a dead one. The clock offset comes from the lowest-RTT sample in the window, is applied immediately the first time, then slews at most 5% of elapsed time and never goes backwards. There is deliberately no wall-clock fallback: until the server has answered, the SDK doesn't know its time. On each ping tick, before sending, the connection is treated as lost once the oldest unanswered Ping is older than max(ping timeout, 4 x RTO) and at least two Pings are unanswered. That goes through the existing #handleConnectionLoss path, so with automatic reconnect it reconnects, and without it onDisconnect fires with the error; either way pending ping() calls reject. For a debug overlay or a "test my connection" button, conn.ping() sends a Ping on demand and returns a promise for a frozen PingResult built from its own Pong: rtt (round trip minus server hold), roundTrip, serverHold, serverReceiveTime, and the sentAt/receivedAt performance.now() readings. The sample feeds networkStats like any other, and ping() works with periodic pinging disabled. - The maths lives in a pure NetworkStatsTracker (network_stats.ts) that takes every time as an argument, so it is unit-tested deterministically. - Pings bypass the outbound queue and v3 coalescing, one per websocket message, so they never wait behind queued frames. - Arrival is timestamped as the first statement of the raw socket onmessage in WebsocketDecompressAdapter, before decompression or waiting on earlier frames, and carried through to the Pong handler as receivedAt. - Each Ping takes a requestId from the same counter as reducer and procedure calls, and Pongs are matched to pending pings by it. - A Pong answers every Ping sent up to its clientSendTime, not just its own, because the server's fast-path Pongs can overtake one another. - Requiring two unanswered Pings stops a stale send time from before a sleep or page freeze tripping the timeout on its own. - The RTO multiple gives slow links room (a steady 3 s round trip gets 36 s). RTO is RFC 6298's SRTT + max(G, 4*RTTVAR) from measured samples only: RFC 6298's 1 s floor would make 4 s the shortest possible timeout, silently overriding a shorter withPingTimeout. - withPingInterval(ms) on the builder sets the interval; 0 disables pinging and the liveness timeout, leaving networkStats undefined until an explicit ping(). withPingTimeout(ms) sets the timeout, defaulting to 10 s. Pinging stops, the stats reset and pending pings reject when the socket closes; ping() rejects before InitialConnection. - NetworkStats and PingResult are exported as types. - The esm (gzip) size-limit budget rises from 50 kB to 55 kB. The new code adds about 1.9 kB gzipped to dist/index.mjs and its doc comments, which this unminified bundle keeps, about 1.7 kB more, taking it to 52.6 kB. The minified budgets still pass with room to spare. - The shared design notes in TODO.md gain serverTimeLatest for every SDK, and the TypeScript items are ticked off. - Unrelated generator drift in client_api/index.ts: the CLI version header moves from 2.0.0 to 2.11.0, and the current template adds a procedures accessor export. - The codegen did not run cleanly: generate-client-api assumes the cargo target dir is ./target, and spacetimedb-cli generate --module-def still demands a module source directory. Both were worked around locally without changes.
A Rust client has no way to tell how far away the server is or what time it is there. Games want the round-trip time to tune input prediction and show a latency readout, and they want the server's clock to schedule things against reducer timestamps. The server answers ClientMessage::Ping with a Pong carrying everything needed to measure both, and the TypeScript SDK already uses it, but the Rust SDK never sends a Ping. Worse, parse_loop meets a Pong with unreachable!("The Rust SDK does not send pings"), so the SDK's protocol handling had a panic waiting for the day one arrived.
Where the work lives is decided by the SDK's queues. A server message passes through the WebSocket task, then the parse loop, before it reaches code the application drives with frame_tick or advance_one_message. Timestamp a Pong anywhere after the socket and the app's own queueing is counted as network latency. Resolve a ping() there and it hangs until the app next advances. So the WebSocket loop owns all Ping timing. It reads the monotonic clock as each binary frame arrives, before decompressing it, consumes Pongs on the spot instead of queueing them, and stamps each Ping just before writing it to the socket in its own WebSocket message. It starts only once it has seen InitialConnection itself. Connection code reads the stats and asks for on-demand Pings through a small shared handle.
The maths is a direct port of the TypeScript NetworkStatsTracker, with its tests: RFC 6298 SRTT and RTTVAR, rtt_min over the last 10 samples, an offset estimate from the lowest-RTT sample that is applied immediately the first time, then slews at most 5% and never goes backwards, and pong_wait, the age of the oldest unanswered Ping.
Dead-connection detection differs by build. Native builds already have a 30 s idle check on WebSocket frames, so they keep it and add nothing. The Pongs now arriving every second don't weaken that check, because on a dead link they stop arriving too. The browser build has no transport keepalive, because the browser's WebSocket hides protocol Ping frames from the page. So, on each tick, it closes the socket once two or more Pings are unanswered past max(ping timeout, 4 x RTO), and reports Error::ConnectionUnresponsive through the existing error-disconnect path. The Rust SDK has no auto-reconnect, so this disconnects, with on_disconnect receiving the error. That exposed a hazard: run_threaded and run_background_task panic on any error that isn't a normal disconnect, which would have turned every network drop in the browser into a wasm trap. They now treat ConnectionUnresponsive as an expected end, since on_disconnect has already seen it. run_async still returns it.
- conn.network_stats() -> Option<NetworkStats> is None until the first Pong and resets when the connection closes. NetworkStats has rtt, rtt_latest, rtt_min, jitter, server_now, server_time_latest and pong_wait.
- conn.ping() returns a future of Result<PingResult>. It sends eagerly, as the TypeScript Promise does, takes its request_id from the shared counter, feeds the stats, works with periodic pinging disabled, and fails with NotYetConnected before InitialConnection or Disconnected when the connection closes.
- The builder gains with_ping_interval(Duration), 1 s by default, where zero disables periodic Pings. Under the browser feature only, it also gains with_ping_timeout(Duration), 10 s by default.
- network_stats() and ping() need a codegen change, because the generated DbConnection keeps its imp private. They are generated as inherent methods, so older generated bindings still compile without them. The codegen snapshot and the checked-in test-client bindings are updated to match.
- Running the SDK test regenerated connect_disconnect_client's bindings, which moves its CLI version header from 2.10.1 to 2.12.0.
- The browser build uses web_time::Instant, because std::time::Instant panics on wasm32-unknown-unknown, and gloo-timers intervals. Both are new optional dependencies under the browser feature.
- The connect_disconnect test client now awaits ping() after connecting, and checks rtt <= round_trip and that network_stats() is Some. That test passes natively for all four module languages. The browser loop is compile-checked natively and for wasm32, but its end-to-end test was not run.
The server now speaks ClientMessage::Ping and ServerMessage::Pong, but the C# SDK's generated protocol types predate them. Before the SDK can measure round trips, it has to be able to say Ping and to decode Pong at all. Worse, both message switches in SpacetimeDBClient.cs end in 'default: throw new InvalidOperationException()', so once the bindings know about Pong, the first one to arrive would kill the parse thread unless it has an arm of its own. The regeneration adds Ping.g.cs and Pong.g.cs, each with a Unity .meta file like its neighbours, and appends Ping as ClientMessage tag 6 and Pong as ServerMessage tag 9, matching the protocol. Nothing else drifted. Both switches in SpacetimeDBClient.cs get a Pong arm that does nothing for now; the next change gives it real work. TrackerMetadataForMessage already has a fallback, and the SDK never switches over ClientMessage, so neither needed changing. tools~/gen-client-api.sh doesn't run as-is, so its steps were run by hand, with no change to the tools: - 'spacetime generate --module-def' still runs the module-source-dir check (crates/cli/src/subcommands/generate.rs), which ignores --module-def, and --module-path is in the same clap group so can't be combined with it. The CLI was run from a scratch directory containing an empty spacetimedb/ subdirectory, via --manifest-path, with the schema from 'cargo run -p spacetimedb-client-api-messages --example get_ws_schema_v2' on stdin. The generated Types/*.g.cs were copied over as the script's mv does. - The script's 'cargo build' of crates/standalone isn't needed for generation and was skipped. Everything runs through cargo run, so a non-default CARGO_TARGET_DIR doesn't matter here. The C# SDK tests pass (110/110, -warnaserror), dotnet format reports no changes, and the Godot project builds.
… detection to the C# SDK.
A Unity or .NET client has no way to tell how far away the server is or what
time it is there. Games want the round-trip time to tune input prediction and
show a latency readout, and they want the server's clock to schedule things
against reducer timestamps. The server answers a Ping with a Pong carrying
everything needed to measure both, and the TypeScript and Rust SDKs already
use it, but the C# SDK never sends one.
Under Unity WebGL there's a second problem. The socket there is the
browser's WebSocket, which can't see a network that fails silently. The
socket stays open with nothing arriving: no close, no error, and an app that
thinks it is still connected. Reconnect.cs can't help, because nothing ever
tells it the connection is gone.
This ports the shared design from TODO.md, using the TypeScript SDK's
network_stats.ts as the reference for the maths:
- conn.stats.Network is a NetworkStats? snapshot with Rtt, RttLatest, RttMin,
Jitter and PongWait as TimeSpans, and ServerNow and ServerTimeLatest as
Timestamps. It is null until the first Pong, and again after the
connection closes or is lost. Each read takes a fresh snapshot under a
lock, so it is safe from any thread.
- conn.Ping() returns a Task<PingResult> with Rtt, RoundTrip, ServerHold,
ServerReceiveTime, and SentAt/ReceivedAt as Stopwatch timestamps. Like a
one-off query, it completes on the parse thread without waiting for
FrameTick, with continuations run asynchronously. It fails at once before
InitialConnection or after close, and every pending Ping() fails when the
connection closes.
- The builder gains WithPingInterval(TimeSpan), 1 s by default, where zero
disables periodic Pings. The timer is a Stopwatch deadline checked in
FrameTick, so it works the same under Unity and plain .NET. The first Ping
goes out on InitialConnection, before the user's connect callback and the
subscription replay.
Pongs are timestamped where they come off the socket: on the native receive
thread as ReceiveAsync returns, and in HandleWebGLMessage before the message
waits in the dispatch queue for the next FrameTick. The stamp travels with
the message, so neither the parse queue nor frame time counts as network
time. ParseMessage pulls Pongs out right after decoding and drops their
parse-queue tracker entry, so Pings never show up in the existing request
trackers or reach the apply queue. Every C# send is already its own
WebSocket message, so Pings go out alone without any change to the send path.
Dead-connection detection (rule 8) is compiled only under
UNITY_WEBGL && !UNITY_EDITOR. Before each periodic Ping, if at least two are
unanswered and the oldest is older than max(ping timeout, 4 x RTO), the
connection is failed through HandleSocketFailure, so it reconnects if
automatic reconnect is on and disconnects with an error otherwise.
WithPingTimeout(TimeSpan), 10 s by default, is declared under UNITY_WEBGL
alone, so user code calling it still compiles in the Unity Editor with the
WebGL target selected; there the value is stored but unused.
Native builds don't get this check, as the design says, but the premise
behind that doesn't hold for C#. The SDK never sets
ClientWebSocket.Options.KeepAliveInterval, so .NET's 30 s default applies, and
that keepalive sends unsolicited Pong frames and expects no reply. It keeps
middleboxes from timing out, but it detects nothing. KeepAliveTimeout, which
would detect a dead connection, needs .NET 9 and isn't available on
netstandard2.1. So native C# builds currently have no dead-connection
detection at all. That's left unchanged here and needs a decision.
Other details:
- Every close path bumps socketGeneration and then resets the stats under
the tracker's lock. HandlePong checks the generation under the same lock,
so a late Pong from a replaced socket can't leak into its successor's
stats.
- Pings take request ids from a counter of their own, shared by periodic
Pings and Ping(). The design asks for the counter that reducer and
procedure calls use, but the C# SDK has none: each takes its ids from its
own NetworkRequestTracker.
- On native builds the send queue is first in, first out, and ClientWebSocket
allows one send at a time, so a Ping can wait behind outbound messages
already queued, and that wait counts towards its RTT. Left as is.
- No codegen change. The generated DbConnection inherits Ping(), and its
body uses only qualified names, so the inherited Ping() can't collide with
a user type called Ping.
- NetworkStatsTests.cs ports all 20 tests from network_stats.test.ts. Four
more in ReconnectTests check the following:
- nothing is sent before InitialConnection, and Pings then repeat;
- Ping() resolves from its own Pong without FrameTick, and Pongs stay out
of the trackers;
- a reconnect fails pending Pings, resets the stats and ignores a late Pong
from the old socket;
- Disconnect does the same without reconnect.
FakeSocket keeps Pings in a queue of their own, so the existing assertions
are unchanged.
Tests pass (134, up from 110, with -warnaserror), dotnet format reports no
changes, and the Godot build has only its existing warnings. The WebGL code
was compile-checked by building the SDK sources with UNITY_WEBGL defined
against stand-ins for the few Unity APIs it uses, with no new warnings; it
wasn't run in a browser. End to end, a net8.0 console client on this SDK
talked to a local standalone server running templates/basic-rs's module.
Ping() measured an rtt of 0.54 ms (a 0.61 ms round trip, 0.07 ms server
hold), stats.Network filled in, PongWait stayed under 0.05 ms, ServerNow
stayed within 0.18 ms of the wall clock and never went backwards, reducers
and subscriptions still worked, and after Disconnect, Network was null and
Ping() failed.
A client measures round-trip time by sending a Ping and reading the Pong. The
server stamps the Pong with how long it held the Ping, and the client subtracts
that from the RTT, so that time spent inside the server doesn't count as
network latency. That only works if the hold duration covers everything the
server did with the Ping.
The fast path in `ws_answer_pings` answers a Ping the moment it's read off the
socket, but only when the Ping is alone in its websocket message. Suppose a
v3 client batches two Pings in one payload, or a Ping with other messages:
the TypeScript SDK micro-batches its outbound messages, and a third-party
client might send its Pings through that same queue. The payload skips the
fast path and goes to the message handler, which answers the Pings in order:
ws_v2::ClientMessage::Ping(ping) => {
let held = timer.elapsed();
...
client.send_message(None, ws_v2::ServerMessage::Pong(pong))
`timer` is stamped when the handler dequeues the payload, not when the socket
read it, and the Pong then goes out through the client's ordered outbound
channel, where `held` misses its wait too. In a local test, a lone Ping
measured 0.5 ms RTT with 50 µs of hold. Two batched Pings measured 2.5 ms RTT
but reported only 3–6 µs of hold. In a smoketest run against a server
without this fix, the second of two batched Pongs arrived 4.4 ms after the
first, yet both reported the same 172 µs of hold. Either way, the client
computes an RTT several milliseconds too high.
Nothing errors and the numbers look plausible, so a client would just report
a worse connection than it has. Our SDKs send each Ping alone, so they never
hit this, and the unit tests only covered the lone case.
Now `ws_answer_pings` answers every Ping it finds in a v3 payload, through
`PendingPong` and `unordered_tx` exactly as in the lone case, and the message
handler ignores Pings.
- A payload holding only Pings is dropped once they are answered. Any other
payload passes through unchanged, so the handler still sees, and ignores,
the Pings in it.
- A payload that fails to decode passes through with none of its Pings
answered, so the handler reports the error exactly as before.
- v2 is one message per payload, so it is only decoded when its first byte is
the Ping tag. Like the handler, it decodes the first message and ignores any
trailing bytes.
- A Pong can now overtake the replies to messages batched ahead of its Ping.
The `Ping` doc comment now says a Ping is a latency probe, not a barrier.
- v3 messages aren't length-prefixed, so finding the Pings means decoding the
whole payload, and the handler decodes it again. Measured in a release
build, the extra decode costs the same as the handler's own: about 14 µs per
MB of reducer arguments, essentially one memcpy. That isn't worth threading
decoded messages through to the handler.
The server answers every protocol-level Ping in `ws_answer_pings`, a stage in `ws_recv_task` that sits in front of the message handler, and the handler itself now ignores Pings. That stage has unit tests, but they exercise it on its own. If it were ever dropped from the pipeline, every existing test would still pass, and every Ping would go unanswered: the SDKs' network stats would never appear, and in browser builds the connection would be closed as dead every ten seconds. These tests open a raw v3 websocket to a published module and check that Pings come back as Pongs: - a Ping sent alone in its own websocket message; - two Pings batched in one payload, which the server answers and then drops; - two Pings batched behind a OneOffQuery, where the payload must also pass through so that the query is still answered. Each Pong must echo its Ping's `client_send_time`, and report a `server_receive_time` close to the local clock and a plausible hold duration. With `ws_answer_pings` removed from the pipeline, both tests fail for lack of a reply. They don't check that a Pong overtakes slow work queued ahead of its Ping. I tried, with a Subscribe to a view that takes about 380 ms to evaluate, but the handler only enqueues a Subscribe (and a OneOffQuery or reducer call) on the module instance and moves on, so even a Ping answered in order beats the SubscribeApplied. The same test passed against a server without the batched-Ping fix, so it couldn't tell the two apart.
…t SDKs. The TypeScript, Rust and C# SDKs now ping the server every second and expose the results: round-trip time, jitter, an estimate of the server's clock, and how long the oldest ping has gone unanswered. They also offer a one-off ping(), and new builder settings for the ping interval and, in browser builds, the ping timeout. None of this is in the docs, so the only way to find it is to read the SDK source, and some of it isn't obvious from the API alone. The stats are deliberately absent until the first reply rather than falling back to the local clock, `serverNow` is on the same clock as `ctx.timestamp`, and whether a silently failed network is ever noticed depends on which SDK and which build you use. This adds a "Connection Metrics" section to the connection page, with tabbed examples like the rest of the page, covering what each metric means, the server clock and how to choose a fallback explicitly, the ping interval, one-off pings, and dead-connection detection per SDK. The reference pages get entries for the new builder methods, the stats accessor and ping(), each linking back to the connection page. - The Unreal tabs say metrics are coming soon, matching how the page already handles automatic reconnect there. The Unreal reference page is unchanged. - The C# tab says plainly that non-WebGL builds don't detect a silently failed network. .NET's ClientWebSocket keepalive only sends unsolicited Pongs, and the SDK has no check of its own there, so `PongWait` is the only signal.
The server now speaks ClientMessage::Ping and ServerMessage::Pong, but the Unreal SDK's generated protocol types predate them, so it can neither say Ping nor decode a Pong. Before it can measure round trips, it needs both. This builds on #6318, which regenerated the bindings for SubscribeBatch and SubscribeBatchApplied. Those come before Ping and Pong in the protocol, so with them in place the regeneration appends Ping as ClientMessage tag 6 and Pong as ServerMessage tag 9, matching the server. ProcessServerMessage gets a Pong arm that does nothing for now; without it, every Pong would log "Unknown server-message tag". The next change gives it real work. The round-trip test now covers Ping and Pong, and also pins their wire encoding: Ping is 13 bytes starting with tag 6, and Pong is 29 bytes starting with tag 9. A round trip alone would pass even if both sides agreed on the wrong tag. The DEVELOP.md recipe doesn't run as written, so its steps were adapted, with no change to the tools: - 'spacetime generate' still requires a module source directory even with --module-def, so it was run from a scratch directory containing an empty spacetimedb/ subdirectory, with the schema from 'cargo run -p spacetimedb-client-api-messages --example get_ws_schema_v2'. - The Unreal generator requires a .uproject in --uproject-dir, and sdks/unreal/src/SpacetimeDbSdk only has a .uplugin. It was pointed at a scratch project instead, and the generated Types, Optionals and Results headers were copied over, leaving out ReducerBase.g.h and SpacetimeDBClient.g.h as DEVELOP.md says. The regeneration also picked up unrelated drift from a codegen change on master: default member initialisers ('= {}') in SpacetimeDbSdkOptionalQueryRows, SpacetimeDbSdkOptionalUInt32 and SpacetimeDbSdkResultQueryRowsString. It's kept, since the next regeneration would bring it in anyway. Not compiled: no Unreal Engine is installed on the machine this was written on.
An Unreal game talking to SpacetimeDB has no way to tell how good its connection is. A developer who wants to show a ping readout, decide how much lag compensation to apply, or line local animation up with a reducer's ctx.timestamp has to build their own probe: call a no-op reducer, time it, and accept that the measurement includes the module's queue and execution time. They have no way to know what time it is on the server. The TypeScript, Rust and C# SDKs now answer both questions from the protocol's Ping and Pong messages; this brings the Unreal SDK level with them. The connection sends a Ping as soon as InitialConnection arrives and then every second, from FrameTick, and turns each Pong into the same statistics as the other SDKs: smoothed RTT, latest and minimum RTT, jitter, the estimated server time, the raw server time of the latest Pong, and how long the oldest Ping has gone unanswered (PongWait, the one value that keeps rising when the network goes silent). They're read with TryGetNetworkStats, a BlueprintPure node that returns false until the first Pong, or GetNetworkStats for C++ callers who prefer a TOptional. Ping() sends a one-off Ping and calls a dynamic delegate with an FSpacetimeDBPingResult, so it can be bound from Blueprints. The interval is set with WithPingInterval on the generated builder; zero disables periodic pinging. Each message's arrival time is taken in HandleWSBinaryMessage, before it is queued for parsing on a worker and released in order by FrameTick, so neither queue counts as network time. The statistics are reset whenever the transport is reset, so a reconnect measures the new socket afresh, and pending Ping() calls fail when the connection is lost or closed. Protocol v2 sends one message per WebSocket frame, so every Ping already goes out alone. - NetworkStats.h and .cpp hold FSpacetimeDBNetworkStats, FSpacetimeDBPingResult, FOnPingCompleteDelegate and the estimator, FSpacetimeDBNetworkStatsTracker, a direct port of the Rust SDK's NetworkStatsTracker, without its is_unresponsive check (see below). It takes every time as microseconds from the caller, so it is pure. - The generated builder gets WithPingInterval(FTimespan), forwarding to UDbConnectionBuilderBase::WithPingIntervalBase. The codegen test checks for it, and the six committed client bindings (four test clients, QuickstartChat and Blackholio) are updated to match. Their new lines were checked against the generator's actual output. - The reconnect tests' fake connection now disables pinging by default, so its existing assertions about exactly what was sent still hold, and gains helpers for the new Ping tests. Things to know: - Timing is coarser than in the other SDKs. FWebSocketsModule hands messages to the game thread on an engine tick, and sends them from its own thread, so measured RTT can include up to a frame of engine-side queueing in each direction. There's no earlier point the SDK can see. RttMin is the best guide to the underlying network. - Pings are sent, and Pongs processed, only while the connection is ticked, by auto-ticking or by the app calling FrameTick. Stats don't advance without it, which matches every other callback in the SDK. - No dead-connection detection (rule 8 in the design notes). It applies only where the transport has no keepalive of its own, and whether FWebSocketsModule has one, on each backend (libwebsockets, WinHTTP on Windows), couldn't be settled without the engine source, so it was left out rather than guessed. PongWait is exposed for apps that want to apply their own threshold. This needs a decision from someone with the engine source before the SDK either adds the check or documents that the transport covers it. Testing: none of this has been compiled, because no Unreal Engine is installed on the machine it was written on. Nothing below involved the engine: - The estimator and its 52 checks (SpacetimeDB.NetworkStats.Tracker and .PongWait, ported from the Rust tests) were built with clang++ -Wshadow against minimal stand-ins for the handful of engine types they use (TArray, TOptional, FTimespan, FMath), and all pass. With the RTTVAR and SRTT updates swapped, the RFC 6298 ordering check fails, as it should. - The codegen test passes with its new WithPingInterval assertions. SpacetimeDB.NetworkStats.Connection, which drives a connection through the fake socket (Ping before and after InitialConnection, the immediate and periodic Pings, Pong timings, Ping() success and failure, reset on connection loss), has not been run. Nor has any of the code that touches UObjects, delegates or the WebSocket, so UHT and compile errors are possible and the whole change needs a build and an automation run before review.
The Connection Metrics section of the connection page showed real examples for TypeScript, C# and Rust, but 'coming soon' in every Unreal tab, and the Unreal reference page didn't mention the API at all. With the Unreal SDK now offering the same metrics, its users need the same guidance. Each Unreal tab now has a real example in the SDK's idiom: reading the stats with TryGetNetworkStats (also a Blueprint node, with GetNetworkStats for C++ callers who want a TOptional), an explicit local-clock fallback for the server time, WithPingInterval on the builder, and Ping() with a dynamic delegate. The metrics table's C# column now covers Unreal too, since the field names are the same, and FTimespan joins the list of duration types. Two things are specific to Unreal, and both are said where users will read them: - Pings are sent and answered as the connection ticks, and the measured round trip can include up to a frame of the engine WebSocket module's own queueing, so RttMin is the best guide to the underlying network. - There is no ping timeout. Whether the engine's WebSocket module detects a silently failed network itself hasn't been established, so the Detecting a Dead Connection tab says so and points at PongWait, rather than claiming either way. The Unreal reference gets WithPingInterval with the other builder methods, and a Measure the connection section with TryGetNetworkStats, Ping and the two structs. The sentence on when stats are absent now also says they reset when a reconnect replaces the connection, which holds for every SDK. The docs build passes with Docusaurus' broken-link and broken-anchor checks.
krisajenkins
force-pushed
the
websocket-ping-pong-rtt-unreal
branch
from
November 17, 2026 08:13
f146259 to
5bd09e3
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
Adds round-trip time, jitter and server-clock estimates, and a one-off
Ping(), to the Unreal SDK, matching what #6087 adds to the TypeScript, Rust and C# SDKs.Part of #5926.
This is a draft, and none of it has been compiled. It was written on a machine without Unreal Engine, so it needs a build and an automation-test run before review. It builds on #6087 (the Ping/Pong protocol and the other SDKs), which is still open, so until that merges this PR's diff includes it. It also builds on #6318 (Unreal automatic reconnect), which is already on master. The Unreal work is the last three commits:
Ping(ClientMessage tag 6) andPong(ServerMessage tag 9), plus aPongarm inProcessServerMessagethat does nothing yet, so Pongs aren't logged as unknown tags.An Unreal game has no way to tell how good its connection is, or what time it is on the server. Today a developer who wants a ping readout times a no-op reducer, which counts the module's queueing and execution as network time. With this change, the connection pings on
InitialConnectionand every second after, fromFrameTick, and turns each Pong into the same statistics as the other SDKs:TryGetNetworkStats(FSpacetimeDBNetworkStats&)is aBlueprintPurenode that returns false until the first Pong.GetNetworkStats()returns the same stats as aTOptional, for C++ callers.FSpacetimeDBNetworkStatshasRtt,RttLatest,RttMin,JitterandPongWaitasFTimespan, andServerNowandServerTimeLatestasFSpacetimeDBTimestamp.Ping(FOnPingCompleteDelegate)sends a one-off Ping and calls a dynamic delegate with anFSpacetimeDBPingResult, so it can be bound from Blueprints. It fails at once if the connection isn't established, and when the connection is lost or closed.WithPingInterval(FTimespan), 1 s by default; zero disables periodic pinging.Each message's arrival time is taken in
HandleWSBinaryMessage, before it's queued for parsing on a worker thread and released in order byFrameTick, so neither queue counts as network time. The stats reset whenever the transport resets, so a reconnect measures the new socket afresh. The estimator,FSpacetimeDBNetworkStatsTracker, is a direct port of the Rust SDK's and takes every time from the caller, so it is pure.Things reviewers should know:
FWebSocketsModulehands messages to the game thread on an engine tick, and sends them from its own thread. So a measured RTT can include up to a frame of engine-side queueing in each direction, and the SDK can't see anything earlier. The docs sayRttMinis the best guide to the underlying network.FWebSocketsModulehas one, on each backend (libwebsockets, and WinHTTP on Windows), needs the engine source to settle, so it was left out rather than guessed.PongWaitis exposed for apps that want their own threshold, and the docs say so. This needs a decision from someone with the engine source.FrameTick, like every other callback in the SDK.API and ABI breaking changes
None. The new methods, structs and builder setting are additive. The generated builder gains
WithPingInterval, and the six committed client bindings are updated to match the generator.As with #6087, a client on this SDK talking to a server without
Pingsupport is disconnected by its first Ping.Rollback safety impact
n/a. This only changes the Unreal SDK. The server-side ordering concern is covered in #6087.
Expected complexity level and risk
UDbConnectionBase's receive path,FrameTick, and the close and reconnect handling that Auto reconnect in the C++ sdk #6318 added. Because none of it has been compiled, UHT or compile errors are likely and would show up in the first engine build.Testing
Added:
SpacetimeDB.NetworkStats.Trackerand.PongWait: the estimator's tests, ported from the Rust SDK. These cover the RFC 6298 SRTT and RTTVAR update order, the minimum-RTT window, offset slewing, the never-backwards clamp onServerNow, Pongs answering earlier Pings, andPongWait.SpacetimeDB.NetworkStats.Connection: drives a connection through Auto reconnect in the C++ sdk #6318's fake socket. It coversPing()before and afterInitialConnection, the immediate and periodic Pings, timings computed from a Pong,Ping()succeeding and failing, and the stats resetting on connection loss.PingandPongthat also pin their wire encoding: a 13-byte Ping starting with tag 6, and a 29-byte Pong starting with tag 9.WithPingInterval.Manual, off-engine:
TArray,TOptional,FTimespan,FMath), and all 52 checks pass. With the RTTVAR and SRTT updates swapped, the update-order check fails.Not tested yet: everything that needs the engine. That includes the connection test above, #6318's existing reconnect tests with this change applied, and the code that touches UObjects, delegates and the WebSocket.
SpacetimeDB.NetworkStats,SpacetimeDB.SerializationandSpacetimeDB.TestClient.Reconnect*automation tests.FWebSocketsModule's keepalive behaviour on each backend.