Skip to content

feat: Add the MegaStream wire codec and session - #895

Draft
keelerm84 wants to merge 1 commit into
feat/megastreamfrom
mk/SDK-3123/megastream-wire
Draft

keelerm84 wants to merge 1 commit into
feat/megastreamfrom
mk/SDK-3123/megastream-wire

Conversation

@keelerm84

Copy link
Copy Markdown
Member

Summary

First increment of MegaStream support, on the long-running feat/megastream branch. Additive only: the new packages are wired to nothing, so the v9 RC train is unaffected.

internal/megastream/wire holds the fifteen v1 message types and their two framings. The handshake is JSON in text frames, so either side can read the other's rejection whatever binary format the session negotiates; everything after it is a MessagePack envelope, optionally prefixed with a compression byte. Hello and Welcome spell the message name out as type while envelopes use t, and Goodbye/ErrorMessage keep t in both framings.

internal/megastream/session owns one connection: dial, handshake, capability negotiation, heartbeat acknowledgment, the two-interval silence timeout, close advice clamped to an hour, and a jittered exponential reconnect schedule that parks on a configuration error. It holds no credential state — a caller-supplied provider assembles the handshake immediately before every attempt, so a reconnect presents the current desired set with the latest recorded selectors. Every attempt reports its outcome, refused handshakes included, so the layer above can surface a configuration error rather than only logging it.

Decisions worth flagging to a reviewer

  • The socket scheme derives from StreamURI (http → ws, https → wss) rather than being hardcoded to wss. Local development against a mock server is impossible otherwise, and every other URI in relay config works this way.
  • wire.PayloadVersion accepts an integer or a float encoding. The schemas type put-object.version, delete-object.version and payload-transferred.version as "number", while the same spec types the protocol version in hello/welcome as "integer". Until that inconsistency is corrected a float is legal, and a strict decoder would fail the whole connection on its first object. The doc comment says to delete the type and use plain int once the schemas say "integer".
  • coder/websocket defaults to a 32 KiB read limit, which a large flag would silently exceed, so the session sets its own.
  • The harness that drives the real megastream-mock server is deliberately not committed. That mock is an unpublished repo for our own development, so a committed test against it could only ever skip in CI — no protection, and it would make a green suite look like it covered more than it did. Tests here drive the session against a scripted in-process peer instead, which keeps them hermetic.

New dependencies

coder/websocket and vmihailenco/msgpack/v5. zstd comes through klauspost/compress, already present. A hand-rolled zero-copy envelope decoder is a benchmark-driven follow-up, contained inside the wire package.

Follow-ups for the spec PR

Two interop ambiguities, both surfaced by checking this against an independently written implementation: the three version fields above should be "type": "integer", and put-object.object still pins MessagePack bin in prose rather than in the schema (the decoder here accepts bin and str).

@keelerm84
keelerm84 force-pushed the mk/SDK-3123/megastream-wire branch from 5d13a74 to f0ccd5e Compare September 25, 2026 13:51
Phase 1 of megastream support: new packages wired to nothing, so the v9 RC
train is unaffected.

`internal/megastream/wire` holds the fifteen v1 message types and their two
framings. The handshake is JSON in text frames, so either side can read the
other's rejection whatever binary format the session negotiates; everything
after it is a MessagePack envelope, optionally prefixed with a compression
byte. Hello and Welcome spell the message name out as `type` while envelopes
use `t`, and Goodbye and ErrorMessage keep `t` in both framings.

`internal/megastream/session` owns one connection: dial, handshake, capability
negotiation, heartbeat acknowledgment, the two-interval silence timeout, close
advice clamped to an hour, and a jittered exponential reconnect schedule that
parks on a configuration error. It holds no credential state -- a
caller-supplied provider assembles the handshake immediately before every
attempt, so a reconnect presents the current desired set with the latest
selectors. Every attempt reports its outcome, including a refused handshake, so
the layer above can surface a configuration error rather than only logging it.

The socket scheme derives from the configured streaming URI rather than being
hardcoded to wss, which is what makes local development against a mock server
possible.

Tests drive the session against a scripted peer in-process, which keeps them
hermetic and runnable in CI. The harness that runs the real megastream-mock
server is deliberately not committed: that mock is an unpublished repo for our
own development, and a committed test that can only ever skip in CI provides no
protection while making a green suite misleading.
@keelerm84
keelerm84 force-pushed the mk/SDK-3123/megastream-wire branch from f0ccd5e to 49a9be3 Compare September 25, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant