feat(p987): add ingest service for the 987 live data path - #226
Conversation
Nothing consumed the p987 topic namespace, so the TCM-987 relay was publishing into the void. This mirrors gr26's live path — subscribe, validate, decode, persist, republish — for the Porsche 987. Decoding works differently, and that is the substance of this service. gr26 describes each frame as a list of whole-byte fields via mapache-go's Message type, which is workable when the CAN layout and the decoder were designed together. Stock Porsche CAN is not ours to design: signals start at arbitrary bit offsets, run arbitrary bit lengths, and carry a scale and offset (SCCM_SteeringAngle is 13 bits starting at bit 2, x0.175 deg). So p987 decodes from the DBC directly, embedded in the binary so a malformed file fails at startup rather than silently decoding nothing. Multiplexed signals are skipped. Resolving them needs the message's multiplexer switch signal, and the 987 DBC declares multiplexed signals without ever declaring the switch — there is no correct way to know which variant a frame carries, and guessing emits three wrong values per right one. 12 of 214 signals are affected, all on DME2/DME3. The bus label takes gr26's node segment, in the topic and in node_id. On stock CAN the sender is implied by the arbitration id; the bus is the only routing fact the id cannot carry, and two buses have independent 11-bit id spaces, so it belongs in the natural key. TCM housekeeping frames are decoded here rather than from the DBC, since they never touched a physical bus. 0x201 follows TCM-987's 29-byte Pi layout — 4 cores, no GPU or power rails, plus the throttle byte — not TCM-26's 44-byte Jetson layout. Frames that cannot be decoded are still persisted with a status in metadata (unknown_can_id, short_frame, invalid_timestamp), which is how an unknown id gets reverse-engineered later. Adds /p987/dbc and /p987/dbc/:id so the decoder registry can be inspected without waiting for a frame to arrive, and a signal-level trace on the frame endpoints showing bit placement and scaling. Excludes the job/batch and shelter cold-storage path — this is the live path only.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
Matches how the rest of the services are set up.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8e5f121c79
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| logger.SugarLogger.Infof("[MQ] Received invalid can id: %s, ignoring", canID) | ||
| return | ||
| } | ||
| go HandleMessage(vehicleID, bus, int(canIDInt), payload) |
There was a problem hiding this comment.
Every valid CAN topic starts a new goroutine, while each handler can block on upload-key HTTP validation, ClickHouse inserts, and MQTT publishing. During normal high-rate CAN traffic, any period where one of those downstream systems processes frames more slowly than they arrive causes goroutines and retained payloads to accumulate without limit, eventually exhausting memory and dropping the entire live feed. Dispatch frames through a bounded worker pool or queue so ingestion has an explicit concurrency limit.
Useful? React with 👍 / 👎.
| return mapache.Vehicle{}, false | ||
| } | ||
|
|
||
| resp, err := http.Get(upstreamURL) |
There was a problem hiding this comment.
When authentication is enabled and the vehicle service accepts a connection but never returns headers, http.Get uses the default client without a request timeout and can block forever. Because the cache is only populated after this call finishes, subsequent frames for the same vehicle start additional stuck lookups, compounding the live-ingest outage and resource leak. Use a client with a finite timeout or a request context deadline.
Useful? React with 👍 / 👎.
…sing a DBC Replaces the runtime DBC parser with hand-declared mp.Message definitions, matching how gr26 describes its messages. Stock Porsche signals do not sit on byte boundaries, so a field here is the contiguous run of bytes its signals occupy and each signal is shifted and masked out of the field value — the same shape as gr26's TCMStatus bitfield, just used more widely. SCCM_SteeringAngle, 13 bits starting at bit 2 with a 0.175 scale, becomes a shift of 2 inside the two-byte field covering bytes 0-1. All 30 messages and 214 signals are declared, split by ECU the way gr26 splits by node. The transcription was checked against the previous DBC decoder before that code was deleted: every message, 200 random payloads each, 39,600 signal values compared, all matching. Four 64-bit signals (ECU_ID1_Bytes, ECU_Coding_Bytes, GW_Config_Data, GW_Network_Data) are declared as fields that export nothing. They are opaque identification and configuration blobs, not physical quantities, and 64 bits fits neither the int a field decodes into nor the float64 a signal carries. The bytes remain in the stored frame, which is where you would read them anyway. Same treatment gr26 gives its ULID field. Multiplexed signals on DME2 and DME3 are still not decoded, for the same reason as before: the DBC declares them without ever declaring the multiplexer switch. Message lookup is keyed on bus as well as id. Shelter injects frames at 0x210 and 0x211 through the relay's virtual CAN port, and 0x210 is also SCCM2 on the car's own bus — same id, different message, told apart only by which bus it arrived on. Drops the /p987/dbc endpoints and restores gr26's field-level trace on the frame endpoints.
1 participant
Mapache had no
p987anything, so the TCM-987 relay has been publishing into the void. This adds the cloud-side ingest for the live data path, mirroring gr26.p987/service: subscribep987/#(shared subscription) → validate upload key → decode → ClickHouse → republish decoded signals toquery/live/{vehicle}/{signal}forlivemapache_okbit and RTT workp987_canClickHouse table, same shape asgr26_can; signals go to the sharedsignaltable/p987/ping,/p987/messages/:id,/p987/signals/:iddocker-compose.yaml(port 7020),kerbecs.yaml, and ap987build workflowSignal definitions
Declared as
mp.Messagevalues the same way gr26 does, split by ECU the way gr26 splits by node —sccm.go,dme.go,psm.go,pdk.go,gateway.go,pas.go,klima.go. 30 messages, 214 signals, transcribed fromcayman_987.dbcin the TCM-987 repo.Stock Porsche signals don't sit on byte boundaries, so a field is the contiguous run of bytes its signals occupy and each signal is shifted and masked out of the field value — the same shape as gr26's
TCMStatusbitfield, used more widely.SCCM_SteeringAngle(13 bits at bit 2, ×0.175) becomes a shift of 2 inside the 2-byte field covering bytes 0–1.The transcription was checked against a DBC-driven decoder before that code was removed: every message, 200 random payloads each, 39,600 signal values compared, all matching.
Things worth knowing
ECU_ID1_Bytes,ECU_Coding_Bytes,GW_Config_Data,GW_Network_Data. They're opaque identification/config blobs, and 64 bits fits neither theinta field decodes into nor thefloat64a signal carries. Bytes stay in the stored frame. Same treatment gr26 gives its ULID field.m0–m3) but never declares the multiplexer switch, so there's no way to know which variant a frame carries. Worth fixing in the DBC.FillFromBytesrequires it, so a frame whose DLC differs from the declared message length recordsdecode_errorand decodes nothing. Stricter than the DBC decoder was.node_id. On stock CAN the sender is implied by the arbitration id — the bus is the only routing fact the id can't carry, and two buses have independent 11-bit id spaces, so it's part of the natural key. Signals are bus-prefixed (pcan_SCCM_SteeringAngle), which keeps the signal→frame join working.Undecodable frames are still persisted with a status in
metadata(unknown_can_id,decode_error,invalid_timestamp) so an unknown id can be reverse-engineered later.Verified end to end
Ran the actual TCM-987 relay against nanomq and this service:
tcm_mapache_ok = 1,tcm_mapache_ping = 2— the relay only sets that bit after receiving a fresh pong, so the round trip works in both directionspcan_SCCM_SteeringAngle = 175from a hand-computed frame, matching the expected 13-bit extraction and ×0.175 scalingRe-run after the refactor with identical results.
Not included
Job/batch handlers, Foreman workers, and the Epic Shelter cold-storage ingest — live path only, per scope.