fix(coordinator): re-arm the self-telemetry interval on failure - #335
Open
pfmos wants to merge 1 commit into
Open
fix(coordinator): re-arm the self-telemetry interval on failure#335pfmos wants to merge 1 commit into
pfmos wants to merge 1 commit into
Conversation
`get_self_telemetry()` sends the four-byte "self" form of CMD_SEND_TELEMETRY_REQ (opcode plus three reserved bytes, no pub key). Firmware answers it (MyMesh.cpp: `CMD_SEND_TELEMETRY_REQ && len == 4`), but a non-firmware companion need not: an openHop virtual companion up to and including 1.1.1 requires the 36-byte contact form and rejects the short frame with ERR_CODE_ILLEGAL_ARG. `_last_self_telemetry_update` advanced only in the success branch, so against such a node the interval gate never re-armed. The configured interval collapsed to the coordinator tick and the request was re-sent every cycle, forever, each one logged at ERROR. Measured on a live instance against an openHop companion: 357 requests in 30 minutes (~5 s apart) on a 300 s setting -- half of every companion frame the integration sent, none able to succeed -- and roughly 17k ERROR-tier log lines a day. The interval gates attempts, not successes, so record the attempt before issuing it. A failure now waits the configured interval like a success does; behaviour on a node that answers normally is unchanged. Also route failures through `_log_self_telemetry_error`, following the existing `_log_get_msg_error` pattern: an error code meaning the node can never satisfy the request (ILLEGAL_ARG, UNSUPPORTED_CMD) is reported once at WARNING naming the cause and the remedy, then demoted to DEBUG. Every other failure keeps ERROR on every occurrence, and a success clears the flag so a genuine later regression is reported afresh. Tests AST-extract the production helper as `test_get_msg_error_log_level` does, and add a structural guard asserting no assignment to `_last_self_telemetry_update` sits inside the request `try` block. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
get_self_telemetry()sends the four-byte "self" form ofCMD_SEND_TELEMETRY_REQ— opcode plus three reserved bytes, no pub key (meshcore_py/commands/device.py). Firmware answers it;MyMesh.cppsplits the command by frame length and has an explicitlen == 4self branch.A non-firmware companion need not. An openHop virtual companion up to and including 1.1.1 requires the 36-byte contact form and rejects the short frame:
_last_self_telemetry_updateadvanced only in the success branch, so against such a node the interval gate never re-armed. The configured interval collapsed to the coordinator tick and the request was re-sent on every cycle, forever, each one logged at ERROR.Measured on a live instance against an openHop companion, from the repeater's own frame log over 30 minutes:
0x27len=40x14len=10x0alen=10x38len=2The configured interval was 300 s. Half of every companion frame the integration sent was a request that could not succeed, and it produced roughly 17k ERROR-tier log lines a day.
Change
Re-arm the interval on failure. The interval gates attempts, not successes, so the attempt is recorded before it is issued. A failure now waits the configured interval exactly as a success does. Behaviour against a node that answers normally is unchanged.
Level the log by whether a retry could ever help. Failures route through a new
_log_self_telemetry_error, following the existing_log_get_msg_errorpattern. An error code meaning the node can never satisfy the request (ERR_CODE_ILLEGAL_ARG,ERR_CODE_UNSUPPORTED_CMD) is reported once at WARNING, naming the cause and the remedy, then demoted to DEBUG. Every other failure keeps ERROR on every occurrence. A success clears the flag, so a genuine later regression is reported afresh.Together: a node that cannot answer produces one actionable WARNING instead of ~17k ERRORs a day, and stops consuming a request slot every tick.
Note on the other side of this
openHop already fixed its half —
3513bab, "answer the self form of telemetry requests", 16 Jul 2026 — but it is ondevonly, not in any tag, anddevis ~340 commits ahead ofmain. Until that ships, released openHop builds still reject the frame. This change is also worth having independently: it makes any peer that refuses any optional command degrade quietly instead of hammering it.Tests
tests/test_self_telemetry_retry.py, followingtest_get_msg_error_log_level.py: the helper is AST-extracted from production source and exercised directly, sincecoordinator.pycannot be imported whole under the conftest stubs.ERR_CODE_UNSUPPORTED_CMDtreated the same wayNone) keep ERROR_last_self_telemetry_updatemay sit inside thetrythat issues the request — this fails on the pre-change source and passes after, so it pins the actual bug rather than the symptomVerified locally: the 5 new tests pass, and the collectible suite is unchanged versus clean
HEAD(104 passed vs 99, same 32 failures and 10 collection errors, all pre-existingTypeError: unsupported operand type(s) for |from PEP 604 syntax on the local Python 3.9 — CI runs a newer interpreter).