Skip to content

Drop blah2 tracker config and let it reach deployed nodes - #40

Merged
Purple10101 merged 2 commits into
mainfrom
20260918-let-tracker-disable-reach-nodes
Sep 19, 2026
Merged

Purple10101 merged 2 commits into
mainfrom
20260918-let-tracker-disable-reach-nodes

Conversation

@Purple10101

Copy link
Copy Markdown
Collaborator

Summary

Makes process.tracker.enable: false actually reach deployed nodes, and drops
the now-dead tracker config entirely. Pairs with
offworldlabs/blah2-arm#69, which removes blah2's built-in tracker.

The problem this fixes

config/default.yml has shipped process.tracker.enable: false since
2026-07-24 (46f7bc2), released in v0.4.2.0 and every tag since — its own
commit message cites memory growth as the reason.

It has reached zero nodes in nearly two months.

The merger seeds user.yml with a whole copy of default.yml on first boot,
and user.yml overrides default.yml. So every node on the estate has
enable: true frozen in its own overlay from whenever it first booted.
Confirmed on fairforest-b and owl-ded9: both merged configs said enable: true.

That tracker was leaking 2.00 MB/h — 100% of measured RSS growth, 18 days to
exhaust a 2 GB board with no swap. See the linked PR for the measurement.

migrate_remove_tracker() strips process.tracker and network.ports.track at
merge time. It joins the three existing migrate_*() functions that exist for
exactly this reason.

General lesson: changing config/default.yml alone changes nothing on any
deployed node.
It needs a matching migrate_*() in merge_config.py.

Design notes

Unconditional, unlike the migrations that compare against a LEGACY_*
value to let a deliberate choice survive. There is no deliberate choice to
protect: the code these keys configured no longer exists, so every value is
equally dead.

Drops the keys rather than rewriting them, leaving user.yml on disk
untouched. That keeps the migration load-bearing rather than a one-shot fixup,
and lets a Mender rollback regenerate the old config from the old
default.yml. Verified live: after the merger ran on owl-ded9, user.yml was
byte-identical to a pre-migration backup.

Verified live on owl-ded9

Dropping first-boot process.tracker from user config: blah2's built-in tracker no longer exists
Dropping first-boot network.ports.track from user config: nothing publishes or listens on it any more

Both keys gone from the merged config, user.yml untouched, retina_tracker
sidecar config intact, and the node ran 12.8 h overnight with the paired blah2
image at 0.167 MB/h against the 2.00 MB/h it replaces.

Deployment: three images move together

config-merger, blah2 and blah2-api must be deployed as a set. Against
a config this migration has stripped:

  • an old blah2 reads process.tracker.enable through ryml, whose default
    error handler calls abort() — it dies rather than warns. The docstring
    covers this.
  • an old blah2-api calls server_tracker.listen(config.network.ports.track)
    with the key gone and binds a random free port, silently. This one is not
    in the docstring and is worth adding: during verification /api/tracker kept
    answering 200 after the strip, purely because the API had not restarted.

Merge note

This branch was 3 behind main and has been merged up. main added
migrate_adsb_truth_server() in the same two places this branch adds
migrate_remove_tracker() — a function body and a call in main() — and git
collapsed two similarly-shaped tests in test_merge_config.py whose "does not
rewrite user.yml" cases share a middle. Resolved by keeping both sides
everywhere. 53 tests pass, covering both migrations.

🤖 Generated with Claude Code

Purple10101 and others added 2 commits September 18, 2026 21:32
blah2's built-in tracker has been removed, so process.tracker and
network.ports.track now configure code that does not exist.

Removing them from default.yml alone reaches no node. The merger seeds user.yml
with a whole copy of default.yml on first boot and user.yml overrides
default.yml, so every node on the estate persists the block that shipped when
it booted. That is exactly how 46f7bc2 came to sit unshipped: it set
process.tracker.enable: false on 2026-07-24, released in v0.4.2.0 and every tag
since, and reached zero nodes in nearly two months, so every node kept running
the tracker and paying for its unbounded nInactive growth.

migrate_remove_tracker() is unconditional, unlike migrate_doppler_span() and
migrate_tracker_forward(), which compare against a LEGACY_* value so a
deliberate choice survives. There is no deliberate choice to protect once the
code the keys configured is gone, and whole-block equality would skip exactly
the nodes whose M/N someone had tuned.

Safe to ship alongside the blah2 change: config-merger has restart "no" and
blah2 depends on it completing, so the merged config is regenerated from the
baked-in default.yml before blah2 may start, on every compose up. Mender does
that on rollback as well, where the old merger restores the old keys for the
old binary, and the migration never writes user.yml to disk. The cost is that
hand-swapping a pre-removal blah2 image onto an updated node will now abort on
the missing key, since ryml's default error handler calls abort().

Verified end to end against fairforest-b's real user.yml: the merged config
loses both keys and keeps fc, location, node_id, tracker_forward, ADS-B truth
and the detection and clutter settings.

Also drops the dead "3003:3003" mapping from the staging compose.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
main added migrate_adsb_truth_server() in the same places this branch adds
migrate_remove_tracker(): a function body and a call in main(). Git merged the
source cleanly but collapsed two similarly-shaped tests in
test_merge_config.py, whose "does not rewrite user.yml" cases have the same
middle. Resolved by keeping both sides everywhere: five migrate_*() functions,
both new calls, and both sets of tests.

53 tests pass on the result.
@Purple10101

Copy link
Copy Markdown
Collaborator Author

Pairs with offworldlabs/blah2-arm#69. Deploy config-merger, blah2 and blah2-api together: against a config this migration has stripped, an old blah2 aborts and an old blah2-api binds a random port.

@Purple10101
Purple10101 merged commit 0dddd31 into main Sep 19, 2026
1 check passed
@Purple10101
Purple10101 deleted the 20260918-let-tracker-disable-reach-nodes branch September 19, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants