imx93-jaguar-eink: BLE advertising 'ghost/on-air starvation' after long Wi-Fi uptime (BlueZ reports active but undiscoverable)

[ajlennon]

Summary

On eink-0167 (eink factory target 25 + PR #38 watchdog), after ~3h of Wi-Fi-connected uptime the board became undiscoverable over BLE from the lab host, while BlueZ still reported ActiveInstances=1. A systemctl restart improv immediately restored a fresh, discoverable advertisement.

This is an on-air / "ghost advertising" state: the advertisement is registered (so is_advertising() looks healthy) but nothing is actually radiating. The current watchdog (PR #38) only recovers from registration drops / stack wedges (ActiveInstances==0 or unresponsive D-Bus), so it cannot detect or recover this case.

Evidence

• Reproduced once at ~3h uptime, Wi-Fi connected. ActiveInstances=1, improv.service healthy, no errors in journal; XPS scanner saw nothing until a service restart.
• Earlier 48-min soak (Wi-Fi on, idle + saturated, both bands, serial-driven) was 24/24 discoverable, 0 misses — so this is a long-uptime effect, not reproducible in short runs.
• With Wi-Fi OFF, advertising has been 100% reliable over long soaks (primary fresh-onboarding path is unaffected). Points at Wi-Fi/BT coexistence on the shared IW612 radio (shared-antenna topology) as a contributing factor, though the ghost state has also been seen around aborted BLE connects / fresh boot with Wi-Fi off.

Impact

Affects re-onboarding an already-provisioned board (Wi-Fi up). Fresh factory onboarding (Wi-Fi off) is unaffected. Onboarding must never depend on a manual restart, so this needs a self-healing fix.

Mitigation in progress

A parallel "ghost-advertising bounce" (ADVERT_BOUNCE_SECS, default 60s) is WIP on the chore/layer-tidy branch: it periodically re-asserts (unregister + re-register) the advertisement even while BlueZ reports it active, since ActiveInstances can't tell us whether the advert is truly radiating. A bounce is a sub-second re-register gap, skipped whenever a central is mid-session. This needs to be reconciled with PR #38's Bugbot-fixed watchdog (the two watchdog lines have diverged) and then soak-tested at multi-hour uptime.

Follow-ups

• Reconcile the chore/layer-tidy bounce work with main's PR python3-improv: self-healing BLE advertising watchdog (imx93-jaguar-eink) #38 watchdog, land as a focused PR.
• Multi-hour (>3h) Wi-Fi-connected soak to confirm the bounce recovers the ghost state.
• Investigate IW612 PTA/coexistence config (shared-antenna) with NXP FAE if the bounce is insufficient.

Related: meta-dynamicdevices PR #38 (watchdog), PR #41 (Device-Status superset).

[ajlennon]

Update: bounce mitigation is merged on main

The periodic re-assert ("bounce") mitigation for this ghost/on-air state is now on main — it landed in commit e6202ac (PR #41), whose branch was built on top of the earlier watchdog WIP, so the merge brought both the Device-Status superset and the bounce.

What the mitigation does (onboarding-server.py, advertising_watchdog):

• ADVERT_BOUNCE_SECS (default 60s, IMPROV_ADVERT_BOUNCE_SECS): even while BlueZ reports ActiveInstances>=1, the watchdog periodically bounces the advertisement (_reassert_advert: stop_advertising → start_advertising), so an on-air stall self-heals within ~a minute without a manual restart.
• Still also recovers registration drops (ActiveInstances==0) and exits for a systemd restart after ADVERT_MAX_FAILURES unresponsive ticks.
• Both re-assertions are skipped while a central is mid-session so onboarding is never disrupted; the bounce clock resets on disconnect.

Why this is a workaround, not a fix: ActiveInstances can't tell us whether the advert is truly radiating, so we can't detect the stall — we just re-assert on a timer. The root cause (why BlueZ ghost-advertises after long Wi-Fi uptime; suspected IW612 Wi-Fi/BT coexistence on the shared-antenna topology) is unresolved.

Remaining work (keeping this issue open):

• Build/OTA: eink-0167 runs target 25 (pre-python3-improv: emit Device-Status JSON superset (net/time/sec) for imx93-jaguar-eink #41) so the running improv.service does not yet have the bounce — needs the next eink image to take effect.
• Long-uptime soak (>3h, Wi-Fi connected) on the new build to confirm the bounce clears the ghost state on-air.
• Root-cause: IW612 coexistence / antenna topology (FAE follow-up).