fix: resolve unbounded get_response() hang on FakeBus - #130
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
I've gathered some intelligence on your latest changes. 🕵️♀️I've aggregated the results of the automated checks for this PR below. 🔍 LintA detailed summary of the latest automation run. 📝 ❌ ruff: issues found — see job log 🏷️ Release PreviewEvaluating the impact of these changes on our release schedule. 📉 Current:
✅ PR title follows conventional commit format. 🚀 Release Channel Compatibility Predicted next version:
🔒 Security (pip-audit)Our digital defenses have been updated. 🛡️ ✅ No known vulnerabilities found (79 packages scanned). 📋 Repo HealthEvaluating the longevity of the project. 🌳 ✅ All required files present. Latest Version: ✅ ⚖️ License CheckScanning for any potential trademark infringements. ™️ ✅ No license violations found. Policy: Apache 2.0 (universal donor). StrongCopyleft / NetworkCopyleft / WeakCopyleft / Other / Error categories fail. MPL allowed. 🔨 Build TestsFrom source to binary, let's see how it holds up. 🧱 ✅ All versions pass
📊 CoverageMeasuring the reach of our test cases. 📏 ❌ 59.0% total coverage Files below 80% coverage (15 files)
Full report: download the Generated with ❤️ by OVOS Automations |
) OVOSSkill.get_response()/ask_yesno() (ovos-workshop 7.0.6) spawn a killable background thread that re-prompts forever when num_retries=-1 (the default) and nothing ever answers it. On a real voice satellite the listener/GUI eventually emits mycroft.skills.abort_question on user silence; ovoscope's synchronous FakeBus never generates one, so any skill handler that calls get_response()/ask_yesno() without a queued follow-up utterance hangs the calling thread indefinitely (OVOSSkill._wait_response(), ovos_workshop/skills/ovos.py ~1802-1809 - "while not ans: time.sleep(0.1)", no deadline). MiniCroft now arms a short watchdog timer on "skill.converse.get_response.enable" that emits the existing "mycroft.skills.abort_question" bus message (no new message type) if ".disable" hasn't fired first - mirroring what a real listener/GUI would send on silence. This lets get_response()/ask_yesno() resolve to None promptly instead of looping every get_response_timeout (20s default) forever. Watchdog timers are keyed by (skill_id, session_id) - the same two-part scope ovos-workshop's own @killable_event("mycroft.skills.abort_question", check_skill_id=True) already uses to decide which stalled thread an abort is for. An adversarial review of an earlier version of this fix caught a real regression here: a flat list of pending timers meant ANY skill's ".disable" cancelled EVERY other skill's still-pending watchdog too, so in a fleet-style MiniCroft running multiple skills concurrently, one skill finishing its (answered) get_response() would silently disarm a different skill's watchdog - if that second question was never answered, it hung forever again, defeating the fix for exactly the multi-skill case it needs to hold up under. Scoping by (skill_id, session_id) and cancelling only the matching entry on ".disable" fixes it; all remaining timers are still cancelled in stop() the same way the existing mock-TTS timers already are. The existing nested-speak_dialog(wait=True) mock-TTS handshake (recognizer_loop:audio_output_start/end) already resolves correctly for nested calls via its own per-call Timer; a regression test pins that behavior alongside the get_response fix and its concurrency scoping. Field evidence: OpenVoiceOS/ovos-skill-alerts#138 "Update 3" - py-spy captures showing OVOSSkill._real_wait_response threads parked forever on the ovoscope FakeBus, and PR #123's CI job silently hanging to the 30-minute kill on 4 of 5 Python versions after the last test finished. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
85a5616 to
7f05d00
Compare
Revised after adversarial review — see "Concurrency defect found in review" below.
Field evidence
OpenVoiceOS/ovos-skill-alerts#138 ("Update 3") root-caused, via py-spy stack
captures on a real CI hang, two mechanisms by which real skill handlers
block forever under ovoscope's synchronous FakeBus:
wait_while_speaking()(ovos_bus_client/session.py:710) burnsits full 15s timeout for nested/cascading
speak_dialog(..., wait=True)calls if
recognizer_loop:audio_output_endnever arrives. Alreadymitigated in
ovoscope/__init__.py's_mock_ttsmock-TTS handler(per-call
threading.Timerunduck) — verified still correct for nestedcalls (see regression test below).
OVOSSkill._wait_response()(ovos_workshop/skills/ovos.py~1802-1809,
ovos-workshop==7.0.6) has no deadline at all:while not ans: time.sleep(0.1). With theget_response()/ask_yesno()default
num_retries=-1, the background_real_wait_responsethreadre-prompts forever if nothing ever answers it. A real voice satellite
eventually emits
mycroft.skills.abort_questionon user silence;ovoscope's FakeBus never did. This is what leaked non-answered-thread
state into
ovos-skill-alerts' shared-MiniCroft e2e suite and made CIhang to the 30-minute job kill on 4 of 5 Python versions on PR fix: Han audit round 2 — SkillApi retention, accuracy gate, false-green assertions, race windows #123.
Fix
MiniCroftarms a short watchdogthreading.Timeronskill.converse.get_response.enablethat emits the existingmycroft.skills.abort_questionbus message (no new message type — this isthe same message a real listener/GUI sends on cancel/silence, and
OVOSSkill._real_wait_responseis already decorated with@killable_event("mycroft.skills.abort_question", ...)to handle it) if.disabledoesn't fire first..disable(emitted onceget_response()actually returns, answered or not) disarms the watchdog. Tracked/cancelled
in
stop()the same way the existing mock-TTS timers already are, so noorphaned timer can fire onto a closed bus and poison a later test.
Concurrency defect found in review
The first version of this fix kept ONE flat list of pending watchdog
timers. Adversarial review reproduced a real regression: with two
concurrent
get_response()flows in flight (skill A and skill B, afleet-style MiniCroft running multiple skills), skill A's
.disabledrained and cancelled the entire list, including skill B's still-armed
watchdog — if B's question was never answered, B hung forever again,
defeating the fix for exactly the multi-skill case it needs to hold up
under. Repro that pinned it:
enable(A),enable(B),disable(A)→0timers left instead of
1.Fixed by scoping each watchdog by
(skill_id, session_id)— the sametwo-part scope
ovos_workshop's own@killable_event("mycroft.skills.abort_question", check_skill_id=True)already uses to decide which stalled thread an abort is actually for.
.disablenow cancels only its own(skill_id, session_id)entry; theabort emit carries the same
skill_id/session context so only thestalled question dies.
stop()still drains and cancels every remainingentry on teardown.
Smallest correct change:
ovoscope/__init__.pyonly (MiniCroft),mirroring the existing
_mock_ttstimer-tracking pattern already in thefile.
Red → green proof
test/unittests/test_tts_lifecycle_nested_and_get_response.py:TestNestedSpeakDialogWait/TestUnansweredGetResponse— the originaltwo regression tests. Before the fix: both hang (a 90s
timeoutkill wasneeded to get a shell back). After: both pass in ~3s.
TestConcurrentGetResponseWatchdogScoping(new, pins the review finding):test_disabling_one_skill_does_not_cancel_another_skills_watchdog—whitebox, drives the
enable/disableprotocol messages directly.On the flat-list version this FAILS:
AssertionError: 1 != 0(skill B's watchdog is gone after only skill A disabled). On the fixed
version it passes.
test_unanswered_skill_still_aborted_when_another_skill_answers_first— end-to-end: two skills, two sessions, concurrent
get_response()calls; skill A is answered via an injected utterance, skill B is
deliberately left unanswered. On the flat-list version this hangs
(verified via a 20s
timeoutkill on the whole test class). On thefixed version: skill A's real answer survives untouched, skill B is
aborted by its own watchdog and resolves to
None— both complete in~7s total for the 4-test file.
Full
ovoscopesuite (after fix)That one failure is a pre-existing order-dependent flake, unrelated to
this change: it passes in isolation and doesn't touch
get_response/speak_dialogmocking at all. Reproduces the same way withthe fix reverted.
Real-world gate: ovos-skill-alerts PR #123
Checked out
OpenVoiceOS/ovos-skill-alertsPR #123(
feat/ovoscope-tests) into a separate throwaway clone/venv (Python3.14 in this sandbox), installed this branch's
ovoscopevia local path,and ran its
test/end2endsuite (ovos-workshop/ovos-core/padaciosopipeline —
ovos-padatiousstill not installable on this box, matchingthe maintainer's own note in #138).
job hangs to the 30-minute
timeout-minuteskill.30 failed, 3 passed, 133 skipped in ~12s— terminates, no hang. The failures are thealready-documented, unrelated auto-generation defects from fix: raise clear TypeError for bare-string expected_messages #138 Update 1
(utterances that don't satisfy the pinned adapt pipeline's
.require(...)vocab) — out of scope here.Note: this sandbox's exact pass/fail counts differ from the numbers seen
in review logs on a Python 3.11 +
pytest-xdistsetup (1 failed, 32 passed, 133 skipped in ~24s) — re-ran withpytest-xdisthere too andstill saw the pre-existing vocab-mismatch failures, just distributed
differently across workers (
30 failed, 2 passed, 52 skipped in ~19s, same"terminates, no hang" outcome). The discrepancy tracks Python version /
worker distribution affecting which of the already-documented, unrelated
vocab-pinning tests execute — it does not affect the property this PR
claims: the suite reliably terminates instead of hanging.
Scope
Only
ovoscope/__init__.py(MiniCroft) changed, plus the regressiontest. No new bus message types. No changes to
ovos-workshop(out ofscope for this repo — the
_wait_response()no-deadline issue is aseparate upstream defect worth its own report there, not fixed here).