Release 1.6.3a1 - #124
Open
github-actions[bot] wants to merge 50 commits into
Open
Conversation
Added AI disclosure section to README.md outlining the use of AI tools in project development and the maintenance of public records.
Add a self-contained probe that drives a real HotWordEngine over a single clip the way the listening loop does: a few seconds of leading silence to warm the engine's rolling feature window (openWakeWord et al. only emit once it is full, ~2.5 s — too little lead silently drops short positives and biases false rejects), then the clip streamed frame by frame, returning a detection decision plus latency and frames-to-detection. Unlike MiniVoiceLoop it needs no bus or [listener] extra — just the [bench] extra (numpy). Tolerates the HotWordEngine(lang) signature and the vestigial found_wake_word(frame) arg. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat: per-clip WakeWordProbe for benchmark/test harnesses
…us-on-stop fix: restore SessionManager.bus when MiniCroft stops
End2EndTest.execute() and from_message() only stopped the MiniCroft on the success path, so a failing assertion left SessionManager.bus, default_session and Configuration patched for every later test. Both now run stop() from a finally block. MiniCroft snapshots the whole default Session at boot and restores it in stop(), so inject_active activations and wire-folded session values no longer outlive the test that made them. Mock-TTS unduck timers are tracked, made daemon and cancelled in stop(). An orphaned timer could otherwise emit onto a closed bus and fold a stale session onto the global SessionManager during a later test. CaptureSession resets its eof state atomically, records a timed_out flag, and returns a copy from finish(). A capture timeout now fails with a clear message instead of surfacing as a message-count mismatch. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BusCoverageTracker snapshotted the session-cumulative global collector and added it into per-test counts, so every later test inherited the invocations of every earlier one. The snapshot is now a baseline and the report uses the delta over the tracker's own lifetime, frozen at start_tracking() so the tracking window is not counted twice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cmd_run booted a MiniCroft but never assigned it to the test, so execute() booted a second managed one and both patched the same globals. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
match() treated mycroft.skill.handler.start as a failure signal, but it fires on a SUCCESSFUL match — so a successful match returned None. It also checked the failure flag before the captured message and spun a watcher thread that polled at 20Hz forever after a timeout. match_result() now returns a discriminated matched/no-match/timeout outcome and waits on the events directly. assert_no_match() fails on a timeout instead of passing vacuously; match() keeps its old signature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The docstring told callers to emit after calling the helper, which is impossible single-threaded because the helper blocks. It now takes an optional emit= message and sends it once the handlers are in place. A match that raced an intent failure could also be dropped; appends are guarded by a lock and re-read once before giving up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The side effect inspected `mock.url` on a MagicMock, so no configured URL
ever matched and json() always returned {}. It now lives on the patched
GET, which receives the URL.
OCPTest also waits for ovos.common_play.query.response instead of sleeping
half the timeout, and stops the MiniCroft from a finally block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AudioServiceHarness.__exit__ skipped bus.close() when shutdown() raised. ListenerHarness and MiniListener left their wildcard "message" capture handler on the bus, so a shared bus kept feeding a dead harness. PlaybackServiceHarness now restores the TTS.queue object it replaced and refuses a second concurrent harness, because TTS.queue is process-wide class state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A load failure was warned about and skipped, then resurfaced much later as an unrelated assert_emitted timeout. Loading now raises by default; pass tolerate_load_errors=True to keep going, in which case the errors are kept in load_errors and quoted in assert_emitted failures. MiniPHAL.__exit__ also detaches its capture handler and closes the bus. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cached MiniCrofts were never stopped and two could be live at once, each clobbering the same globals. At most one stays live now, and an atexit hook stops the rest. _wait_for_m2v_sync removes its three listeners in a finally block and only pays the 3.5s pad when no m2v activity was observed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feed_file ignored the join() result, so a listener thread that outlived its stop() kept appending to _messages during the next run. A still-alive thread is now logged and replaced with a fresh listener object. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`except (ImportError, Exception): pass` turned a malformed pyproject.toml into an understated coverage number. TOMLDecodeError and OSError are now caught explicitly and recorded in EcosystemCoverageReport.parse_errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The client was left in place on a ConnectionError, so its reconnect thread lived for the rest of the process. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One test per defect, each written to fail against the pre-fix code: teardown on the failure path, default-session isolation, TTS timer lifecycle, bus-coverage deltas, CaptureSession races, pipeline match verdicts, wait_for_match subscription order, the OCP HTTP mock, harness teardown, PHAL load errors, coverage parse errors and the RemoteRecorder connect leak. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
raise-by-default made the old warn-and-skip expectation wrong; cover both the default raise and the tolerate_load_errors opt-out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI showed two gaps: the session restore bailed out when boot replaced the default-session singleton, leaking exactly the state it exists to scrub — restore now targets whatever object holds the role at stop() time. And on Python 3.10 there is no stdlib tomllib, so a malformed pyproject.toml was silently ignored — depend on the tomli backport there. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ovos-bus-client 1.x has serialize/deserialize, 2.x to_dict/from_dict; the snapshot silently became None on 1.x and the restore no-opped. Support both and warn instead of failing silently. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix: Han audit round 1 — teardown safety, session isolation, harness lifecycle
…n available cmd_validate always ran _basic_validate, contradicting docs/cli.md which documented it as preferring pydantic_helpers.validate_fixture when the pydantic extra is importable. Match the code to the documented behaviour instead of weakening the docs, and add tests covering both the pydantic-available and fallback paths. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- cli.md: document the bus-coverage subcommand and the ovoscope-setup console script - minicroft.md: add modernize/emit_legacy to the MiniCroft constructor table - add docs/e2e-pipeline-harness.md (E2EPipelineHarness, bus helpers, intent-registration shims) and docs/intent-cases.md (IntentCase/register_intent_case_tests), linked from docs/index.md, README, and setup_skill.py's downloaded docs list - add FAQ.md and CONTRIBUTING.md; repair the truncated AI Disclosure section in README (it referenced a changelog file that never shipped) - index.md: scope the "does not load PHAL/audio" claim to MiniCroft/End2EndTest and point to the dedicated phal.md/audio-testing.md harnesses, which do cover them - replace stale ovoscope/*.py:<line> citations across docs/*.md with symbol references so they can't drift out of sync with the source Automated documentation audit provenance: gaps found by a Han documentation audit (round 1) against current dev source. Docs fixes by Claude (sonnet), orchestrated by Claude Fable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…idate validate_fixture skips absent sections entirely, so replacing the basic checks with it let a fixture with no expected_messages pass; and CI (where pydantic is installed) rejected the legacy dict-shaped source_message. Basic checks always run first, pydantic validates each message on top, and a legacy dict section is treated as a one-message list. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs: close the documentation audit gaps (Han audit round 1)
Memory (BLOCKER): SkillApi.bus is a process-wide class attribute that
SkillManager points at the harness FakeBus during boot. Left there after
stop(), it pinned the whole MiniCroft object graph — ~633MB per stopped
instance — so a suite that boots several harnesses kept all of them alive.
stop() now restores it and clears the handlers still on the harness bus.
Bus-coverage tracker: start_tracking() wraps bus.emit. An assertion failure
between start and stop left the wrapper installed for the rest of the process
and stacked one more wrapper per test. The tracked block is now in a
try/finally.
Session chaining: a response with no session in its context raised a bare
KeyError. It now says which source_message cannot be chained, and why.
CaptureSession: an end-of-test message emitted OUTSIDE a capture window
counted towards the next capture, which then returned at once with an empty
message list — a vacuous pass. A capture is now armed under the same lock
that resets the counter, and only an armed generation counts. __del__ is a
no-op when the MiniCroft has no bus.
Mock TTS: `if not self._stopped: bus.emit(...)` was a TOCTOU against stop().
Both sides now hold one lock, so the flag flip and the emit are exclusive.
Default session: the snapshot and the restore now use the SAME bus-client API
family (to_dict/from_dict or serialize/deserialize) — pairing them across
families rebuilt a wrong session. When the snapshot itself failed, the restore
no longer degrades to a total no-op: active_skills is put back explicitly.
GUI assertions: namespace and page comparisons used substring matching, so
they could not fail on a near match ("weather" passed on any namespace
containing it). They compare by equality now, with opt-in exact=False for the
old behaviour. assert_namespace_cleared also matches gui.clear.namespace, the
topic the GUI service really emits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The --ovoscope-accuracy-min gate set its failure flag in pytest_terminal_summary and read it in pytest_sessionfinish. sessionfinish runs FIRST, so the flag was always read before it was set and the gate could never change the exit status — a CI job with 0% accuracy still exited 0. The gate is now computed (and cached on the config) in sessionfinish; terminal_summary only prints the cached result. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
diff: a file without "expected_messages" fell back to an empty list, so two unrelated JSON files compared []-vs-[] and the CLI reported "Identical" with exit 0. It now raises ValueError, and the CLI turns that into a clean error. _dict_diff uses a sentinel, so an expected None no longer matches an absent key. bus_coverage: stop_tracking() restores bus.emit only while it is still this tracker's own wrapper, so it cannot clobber another tracker's wrapper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
OCPPlayerHarness started eight mock.patches before constructing the real player; PlaybackServiceHarness claimed TTS.queue and the _active singleton before its patcher started; PipelineHarness booted a MiniCroft and then wired the sink outside any guard. A failure in the middle left process-wide state patched for the rest of the run. Each __enter__ now unwinds through its own teardown before propagating. PipelineHarness also clears the sink's stale verdict: an explicit intent failure left the PREVIOUS match on _last_match, and match_result did not reset it between utterances. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- wakeword_probe: apply_hotword_compat() patched HotWordEngine.__init__ for the rest of the process. It is now a `hotword_compat()` context manager scoped to the engine construction it exists for. - listener: `any(e.found_wake_word() ...)` short-circuits, and found_wake_word() is a destructive read — later engines stayed latched and reported a stale detection on the NEXT call. Every latch is read first. - listener: the except ImportError around AudioTransformersService covered the constructor too, so a constructor bug surfaced as "install ovos-dinkum-listener". The guard now covers only the import. - listener: a WAV that will not parse is logged before falling back to raw PCM. - voice_loop: shutdown() detaches the bus capture handler, so a harness torn down without the context manager stops collecting messages. - media_provider: real-provider calls run under a timeout (call_timeout, default 30s) instead of hanging the run on an unresponsive server. - tts_intelligibility: a reference-STT failure is logged and marked (transcript=None plus transcribe_failed / transcribe_error in the report) instead of silently scoring wer=1.0 as if the TTS were unintelligible. Rendered file names use a sha1 prefix, not a randomised 32-bit hash. - setup_skill: a failed SKILL.md download reports an error and exits 1 instead of leaving a half-installed skill and claiming success. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds test_audit_round2.py — one adversarial test per audit finding, plus direct coverage of helpers that had none: E2EPipelineHarness config restore (including two subclasses back to back), wait_for_failure on timeout, the adapt register/detach round trip, diff._dict_diff on nested dicts and lists of dicts, the CLI on a missing and on a failing fixture, _parse_setup_py_entry_points on good and malformed input, the PipelineHarness match/failure sequence, and an anti-vacuity guard that an injected failure really does propagate out of End2EndTest. test_capture_session.py and TestCaptureSessionDel now drive a SimpleNamespace(bus=FakeBus()) stub instead of booting a MiniCroft: the class only touches minicroft.bus, and each boot retained hundreds of MB. The GUI tests asserted namespaces that only CONTAINED the expected value — exactly the false green being fixed — so they now assert the real namespace, with new cases proving a near match fails. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The test bypassed __init__ with __new__ and set only the attributes match_result used at the time, so it broke the moment match_result read one more attribute. Constructing the object normally keeps it honest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix: Han audit round 2 — SkillApi retention, accuracy gate, false-green assertions, race windows
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Human review requested!