fix: Han audit round 2 — SkillApi retention, accuracy gate, false-green assertions, race windows

[JarbasAl]

Round 2 of the Han audit. Provenance: adversarial validation of the round-1
fixes, a resilience sweep of the harness modules, and a test-quality audit of
the suite itself.

Every finding below has a regression test in test/unittests/test_audit_round2.py
that fails on the pre-fix code (verified by re-running the new suite against
the base commit: 18 of them fail there).

A. Memory retention (BLOCKER — root-caused and measured)

SkillApi.bus is a process-wide class attribute. SkillManager points it at
the harness FakeBus during boot and nothing ever put it back, so a stopped
MiniCroft stayed reachable through that bus and its handlers.

Measured cost: ~633MB retained per stopped MiniCroft. A suite that boots
several harnesses kept every one of them in memory until the process ended.

MiniCroft.__init__ now snapshots SkillApi.bus next to _original_sm_bus,
and stop() restores it. As defence in depth, stop() also removes the
listeners on its own bus and clears _handler_guards / _dedup_registrations.

Regression test: boot and stop two MiniCrofts, collect, and assert at most one
MiniCroft object survives (the first-boot residue from lazily built module
singletons is known and allowed).

B. Dead CI gate (BLOCKER)

--ovoscope-accuracy-min set config._ovoscope_accuracy_gate_failed in
pytest_terminal_summary and read it in pytest_sessionfinish. sessionfinish
runs FIRST, so the flag was always read before it was set: the gate could
never fail a CI run. A job at 0% accuracy printed "gate FAILED" and exited 0.

The gate is computed (and cached on the config) in pytest_sessionfinish;
pytest_terminal_summary only prints the cached result. The regression test
runs pytest in a subprocess with a seeded result and asserts the exit status.

C. Silent state corruption / false green

• The bus-coverage tracker wrapped bus.emit outside any try/finally. An
  assertion failure left the wrapper installed for the rest of the process and
  stacked one more per test. Now in a finally, and stop_tracking() restores
  only while bus.emit is still that tracker's own wrapper.
• Chaining a source_message onto a response with no session raised a bare
  KeyError; it now names the message that cannot be chained.
• OCPPlayerHarness started eight mock.patches before an unguarded
  OCPMediaPlayer(...), and PlaybackServiceHarness claimed TTS.queue and
  the _active singleton before its patcher started. A failure in the middle
  left process-wide state patched for the rest of the run. Both unwind now.
• ovoscope diff on a file with no expected_messages compared [] vs []
  and reported "Identical" with exit 0. It raises ValueError now, and
  _dict_diff uses a sentinel so an expected None differs from an absent key.
• The GUI assertions matched namespaces and pages by substring, so they could
  not fail on a near match. They compare by equality now (page by basename),
  with opt-in exact=False for the old behaviour. assert_namespace_cleared
  also matches gui.clear.namespace, the topic the GUI service really emits —
  it could not match any real message before.

D. Race windows (from adversarial validation of round 1)

• The mock TTS did if not self._stopped: bus.emit(...), which stop() could
  interleave. Both sides hold one lock now, so the flag flip and the emit are
  exclusive.
• CaptureSession counted an end-of-test message emitted OUTSIDE a capture
  window, so the next capture returned at once with an empty message list — a
  vacuous pass. A capture is armed under the same lock that resets the counter,
  and only the armed generation counts. The public API is unchanged.
• The default-session snapshot and restore could pair to_dict() output with
  deserialize(). The API family used at snapshot time is recorded and the
  matching loader is used. When the snapshot itself failed, the restore no
  longer degrades to a total no-op: active_skills is put back explicitly.
• PipelineHarness.__enter__ left a booted MiniCroft running if the wiring
  after the boot raised.

E. Resilience sweep

• apply_hotword_compat() patched HotWordEngine.__init__ permanently; it is
  now the hotword_compat() context manager.
• any(e.found_wake_word() ...) short-circuited over a destructive read, so
  later engines stayed latched and reported a stale detection on the next call.
• A reference-STT failure was silently scored wer=1.0, which reads as a TTS
  regression. It is logged and marked (transcript=None, transcribe_failed,
  transcribe_error in the report). Rendered file names use a sha1 prefix
  instead of a randomised 32-bit hash().
• A failed SKILL.md download now reports an error and exits 1.
• The except ImportError around AudioTransformersService covered the
  constructor too; it now covers only the import.
• MiniVoiceLoop.shutdown() detaches the bus capture handler.
• _wav_to_audio_data logs a WAV parse failure before falling back to raw PCM.
• CaptureSession.__del__ is a no-op when the MiniCroft has no bus.
• Real media-provider calls run under a timeout (call_timeout, default 30s).
• PipelineHarness: an explicit intent failure left the PREVIOUS match on the
  sink's _last_match, and match_result did not reset it between utterances.

F. Test-suite quality

test_capture_session.py and TestCaptureSessionDel booted a real MiniCroft
to exercise a class that only touches minicroft.bus; both use a
SimpleNamespace(bus=FakeBus()) stub now, matching the canonical race tests in
test_audit_round1.py.

New direct coverage for helpers that had none: E2EPipelineHarness
setUpClass/tearDownClass config restore (including two subclasses back to
back), wait_for_failure on timeout, the adapt register/detach round trip,
_dict_diff on nested dicts and lists of dicts, the CLI on a missing and on a
failing fixture, _parse_setup_py_entry_points on good and malformed input,
the PipelineHarness match/failure sequence, and an anti-vacuity guard that an
injected failure really does propagate out of End2EndTest.execute().

---

Fixes by Claude (opus), orchestrated by Claude Fable.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added configurable media-provider operation timeouts, with a 30-second default and an option to disable them.
  • Added richer transcription results, including failure status and error details.
  • Added flexible GUI assertion matching with exact, prefix, and substring options.

• Bug Fixes

  • Improved cleanup and state restoration after failed or interrupted audio, media, pipeline, and session operations.
  • Corrected accuracy-gate exit statuses, fixture validation, download failures, wake-word handling, and diff error reporting.
  • Improved handling of capture-session races and media-provider failures.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6d07d923-6ec7-4049-8c08-6eca50a1a249

📥 Commits

Reviewing files that changed from the base of the PR and between 3b83e76 and 60c8b1f.

📒 Files selected for processing (20)

• CHANGELOG.md
• ovoscope/__init__.py
• ovoscope/audio.py
• ovoscope/bus_coverage.py
• ovoscope/cli.py
• ovoscope/diff.py
• ovoscope/listener.py
• ovoscope/media.py
• ovoscope/media_provider.py
• ovoscope/pipeline.py
• ovoscope/pytest_plugin.py
• ovoscope/setup_skill.py
• ovoscope/tts_intelligibility.py
• ovoscope/voice_loop.py
• ovoscope/wakeword_probe.py
• test/unittests/test_audit_round1.py
• test/unittests/test_audit_round2.py
• test/unittests/test_capture_session.py
• test/unittests/test_end2end_extended.py
• test/unittests/test_gui_capture.py

---

📝 Walkthrough

Walkthrough

The change hardens ovoscope lifecycle cleanup, session restoration, capture handling, fixture validation, accuracy-gate exit status, GUI assertions, provider timeouts, wake-word processing, TTS reporting, setup failures, and regression coverage.

Changes

Audit round 2 hardening

Layer / File(s)	Summary
Lifecycle, session, and capture management ovoscope/__init__.py, test/unittests/test_capture_session.py, test/unittests/test_end2end_extended.py	MiniCroft restores bus and session state during teardown. Capture generations reject stale EOF events. End-to-end execution validates chained session propagation and stops coverage tracking.
Harness cleanup and state clearing ovoscope/audio.py, ovoscope/bus_coverage.py, ovoscope/media.py, ovoscope/pipeline.py, ovoscope/voice_loop.py, test/unittests/test_audit_round1.py	Harnesses clean up partial setup and clear stale state. Bus coverage preserves replacement wrappers. Voice-loop shutdown detaches capture handlers.
GUI matching assertions ovoscope/__init__.py, test/unittests/test_gui_capture.py	GUI assertions use exact namespace and page matching by default. Callers can request prefix or substring matching. Clear messages include clear.namespace.
Validation and exit-status handling ovoscope/diff.py, ovoscope/cli.py, ovoscope/pytest_plugin.py, ovoscope/setup_skill.py	Fixture validation distinguishes missing keys from None. CLI and skill installation failures return explicit errors. Accuracy-gate evaluation sets the session exit status during session finishing.
Provider, listener, and audio resilience ovoscope/media_provider.py, ovoscope/listener.py, ovoscope/tts_intelligibility.py, ovoscope/wakeword_probe.py	Provider operations support configurable timeouts. Listener errors are logged and constructor failures propagate. Wake-word latches are consumed across engines. TTS scores preserve transcription failure metadata and stable filenames.
Round-two regression coverage test/unittests/test_audit_round2.py, CHANGELOG.md	The regression suite covers lifecycle, validation, cleanup, matching, timeout, setup, and failure behavior. The changelog records the audit fixes.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• OpenVoiceOS/ovoscope#118: Related MiniCroft, session-restoration, bus, capture, and harness-cleanup changes.
• OpenVoiceOS/ovoscope#60: Related accuracy-gate and pytest exit-status handling.
• OpenVoiceOS/ovoscope#44: Related audio and MiniCroft teardown changes.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch audit/han-round2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

The results of your automated verification are here! 📜

I've aggregated the results of the automated checks for this PR below.

🔍 Lint

The automated pipeline is running smoothly. 🚂

❌ ruff: issues found — see job log

🏷️ Release Preview

Checking if we're ready for the big release. 🏁

Current: 1.6.2a2 → Next: 1.6.3a1

Signal	Value
Label	fix
PR title	fix: Han audit round 2 — SkillApi retention, accuracy gate, false-green assertions, race windows
Bump	build

✅ PR title follows conventional commit format.

---

🚀 Release Channel Compatibility

Predicted next version: 1.6.3a1

Channel	Status	Note	Current Constraint
Stable	⚪	Not in channel	-
Testing	❌	Too new (must be <1.0.0)	ovoscope>=0.7.2,<1.0.0
Alpha	✅	Compatible	ovoscope>=1.6.1a1

🔒 Security (pip-audit)

Checking for any insecure data transmissions. 📡

✅ No known vulnerabilities found (79 packages scanned).

⚖️ License Check

Legal eagle here! Checking those licenses. ⚖️

✅ No license violations found.

Policy: Apache 2.0 (universal donor). StrongCopyleft / NetworkCopyleft / WeakCopyleft / Other / Error categories fail. MPL allowed.

📋 Repo Health

A quick checkup for the repository! 🩺

✅ All required files present.

Latest Version: 1.6.2a2

✅ ovoscope/version.py — Version file
✅ README.md — README
✅ LICENSE — License file
✅ pyproject.toml — pyproject.toml
⚠️ setup.py — setup.py
✅ CHANGELOG.md — Changelog
✅ ovoscope/version.py has valid version block markers

🔨 Build Tests

Construction of your features is officially finished. 🏠

✅ All versions pass

Python	Build	Install	Tests
3.10	✅	✅	✅
3.11	✅	✅	✅
3.12	✅	✅	✅
3.13	✅	✅	✅
3.14	✅	✅	✅

📊 Coverage

Coverage report incoming! Every line counts. 🎯

❌ 58.6% total coverage

Files below 80% coverage (15 files)

File	Coverage	Missing lines
ovoscope/simple_listener.py	0.0%	63
ovoscope/tts_intelligibility.py	0.0%	200
ovoscope/version.py	0.0%	5
ovoscope/classic_listener.py	18.2%	117
ovoscope/intent_cases.py	21.4%	151
ovoscope/pytest_plugin.py	39.8%	237
ovoscope/cli.py	47.8%	132
ovoscope/ocp.py	47.9%	61
ovoscope/e2e.py	53.5%	66
ovoscope/media.py	57.0%	104
ovoscope/listener.py	57.1%	127
ovoscope/voice_loop.py	59.2%	118
ovoscope/__init__.py	62.3%	345
ovoscope/audio.py	63.4%	126
ovoscope/media_provider.py	67.6%	23

Full report: download the coverage-report artifact.

---

Your automated guardian for repository health 🛡️