Skip to content

Mutate plugins - #2295

Merged
amilcarlucas merged 6 commits into
masterfrom
mutate_plugins
Oct 30, 2026
Merged

amilcarlucas merged 6 commits into
masterfrom
mutate_plugins

Conversation

@amilcarlucas

Copy link
Copy Markdown
Collaborator

Description

Mutate plugins

Checklist

  • Run pre-commit checks locally
  • Verified by a human programmer
  • All commits are signed off (use git commit --signoff)
  • Code follows our coding standards
  • Documentation updated if needed
  • No breaking changes or properly documented

Testing

Describe how you tested these changes:

  • Unit tests pass
  • Integration tests pass
  • Manual testing performed
  • Tested on flight controller hardware

@amilcarlucas amilcarlucas added the AIReview Request an automated AI review; picked up by the reviewprs sweep label Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

☂️ Code Coverage

current status: ✅

Overall Coverage

Statements Covered Coverage Threshold Status
29851 28287 95% 89% 🟢

New Files

No new covered files...

Modified Files

No covered modified files...

updated for commit: dd767ed by action🐍

@github-actions

Copy link
Copy Markdown
Contributor

Test Results

     4 files  ±    0       4 suites  ±0   1h 11m 6s ⏱️ - 17m 24s
 8 855 tests +  275   8 846 ✅ +  272   3 💤 ±0  5 ❌ +2  1 🔥 +1 
35 212 runs  +1 100  35 145 ✅ +1 092  61 💤 +5  5 ❌ +2  1 🔥 +1 

For more details on these failures and errors, see this check.

Results for commit 6f2aa84. ± Comparison against base commit acad9f5.

This pull request removes 5 and adds 280 tests. Note that renamed tests count towards both.
tests.plugins.test_data_model_ahrs_orientation.TestAhrsOrientationDataModelCaptureAndEstimate ‑ test_model_identifies_ardupilot_preset_from_independent_samples[16-samples2]
tests.plugins.test_data_model_ahrs_orientation.TestAhrsOrientationDataModelCaptureAndEstimate ‑ test_model_identifies_ardupilot_preset_from_independent_samples[24-samples3]
tests.plugins.test_data_model_ahrs_orientation.TestAhrsOrientationDataModelCaptureAndEstimate ‑ test_model_identifies_ardupilot_preset_from_independent_samples[38-samples4]
tests.plugins.test_data_model_rc_calibration.TestRCMappingStaging ‑ test_failed_parameter_addition_rolls_back_preceding_additions
tests.plugins.test_data_model_servo_out ‑ test_a_frame_needing_too_many_outputs_is_rejected_before_assigning_any
tests.bdd_backend_safe_file_io ‑ test_denied_metadata_preservation_does_not_discard_the_saved_content[chmod]
tests.bdd_backend_safe_file_io ‑ test_denied_metadata_preservation_does_not_discard_the_saved_content[chown]
tests.bdd_backend_safe_file_io ‑ test_directory_open_failure_does_not_discard_a_completed_save
tests.bdd_backend_safe_file_io ‑ test_failed_update_preserves_the_original_and_removes_temporary_files[fsync]
tests.bdd_backend_safe_file_io ‑ test_failed_update_preserves_the_original_and_removes_temporary_files[replace]
tests.bdd_backend_safe_file_io ‑ test_failed_update_preserves_the_original_and_removes_temporary_files[writer]
tests.bdd_backend_safe_file_io ‑ test_new_file_uses_the_current_umask_without_changing_it[18-420]
tests.bdd_backend_safe_file_io ‑ test_new_file_uses_the_current_umask_without_changing_it[23-416]
tests.bdd_backend_safe_file_io ‑ test_overwrite_preserves_custom_permission_bits
tests.bdd_backend_safe_file_io ‑ test_relative_destination_is_saved_in_the_current_directory
…

@AP-Review

AP-Review commented Oct 28, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-10-28)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head 6f2aa84f8a.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/methodicconfigurator/2295/1.html#prMethodicConfigurator-2295

Test-only PR that widens mutation-testing scope. No blocking problems were found at 6f2aa84, and the changed tests pass. One new test cannot detect the failure it is written to catch.

Should fix

  • tests/test_backend_data_telemetry.py:887: the no-port shutdown test asserts not recwarn.list, but pytest records worker-thread exceptions only after the test body has finished. A crash in the heartbeat worker's port cleanup (e.g. getattr(master.port, "close", None)) still passes. Record exceptions with monkeypatch.setattr(threading, "excepthook", ...) and assert that list is empty.
Minor (2)
  • tests/test_backend_data_telemetry.py:842: rare flake. A periodic gcs-heartbeat tick between the forced MAVLink 1 send and the call_args check makes it read a MAVLink 2 packet. Hold master.mav.amc_send_lock across the send and the assert. The same applies at line 217: enter caplog.at_level("DEBUG") before installing flaky_write.
  • pyproject.toml:331: note. forkserver_warmup = "none" roughly halved mutmut throughput in a local two-module sample, and the configured suite is now ~32k mutants against the 230-minute step limit in mutation.yml. A workflow_dispatch run would confirm the weekly job still finishes.

Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
Catch inspector sorting, diagnostic assembly and file-save regressions,
and make asynchronous heartbeat assertions deterministic.

- Add the MAVLink inspector model and dedicated tests to mutation testing,
  and document the expanded scope.
- Distinguish every inspector sort key with independently ordered data.
  Cover empty startup, invalid headings in both directions, and stable
  tied rows across snapshot updates.
- Test legacy STATUSTEXT metadata, malformed chunk numbers preserving
  pending messages, and completion at chunk 255.
- Verify replacement-file ownership requests, contain directory-close
  errors, and preserve writer errors when temporary-file cleanup fails.
- Distinguish local and remote upload filenames in error messages and
  outcomes while verifying processing continues with later files.
- Check ESC requests at the exact one-second deadline and prevent
  catch-up bursts after polling pauses.
- Require nonblocking receives from both telemetry backends and isolate
  heartbeat scheduling from receive-contract tests.
- Capture heartbeat worker exceptions with threading.excepthook.
- Hold the encoder lock across MAVLink version assertions and enable
  DEBUG capture before installing the transient write failure.

Validation:
- 306 affected and related tests pass under Xvfb.
- Inspector audit improves from 38 to 40 killed out of 42 mutants,
  with two survivors and no timeouts or crashes.
- Focused checks kill 11 previously surviving file/diagnostic mutations.
- Separate focused proofs detect the ESC deadline survivor and
  blocking-read timeout mutation.

Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
@AP-Review

AP-Review commented Oct 30, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-10-30)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: ACCEPT

Reviewed at head 92d4daeb5d.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/methodicconfigurator/2295/2.html#prMethodicConfigurator-2295

No blocking problems at the new head 92d4dae, which still changes only tests, mutmut configuration and CONTRIBUTING.md.
Since the last review at 6f2aa84, the no-port heartbeat shutdown test now catches worker-thread crashes, and both heartbeat-test races are closed.

Minor (1)
  • pyproject.toml:333: mutation runtime note (still open). It stays informational: a manual workflow_dispatch run would show whether the weekly job fits in 230 minutes.

Add focused behavior tests to the largest remaining coverage gaps in the
full-suite HTML report, then strengthen assertions against mutation
survivors. Preserve production behavior and the original coverage data.

File browser:
- Add 72 cases for rename collisions and permissions, directory creation
  and cancellation, selection events, keyboard shortcuts and task errors.
- Prevent overlapping operations and transfers after planning failures.
- Report listing and deletion errors without actionable stale rows.
- Cover 99 missed lines and 43 branches; line coverage rises from 86.3%
  to 95.1%. Kill all 153 mutations audited across four handlers.

ESC availability and analysis:
- Add 84 cases using real in-memory log records and firmware metadata.
- Check missing and legacy telemetry, small positive error rates, report
  names, per-instance error peaks and their first matching timestamps.
- Test exact RPM observation and armed-coverage thresholds, zero initial
  timestamps, unclosed intervals and later flights without earlier data.
- Compare current across armed windows while excluding disarmed samples.
- Check motor-spin tolerances, suggested values and documented DShot
  multipliers, including the exact 1000 Hz warning boundary.
- Cover all 118 missed lines and 61 branches, bringing both to 100%.
- Kill 861 of 878 module mutations, retaining 17 reviewed equivalent
  survivors without suppressions.

Parameter editor table:
- Add 58 cases using real Tk controls and controlled event delivery.
- Verify case-insensitive filtering, single-match Enter behavior,
  explicit selection, shortcuts, double-click addition and cancellation.
- Check the exact bulk-add confirmation boundary and partial feedback.
- Verify initial bitmask selections, decimal previews, invalid/empty
  values, focus handling, reopening and replacement of row widgets.
- Clear the difference marker when an edit matches the FC value.
- Check manual locking of choice widgets and model-rejected additions.
- Verify row reservation, ordering, the final Add action, completion
  callbacks and rejection of obsolete render generations.
- Respect dismissed bitmask guidance and retain exception context in
  rendering diagnostics.
- Bound the existing batch-drain test so endless scheduling fails by
  assertion instead of timing out.
- Cover 98 missed lines; line coverage rises from 85.6% to 99.2%.
- Kill 541 of 632 audited dialog/rendering mutations, up from 114.
  The baseline rendering timeout is killed and no audited mutation lacks
  test associations. Leave 91 survivors unsuppressed for further review.

Add ESC analysis and the parameter table, together with their existing
unit/integration/acceptance tests, to the mutation configuration and
document the expanded scope.

Validation:
- 294 browser tests, 122 log-analysis tests and 190 parameter-table tests
  pass in separate focused runs under Xvfb.
- Pre-commit, Ruff, Pylint and git diff --check pass.
- Nine parallel mutation timeouts survive when rerun serially; final
  table results contain no timeouts or crashes.
- Preserve combined coverage reports and mutation review artifacts.

The table audit covers five methods rather than all 2336 generated
module mutations. Its survivors include presentation, platform and
uncovered defensive paths, as well as equivalent changes. The full
repository test suite and weekly mutation job were not rerun.

Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
@AP-Review

AP-Review commented Oct 30, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-10-30)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: REQUEST CHANGES

Reviewed at head aab17af48e.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/methodicconfigurator/2295/3.html#prMethodicConfigurator-2295

Two blocking problems at the new head aab17af: the new parameter-table tests fail in CI, and the new tests break the ty type-check gate.
Since the last review at 92d4dae, nothing earlier has regressed. The new commit adds tests for the file browser, ESC analysis and parameter table.

Blocking

  1. tests/test_frontend_tkinter_parameter_editor_table.py:108: 20 new add-dialog and bitmask tests depend on test order. They pass alone but fail in Ubuntu 3.11 and Docker CI and in mutation.yml's -x baseline step. The dialogs' tk.StringVar()/tk.BooleanVar() take no master and bind to tkinter._default_root, but mock_master makes a fresh tk.Tk() that becomes the default root only when no session root exists yet. As a result the search trace never filters (:2997, :3079) and the bitmask checkboxes never update (:3444). Reuse the session root, or monkeypatch tkinter._default_root to the fixture's Tk.
  2. tests/test_data_model_availability_models.py:315: the new tests add 8 ty check errors, so the Pylint workflow fails at its ty step before pylint runs. The errors are at :463 and :737 (dtype.names may be None), :512 and :548 (apm_doc may be None), file_browser.py:2952 (patch(..., **settings) overload), parameter_editor_table.py:136 (Misc.bind overload) and :3689 (minsize > 0 on float | str | Any). Fix with assert ... is not None, explicit patch keywords, a cast, and int(...).
Minor (1)
  • pyproject.toml:340: mutation runtime note (still open). This commit adds two more mutated modules, and no run has yet shown that the weekly job fits in 230 minutes.

Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
Make the table fixture's Tk interpreter the default while tests run,
then destroy it and restore the previous default root during cleanup.

Add a regression that initializes the session root first and verifies
that masterless search and bitmask variables update their controls.
This prevents dialog tests from depending on GUI test execution order.

Validate with all 4,642 mutation-baseline tests, the 191-test table
suite, ty, Ruff, Pylint, and pre-commit.

Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
@AP-Review

Copy link
Copy Markdown

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: ACCEPT

Reviewed at head dd767ed7d0.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/methodicconfigurator/2295/4.html#prMethodicConfigurator-2295

No blocking problems at the new head dd767ed, which still changes only tests, mutmut configuration and CONTRIBUTING.md.
Since the last review at aab17af, both blockers are fixed: the parameter-table tests now pass after a session root (and hosted Ubuntu 3.11 pytest is green), and ty check is clean on the changed tests.

Minor (1)

@amilcarlucas
amilcarlucas merged commit ea6bd3f into master Oct 30, 2026
28 of 31 checks passed
@amilcarlucas
amilcarlucas deleted the mutate_plugins branch October 30, 2026 23:07
@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 41111479025

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage increased (+1.2%) to 94.791%

Details

  • Coverage increased (+1.2%) from the base build.
  • Patch coverage: No coverable lines changed in this PR.
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 29851
Covered Lines: 28296
Line Coverage: 94.79%
Relevant Branches: 9002
Covered Branches: 8030
Branch Coverage: 89.2%
Branches in Coverage %: No
Coverage Strength: 2.83 hits per line

💛 - Coveralls

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIReview Request an automated AI review; picked up by the reviewprs sweep

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants