Skip to content

Mutate plugins #2295

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
amilcarlucas merged 6 commits into from
Oct 30, 2026
Merged

Mutate plugins #2295

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c30b931
test(plugins): Added mutation tests to backends and data_models
amilcarlucas Oct 28, 2026
380f89f
test(data_models): Added more mutation tests
amilcarlucas Oct 28, 2026
92d4dae
test: expand mutation coverage for telemetry and file workflows
amilcarlucas Oct 30, 2026
aab17af
test: harden file browser, ESC analysis and parameter table workflows
amilcarlucas Oct 30, 2026
6f5afbe
fix(ty): fix ty linting issues
amilcarlucas Oct 30, 2026
dd767ed
test: isolate parameter table Tk roots
amilcarlucas Oct 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 47 additions & 9 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,15 +87,25 @@ SITL_BINARY="$PWD/sitl/arducopter" PATH="$PWD/.venv/bin:$PATH" \
### Mutation testing

The optional [Mutmut](https://mutmut.readthedocs.io/en/latest/) suite checks whether
existing tests detect small changes to thirty-eight modules covering battery voltages,
existing tests detect small changes to modules covering battery voltages,
parameter validation and serialization, firmware version parsing, parameter
history, AutoTune acceleration/gain backoff, IMU helpers, firmware upload,
log-analysis orchestration, usage popups, file browsing and task scheduling, motor-test UI,
bootloader protocol handling, schema-validated filesystem IO, IMU log analysis,
vehicle-component validation/display, flight-controller business rules, transfer
progress accounting, configuration expression evaluation, firmware parameter
migrations, battery-monitor telemetry and status handling, and configuration-step
processing. It runs on Linux, macOS, or Windows
processing. All plugin backends and data models are included, together with shared
data/ESC telemetry models, telemetry and radio backends, and flight-controller
command handling used by calibration and motor-testing plugins. External parameter
upload selection, configured file-upload policy, atomic file replacement, and
flight-controller diagnostic assembly and device rows are also included, together
with MAVLink inspector sorting, source identifiers, and decoded-field presentation.
ESC log analysis covers armed-period RPM and current comparisons, per-output error
peaks, motor-spin margins, and documented DShot output rates.
Parameter-table tests cover add-dialog filtering and selection, bulk-add feedback,
bitmask editing, and completion of batched rendering.
It runs on Linux, macOS, or Windows
through WSL.

Install it into the project virtual environment:
Expand All @@ -119,6 +129,14 @@ and a working desktop display, without `xvfb-run`.

The `[tool.mutmut]` section in `pyproject.toml` selects both the modules to mutate
and the tests to run. Update those lists together when expanding the suite.
Plugin `backend_*.py` and `data_model_*.py` targets use globs, and the entire
`tests/plugins/` directory is selected, so newly added plugin implementations and
their tests are included automatically. Shared implementations outside the plugin
directory and their tests must still be added explicitly.
The command module's source-lint regression is deselected only for mutation runs:
it checks the hand-written source, while Mutmut generates thousands of function
variants that make that subprocess exceed the normal test timeout. Normal pytest
and the pylint CI job continue to run that check.
Include the module's existing behavior, BDD, and unit tests when auditing the
selection. A "no tests" result can indicate an omitted test file or missing
association rather than missing tests in the repository.
Expand All @@ -130,6 +148,8 @@ Add repository resources and test helpers used by selected tests to `also_copy`;
the AutoTune tests need `TUNING_GUIDE_Heli.md`, component-model tests need the
`tests/` directory for shared fixtures, and MAVLink inspector tests need the
PowerShell completion module copied into Mutmut's test tree.
Plugin packaging regressions also require the `windows/`, `linux/`, and `macos/`
directories containing the frozen-build specifications.
Generated copies and results are stored in the ignored `mutants/` directory;
Mutmut resumes previous work there. For a fresh baseline, remove that generated
directory first, especially after changing tests or dependencies. Run from the
Expand Down Expand Up @@ -194,6 +214,18 @@ evaluation context. The current rename planner returns no duplicate removals,
so that feedback branch is unreachable through normal processing. Review these
separately from lost metadata, incorrect live inputs, and conflicting renames.

For external parameter and configured file uploads, check omitted-edit warnings,
verified FC snapshots, complete per-file outcomes including filenames, and executor
progress callbacks. For atomic writes, verify that failures preserve the original
and remove temporary files, and that content is synced before replacement.
Encoding aliases and native newline defaults can behave identically on Linux;
directory-open flag changes can also be equivalent where `O_RDONLY` is zero.
For diagnostics, cover timeout boundaries, component isolation, pending-message
eviction, malformed text, and invalid entries before valid devices. Some split-limit
changes preserve the first token, category lookups normalize case, and the decoder
returns an error together with a missing decoded value. Review these separately
from device-ID and chunk-sequence boundary survivors and platform-specific I/O paths.

For a focused rerun, pass quoted module or function patterns to `mutmut run`,
for example:

Expand All @@ -206,13 +238,19 @@ Keep a copy of previous results before refreshing `mutants/`, and report killed,
surviving, unassociated, timed-out and crashed mutants separately. A partial run
does not establish a new score for the full configured suite.

If a worker crashes or times out while the same mutant fails normally in a direct
pytest run, compare it in a separate temporary checkout with
`forkserver_warmup = "none"` under `[tool.mutmut]`. This initializes test-module
imports inside each worker instead of preloading them in the fork server and
can distinguish shared GUI import state from an actual mutant-induced hang.
It adds worker startup cost, so use it for diagnosis before changing the normal
suite configuration. Retain both runs' execution outcomes in the report.
The suite uses `forkserver_warmup = "none"` under `[tool.mutmut]` so test-module
imports, including Tk, initialize inside each worker. Preloading GUI modules in
the fork server produced worker crashes and timeouts during plugin mutation runs;
the same mutants were killed with worker-local imports. This costs additional
worker startup time. Keep worker counts modest and retain crashed or timed-out
outcomes separately when comparing runs; a worker failure alone does not show
that a mutant changes application behavior.
The shared plugin conftest runs direct backend/model assertions before Tk callers
when `MUTANT_UNDER_TEST` is set. GUI tests remain selected; this ordering lets
incorrect domain results fail before a GUI callback can stall on them.
Mutation jobs use pytest-timeout's thread watchdog so a blocked Tk C call cannot
outlive the per-test timeout while consuming too little CPU for Mutmut's CPU
limit. Normal pytest runs keep their existing timeout method.

### Optional local code graph

Expand Down
70 changes: 52 additions & 18 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,10 +190,25 @@ skip = "*.pdef.xml,*.pdf,*.po"

[tool.mutmut]
source_paths = ["ardupilot_methodic_configurator/"]
# Focus on parameter handling, parsing, and calculations with targeted tests.
# Cover parameter handling, parsing, calculations, and every plugin backend/data model.
only_mutate = [
# Globs keep new plugin backends and data models in scope automatically.
"ardupilot_methodic_configurator/plugins/backend_*.py",
"ardupilot_methodic_configurator/plugins/data_model_*.py",
# External parameter selection, file-upload policy, atomic writes, and diagnostics.
"ardupilot_methodic_configurator/data_model_parameter_compare_and_upload.py",
"ardupilot_methodic_configurator/data_model_file_upload.py",
"ardupilot_methodic_configurator/backend_safe_file_io.py",
"ardupilot_methodic_configurator/data_model_fc_diagnostics.py",
# Shared implementations used by telemetry and calibration plugins.
"ardupilot_methodic_configurator/backend_data_telemetry.py",
"ardupilot_methodic_configurator/backend_esc_telemetry.py",
"ardupilot_methodic_configurator/backend_radio_status.py",
"ardupilot_methodic_configurator/backend_flightcontroller_commands.py",
"ardupilot_methodic_configurator/data_model_data_telemetry.py",
"ardupilot_methodic_configurator/data_model_esc_telemetry.py",
"ardupilot_methodic_configurator/data_model_mavlink_inspector.py",
"ardupilot_methodic_configurator/data_model_configuration_step.py",
"ardupilot_methodic_configurator/plugins/data_model_battery_monitor.py",
"ardupilot_methodic_configurator/data_model_parameter_upgrade.py",
"ardupilot_methodic_configurator/backend_flightcontroller_business_logic.py",
"ardupilot_methodic_configurator/data_model_transfer_progress.py",
Expand All @@ -209,25 +224,23 @@ only_mutate = [
"ardupilot_methodic_configurator/data_model_vehicle_components_import.py",
"ardupilot_methodic_configurator/data_model_vehicle_components_json_schema.py",
"ardupilot_methodic_configurator/backend_firmware_upload.py",
"ardupilot_methodic_configurator/plugins/data_model_motor_test.py",
"ardupilot_methodic_configurator/frontend_tkinter_usage_popup_windows.py",
"ardupilot_methodic_configurator/frontend_tkinter_log_analysis.py",
"ardupilot_methodic_configurator/backend_flightcontroller_connection.py",
"ardupilot_methodic_configurator/frontend_tkinter_fc_banner_window.py",
"ardupilot_methodic_configurator/frontend_tkinter_mavlink_inspector.py",
"ardupilot_methodic_configurator/log_analysis/data_model_plane_landing.py",
"ardupilot_methodic_configurator/log_analysis/data_model_availability_imu.py",
"ardupilot_methodic_configurator/log_analysis/data_model_availability_esc.py",
"ardupilot_methodic_configurator/log_analysis/data_model_firmware_version.py",
"ardupilot_methodic_configurator/log_analysis/data_model_parameter_history.py",
"ardupilot_methodic_configurator/plugins/imu_helpers.py",
"ardupilot_methodic_configurator/plugins/data_model_autotune_backoff.py",
"ardupilot_methodic_configurator/plugins/data_model_autotune_accel_backoff.py",
"ardupilot_methodic_configurator/plugins/data_model_autotune_gain_backoff.py",
"ardupilot_methodic_configurator/data_model_firmware_upload.py",
"ardupilot_methodic_configurator/log_analysis/backend_log_analysis.py",
"ardupilot_methodic_configurator/frontend_tkinter_usage_popup_window.py",
"ardupilot_methodic_configurator/frontend_tkinter_file_browser.py",
"ardupilot_methodic_configurator/frontend_tkinter_file_browser_tasks.py",
"ardupilot_methodic_configurator/frontend_tkinter_parameter_editor_table.py",
"ardupilot_methodic_configurator/plugins/frontend_tkinter_motor_test.py",
"ardupilot_methodic_configurator/backend_filesystem_json_with_schema.py",
]
Expand All @@ -243,11 +256,19 @@ do_not_mutate = [
"ardupilot_methodic_configurator/decode_devid.py",
]
pytest_add_cli_args_test_selection = [
# Include model, backend, unit, behavior, and real-widget plugin regressions.
"tests/plugins/",
"tests/test_data_model_parameter_compare_and_upload.py",
"tests/test_data_model_file_upload.py",
"tests/unit_backend_safe_file_io.py",
"tests/bdd_backend_safe_file_io.py",
"tests/test_backend_safe_file_io_write_capture.py",
"tests/test_data_model_fc_diagnostics.py",
"tests/test_backend_data_telemetry.py",
"tests/test_esc_telemetry.py",
"tests/test_backend_flightcontroller_commands.py",
"tests/unit_backend_flightcontroller_commands.py",
"tests/test_data_model_configuration_step.py",
"tests/plugins/unit_data_model_battery_monitor.py",
"tests/plugins/unit_frontend_tkinter_battery_monitor.py",
"tests/plugins/bdd_battery_monitor.py",
"tests/plugins/acceptance_battery_monitor.py",
"tests/test_data_model_parameter_upgrade.py",
"tests/test_backend_flightcontroller_business_logic.py",
"tests/test_data_model_transfer_progress.py",
Expand All @@ -263,6 +284,8 @@ pytest_add_cli_args_test_selection = [
"tests/test_bootloader_identity.py",
"tests/test_bootloader_partial_read.py",
"tests/test_data_model_imu_analysis.py",
"tests/test_data_model_availability_models.py",
"tests/test_data_model_log_analysis.py",
"tests/test_bus_labels.py",
"tests/test_data_model_vehicle_components_display.py",
"tests/unit_data_model_vehicle_components_display.py",
Expand All @@ -279,31 +302,42 @@ pytest_add_cli_args_test_selection = [
"tests/unit_data_model_vehicle_components_json_schema.py",
"tests/test_backend_firmware_upload.py",
"tests/test_data_model_firmware_upload.py",
"tests/plugins/test_data_model_motor_test.py",
"tests/test_frontend_tkinter_usage_popup_windows.py",
"tests/test_frontend_tkinter_log_analysis.py",
"tests/test_backend_flightcontroller_connection.py",
"tests/test_frontend_tkinter_fc_banner_window.py",
"tests/test_frontend_tkinter_mavlink_inspector.py",
"tests/test_data_model_mavlink_inspector.py",
"tests/test_data_model_plane_landing.py",
"tests/test_data_model_firmware_version.py",
"tests/test_data_model_parameter_history.py",
"tests/plugins/test_imu_helpers.py",
"tests/plugins/test_autotune_accel_backoff.py",
"tests/plugins/test_autotune_gain_backoff.py",
"tests/test_backend_log_analysis.py",
"tests/test_backend_fc_diagnostics.py",
"tests/test_frontend_tkinter_usage_popup_window.py",
"tests/test_frontend_tkinter_file_browser.py",
"tests/bdd_frontend_tkinter_file_browser.py",
"tests/test_frontend_tkinter_file_browser_tasks.py",
"tests/plugins/test_frontend_tkinter_motor_test.py",
"tests/test_frontend_tkinter_parameter_editor_table.py",
"tests/integration_frontend_tkinter_parameter_editor_table.py",
"tests/acceptance_bitmask_parameter_editor_usage_popup.py",
"tests/test_backend_filesystem_json_with_schema.py",
]
pytest_add_cli_args = ["-q", "-x", "-p", "tests.plugins.conftest"]
# The AutoTune tests verify helicopter-stage guidance as well as package data.
also_copy = ["pytest.ini", "TUNING_GUIDE_Heli.md", "tests", "ardupilot_methodic_configurator_command_line_completion.psm1"]
pytest_add_cli_args = [
"-q", "-x", "-p", "tests.plugins.conftest",
# A signal timeout cannot interrupt a blocked Tk C call.
"--timeout-method=thread",
# Lint the original source in normal pytest/CI, not Mutmut's generated variants.
"--deselect=tests/test_backend_flightcontroller_commands.py::test_command_ack_meets_pylint_statement_and_return_limits",
]
# Plugin tests also verify helicopter guidance and all three frozen-build specifications.
also_copy = [
"pytest.ini", "TUNING_GUIDE_Heli.md", "tests",
"ardupilot_methodic_configurator_command_line_completion.psm1",
"windows", "linux", "macos",
]
process_isolation = "forkserver"
# Tk imports must initialize inside each worker, not the fork server.
forkserver_warmup = "none"

[tool.ruff]
target-version = "py311"
Expand Down
Loading
Loading