Skip to content

HOW_TO_RUN_TESTS.md

Latest commit

 

History

History
375 lines (269 loc) · 9.93 KB

HOW_TO_RUN_TESTS.md

File metadata and controls

375 lines (269 loc) · 9.93 KB
Raw

How to Run Tests for LEDMatrix

This guide explains how to use the test suite for the LEDMatrix project.

Prerequisites

1. Install Test Dependencies

Make sure you have the testing packages installed:

# Install all dependencies including test packages
pip install -r requirements.txt -r requirements-test.txt

2. Set Environment Variables

For tests that don't require hardware, set the emulator mode:

export EMULATOR=true

This ensures tests use the emulator instead of trying to access actual hardware.

Running Tests

Run All Tests

# From the project root directory
pytest

# Or with more verbose output
pytest -v

# Or with even more detail
pytest -vv

Run Specific Test Files

# Run a specific test file
pytest test/test_display_controller.py

# Run multiple specific files
pytest test/test_display_controller.py test/test_plugin_system.py

Run Specific Test Classes or Functions

# Run a specific test class
pytest test/test_display_controller.py::TestDisplayControllerLivePriority

# Run a specific test function
pytest test/test_display_controller.py::TestDisplayControllerSchedule::test_active_hours

Run Tests by Marker

pytest.ini declares the markers unit, integration, hardware, slow and plugin (with --strict-markers, so a typo in a marker name is an error). Few tests are marked: only a handful carry unit, and none currently carry integration, slow or hardware, so -m integration and -m slow select nothing. Select tests by file, directory or -k instead.

# What CI runs for the core suites (excludes anything marked hardware)
pytest -m "not hardware" test/ --ignore=test/plugins

# Tests whose name matches an expression
pytest -k "config and not secrets"

Run Tests in a Directory

# Run all tests in the test directory
pytest test/

# Run plugin tests only
pytest test/plugins/

# Run web interface tests only
pytest test/web_interface/

# Run web interface integration tests
pytest test/web_interface/integration/

Understanding Test Output

Basic Output

When you run pytest, you'll see:

test/test_display_controller.py::TestDisplayControllerInitialization::test_init_success PASSED
test/test_display_controller.py::TestDisplayControllerOnDemand::test_activate_on_demand PASSED
...
  • PASSED - Test succeeded
  • FAILED - Test failed (check the error message)
  • SKIPPED - Test was skipped (usually due to missing dependencies or conditions)
  • ERROR - Test had an error during setup

Verbose Output

Use -v or -vv for more detail:

pytest -vv

This shows:

  • Full test names
  • Setup/teardown information
  • More detailed failure messages

Show Print Statements

To see print statements and logging output:

pytest -s

Or combine with verbose:

pytest -sv

Coverage Reports

Coverage is not collected by a plain pytest run: pytest.ini deliberately has no coverage flags, so local runs stay fast. Ask for it explicitly (needs pytest-cov, which is in requirements-test.txt):

# Terminal summary
pytest --cov=src --cov=web_interface --cov-report=term test/ --ignore=test/plugins

# HTML report in htmlcov/
pytest --cov=src --cov=web_interface --cov-report=html test/ --ignore=test/plugins

Then open htmlcov/index.html in your browser (xdg-open on Linux, open on macOS, start on Windows).

Coverage Threshold

The only threshold is in CI: the core unit-test job in .github/workflows/test.yml runs with --cov-fail-under=52. To check it locally, add that flag to the command above.

Common Test Scenarios

Run Tests After Making Changes

# Quick run: just the tests for the area you changed
pytest test/test_config_manager.py

# Full test suite
pytest

Web UI JavaScript Tests

The suites in test/js need node; the DOM ones also need jsdom and a running web interface (details in test/js/README.md):

npm install --no-audit --no-fund --prefix test/js   # jsdom; node_modules is gitignored
EMULATOR=true python3 web_interface/app.py           # in another shell
BASE=http://localhost:5000 REQUIRE_DOM=1 node test/js/run_all.js

pytest test/test_js_unit_suites.py runs just the unit suites.

Plugin Config Form Parity

test/test_field_model_parity.py checks build_field_model against the render_field macro for every plugin schema it finds (WEB_FRONTEND_ARCHITECTURE.md). It always covers plugin-repos/ and the test fixtures; point it at a checkout of the official plugins to cover those too:

LEDMATRIX_MONOREPO_PLUGINS=../ledmatrix-plugins/plugins pytest test/test_field_model_parity.py

Debug a Failing Test

# Run with maximum verbosity and show print statements
pytest -vv -s test/test_display_controller.py::TestDisplayControllerSchedule::test_active_hours

# Run with Python debugger (pdb)
pytest --pdb test/test_display_controller.py::TestDisplayControllerSchedule::test_active_hours

Run Tests in Parallel (Faster)

# Install pytest-xdist first
pip install pytest-xdist

# Run tests in parallel (4 workers)
pytest -n 4

# Auto-detect number of CPUs
pytest -n auto

Stop on First Failure

# Stop immediately when a test fails
pytest -x

# Stop after N failures
pytest --maxfail=3

Test Organization

Test Files Structure

test/
├── conftest.py                          # Shared fixtures and configuration
├── test_display_controller.py           # Display controller tests
├── test_display_manager.py              # Display manager tests
├── test_plugin_system.py                # Plugin system tests
├── test_plugin_loader.py                # Plugin discovery/loading tests
├── test_plugin_loading_failures.py      # Plugin failure-mode tests
├── test_cache_manager.py                # Cache manager tests
├── test_config_manager.py               # Config manager tests
├── test_config_service.py               # Config service tests
├── test_config_validation_edge_cases.py # Config edge cases
├── test_font_manager.py                 # Font manager tests
├── test_text_helper.py                  # Text helper tests
├── test_error_handling.py               # Error handling tests
├── test_error_aggregator.py             # Error aggregation tests
├── test_schema_manager.py               # Schema manager tests
├── test_web_api.py                      # Web API tests
├── plugins/                             # Plugin rendering suites
│   ├── test_plugin_matrix.py            # Every discovered plugin, across panel sizes
│   ├── test_harness.py
│   └── test_visual_rendering.py
└── web_interface/
    ├── test_config_manager_atomic.py
    ├── test_state_reconciliation.py
    ├── test_plugin_operation_queue.py
    ├── test_dedup_unique_arrays.py
    └── integration/                     # Web interface integration tests
        ├── test_config_flows.py
        └── test_plugin_operations.py

Test Categories

  • Unit Tests: Fast, isolated tests for individual components
  • Integration Tests: Tests that verify components work together
  • Error Scenarios: Tests for error handling and edge cases
  • Edge Cases: Boundary conditions and unusual inputs

Troubleshooting

Import Errors

If you see import errors:

# Make sure you're in the project root (wherever you cloned it)
cd ~/LEDMatrix

# Check Python path
python -c "import sys; print(sys.path)"

# Run pytest from project root
pytest

Missing Dependencies

If tests fail due to missing packages:

# Install all dependencies
pip install -r requirements.txt -r requirements-test.txt

# Or install specific missing package
pip install <package-name>

Hardware Tests Failing

If tests that require hardware are failing:

# Set emulator mode
export EMULATOR=true

# Or skip hardware tests
pytest -m "not hardware"

Coverage Not Working

If coverage reports aren't generating:

# Make sure pytest-cov is installed
pip install pytest-cov

# Coverage is opt-in; ask for it explicitly
pytest --cov=src --cov=web_interface --cov-report=html

Continuous Integration

The repo runs the pytest suite via .github/workflows/test.yml on every push and pull request: a plugin-safety job that runs test/plugins/, and a core unit-test job that runs the whole test/ tree except test/plugins/ with -m "not hardware" and enforces coverage (--cov-fail-under=52). New test files are picked up automatically. Release version consistency is checked by .github/workflows/release-version-check.yml. Bandit, flake8, mypy and gitleaks run as pre-commit hooks (see .pre-commit-config.yaml), not in CI.

Best Practices

  1. Run tests before committing:

    pytest test/test_<area>.py  # Quick check of what you touched

  2. Run full suite before pushing:

    pytest  # Full test suite (add --cov flags for coverage)

  3. Fix failing tests immediately - Don't let them accumulate

  4. Keep coverage above threshold - CI fails below 52%

  5. Write tests for new features - Add tests when adding new functionality

Quick Reference

# Most common commands
pytest                    # Run all tests (no coverage)
pytest -v                 # Verbose output
pytest test/test_x.py     # Run one file
pytest -k "test_name"    # Run tests matching pattern
pytest --cov=src         # Generate coverage report
pytest -x                # Stop on first failure
pytest --pdb              # Drop into debugger on failure

Getting Help

  • Check test output for error messages
  • Look at the test file to understand what's being tested
  • Check conftest.py for available fixtures
  • Review pytest.ini for configuration options