Skip to content

Use pytest only and run PhysioNet checks as separate tests - #588

Merged
cbrnr merged 5 commits into
MIT-LCP:mainfrom
cbrnr:pytest-tests
Oct 28, 2026
Merged

cbrnr merged 5 commits into
MIT-LCP:mainfrom
cbrnr:pytest-tests

Conversation

@cbrnr

@cbrnr cbrnr commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Fifth PR of the tooling series. It moves the test suite to plain pytest and makes tests that depend on PhysioNet independent from the local checks, so a slow or unreachable PhysioNet no longer fails (or hangs) CI. It consists of five commits (the last two were added after CI showed 502 Bad Gateway errors, and after a review comment about hangs):

  1. Convert tests from unittest to pytest. Test classes are plain pytest classes, self.assert*() calls are plain asserts (pytest.raises for assertRaises), setUpClass() is replaced by a class-scoped temp_path fixture in conftest.py (tmp_path for the CSV conversion test, which used to write to test_output/), and the unittest.main() footers are gone. Nothing else changes: the same 82 tests are collected with the same IDs. (unittest.mock is still used in test_processing.py.)

  2. Run local and PhysioNet checks as separate tests. See below.

  3. Format section comments in test_record.py consistently. The four section banners had different widths and spacing.

  4. Skip network tests on HTTP server errors, too. PhysioNet regularly answers with 502 Bad Gateway, which wfdb raises as NetFileError. The skip hook now also treats a NetFileError with a 5xx status or 429 as a network error (404 and 403 still fail).

  5. Add timeouts to requests for remote files. This is the only change to library code. wfdb/io/_url.py did not set a timeout, so a server that accepts the connection but does not respond blocked forever (scripts hung, and CI jobs waited until the job timeout). Requests now time out after 10 seconds to connect and 60 seconds to wait for data (not for the whole transfer, so slow but steady downloads still work), and a requests.exceptions.Timeout is raised, which the hook above reports as a skip for tests marked network. The constant is wfdb.io._url.TIMEOUT. A new test uses a dummy server with a delayed response. There is also a changelog entry.

PhysioNet tests

Most tests that use PhysioNet first check the local sample data and then compare it with the same data streamed from PhysioNet. If PhysioNet is slow or down, the whole test failed (or hung), including the local checks.

  • 15 such tests now take a source fixture that is parametrized with local and physionet, e.g. test_1a[local] and test_1a[physionet]. The local variant never uses the network. The sequence of operations within each test is unchanged (the PhysioNet read and its comparison are just conditional).
  • The physionet variants and the two dl_database tests are marked with network (registered in pyproject.toml).
  • A small hook in tests/conftest.py reports a network test as skipped (with the reason) if it fails with a connection error, a timeout or an interrupted transfer. Failed assertions, and the same errors in tests that are not marked network, still fail. The skip reasons are shown in the summary (-r s), so an outage is visible.
  • pytest -m "not network" runs only the tests that work offline (documented in DEVELOPING.md). 80 of the 97 tests work offline, compared to 65 of 82 before.

Testing

  • Without access to PhysioNet, the suite gives 80 passed and 17 skipped on Python 3.11 with the lowest allowed dependency versions and on 3.15 with the latest ones (before: 65 passed and 17 failed).
  • I ran the 15 physionet variants against the real PhysioNet, and they all pass. I did not run the two dl_database tests (about 27 MB each).
  • I checked the skip hook with a connection error and a timeout in marked tests (skipped, also with -n 2), and with a failed assertion in a marked test and a connection error in an unmarked test (both fail).
  • ruff check and ruff format --check pass.

cbrnr added 4 commits October 28, 2026 20:42
Remove the unittest test framework from the test suite:

- Test classes are plain pytest classes instead of unittest.TestCase
  subclasses.
- self.assert*() calls are plain asserts (pytest.raises for assertRaises).
- The temporary directory that used to be created in setUpClass() is a
  class-scoped fixture (temp_path) in conftest.py, and the CSV conversion
  test uses tmp_path instead of a "test_output" directory in the working
  directory.
- assertAxesMatchSignal() in test_plot.py is a plain function.
- The unittest.main() footers are removed.

The tests are unchanged otherwise: the same 82 tests are collected.
(unittest.mock is still used in test_processing.py.)
Tests that stream or download data from PhysioNet are now marked with
"network". If such a test fails with a connection error, a timeout or an
interrupted transfer, it is reported as skipped instead of failing (the
reason is shown in the summary), so a slow or unreachable PhysioNet no
longer breaks CI. Other failures, including failed assertions, still fail.

15 tests compared local sample data with the same data streamed from
PhysioNet. They now take a "source" fixture that is parametrized with
"local" and "physionet" (the latter marked "network"), so test_X[local]
only uses local files and test_X[physionet] additionally streams from
PhysioNet. The sequence of operations within each test is unchanged. A
PhysioNet outage therefore only skips the [physionet] variants, and the
local reading and writing checks always run (80 tests work offline instead
of 65). Use `pytest -m "not network"` to run only these tests.
Use the same width (72 columns) and spacing ("# --- title --- #") for all
section banners.
PhysioNet regularly answers with "502 Bad Gateway", which is raised as a
NetFileError and was not covered by the skip hook (only connection errors,
timeouts and interrupted transfers were), so the affected [physionet] tests
still failed in CI.

Treat NetFileError with a 5xx status code or 429 (too many requests) like
the other network errors. Client errors such as 404 or 403 still fail.
@tompollard

Copy link
Copy Markdown
Member

Thanks, this looks good! One case the hook can't cover yet is a hang: wfdb doesn't set a timeout on its requests (session.request(...) in wfdb/io/_url.py), so if PhysioNet accepts the connection but doesn't respond, a physionet test waits until the 45-minute job timeout instead of being skipped.

wfdb did not set a timeout for its HTTP requests, so a server that
accepts the connection but does not respond (for example, PhysioNet when
it is overloaded) blocked forever: scripts hung, and in the tests a
[physionet] variant waited until the CI job timeout instead of being
skipped.

Use a timeout of 10 seconds to connect and 60 seconds to wait for data
(not for the whole transfer, so slow but steady downloads still work). A
requests.exceptions.Timeout is raised if the server does not respond,
which the test suite already reports as a skip for tests marked "network".

Add a delay option to the dummy HTTP server in the tests to test this.

@tompollard tompollard left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, thanks again!

@cbrnr

cbrnr commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

Now let's wait for the tests to come back green. This time they should!

@cbrnr
cbrnr merged commit bae6bff into MIT-LCP:main Oct 28, 2026
18 checks passed
@cbrnr
cbrnr deleted the pytest-tests branch October 28, 2026 20:38
tompollard added a commit that referenced this pull request Oct 28, 2026
#589)

This is the last part of the cleanup that started with #578, #580, #581,
#582 and #588. It builds the documentation without warnings, applies one
docstring style everywhere, and moves the package to a `src` layout.
Each commit can be reviewed on its own.

1. **Fix all Sphinx documentation warnings.** The docs build reported 94
warnings. They are fixed, and the docs workflow now builds with `-W
--keep-going` so they stay fixed.
- Duplicate object descriptions: repeated `automodule` directives and
the classes documented on both the `wfdb` and `io` pages are marked
`:no-index:`, so each object is indexed once.
- `numpydoc_class_members_toctree = False`, because the autosummary
stubs it links to do not exist.
- Too short title underlines, and reStructuredText errors in docstrings
(lists, literal blocks for the EDF/MATLAB/WAV format descriptions and
tables, doctest continuation lines).
- The `:no-index:` option needs Sphinx 7.2, so the pinned Sphinx version
of the `docs` group goes from 7.0.0 to 7.2.6.
2. **Wrap docstrings and comments at 88 columns**, the same width as the
code, and enable W505 in Ruff so it stays that way. This is a mechanical
change, and the commit is listed in `.git-blame-ignore-revs`.
3. **Start docstrings on the opening quote line with a one-line
summary.** Summaries that were wrapped over several lines or too long
for one line are shortened, and the rest of the text moves into the
description below. No code changes: comparing the ASTs with docstrings
removed shows no difference. A few docstrings were reworded to fit, and
the descriptions of `csv_to_wfdb` and `csv2ann` (which described reading
a WFDB header) now match what the functions do. Also listed in
`.git-blame-ignore-revs`.
4. **Move the package to `src/wfdb`.** This is the default module root
of `uv_build`, so the `module-root` setting is removed. The tests now
always run against the installed package, and `docs/conf.py` adds `src`
to `sys.path`. Files are moved without changes, so `git blame` and `git
log --follow` keep working. The wheel and the sdist build as expected.
5. **Ignore the two reformatting commits in git blame**, and add a note
about the docstring style to `DEVELOPING.md`. The hashes stay valid with
a merge commit, as in #582.
6. **Add `AGENTS.md` and `CLAUDE.md`** with short guidance for coding
agents (commands and conventions). `CLAUDE.md` only points to
`AGENTS.md`.

Checked locally: Ruff lint and format, the docs build with `-W
--keep-going`, and the tests that work offline on Python 3.11 with the
lowest dependencies and on 3.15.

@tompollard this concludes the tooling and infrastructure refresh. Any
remaining changes are minor, such as fixing the ignored Ruff rules,
which can be done later in follow-up PRs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants