ci: drop pytest in favor of Sprocket test by a-frantz · Pull Request #263 · stjudecloud/workflows

a-frantz · 2025-09-03T21:03:02Z

~~This PR is for demonstration purposes of the RFC document here~~

this PR has morphed from a demonstration into a real PR for swapping our test infra from pytest to Sprocket. Still a WIP as not all the tests are written yet and it has not been hooked into CI yet

github-advanced-security · 2025-10-08T13:15:06Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

This reverts commit d89fd65.

Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.5.0 to 2.6.3. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/urllib3/urllib3/releases">urllib3's releases</a>.</em></p> <blockquote> <h2>2.6.3</h2> <h2>🚀 urllib3 is fundraising for HTTP/2 support</h2> <p><a href="https://sethmlarson.dev/urllib3-is-fundraising-for-http2-support">urllib3 is raising ~$40,000 USD</a> to release HTTP/2 support and ensure long-term sustainable maintenance of the project after a sharp decline in financial support. If your company or organization uses Python and would benefit from HTTP/2 support in Requests, pip, cloud SDKs, and thousands of other projects <a href="https://opencollective.com/urllib3">please consider contributing financially</a> to ensure HTTP/2 support is developed sustainably and maintained for the long-haul.</p> <p>Thank you for your support.</p> <h2>Changes</h2> <ul> <li>Fixed a security issue where decompression-bomb safeguards of the streaming API were bypassed when HTTP redirects were followed. (CVE-2026-21441 reported by <a href="https://github.com/D47A"><code>@D47A</code></a>, 8.9 High, GHSA-38jv-5279-wg99)</li> <li>Started treating <code>Retry-After</code> times greater than 6 hours as 6 hours by default. (<a href="https://redirect.github.com/urllib3/urllib3/issues/3743">urllib3/urllib3#3743</a>)</li> <li>Fixed <code>urllib3.connection.VerifiedHTTPSConnection</code> on Emscripten. (<a href="https://redirect.github.com/urllib3/urllib3/issues/3752">urllib3/urllib3#3752</a>)</li> </ul> <h2>2.6.2</h2> <h2>🚀 urllib3 is fundraising for HTTP/2 support</h2> <p><a href="https://sethmlarson.dev/urllib3-is-fundraising-for-http2-support">urllib3 is raising ~$40,000 USD</a> to release HTTP/2 support and ensure long-term sustainable maintenance of the project after a sharp decline in financial support. If your company or organization uses Python and would benefit from HTTP/2 support in Requests, pip, cloud SDKs, and thousands of other projects <a href="https://opencollective.com/urllib3">please consider contributing financially</a> to ensure HTTP/2 support is developed sustainably and maintained for the long-haul.</p> <p>Thank you for your support.</p> <h2>Changes</h2> <ul> <li>Fixed <code>HTTPResponse.read_chunked()</code> to properly handle leftover data in the decoder's buffer when reading compressed chunked responses. (<a href="https://redirect.github.com/urllib3/urllib3/issues/3734">urllib3/urllib3#3734</a>)</li> </ul> <h2>2.6.1</h2> <h2>🚀 urllib3 is fundraising for HTTP/2 support</h2> <p><a href="https://sethmlarson.dev/urllib3-is-fundraising-for-http2-support">urllib3 is raising ~$40,000 USD</a> to release HTTP/2 support and ensure long-term sustainable maintenance of the project after a sharp decline in financial support. If your company or organization uses Python and would benefit from HTTP/2 support in Requests, pip, cloud SDKs, and thousands of other projects <a href="https://opencollective.com/urllib3">please consider contributing financially</a> to ensure HTTP/2 support is developed sustainably and maintained for the long-haul.</p> <p>Thank you for your support.</p> <h2>Changes</h2> <ul> <li>Restore previously removed <code>HTTPResponse.getheaders()</code> and <code>HTTPResponse.getheader()</code> methods. (<a href="https://redirect.github.com/urllib3/urllib3/issues/3731">#3731</a>)</li> </ul> <h2>2.6.0</h2> <h2>🚀 urllib3 is fundraising for HTTP/2 support</h2> <p><a href="https://sethmlarson.dev/urllib3-is-fundraising-for-http2-support">urllib3 is raising ~$40,000 USD</a> to release HTTP/2 support and ensure long-term sustainable maintenance of the project after a sharp decline in financial support. If your company or organization uses Python and would benefit from HTTP/2 support in Requests, pip, cloud SDKs, and thousands of other projects <a href="https://opencollective.com/urllib3">please consider contributing financially</a> to ensure HTTP/2 support is developed sustainably and maintained for the long-haul.</p> <p>Thank you for your support.</p> <h2>Security</h2> <ul> <li>Fixed a security issue where streaming API could improperly handle highly compressed HTTP content ("decompression bombs") leading to excessive resource consumption even when a small amount of data was requested. Reading small chunks of compressed data is safer and much more efficient now. (CVE-2025-66471 reported by <a href="https://github.com/Cycloctane"><code>@Cycloctane</code></a>, 8.9 High, GHSA-2xpw-w6gg-jr37)</li> <li>Fixed a security issue where an attacker could compose an HTTP response with virtually unlimited links in the <code>Content-Encoding</code> header, potentially leading to a denial of service (DoS) attack by exhausting system resources during decoding. The number of allowed chained encodings is now limited to 5. (CVE-2025-66418 reported by <a href="https://github.com/illia-v"><code>@illia-v</code></a>, 8.9 High, GHSA-gm62-xv2j-4w53)</li> </ul> <blockquote> <p>[!IMPORTANT]</p> <ul> <li>If urllib3 is not installed with the optional <code>urllib3[brotli]</code> extra, but your environment contains a Brotli/brotlicffi/brotlipy package anyway, make sure to upgrade it to at least Brotli 1.2.0 or brotlicffi 1.2.0.0 to benefit from the security fixes and avoid warnings. Prefer using <code>urllib3[brotli]</code> to install a compatible Brotli package automatically.</li> </ul> </blockquote>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/urllib3/urllib3/blob/main/CHANGES.rst">urllib3's changelog</a>.</em></p> <blockquote> <h1>2.6.3 (2026-01-07)</h1> <ul> <li>Fixed a high-severity security issue where decompression-bomb safeguards of the streaming API were bypassed when HTTP redirects were followed. (<code>GHSA-38jv-5279-wg99 <https://github.com/urllib3/urllib3/security/advisories/GHSA-38jv-5279-wg99></code>__)</li> <li>Started treating <code>Retry-After</code> times greater than 6 hours as 6 hours by default. (<code>[#3743](urllib3/urllib3#3743) <https://github.com/urllib3/urllib3/issues/3743></code>__)</li> <li>Fixed <code>urllib3.connection.VerifiedHTTPSConnection</code> on Emscripten. (<code>[#3752](urllib3/urllib3#3752) <https://github.com/urllib3/urllib3/issues/3752></code>__)</li> </ul> <h1>2.6.2 (2025-12-11)</h1> <ul> <li>Fixed <code>HTTPResponse.read_chunked()</code> to properly handle leftover data in the decoder's buffer when reading compressed chunked responses. (<code>[#3734](urllib3/urllib3#3734) <https://github.com/urllib3/urllib3/issues/3734></code>__)</li> </ul> <h1>2.6.1 (2025-12-08)</h1> <ul> <li>Restore previously removed <code>HTTPResponse.getheaders()</code> and <code>HTTPResponse.getheader()</code> methods. (<code>[#3731](urllib3/urllib3#3731) <https://github.com/urllib3/urllib3/issues/3731></code>__)</li> </ul> <h1>2.6.0 (2025-12-05)</h1> <h2>Security</h2> <ul> <li>Fixed a security issue where streaming API could improperly handle highly compressed HTTP content ("decompression bombs") leading to excessive resource consumption even when a small amount of data was requested. Reading small chunks of compressed data is safer and much more efficient now. (<code>GHSA-2xpw-w6gg-jr37 <https://github.com/urllib3/urllib3/security/advisories/GHSA-2xpw-w6gg-jr37></code>__)</li> <li>Fixed a security issue where an attacker could compose an HTTP response with virtually unlimited links in the <code>Content-Encoding</code> header, potentially leading to a denial of service (DoS) attack by exhausting system resources during decoding. The number of allowed chained encodings is now limited to 5. (<code>GHSA-gm62-xv2j-4w53 <https://github.com/urllib3/urllib3/security/advisories/GHSA-gm62-xv2j-4w53></code>__)</li> </ul> <p>.. caution::</p> <ul> <li>If urllib3 is not installed with the optional <code>urllib3[brotli]</code> extra, but your environment contains a Brotli/brotlicffi/brotlipy package anyway, make sure to upgrade it to at least Brotli 1.2.0 or brotlicffi 1.2.0.0 to benefit from the security fixes and avoid warnings. Prefer using</li> </ul>  </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/urllib3/urllib3/commit/0248277dd7ac0239204889ca991353ad3e3a1ddc"><code>0248277</code></a> Release 2.6.3</li> <li><a href="https://github.com/urllib3/urllib3/commit/8864ac407bba8607950025e0979c4c69bc7abc7b"><code>8864ac4</code></a> Merge commit from fork</li> <li><a href="https://github.com/urllib3/urllib3/commit/70cecb27ca99d56aaaeb63ac27ee270ef2b24c5c"><code>70cecb2</code></a> Fix Scorecard issues related to vulnerable dev dependencies (<a href="https://redirect.github.com/urllib3/urllib3/issues/3755">#3755</a>)</li> <li><a href="https://github.com/urllib3/urllib3/commit/41f249abe1ef3e20768588969c4035aba060a359"><code>41f249a</code></a> Move "v2.0 Migration Guide" to the end of the table of contents (<a href="https://redirect.github.com/urllib3/urllib3/issues/3747">#3747</a>)</li> <li><a href="https://github.com/urllib3/urllib3/commit/fd4dffd2fc544166b76151a2fa3d7b7c0eab540c"><code>fd4dffd</code></a> Patch <code>VerifiedHTTPSConnection</code> for Emscripten (<a href="https://redirect.github.com/urllib3/urllib3/issues/3752">#3752</a>)</li> <li><a href="https://github.com/urllib3/urllib3/commit/13f0bfd55e4468fe1ea9c6f809d3a87b0f93ebab"><code>13f0bfd</code></a> Handle massive values in Retry-After when calculating time to sleep for (<a href="https://redirect.github.com/urllib3/urllib3/issues/3743">#3743</a>)</li> <li><a href="https://github.com/urllib3/urllib3/commit/8c480bf87bcefd321b3a1ae47f04e908b6b2ed7b"><code>8c480bf</code></a> Bump actions/upload-artifact from 5.0.0 to 6.0.0 (<a href="https://redirect.github.com/urllib3/urllib3/issues/3748">#3748</a>)</li> <li><a href="https://github.com/urllib3/urllib3/commit/4b40616e959c0a2c466e8075f2a785a9f99bb0c1"><code>4b40616</code></a> Bump actions/cache from 4.3.0 to 5.0.1 (<a href="https://redirect.github.com/urllib3/urllib3/issues/3750">#3750</a>)</li> <li><a href="https://github.com/urllib3/urllib3/commit/82b8479663d037d220c883f1584dd01a43bb273b"><code>82b8479</code></a> Bump actions/download-artifact from 6.0.0 to 7.0.0 (<a href="https://redirect.github.com/urllib3/urllib3/issues/3749">#3749</a>)</li> <li><a href="https://github.com/urllib3/urllib3/commit/34284cb01700bb7d4fdd472f909e22393e9174e2"><code>34284cb</code></a> Mention experimental features in the security policy (<a href="https://redirect.github.com/urllib3/urllib3/issues/3746">#3746</a>)</li> <li>Additional commits viewable in <a href="https://github.com/urllib3/urllib3/compare/2.5.0...2.6.3">compare view</a></li> </ul> </details> <br /> [![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=urllib3&package-manager=uv&previous-version=2.5.0&new-version=2.6.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/stjudecloud/workflows/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

_Describe the problem or feature in addition to a link to the issues._ the non summary mode of validate_bam can output multiple lines _per read_ in the input BAM, using more disk than the input file. This passing that output through gzip. This also removes the x2 factor on disk size allocation for this task. We may instead want to keep that x2 factor but only if we're in full report mode? Before submitting this PR, please make sure: - [ ] You have added a few sentences describing the PR here. - [ ] The code passes all CI tests without any errors or warnings. - [ ] You have added tests (when appropriate). - [ ] You have added an entry in any relevant CHANGELOGs (when appropriate). - [ ] If you have made any changes to the `scripts/` or `docker/` directories, please ensure any image versions have been incremented accordingly! - [ ] You have updated the README or other documentation to account for these changes (when appropriate). --------- Co-authored-by: Andrew Thrasher <adthrasher@gmail.com>

Add filtering of sex chromosomes to the UMAP generation. Also generate a list of probes that have SNPs. Before submitting this PR, please make sure: - [x] You have added a few sentences describing the PR here. - [x] The code passes all CI tests without any errors or warnings. - [ ] You have added tests (when appropriate). - [x] You have added an entry in any relevant CHANGELOGs (when appropriate). - [x] If you have made any changes to the `scripts/` or `docker/` directories, please ensure any image versions have been incremented accordingly! - [ ] You have updated the README or other documentation to account for these changes (when appropriate). --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Ari Frantz <ari.frantz@stjude.org> Co-authored-by: Jobin Sunny <38107318+jsunny23@users.noreply.github.com>

Currently, we compute file sizes in `GiB`. We then use those to request disk space, but in `GB`. This is silently undercounting the amount of disk space we need for these tasks. `GB` was chosen to be the common unit because our variables already end in `_gb` so this is the least disruptive change. Before submitting this PR, please make sure: - [x] You have added a few sentences describing the PR here. - [ ] The code passes all CI tests without any errors or warnings. - [ ] You have added tests (when appropriate). - [x] You have added an entry in any relevant CHANGELOGs (when appropriate). - [ ] If you have made any changes to the `scripts/` or `docker/` directories, please ensure any image versions have been incremented accordingly! - [ ] You have updated the README or other documentation to account for these changes (when appropriate).

_Describe the problem or feature in addition to a link to the issues._ Before submitting this PR, please make sure: - [ ] You have added a few sentences describing the PR here. - [ ] The code passes all CI tests without any errors or warnings. - [ ] You have added tests (when appropriate). - [ ] You have added an entry in any relevant CHANGELOGs (when appropriate). - [ ] If you have made any changes to the `scripts/` or `docker/` directories, please ensure any image versions have been incremented accordingly! - [ ] You have updated the README or other documentation to account for these changes (when appropriate).

github-advanced-security

Snyk Container found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

a-frantz · 2026-04-15T16:12:04Z

all tests passing 🎉
@adthrasher for local testing, I picard::mark_duplicates uses more memory than available on my MPB, so I've annotated that test with high_mem. Running sprocket dev test -f high_mem at the root of this branch should be all green. I'm seeing all green on the cluster as well. For executing tests there, try a parallelism = 300 to get a good balance of speed and resource utilization. See stjude-rust-labs/sprocket#821 for why just lifting the parallelism throttle completely doesn't work

adthrasher · 2026-04-17T17:07:20Z

I haven't gotten further, but my assumption was that we'd call sprocket dev test as the second part of this to test the newly built images.

adthrasher · 2026-04-17T17:08:42Z

-          with:
-            lint: true
-            except: KnownRules
+  sprocket_lint:


If we're doing lint and format, I'd probably just shorten this to sprocket.

Suggested change

sprocket_lint:

sprocket:

Or maybe lengthen to sprocket_lint_format

adthrasher · 2026-04-17T17:17:09Z

-    String stringency_arg = if index_validation_stringency_less_exhaustive
-        then "--INDEX_VALIDATION_STRINGENCY LESS_EXHAUSTIVE"
-        else ""


I don't recall what this parameter does. Why are we removing the argument?

a-frantz mentioned this pull request Sep 3, 2025

rfc: sprocket test stjude-rust-labs/rfcs#1

Merged

a-frantz added 2 commits October 8, 2025 09:12

init demo

3285664

revise: mv TOML out of tests dir and into main workspace

7bc1e3a

a-frantz force-pushed the demonstration/sprocket-test branch from 535b997 to 7bc1e3a Compare October 8, 2025 13:13

a-frantz added 2 commits November 18, 2025 09:14

Merge branch 'main' into demonstration/sprocket-test

cdb13f7

switch from TOML to YAML

6438780

a-frantz force-pushed the demonstration/sprocket-test branch from 724fff6 to 6438780 Compare November 19, 2025 16:04

a-frantz mentioned this pull request Nov 19, 2025

feat: first step towards the test command stjude-rust-labs/sprocket#468

Merged

12 tasks

a-frantz added 8 commits November 19, 2025 14:54

WIP

62744f2

fix: tests -> assertions

a9cc715

Update flag_filter.yaml

e09621c

Merge branch 'main' into demonstration/sprocket-test

75b39f7

feat: use Peter's top level mapping representation

0be911a

set up sprocket test infra

374392a

picard: lower merge mem for testing

d89fd65

point to new fixtures

81d9608

a-frantz mentioned this pull request Dec 11, 2025

feat: execute tests stjude-rust-labs/sprocket#489

Merged

12 tasks

a-frantz added 11 commits December 15, 2025 07:36

Revert "picard: lower merge mem for testing"

5d97f28

This reverts commit d89fd65.

WIP

1b7235d

delete pytest folder

881fa7a

revise: change validate_read_group logic

d2898f0

test data_structures

6c910ce

chore: log TODO

64476ad

more tests

c76f201

WIP: remove broken CI elements

2fb9d23

add test depth (more bams and fastqs tested)

d1e0d99

Update read_group.yaml

51ad295

nesting for test fixtures

2b7d0dd

a-frantz changed the title ~~DEMO: sprocket test example~~ ci: drop pytest in favor of Sprocket test Dec 29, 2025

dependabot bot and others added 11 commits March 27, 2026 09:46

ci: rm redundant or unneeded actions

0e4f7a8

ci: add a format check

4f3db73

sprocket lints

08d5a59

ignore failing methylation files

c39480b

ci fix

21a7752

wip: resolving bad merge

a7b661d

a-frantz force-pushed the demonstration/sprocket-test branch from 824d01b to a7b661d Compare March 27, 2026 13:49

github-advanced-security AI found potential problems Mar 27, 2026

View reviewed changes

a-frantz added 10 commits March 27, 2026 09:51

Merge branch 'main' into demonstration/sprocket-test

6842420

more bad merging resolutions

b974e4e

WIP

04324b9

WIP for building a new docker image

0fda8e4

fix

65cfbb8

finish all tools

e6bdb41

typo

b2c7fe2

lint fixes

90f011f

WIP

08aca48

fix ci

edfbaf7

a-frantz requested a review from adthrasher April 15, 2026 16:08

Merge branch 'main' into demonstration/sprocket-test

ac53096

a-frantz marked this pull request as ready for review April 15, 2026 16:08

a-frantz added 2 commits April 15, 2026 12:49

fix: sprocket lint fixes

a044b1c

use main of sprocket-action

9290baa

adthrasher reviewed Apr 17, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: drop pytest in favor of Sprocket test#263

ci: drop pytest in favor of Sprocket test#263
a-frantz wants to merge 63 commits intomainfrom
demonstration/sprocket-test

a-frantz commented Sep 3, 2025 •

edited

Loading

Uh oh!

github-advanced-security AI commented Oct 8, 2025

Uh oh!

github-advanced-security AI left a comment

Uh oh!

a-frantz commented Apr 15, 2026

Uh oh!

adthrasher Apr 17, 2026

Uh oh!

adthrasher Apr 17, 2026

Uh oh!

adthrasher Apr 17, 2026

Uh oh!

adthrasher Apr 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

a-frantz commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-advanced-security AI commented Oct 8, 2025

Uh oh!

github-advanced-security AI left a comment

Choose a reason for hiding this comment

Uh oh!

a-frantz commented Apr 15, 2026

Uh oh!

adthrasher Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

adthrasher Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

adthrasher Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

adthrasher Apr 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

a-frantz commented Sep 3, 2025 •

edited

Loading