Skip to content

feat(file-processing): prevent submission of interleaved fastq files - #7001

Draft
anna-parker wants to merge 6 commits into
mainfrom
interleaved
Draft

feat(file-processing): prevent submission of interleaved fastq files#7001
anna-parker wants to merge 6 commits into
mainfrom
interleaved

Conversation

@anna-parker

@anna-parker anna-parker commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

This solves #6856 but is too inefficient atm to merge - we should use a bloomfilter like in the ena code base

I uploaded some mpox SRA fastq files and these lead preprocessing to fail with a timeout - 5min was not enough! Checking the logs it takes almost 3 whole minutes to download the S3 files... even when gzipped (80M)
image

Screenshot

PR Checklist

  • All necessary documentation has been adapted.
  • The implemented feature is covered by appropriate, automated tests.
  • Any manual testing that has been done is documented (i.e. what exactly was tested?)

🚀 Preview: Add preview label to enable

@claude claude Bot added backend related to the loculus backend component website Tasks related to the web application deployment Code changes targetting the deployment infrastructure preprocessing Issues related to the preprocessing component labels Jul 28, 2026
@claude

claude Bot commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

This PR may be related to: #6856 (MVP: Handle interleaved fastq submissions) — the diff explicitly rejects interleaved FASTQ files (both same-name and CASAVA-style mate-number formats) in file_validation.py, which matches that issue's request. Also possibly related: #6906 (MVP: Validate fastq files) and #6934 (Update ENA deposition and file validation code to allow BAM file submission), given the new file-processing service also validates FASTQ/BAM formats.

@anna-parker anna-parker changed the title Interleaved feat(file-processing): prevent submission of interleaved fastq files Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend related to the loculus backend component deployment Code changes targetting the deployment infrastructure preprocessing Issues related to the preprocessing component website Tasks related to the web application

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant