feat!(website, backend): Update bulk file submission/revision format to use metadata file

[tombch]

resolves #6883 and #5820

Breaking changes

Loculus deployments that carry out bulk submissions with extra files enabled will require users to update their bulk submission metadata file to include files.{category} column(s) that list the files for each submission. Details for how to do this for the current subfolder structure are described below in the 'Examples' section under 'Bulk uploads with subfolders'.

Summary

This PR restructures how files are linked to submissions by the website, as well as how file mappings are sent from the website to the backend, in both bulk and form submissions/revisions. Previously, bulk and form website submission/revisions attached a separate file mapping object. Now, users add a list of files to the submitted metadata tsv and the website adds file mappings for each submission before sending the files to the backend for submission/revision.

New files.{category} columns are introduced within the metadata, and uploaded files must be linked to within here - this can be done simply by adding a space separated list of: the file names (name1 name2 name3, if uploading a flat folder of files) or the file names with the file paths ( name1::path1 name2::path2 name3::path3, if uploading a folder containing subfolders - allowing for different submissions to use the same file names within a bulk upload).

Examples

Bulk uploads with a flat folder of files

Bulk uploads of files can be carried out with a single flat folder of uniquely named files - the user just needs to link these files to the correct submission in the metadata.tsv. For example, with this folder of rawReads:

rawReads /
  id_1_reads_1.fastq
  id_1_reads_2.fastq
  id_2_reads_1.fastq
  id_2_reads_2.fastq
  ...

The user must add the following files.rawReads column to their metadata.tsv:

id    ...  files.rawReads
id_1  ...  id_1_reads_1.fastq id_1_reads_2.fastq
id_2  ...  id_2_reads_1.fastq id_2_reads_2.fastq
...

A user could also choose to give multiple output files the same name across submissions using the name::path syntax:

id    ...  files.rawReads
id_1  ...  reads_1.fastq::id_1_reads_1.fastq reads_2.fastq::id_1_reads_2.fastq
id_2  ...  reads_1.fastq::id_2_reads_1.fastq reads_2.fastq::id_2_reads_2.fastq
...

Bulk uploads with subfolders

The existing bulk uploads mechanism for extra files relied on (for each category) the user uploading a folder containing subfolders, where each subfolder was the submission ID for the files. An example folder for bulk rawReads upload:

rawReads /
  id_1 /
    reads_1.fastq
    reads_2.fastq
  id_2 /
    reads_1.fastq
    reads_2.fastq
  ...

This structure can still remain the same - or a user can now upload any subfolder structure. But what is now required is a files.rawReads column in the metadata.tsv, with each file in a submission referenced by name::path where name is the desired output file name in Loculus, and path is the path to the file in the uploaded folder:

id    ...  files.rawReads
id_1  ...  reads_1.fastq::id_1/reads_1.fastq reads_2.fastq::id_1/reads_2.fastq
id_2  ...  reads_1.fastq::id_2/reads_1.fastq reads_2.fastq::id_2/reads_2.fastq
...

Screenshots

Submission

1. At initial file submission, users can now choose between uploading a folder, or select individual files:

2. Subfolders can be submitted using fileName::filePath in the metadata file:

3. Files referenced in metadata but not uploaded are indicated with a warning and files that are not referenced in the metadata but are uploaded are also warned on:

This 'warning' is blocking - submission can't continue until they are resolved (an error toast would be displayed at submit).

Revision

1. The files.{category} field is included in the downloaded original metadata for revisions, each file has a linked fileId:

2. Single sequences can be revised via the edit page: can add more files and delete individual files:

3. Bulk revision of sequences and metadata without modifying the files just by not modifying the files.{category} field. Allows deletion of files by removing files from the files.{category} field, allows renaming of a file by reusing the fileId, allows adding new files (similar to initial submission).

original:

revised:

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by appropriate, automated tests.
• Any manual testing that has been done is documented (i.e. what exactly was tested?)

🚀 Preview: https://bulk-file-submissions.loculus.org

[claude]

This PR may be related to: #6883, #5820

[anna-parker]

hmm... I dont think we need to look at the values of each submissionId at this point as we already ensured that each submissionId will have each category in its fileMapping in resolveFileMappings, we could just look at the first entry to save computation. Alternatively, this information should be in the config submissionDataTypes.files?.categories?.map((c) => c.name)- I wonder if generally the list of categories should always come from the config to reduce needing to get the categories in each mapping.

[tombch]

ah yeah good point, I will update to use the submissionDataTypes categories

[tombch]

I updated the EditableMetadata.getMetadataTsv to add empty file categories to its tsv so we don't need this function to add them anymore. I have kept the check for file categories being present, as i think its useful as a stopgap - its not expensive to check and if we ever somehow had file mapping categories not present in the metadata it would mean those mappings weren't included, which would be really bad

[anna-parker]

hmm... do we actually want this behavior? this should never happen right? either the metadata referenced a reused file or parseSubmissionFileMapping should have failed?

[anna-parker]

hmm shouldnt we be checking for collisions before we set the state to awaitingUrls?

[anna-parker]

side thought: all the steps after the state awaitingUrls are handled in the useEffect for the folder component, it seems a bit odd to do this separately for the files - this can be done in a follow up PR

[tombch]

We do in this case? For the handleFileSelect, the file upload state only gets set to awaitingUrls if it was undefined, so no collisions can occur. If the file upload state wasn't undefined we check for collisions, and if its > 0, display confirmation dialog, only then does the inline addFiles function run that requests the file urls and then sets the state to uploadInProgress. I guess what this does mean is that when hitting add additional files there is no visual state currently for those files awaiting urls - i think this could be updated in the future