Skip to content

Add deterministic parallel GFF processing - #18

Open
Megachile wants to merge 1 commit into
usadellab:mainfrom
Megachile:pr/deterministic-parallel-post
Open

Add deterministic parallel GFF processing#18
Megachile wants to merge 1 commit into
usadellab:mainfrom
Megachile:pr/deterministic-parallel-post

Conversation

@Megachile

Copy link
Copy Markdown

Parallelize HelixerPost by complete input sequence
Summary
This change adds optional parallelism to HelixerPost:

--workers N
HelixerPost work is divided only at complete sequence boundaries. Each sequence
is processed with the existing algorithm and parameters, and the resulting GFF
blocks are merged in original input order.

The default remains one worker, so existing commands and output are unchanged.

Why
Postprocessing can become a substantial bottleneck for assemblies containing
many thousands of contigs. Those sequences are independent at this stage, so
processing them serially leaves an uncomplicated source of parallelism unused.

This proposal is deliberately limited to postprocessing. It does not change
the neural-network input, model predictions, thresholds, or sequence scope.

Behavior
A sequence is never split between workers.
Work assignment is deterministic.
All workers use the same HelixerPost parameters.
Output is merged in the original sequence order.
The final GFF is written only after every worker succeeds.
A worker failure leaves no apparently complete final output.
--workers 1 produces the same output as the current serial invocation.
Output compatibility
Parallel and serial execution should produce exactly the same GFF, including
feature attributes and ordering. This is enforced as a regression-test
requirement rather than treated as approximate biological equivalence.

In our motivating fragmented test, four workers reduced postprocessing from
about 61 minutes to about 12 minutes while producing a byte-identical GFF. A
smaller, less fragmented test improved only from about 10 minutes to about
8 minutes. The benefit is therefore input-dependent, which is why the option
is explicit and the default is unchanged.

Resource considerations
Additional workers use additional CPU, memory, and temporary output space.
Worker count should remain user-controlled so local and scheduled runs can
avoid oversubscription. Load balancing should use complete-sequence size or row
count while retaining deterministic output order.

Tests
Serial versus parallel whole-file GFF equality.
One, two, and multiple workers.
Sequences spanning multiple HDF5 rows.
Unequal sequence sizes and strands.
Stable ordering across repeated runs.
Empty and single-sequence inputs.
Worker failure and partial-output cleanup.
Invalid worker counts.
Scope
This change does not:

filter or concatenate sequences;
modify inference;
introduce variable-length model input;
change HelixerPost thresholds;
alter default behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant