Add deterministic parallel GFF processing - #18
Open
Megachile wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parallelize HelixerPost by complete input sequence
Summary
This change adds optional parallelism to HelixerPost:
--workers N
HelixerPost work is divided only at complete sequence boundaries. Each sequence
is processed with the existing algorithm and parameters, and the resulting GFF
blocks are merged in original input order.
The default remains one worker, so existing commands and output are unchanged.
Why
Postprocessing can become a substantial bottleneck for assemblies containing
many thousands of contigs. Those sequences are independent at this stage, so
processing them serially leaves an uncomplicated source of parallelism unused.
This proposal is deliberately limited to postprocessing. It does not change
the neural-network input, model predictions, thresholds, or sequence scope.
Behavior
A sequence is never split between workers.
Work assignment is deterministic.
All workers use the same HelixerPost parameters.
Output is merged in the original sequence order.
The final GFF is written only after every worker succeeds.
A worker failure leaves no apparently complete final output.
--workers 1 produces the same output as the current serial invocation.
Output compatibility
Parallel and serial execution should produce exactly the same GFF, including
feature attributes and ordering. This is enforced as a regression-test
requirement rather than treated as approximate biological equivalence.
In our motivating fragmented test, four workers reduced postprocessing from
about 61 minutes to about 12 minutes while producing a byte-identical GFF. A
smaller, less fragmented test improved only from about 10 minutes to about
8 minutes. The benefit is therefore input-dependent, which is why the option
is explicit and the default is unchanged.
Resource considerations
Additional workers use additional CPU, memory, and temporary output space.
Worker count should remain user-controlled so local and scheduled runs can
avoid oversubscription. Load balancing should use complete-sequence size or row
count while retaining deterministic output order.
Tests
Serial versus parallel whole-file GFF equality.
One, two, and multiple workers.
Sequences spanning multiple HDF5 rows.
Unequal sequence sizes and strands.
Stable ordering across repeated runs.
Empty and single-sequence inputs.
Worker failure and partial-output cleanup.
Invalid worker counts.
Scope
This change does not:
filter or concatenate sequences;
modify inference;
introduce variable-length model input;
change HelixerPost thresholds;
alter default behavior.