Skip to content

[OOC] IO Handler Performance Improvements and Generalization - #2532

Merged
janniklinde merged 1 commit into
apache:mainfrom
janniklinde:ooc/planner-merge-3
Jul 5, 2026
Merged

janniklinde merged 1 commit into
apache:mainfrom
janniklinde:ooc/planner-merge-3

Conversation

@janniklinde

Copy link
Copy Markdown
Contributor

This PR improves performance and generality of OOCMatrixIOHandler. We introduce experimental input/output streams that worked better in practice for OOC spilling in local experiments. We also introduce a lightweight OOCFuture to replace CompletableFuture in hot paths.

@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 53.49794% with 226 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.60%. Comparing base (85ea875) to head (a0cbd53).

Files with missing lines Patch % Lines
...time/ooc/cache/io/OOCBufferedDataOutputStream.java 46.30% 73 Missing and 7 partials ⚠️
.../org/apache/sysds/runtime/ooc/cache/OOCFuture.java 33.89% 72 Missing and 6 partials ⚠️
...ntime/ooc/cache/io/OOCBufferedDataInputStream.java 72.72% 28 Missing and 11 partials ⚠️
...sysds/runtime/ooc/cache/io/OOCMatrixIOHandler.java 57.44% 13 Missing and 7 partials ⚠️
...me/instructions/spark/data/IndexedMatrixValue.java 66.66% 4 Missing and 1 partial ⚠️
.../runtime/ooc/cache/io/SpillableObjectRegistry.java 75.00% 1 Missing and 2 partials ⚠️
...he/sysds/runtime/ooc/cache/io/SpillableObject.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2532      +/-   ##
============================================
- Coverage     71.65%   71.60%   -0.05%     
- Complexity    49328    49395      +67     
============================================
  Files          1580     1585       +5     
  Lines        190516   190967     +451     
  Branches      37364    37424      +60     
============================================
+ Hits         136507   136746     +239     
- Misses        43486    43660     +174     
- Partials      10523    10561      +38     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@janniklinde
janniklinde merged commit c0d1229 into apache:main Jul 5, 2026
50 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in SystemDS PR Queue Jul 5, 2026
Bruno-Ignaz pushed a commit to Bruno-Ignaz/systemds that referenced this pull request Sep 1, 2026
…2532)

Improve performance and generality of OOCMatrixIOHandler. 
Introduce experimental input/output streams that worked better in practice for OOC spilling in local experiments.
Add lightweight OOCFuture to replace CompletableFuture in hot paths.
@janniklinde
janniklinde deleted the ooc/planner-merge-3 branch October 1, 2026 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant