fix(amber): enable checkpoint serialization - #7996
Conversation
Automated Reviewer SuggestionsBased on the
|
Backport auto-label reportThis
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🟢 | bs=10 sw=10 sl=64 | 375 | 0.229 | 25,561/33,383/33,383 us | 🟢 -9.0% / 🔴 +141.2% |
| 🔴 | bs=100 sw=10 sl=64 | 767 | 0.468 | 125,793/196,040/196,040 us | 🔴 +28.8% / 🔴 +109.3% |
| ⚪ | bs=1000 sw=10 sl=64 | 912 | 0.557 | 1,084,833/1,207,128/1,207,128 us | ⚪ within ±5% / 🔴 +33.7% |
Baseline details
Latest main 444fc58 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 375 tuples/sec | 373 tuples/sec | 894.86 tuples/sec | +0.5% | -58.1% |
| bs=10 sw=10 sl=64 | MB/s | 0.229 MB/s | 0.228 MB/s | 0.546 MB/s | +0.4% | -58.1% |
| bs=10 sw=10 sl=64 | p50 | 25,561 us | 24,494 us | 11,139 us | +4.4% | +129.5% |
| bs=10 sw=10 sl=64 | p95 | 33,383 us | 36,683 us | 13,843 us | -9.0% | +141.2% |
| bs=10 sw=10 sl=64 | p99 | 33,383 us | 36,683 us | 16,839 us | -9.0% | +98.2% |
| bs=100 sw=10 sl=64 | throughput | 767 tuples/sec | 815 tuples/sec | 1,166 tuples/sec | -5.9% | -34.2% |
| bs=100 sw=10 sl=64 | MB/s | 0.468 MB/s | 0.498 MB/s | 0.711 MB/s | -6.0% | -34.2% |
| bs=100 sw=10 sl=64 | p50 | 125,793 us | 121,321 us | 87,509 us | +3.7% | +43.7% |
| bs=100 sw=10 sl=64 | p95 | 196,040 us | 152,247 us | 93,675 us | +28.8% | +109.3% |
| bs=100 sw=10 sl=64 | p99 | 196,040 us | 152,247 us | 102,153 us | +28.8% | +91.9% |
| bs=1000 sw=10 sl=64 | throughput | 912 tuples/sec | 897 tuples/sec | 1,198 tuples/sec | +1.7% | -23.9% |
| bs=1000 sw=10 sl=64 | MB/s | 0.557 MB/s | 0.547 MB/s | 0.731 MB/s | +1.8% | -23.8% |
| bs=1000 sw=10 sl=64 | p50 | 1,084,833 us | 1,118,932 us | 859,766 us | -3.0% | +26.2% |
| bs=1000 sw=10 sl=64 | p95 | 1,207,128 us | 1,198,616 us | 902,897 us | +0.7% | +33.7% |
| bs=1000 sw=10 sl=64 | p99 | 1,207,128 us | 1,198,616 us | 937,957 us | +0.7% | +28.7% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,533.62,200,128000,375,0.229,25560.62,33383.28,33383.28
1,100,10,64,20,2607.38,2000,1280000,767,0.468,125793.43,196040.00,196040.00
2,1000,10,64,20,21921.87,20000,12800000,912,0.557,1084833.39,1207128.03,1207128.03
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7996 +/- ##
============================================
+ Coverage 93.21% 93.25% +0.04%
- Complexity 4664 4678 +14
============================================
Files 1175 1177 +2
Lines 47646 47685 +39
Branches 5319 5312 -7
============================================
+ Hits 44413 44469 +56
+ Misses 1752 1740 -12
+ Partials 1481 1476 -5
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
What changes were proposed in this PR?
Make
CheckpointStateimplementSerializable, so the existing Kryo binding can persist worker checkpoints.Before: finalize checkpoint → serialize
CheckpointState→ no matching binding →NotSerializableExceptionAfter: finalize checkpoint → Kryo binding → flush/close → restore the original checkpoint payload
The regression test uses a real worker write path, stores a non-empty Unicode payload, reads the record back from sequential storage, and asserts the returned size and restored value. Two existing write-branch tests now await successful completion instead of discarding failures.
Any related issues, documentation, discussions?
Closes #7907
How was this PR tested?
Added positive persistence/round-trip coverage while retaining the existing estimate-only and checkpoint-isolation cases.
Result: 6 tests passed; Scalafix and Scalafmt checks passed. The focused suite used an isolated embedded PostgreSQL database and port.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)