Skip to content

feat(check): add erasure-coding threshold check - #632

Draft
aloknerurkar wants to merge 3 commits into
masterfrom
feat/erasure-coding-check
Draft

aloknerurkar wants to merge 3 commits into
masterfrom
feat/erasure-coding-check

Conversation

@aloknerurkar

Copy link
Copy Markdown
Contributor

Description

Adds an erasure-coding check for #624. It tests Bee's erasure-coded retrieval against the exact recovery threshold of each redundancy level.

How it works:

  • Random data is split with Bee's own splitter into an in-memory chunk store.
  • Every chunk except a computed set is uploaded through the chunk endpoint from one node, and the file is downloaded from a different node.
  • The root chunk is always uploaded. The splitter's signed root replicas are not uploaded as plain chunks.
  • It runs for MEDIUM, STRONG, INSANE and PARANOID, with the download's redundancy fallback header both on and off.

Cases per intermediate chunk (p = its parity count):

Case Withheld Fallback on Fallback off
parity-only all parities succeeds succeeds
at-threshold p children, data chunks first succeeds fails
over-threshold p+1 children fails fails

Options: levels, data-sizes, all-depths, postage-*, download-timeout and seed. Defaults use 32 KiB data. Config entries are added to config.yaml, local.yaml and testnet-bee-playground.yaml.

Testing:

  • Unit tests run the whole case table through Bee's own joiner with an in-memory store, at every level (go test -race passes).
  • Run on a real bee-local cluster (bootnode plus 5 nodes, Bee 2.8.2): all 24 scenarios passed in about 42 minutes.
  • Earlier runs had two download timeouts that passed on retry. They happened on the same cluster with a 60s download timeout, and passed at 180s. I couldn't prove the timeout was the cause.

Notes for reviewers:

  • go.mod and go.sum change, because the tests import Bee's joiner and getter packages. This includes bumping golang.org/x/oauth2 from 0.30.0 to 0.36.0.
  • config/local.yaml still has download-timeout: 60s and timeout: 1h, which are too short for a slow cluster.
  • Open items: a per-upload timeout, telling download timeouts apart from wrong data in the failure message, and merging with feat/erasure-coding-test-scenario, which tests the same area differently.

@aloknerurkar aloknerurkar changed the title Feat/erasure coding check feat(check): add erasure-coding threshold check Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant