ChunkSplitter computes every chunk address but does not expose it, so consumers hash each chunk twice

Summary

ChunkSplitter BMT-hashes every sealed chunk to get the reference it writes into the parent node, then drops that address. onBatch only receives { chunk, key? }. Any consumer that needs the address, which is every consumer that uploads or stamps chunks, has to call calculateChunkAddress(chunk.build()) again. Erasure-coded parity chunks have the same problem the other way round: makeErasureBatch hands them to onChunk before they are hashed, and sealParities hashes them again afterwards.

BMT hashing is pure-JS keccak (127 keccak calls per chunk plus one for the span), so in our measurements the second hash adds about as much time as the whole split.

Where

Version 0.2.2.

Data and intermediate chunks: src/chunk/splitter.ts

elevate() computes the address and keeps it only as ref:

this.pending[level]!.push({
  entry: { chunk: this.chunks[level]! },
  ref: this.chunks[level]!.hash().toUint8Array(),
  span: originalSpan,
})

flushBatch() then forwards only the entry, so ref never reaches onBatch:

this.pendingEntries[level + 1]!.push(...batch.map(p => p.entry))

Parity chunks: src/erasure-coding/batch.ts and sealParities()

makeErasureBatch passes each parity chunk to onChunk unhashed, so the consumer hashes it:

const chunk = chunkFromBytes(bytes)
await onChunk(chunk)
parityEntries.push({ chunk })

sealParities() then hashes the same parity chunk again to write its reference:

for (const { chunk } of parities) {
  this.chunks[level]!.writer.write(new Uint8ArrayReader(chunk.hash().toUint8Array()))
}

The root chunk returned by finalize() is not affected. The splitter never hashes it, so the consumer's hash is the only one.

Impact

Standalone repro (below): 32 MiB of random data, 8,256 chunks, Apple M1, Node 24.13.0.

In an application that splits, wraps in a manifest, and stamps 100 MiB (25,806 chunks), the repeated calculateChunkAddress took 7.8 s of a 30.1 s total. With erasure coding (medium level, 27,780 chunks) it took 8.6 s of 37.0 s. That's about a quarter of the total, spent recomputing values the library already had.

Repro

import { calculateChunkAddress, ChunkSplitter } from '@ethersphere/core-sdk/chunk'

const data = new Uint8Array(32 * 1024 * 1024)
for (let i = 0; i < data.length; i += 65536) crypto.getRandomValues(data.subarray(i, i + 65536))

async function split(onBatch) {
  const start = performance.now()
  const splitter = new ChunkSplitter(onBatch)
  await splitter.append(data)
  await splitter.finalize()
  return Math.round(performance.now() - start)
}

let chunks = 0
const noop = await split(async () => [])
const withAddress = await split(async entries => {
  for (const { chunk } of entries) {
    calculateChunkAddress(chunk.build())
    chunks++
  }
  return []
})
console.log({ chunks, splitOnlyMs: noop, splitAndAddressMs: withAddress })

Output:

{ chunks: 8256, splitOnlyMs: 2607, splitAndAddressMs: 5062 }

Suggested fix

Carry the address on ChunkEntry. The new field is optional, so custom onBatch callbacks that build their own entries keep working.

export type ChunkEntry = { chunk: ChunkBuilder; address?: Reference; key?: Uint8Array }

elevate():

if (this.encrypted) {
  const { address, key } = this.chunks[level]!.encryptedHash()
  const ref = new Uint8Array(64)
  ref.set(address.toUint8Array())
  ref.set(key, 32)
  this.pending[level]!.push({ entry: { chunk: this.chunks[level]!, address, key }, ref, span: originalSpan })
} else {
  const address = this.chunks[level]!.hash()
  this.pending[level]!.push({
    entry: { chunk: this.chunks[level]!, address },
    ref: address.toUint8Array(),
    span: originalSpan,
  })
}

sealParities() reuses the address when there is one:

for (const { chunk, address } of parities) {
  this.chunks[level]!.writer.write(new Uint8ArrayReader((address ?? chunk.hash()).toUint8Array()))
}

makeErasureBatch hashes each parity chunk once and passes the address both ways. An extra trailing argument on onChunk is backward compatible:

onChunk: (chunk: ChunkBuilder, key?: Uint8Array, address?: Reference) => Promise<void>

for (const { chunk, key, address } of batch) {
  await onChunk(chunk, key, address)
}

const chunk = chunkFromBytes(bytes)
const address = chunk.hash()
await onChunk(chunk, undefined, address)
parityEntries.push({ chunk, address })

Each chunk is then hashed exactly once, whether it's a data, intermediate, or parity chunk. Consumers read entry.address or the address argument instead of recomputing it.