Memory Benchmark Matrix

Memory Benchmark Matrix

This page records peak RSS across file sizes and worker counts.

Tracking issue: #183

Related adaptive concurrency work: #78

Test conditions

• Platform: Linux x64
• Node.js: v24.15.0
• Host memory: 29 GiB RAM and 8 GiB swap
• Tool: GNU /usr/bin/time -v
• Benchmark: scripts/benchmark-memory.mjs
• Fixture shape: one title and one long paragraph
• Modes: lint and --fix
• Worker counts: 1, 2, and 4
• File sizes: 1, 3, 5, and 10 MiB
• Adaptive mode: disabled
• Size filtering: disabled

Piscina uses worker threads. GNU time therefore measures the total Node process RSS.

Test process

The first plan used eight files and five runs per cell.

The 1 MiB and one-worker cell produced these results:

• Median peak RSS: 852.7 MiB
• Maximum peak RSS: 887.2 MiB
• Median wall time: 32.44 seconds

The initial result made the full plan unsafe. The test stopped before larger cells ran.

The exploratory matrix used one file per worker. This isolated the active concurrent working set.

Each exploratory cell used a fresh CLI process and one run.

Command shape:

npm run benchmark:memory -- \
  --files <worker-count> \
  --bytes-per-file <bytes> \
  --threads <worker-count> \
  --runs 1

The fix matrix added --fix.

Lint peak RSS

File size	1 worker	2 workers	4 workers
1 MiB	745 MiB	1,506 MiB	2,674 MiB
3 MiB	1,603 MiB	2,951 MiB	5,118 MiB
5 MiB	1,833 MiB	3,875 MiB	6,327 MiB
10 MiB	1,967 MiB	3,487 MiB	7,104 MiB

Fix peak RSS

File size	1 worker	2 workers	4 workers
1 MiB	696 MiB	1,644 MiB	2,741 MiB
3 MiB	1,773 MiB	2,616 MiB	5,351 MiB
5 MiB	1,813 MiB	3,969 MiB	5,925 MiB
10 MiB	1,843 MiB	3,677 MiB	7,249 MiB

Findings

• Peak RSS grows approximately with worker count.
• The worker relationship is noisy but not severely superlinear.
• File-size growth is not linear.
• Single-worker RSS starts to level near 3 to 5 MiB.
• Fix mode does not consistently exceed lint mode.
• Four 10 MiB files need approximately 7.1 GiB peak RSS.

The current adaptive policy prevents the largest measured four-worker peaks.

max < 1 MiB       -> availableParallelism()
1 MiB <= max < 5  -> at most 2 workers
max >= 5 MiB      -> 1 worker

Measured examples:

• 3 MiB with two workers used 2.6 to 3.0 GiB.
• 5 MiB with one worker used approximately 1.8 GiB.
• 10 MiB with one worker used approximately 1.9 GiB.

Important boundary

The policy has a sharp boundary below 1 MiB.

A 0.99 MiB file can still use every available CPU.

The next matrix should test 256 KiB, 512 KiB, and 900 KiB.

It should test 1, 2, 4, 8, and 16 workers.

Limitations

• Each exploratory cell has one run.
• The data shows trends, not precise thresholds.
• The fixture is a synthetic long paragraph.
• Other Markdown shapes can produce different memory use.
• The test covers one Linux host.
• The first five-run cell used eight files.
• Its result is not directly comparable with the exploratory cells.

Run important boundary cells five times before changing policy thresholds.