borg2 b24: benchmarking needed

[ThomasWaldmann]

borg2 chunkers got another round of tuning recently, but it depends a lot on the CPU of the host system.

To select the algorithm it just uses env vars now.
It should also select a hardcoded fastest algorithm per platform now (x86-64 and arm64) when no env var is set.

You can help by doing some measurements on your machine(s) - x86-64, arm64 and also others (others will run the generic code that works on any platform). Try to have a relatively idle machine.

Use borg 2.0.0b24.

These are some command lines that can be used for benchmarking, I put a comment into the line I consider fastest currently:

# Normal chunking - only fastcdc and buzhash64 performance is interesting here.
# X86-64 (AMD/Intel)
BORG_FASTCDC_KERNEL=scalar BORG_BUZHASH64_KERNEL=scalar borg benchmark cpu --chunking  # fastest!?
BORG_FASTCDC_KERNEL=blockwise BORG_BUZHASH64_KERNEL=blockwise borg benchmark cpu --chunking
BORG_FASTCDC_KERNEL=avx2 BORG_BUZHASH64_KERNEL=avx2 borg benchmark cpu --chunking
BORG_FASTCDC_KERNEL=avx512 BORG_BUZHASH64_KERNEL=avx512 borg benchmark cpu --chunking
BORG_FASTCDC_KERNEL= BORG_BUZHASH64_KERNEL= borg benchmark cpu --chunking  # correct autoselect?

# Cryptographic chunking - only the *-aes chunker performance is interesting here.
# X86-64 (AMD/Intel)
BORG_AES_CHUNKER_KERNEL=evp borg benchmark cpu --chunking
BORG_AES_CHUNKER_KERNEL=aes-ni borg benchmark cpu --chunking
BORG_AES_CHUNKER_KERNEL=vaes borg benchmark cpu --chunking  # fastest!?
BORG_AES_CHUNKER_KERNEL= borg benchmark cpu --chunking  # correct autoselect?

# Normal chunking - only fastcdc and buzhash64 performance is interesting here.
# ARM64 CPUs (incl. Apple Silicon)
BORG_FASTCDC_KERNEL=scalar BORG_BUZHASH64_KERNEL=scalar borg benchmark cpu --chunking
BORG_FASTCDC_KERNEL=blockwise BORG_BUZHASH64_KERNEL=blockwise borg benchmark cpu --chunking  # fastest!?
BORG_FASTCDC_KERNEL=neon BORG_BUZHASH64_KERNEL=neon borg benchmark cpu --chunking
BORG_FASTCDC_KERNEL= BORG_BUZHASH64_KERNEL= borg benchmark cpu --chunking  # correct autoselect?

# Cryptographic chunking - only the *-aes chunker performance is interesting here.
# ARM64 CPUs (incl. Apple Silicon)
BORG_AES_CHUNKER_KERNEL=evp borg benchmark cpu --chunking
BORG_AES_CHUNKER_KERNEL=aes-arm64 borg benchmark cpu --chunking  # fastest!?
BORG_AES_CHUNKER_KERNEL= borg benchmark cpu --chunking  # correct autoselect?

[ThomasWaldmann]

Excerpt from the change log, new in b24:

Chunkers:

• goldilocks-aes: fix build on 32-bit archs (no __uint128_t there)

• *-aes chunkers: use the aes-arm64 instructions on Linux and FreeBSD.

• speed up the *-aes chunkers on x86-64

• speed up the buzhash64 kernel on arm64 / Apple Silicon, use the blockwise kernel on aarch64 by default (now faster than NEON)

• speed up the fastcdc blockwise kernel (default kernel on platforms other than x86-64 and aarch64)

[ThomasWaldmann]

Results from Apple M3 Pro CPU:

m3.log

[PhrozenByte]

No compression benchmarks with our fancy new --data? 🥲

Here are my results, again running on live systems:

• Intel Core i5-8265U (4 cores, 8 threads): borg2b24-benchmark_i5-8265U.txt
• AMD Ryzen 5700X3D (8 cores, 16 threads): borg2b24-benchmark_5700X3D.txt

The new benchmark results essentially confirm my findings from #10160 (comment):

• The fastcdc chunker with BORG_FASTCDC_KERNEL=scalar performs best
• On my Intel machine, BORG_BUZHASH64_KERNEL=blockwise performs about 15% better than scalar, with scalar being the next-best option; on my AMD machine, scalar performs best; on both machines, buzhash64 is slower than fastcdc
• BORG_AES_CHUNKER_KERNEL=aes-ni performs better than evp, but unfortunately, neither machine supports vaes
• A fixed BORG_ZSTD_MT_WORKERS=4 seems to be best regardless of the core count
  • However, even with high worker counts and long-running tests such as zstd,22 (2 MiB), I still only see one core at 100% utilization and a second core at around 40% utilization - nothing more. Is this expected behavior?
• There is no consistent picture regarding zstd,-4 vs. lz4:
  • zstd,-4 performs better with large chunks in terms of runtime, lz4 performs better with small chunks
  • zstd,-4 generally achieves a better compression ratio than lz4, though the difference is small
  • I'm wondering again whether we should let Borg chunk the Silesia corpus - using the individual files rather than treating it as a single 201 MiB blob - and feed those chunks into the compression benchmark instead, as discussed in borg2 benchmark cpu: Add --data CLI option #10265 (comment); this might give us a clearer picture of whether performance on small or large chunks matters more
  • Have you looked into whether multithreading could also be used with lz4? AFAIK, lz4 supports multithreading as well (according to Phoronix); there is just no simple threads= option like with zstd, so it would be much harder to integrate. But perhaps it would still be worth considering?

For anyone who also wants to run compression benchmarks using the Silesia corpus (note, however, that this will take quite some time since it uses 200 MiB instead of just 10 MiB of test data, but the results will be much more realistic):

curl -o ./silesia.zip https://sun.aei.polsl.pl/~sdeor/corpus/silesia.zip
unzip ./silesia.zip -d ./silesia
mv ./silesia/mozilla ./silesia/mozilla.tar; tar -C ./silesia -xf ./silesia/mozilla.tar; rm ./silesia/mozilla.tar
mv ./silesia/samba ./silesia/samba.tar; tar -C ./silesia -xf ./silesia/samba.tar; rm ./silesia/samba.tar
mv ./silesia/xml ./silesia/xml.tar; mkdir ./silesia/xml; tar -C ./silesia/xml -xf ./silesia/xml.tar; rm ./silesia/xml.tar

BORG_ZSTD_MT_WORKERS=1 borg2 benchmark cpu --compressing --data ./silesia
BORG_ZSTD_MT_WORKERS=2 borg2 benchmark cpu --compressing --data ./silesia
BORG_ZSTD_MT_WORKERS=4 borg2 benchmark cpu --compressing --data ./silesia
BORG_ZSTD_MT_WORKERS=6 borg2 benchmark cpu --compressing --data ./silesia
BORG_ZSTD_MT_WORKERS=8 borg2 benchmark cpu --compressing --data ./silesia

See a few other, partially related findings below. Let me know if you think I should open separate issues for them.

1. If avx512 isn't supported on the local machine, Borg reports an error but still lists it as a "valid value", which is somewhat confusing:

   $ BORG_FASTCDC_KERNEL=avx512 BORG_BUZHASH64_KERNEL=avx512 borg benchmark cpu --chunking
   Chunkers =============================================================
   fastcdc,19,23,21,2         failed: ValueError: BORG_FASTCDC_KERNEL='avx512': this CPU does not support it. Valid values: avx512,avx2,blockwise,scalar
   buzhash64,19,23,21,4095,2  failed: ValueError: BORG_BUZHASH64_KERNEL='avx512': this CPU does not support it. Valid values: avx512,avx2,blockwise,scalar
   

2. Same with vaes:

   $ BORG_AES_CHUNKER_KERNEL=vaes borg benchmark cpu --chunking
   Chunkers =============================================================
   toeplitz-aes,19,23,21,2    failed: ValueError: BORG_AES_CHUNKER_KERNEL='vaes': this CPU does not support it. Valid values: vaes,aes-ni,evp
   

3. If BORG_ZSTD_MT_WORKERS= is used to explicitly select Borg's default number of workers, Borg reports an error, unlike the other environment variables:

   $ BORG_ZSTD_MT_WORKERS= borg benchmark cpu --compressing --data ~/silesia
   Compression ==========================================================
   zstd,-4 (2MiB)             failed: Error: Error: BORG_ZSTD_MT_WORKERS must be an integer, but is: ''
   

4. The runtime duration column is misaligned when a compression benchmark takes >= 10 seconds; it should probably reserve 8 chars and be right-aligned, as some algorithms took just under 100 seconds:

   $ borg benchmark cpu --compressing --data ~/silesia
   Compression ==========================================================
   zstd,5 (128kiB)            200.38 MiB  1.823s     115.3 MB/s    3.04x
   zstd,10 (128kiB)           200.38 MiB  5.458s      38.5 MB/s    3.16x
   zstd,16 (128kiB)           200.38 MiB  36.432s       5.8 MB/s    3.37x
   zstd,22 (128kiB)           200.38 MiB  92.361s       2.3 MB/s    3.38x
   

[ThomasWaldmann]

However, even with high worker counts and long-running tests such as zstd,22 (2 MiB), I still only see one core at 100% utilization and a second core at around 40% utilization - nothing more. Is this expected behavior?

When processing small chunks: yes. When processing big (2MiB) chunks: no.

[ThomasWaldmann]

lz4 multithreading?

Looks like lz4/lz4#1336 added that to the CLI, not to the library.

I will make an experiment what it takes to add this to borg...

[ThomasWaldmann]

failed: Error: Error: BORG_ZSTD_MT_WORKERS must be an integer, but is: ''

Fixed: #10323

[ThomasWaldmann]

Followup to my answer above — it was wrong: one core is expected for zstd,22 (2 MiB).

borg does ask for MT above ZSTD_MT_MIN_SIZE (768 KiB) with job_size=512 KiB, but libzstd raises the effective job size to the overlap size, which for btultra2/btultra (levels 19+) is a whole window — and the window is shrunk to the source size. So a 2 MiB buffer at level 22 becomes exactly one job: one worker, main thread waiting (your ~40% second core).

Measured here (M3 Pro, nb_workers=4, cpu-time/wall-time, 4.0 = perfect):

 level    2 MiB    8 MiB
    -4     2.24     3.59
     3     3.19     3.68
    10     3.42     3.83
    16     1.46     3.43
    19     1.00     1.00
    22     1.00     1.00

Levels <= 10 scale fine at borg chunk sizes, 16 partially, 19+ never (19 needs > 8 MiB, 22 > 128 MiB). BORG_ZSTD_MT_WORKERS can't help there.

Followup for us: the ZSTD_MT_MIN_SIZE gate only looks at job size, so at levels >= 19 we build and tear down a thread pool per chunk for nothing.