Skip to content

benchmarks.md

Latest commit

 

History

History
148 lines (132 loc) · 31 KB

benchmarks.md

File metadata and controls

148 lines (132 loc) · 31 KB
Raw

Masking Benchmarks Report (.NET 10.0)

Introduction

This report provides reproducible BenchmarkDotNet measurements comparing DragoAnt.System.Text.Json.Observer 2.0 against standard baseline implementations and the most popular open-source JSON masking alternative (JsonMasking by Thiago Barradas).

Evaluation Goals

  1. Memory Allocation: Measure managed heap allocations across different payload shapes (Flat, Nested, Array) and sizes (1 KB, 8 KB, 64 KB).
  2. Throughput & Speed: Compare execution latency under high-load request/response logging conditions.
  3. Large Object Heap (LOH) Behavior: Detect allocations crossing the 85,000-byte threshold that trigger expensive Gen2 garbage collection cycles.

Test Environment

  • BenchmarkDotNet: v0.15.8, InProcessEmitToolchain
  • OS: Windows 11 Pro (25H2)
  • CPU: 11th Gen Intel Core i7-11800H @ 2.30 GHz (8 physical / 16 logical cores)
  • Runtime: .NET 10.0.12 (X64 RyuJIT x86-64-v4)
  • Harness Source: MaskBenchmarks.cs

Executive Summary

Figures from the raw table below (bytes path = ReadOnlySpan<byte> into a reused IBufferWriter<byte>, observer built once):

Payload Shape & Size Observer (bytes path) DOM JsonNode JsonMasking 2.0.0 Speedup vs JsonMasking Memory vs JsonMasking
1 KB Flat 664 B / 8.3 µs 11,000 B / 8.0 µs 57,633 B / 98.0 µs 11.8× faster 87× less
8 KB Flat 665 B / 70.8 µs 86,865 B / 61.8 µs 468,078 B / 902.0 µs 12.7× faster 704× less
64 KB Flat 670 B / 534.0 µs 709,628 B (LOH) 3,594,477 B (LOH) / 8,173.0 µs 15.3× faster 5,364× less
64 KB Array 60,726 B / 639.6 µs 923,927 B (LOH) 3,904,063 B (LOH) / 7,538.2 µs 11.8× faster 64× less

Key Findings

  1. Small, bounded allocations on the bytes path: a constant ~0.7 KB per call for flat bodies; it grows with nesting and array length in this run (1.2–44 KB nested, 0.9–61 KB arrays) because the benchmark's rules hand decoded values to masking functions.
  2. No Large Object Heap: DOM maskers allocate 0.7–3.9 MB on 64 KB bodies, straight into the LOH; the observer stays far below the 85,000-byte threshold.
  3. Linear speed: time grows linearly with body size, at 2.0–3.3× a bare unmasked reader-writer copy.

These numbers were measured before the final 2.0 changes. Since then, rules with a constant or tag mask no longer decode the value, and the repository's allocation test (AllocationTests) pins 0 B per warm call on the bytes path for 1 KB to 64 KB bodies, nested and arrays included. Re-run the benchmarks to refresh this report:

dotnet run -c Release --project DragoAnt.System.Text.Json.Observer.Benchmarks

Full BenchmarkDotNet Results


BenchmarkDotNet v0.15.8, Windows 11 (10.0.26200.9457/25H2/2025Update/HudsonValley2)
11th Gen Intel Core i7-11800H 2.30GHz, 1 CPU, 16 logical and 8 physical cores
.NET SDK 10.0.401
  [Host] : .NET 10.0.12 (10.0.12, 10.0.1226.42308), X64 RyuJIT x86-64-v4

Job=Audit  Toolchain=InProcessEmitToolchain  IterationCount=5  
LaunchCount=1  WarmupCount=2
Method Shape Size Mean Error StdDev Median Ratio RatioSD Gen0 Gen1 Gen2 Allocated Alloc Ratio
'LowerBound copy string->string' Flat 1024 2.992 μs 0.3294 μs 0.0510 μs 2.984 μs 1.00 0.02 0.2670 - - 3352 B 1.00
'LowerBound copy bytes->pooled IBufferWriter' Flat 1024 2.708 μs 0.0316 μs 0.0049 μs 2.709 μs 0.91 0.01 - - - - 0.00
'Lib JsonObserver.Mask string->string (built once)' Flat 1024 8.578 μs 0.7758 μs 0.1201 μs 8.613 μs 2.87 0.06 0.1831 - - 2472 B 0.74
'Lib JsonObserver.Mask bytes->pooled IBufferWriter (built once)' Flat 1024 8.302 μs 1.8867 μs 0.4900 μs 8.408 μs 2.78 0.16 0.0458 - - 664 B 0.20
'Hand-rolled span masker string->string' Flat 1024 3.097 μs 0.3749 μs 0.0580 μs 3.123 μs 1.04 0.02 0.2480 - - 3128 B 0.93
'Hand-rolled span masker bytes->pooled IBufferWriter' Flat 1024 3.121 μs 0.1707 μs 0.0443 μs 3.137 μs 1.04 0.02 - - - - 0.00
'DOM JsonNode.Parse->replace->ToJsonString' Flat 1024 8.041 μs 0.4421 μs 0.1148 μs 8.003 μs 2.69 0.05 0.8698 0.0153 - 11000 B 3.28
'JsonMasking 2.0.0' (DOM ThiagoBarradas) Flat 1024 98.029 μs 7.5102 μs 1.9504 μs 98.333 μs 32.77 0.78 4.5166 - - 57633 B 17.19
'Regex [GeneratedRegex] over UTF-16' Flat 1024 1.666 μs 0.1972 μs 0.0512 μs 1.644 μs 0.56 0.02 0.1469 - - 1848 B 0.55
'LowerBound copy string->string' Flat 8192 22.053 μs 1.8699 μs 0.4856 μs 21.812 μs 1.00 0.03 1.9531 - - 24880 B 1.00
'LowerBound copy bytes->pooled IBufferWriter' Flat 8192 22.128 μs 3.8387 μs 0.5940 μs 22.124 μs 1.00 0.03 - - - - 0.00
'Lib JsonObserver.Mask string->string (built once)' Flat 8192 64.473 μs 6.3769 μs 1.6561 μs 63.792 μs 2.92 0.09 1.2207 - - 16817 B 0.68
'Lib JsonObserver.Mask bytes->pooled IBufferWriter (built once)' Flat 8192 70.760 μs 16.7709 μs 4.3553 μs 70.909 μs 3.21 0.19 - - - 665 B 0.03
'Hand-rolled span masker string->string' Flat 8192 25.448 μs 2.2305 μs 0.5793 μs 25.365 μs 1.15 0.03 1.9531 0.0305 - 24648 B 0.99
'Hand-rolled span masker bytes->pooled IBufferWriter' Flat 8192 24.820 μs 2.3048 μs 0.5985 μs 24.622 μs 1.13 0.03 - - - - 0.00
'DOM JsonNode.Parse->replace->ToJsonString' Flat 8192 61.758 μs 9.1446 μs 1.4151 μs 61.475 μs 2.80 0.08 6.8359 0.8545 - 86865 B 3.49
'JsonMasking 2.0.0' (DOM ThiagoBarradas) Flat 8192 902.005 μs 82.7119 μs 21.4800 μs 901.417 μs 40.92 1.21 37.1094 5.8594 - 468078 B 18.81
'Regex [GeneratedRegex] over UTF-16' Flat 8192 2.964 μs 0.3277 μs 0.0507 μs 2.973 μs 0.13 0.00 1.2856 - - 16192 B 0.65
'LowerBound copy string->string' Flat 65536 264.874 μs 25.3715 μs 6.5889 μs 265.202 μs 1.00 0.03 41.5039 41.5039 41.5039 197180 B 1.000
'LowerBound copy bytes->pooled IBufferWriter' Flat 65536 167.302 μs 13.6379 μs 3.5417 μs 168.728 μs 0.63 0.02 - - - 1 B 0.000
'Lib JsonObserver.Mask string->string (built once)' Flat 65536 576.396 μs 83.6984 μs 21.7362 μs 564.653 μs 2.18 0.09 39.0625 39.0625 39.0625 131752 B 0.668
'Lib JsonObserver.Mask bytes->pooled IBufferWriter (built once)' Flat 65536 534.003 μs 81.7341 μs 21.2261 μs 528.342 μs 2.02 0.09 - - - 670 B 0.003
'Hand-rolled span masker string->string' Flat 65536 264.493 μs 10.0491 μs 2.6097 μs 263.954 μs 1.00 0.02 39.5508 39.5508 39.5508 196936 B 0.999
'Hand-rolled span masker bytes->pooled IBufferWriter' Flat 65536 193.701 μs 33.0041 μs 8.5711 μs 191.940 μs 0.73 0.03 - - - 1 B 0.000
'DOM JsonNode.Parse->replace->ToJsonString' Flat 65536 742.092 μs 94.9202 μs 24.6505 μs 747.238 μs 2.80 0.11 71.2891 71.2891 71.2891 709628 B 3.599
'JsonMasking 2.0.0' (DOM ThiagoBarradas) Flat 65536 8,172.993 μs 1,029.1201 μs 267.2592 μs 8,064.694 μs 30.87 1.15 296.8750 125.0000 62.5000 3594477 B 18.229
'Regex [GeneratedRegex] over UTF-16' Flat 65536 48.677 μs 10.5725 μs 1.6361 μs 48.820 μs 0.18 0.01 39.9780 39.9780 39.9780 131128 B 0.665
'LowerBound copy string->string' Nested 1024 3.388 μs 0.3937 μs 0.1022 μs 3.322 μs 1.00 0.04 0.3052 - - 3840 B 1.00
'LowerBound copy bytes->pooled IBufferWriter' Nested 1024 3.350 μs 0.3956 μs 0.1027 μs 3.369 μs 0.99 0.04 - - - - 0.00
'Lib JsonObserver.Mask string->string (built once)' Nested 1024 9.602 μs 0.8120 μs 0.2109 μs 9.679 μs 2.84 0.10 0.2289 - - 3056 B 0.80
'Lib JsonObserver.Mask bytes->pooled IBufferWriter (built once)' Nested 1024 9.656 μs 0.6462 μs 0.1678 μs 9.660 μs 2.85 0.09 0.0916 - - 1224 B 0.32
'Hand-rolled span masker string->string' Nested 1024 3.740 μs 0.4626 μs 0.1201 μs 3.749 μs 1.10 0.04 0.2594 - - 3320 B 0.86
'Hand-rolled span masker bytes->pooled IBufferWriter' Nested 1024 3.631 μs 0.4597 μs 0.0711 μs 3.628 μs 1.07 0.03 - - - - 0.00
'DOM JsonNode.Parse->replace->ToJsonString' Nested 1024 10.164 μs 0.2979 μs 0.0461 μs 10.186 μs 3.00 0.08 1.1292 0.0305 - 14192 B 3.70
'JsonMasking 2.0.0' (DOM ThiagoBarradas) Nested 1024 154.137 μs 11.8757 μs 3.0841 μs 153.302 μs 45.53 1.49 5.1270 - - 66401 B 17.29
'Regex [GeneratedRegex] over UTF-16' Nested 1024 3.086 μs 0.0838 μs 0.0130 μs 3.084 μs 0.91 0.03 0.1488 - - 1872 B 0.49
'LowerBound copy string->string' Nested 8192 22.014 μs 3.0711 μs 0.7975 μs 22.287 μs 1.00 0.05 2.0447 0.0305 - 25840 B 1.00
'LowerBound copy bytes->pooled IBufferWriter' Nested 8192 22.608 μs 1.4847 μs 0.2298 μs 22.661 μs 1.03 0.04 - - - - 0.00
'Lib JsonObserver.Mask string->string (built once)' Nested 8192 62.873 μs 1.6150 μs 0.2499 μs 62.814 μs 2.86 0.10 1.5869 - - 20081 B 0.78
'Lib JsonObserver.Mask bytes->pooled IBufferWriter (built once)' Nested 8192 66.633 μs 6.7674 μs 1.0473 μs 66.755 μs 3.03 0.11 0.4883 - - 6265 B 0.24
'Hand-rolled span masker string->string' Nested 8192 24.437 μs 2.4713 μs 0.3824 μs 24.438 μs 1.11 0.04 1.8005 - - 22632 B 0.88
'Hand-rolled span masker bytes->pooled IBufferWriter' Nested 8192 26.016 μs 1.4209 μs 0.3690 μs 26.171 μs 1.18 0.04 - - - - 0.00
'DOM JsonNode.Parse->replace->ToJsonString' Nested 8192 71.130 μs 4.8641 μs 1.2632 μs 70.570 μs 3.23 0.12 7.8125 0.9766 - 99129 B 3.84
'JsonMasking 2.0.0' (DOM ThiagoBarradas) Nested 8192 1,309.808 μs 114.4076 μs 29.7113 μs 1,309.034 μs 59.56 2.37 37.1094 5.8594 - 478532 B 18.52
'Regex [GeneratedRegex] over UTF-16' Nested 8192 17.945 μs 0.8432 μs 0.2190 μs 17.991 μs 0.82 0.03 1.0986 - - 13856 B 0.54
'LowerBound copy string->string' Nested 65536 251.332 μs 33.9520 μs 8.8172 μs 249.881 μs 1.00 0.05 41.5039 41.5039 41.5039 197372 B 1.000
'LowerBound copy bytes->pooled IBufferWriter' Nested 65536 157.883 μs 7.2589 μs 1.1233 μs 157.825 μs 0.63 0.02 - - - 1 B 0.000
'Lib JsonObserver.Mask string->string (built once)' Nested 65536 552.029 μs 33.5157 μs 8.7039 μs 551.423 μs 2.20 0.08 33.2031 33.2031 33.2031 152037 B 0.770
'Lib JsonObserver.Mask bytes->pooled IBufferWriter (built once)' Nested 65536 485.504 μs 30.1312 μs 7.8250 μs 487.211 μs 1.93 0.07 3.4180 - - 44346 B 0.225
'Hand-rolled span masker string->string' Nested 65536 279.038 μs 18.4593 μs 4.7938 μs 276.245 μs 1.11 0.04 33.2031 33.2031 33.2031 173601 B 0.880
'Hand-rolled span masker bytes->pooled IBufferWriter' Nested 65536 199.250 μs 39.1222 μs 10.1599 μs 196.758 μs 0.79 0.04 - - - 1 B 0.000
'DOM JsonNode.Parse->replace->ToJsonString' Nested 65536 665.805 μs 91.8661 μs 23.8573 μs 659.861 μs 2.65 0.12 66.4063 33.2031 33.2031 747717 B 3.788
'JsonMasking 2.0.0' (DOM ThiagoBarradas) Nested 65536 10,553.161 μs 593.0347 μs 154.0092 μs 10,552.254 μs 42.03 1.46 281.2500 140.6250 46.8750 3603999 B 18.260
'Regex [GeneratedRegex] over UTF-16' Nested 65536 175.899 μs 14.0589 μs 3.6510 μs 177.243 μs 0.70 0.03 33.2031 33.2031 33.2031 107728 B 0.546
'LowerBound copy string->string' Array 1024 2.377 μs 0.2199 μs 0.0571 μs 2.382 μs 1.00 0.03 0.1945 - - 2464 B 1.00
'LowerBound copy bytes->pooled IBufferWriter' Array 1024 2.332 μs 0.1584 μs 0.0411 μs 2.327 μs 0.98 0.03 - - - - 0.00
'Lib JsonObserver.Mask string->string (built once)' Array 1024 7.171 μs 0.6542 μs 0.1699 μs 7.054 μs 3.02 0.09 0.1526 - - 2008 B 0.81
'Lib JsonObserver.Mask bytes->pooled IBufferWriter (built once)' Array 1024 8.403 μs 4.4652 μs 0.6910 μs 8.548 μs 3.54 0.27 0.0610 - - 912 B 0.37
'Hand-rolled span masker string->string' Array 1024 2.762 μs 0.1185 μs 0.0183 μs 2.759 μs 1.16 0.03 0.1678 - - 2120 B 0.86
'Hand-rolled span masker bytes->pooled IBufferWriter' Array 1024 2.828 μs 0.0983 μs 0.0255 μs 2.841 μs 1.19 0.03 - - - - 0.00
'DOM JsonNode.Parse->replace->ToJsonString' Array 1024 8.258 μs 0.8356 μs 0.2170 μs 8.151 μs 3.47 0.11 0.8392 0.0153 - 10560 B 4.29
'JsonMasking 2.0.0' (DOM ThiagoBarradas) Array 1024 77.036 μs 6.7080 μs 1.7420 μs 77.934 μs 32.42 0.98 3.4180 - - 43537 B 17.67
'Regex [GeneratedRegex] over UTF-16' Array 1024 2.312 μs 0.2782 μs 0.0722 μs 2.285 μs 0.97 0.04 0.0877 - - 1136 B 0.46
'LowerBound copy string->string' Array 8192 24.804 μs 2.5268 μs 0.3910 μs 24.850 μs 1.00 0.02 1.9226 - - 24424 B 1.00
'LowerBound copy bytes->pooled IBufferWriter' Array 8192 25.700 μs 3.4573 μs 0.8979 μs 25.347 μs 1.04 0.04 - - - - 0.00
'Lib JsonObserver.Mask string->string (built once)' Array 8192 75.644 μs 9.1097 μs 2.3658 μs 74.414 μs 3.05 0.10 1.4648 - - 19881 B 0.81
'Lib JsonObserver.Mask bytes->pooled IBufferWriter (built once)' Array 8192 81.245 μs 9.2957 μs 2.4141 μs 80.575 μs 3.28 0.10 0.4883 - - 7633 B 0.31
'Hand-rolled span masker string->string' Array 8192 28.514 μs 3.4918 μs 0.9068 μs 28.666 μs 1.15 0.04 1.6174 - - 20592 B 0.84
'Hand-rolled span masker bytes->pooled IBufferWriter' Array 8192 29.076 μs 2.9791 μs 0.7737 μs 29.279 μs 1.17 0.03 - - - - 0.00
'DOM JsonNode.Parse->replace->ToJsonString' Array 8192 79.305 μs 3.7261 μs 0.5766 μs 79.185 μs 3.20 0.05 8.9111 - - 112953 B 4.62
'JsonMasking 2.0.0' (DOM ThiagoBarradas) Array 8192 834.158 μs 58.3273 μs 15.1474 μs 831.857 μs 33.64 0.73 37.1094 5.8594 - 476382 B 19.50
'Regex [GeneratedRegex] over UTF-16' Array 8192 22.978 μs 2.0121 μs 0.3114 μs 22.995 μs 0.93 0.02 0.9766 - - 12288 B 0.50
'LowerBound copy string->string' Array 65536 293.690 μs 23.0026 μs 5.9737 μs 294.142 μs 1.00 0.03 41.5039 41.5039 41.5039 200612 B 1.000
'LowerBound copy bytes->pooled IBufferWriter' Array 65536 203.025 μs 18.8965 μs 4.9074 μs 202.578 μs 0.69 0.02 - - - 1 B 0.000
'Lib JsonObserver.Mask string->string (built once)' Array 65536 723.163 μs 95.9770 μs 24.9249 μs 714.219 μs 2.46 0.09 32.2266 32.2266 32.2266 162321 B 0.809
'Lib JsonObserver.Mask bytes->pooled IBufferWriter (built once)' Array 65536 639.591 μs 86.6586 μs 13.4105 μs 645.195 μs 2.18 0.06 3.9063 - - 60726 B 0.303
'Hand-rolled span masker string->string' Array 65536 315.506 μs 50.0416 μs 12.9957 μs 310.189 μs 1.07 0.05 32.2266 32.2266 32.2266 168596 B 0.840
'Hand-rolled span masker bytes->pooled IBufferWriter' Array 65536 241.157 μs 11.2084 μs 2.9108 μs 240.868 μs 0.82 0.02 - - - 1 B 0.000
'DOM JsonNode.Parse->replace->ToJsonString' Array 65536 827.664 μs 50.1897 μs 13.0341 μs 828.096 μs 2.82 0.07 75.1953 52.7344 32.2266 923927 B 4.606
'JsonMasking 2.0.0' (DOM ThiagoBarradas) Array 65536 7,538.191 μs 351.1544 μs 91.1937 μs 7,569.994 μs 25.68 0.55 296.8750 156.2500 46.8750 3904063 B 19.461
'Regex [GeneratedRegex] over UTF-16' Array 65536 234.401 μs 6.4708 μs 1.6805 μs 234.726 μs 0.80 0.02 32.2266 32.2266 32.2266 101643 B 0.507