Leios: add LeiosTxCache, but don't rely on it at all yet - #2188
Draft
nfrisby wants to merge 52 commits into
Draft
Conversation
Contributor
Author
|
Most of the lines of code are: the big comments in |
Contributor
Author
|
The same
|
Contributor
Author
|
Benchmarks, on my laptop:
|
nfrisby
commented
Aug 6, 2026
| {-# LANGUAGE BangPatterns #-} | ||
| {-# OPTIONS_GHC -O2 #-} | ||
|
|
||
| -- | A mutable, salted open-addressing hash table for the Leios tx cache: linear |
Contributor
Author
There was a problem hiding this comment.
Issue input-output-hk/ouroboros-leios#1021 says:
- work with modest Leios parameters
- 100kB max EB body
- 1MB max EB closure
I didn't do that; I targeted the full 512 kB body and 12 MB closure case instead.
Thus:
- Maybe the
Referenceimpl is fine, no hash table needed? If the EB body is only 20% of the size this PR was expecting (so ≤~3000 txs instead of ≤~15000), then theMap TxHashworst-case will be 20% of what the benchmark reports. Maybe that's fine? (128 * ~3000 = ~384000, which is still a pretty bigMap, and every major GC has to walk the whole thing). - The 12 MB versus 1 MB closure doesn't relate to the hash table... except for the fact that if we don't need the hash table, then we could just have the
Mapcontain the txs' bytes directly, since that would only be at most ~128 MB (+ GHC heap overhead).- But that doesn't actually save much; merely side-steps having to read the txs from the LeiosDb.
- If we want that, it'd be easy: the
Referencemap is already parameterized bya=(); we'd simply change that toa=ShortByteString, I think? (This PR doesn't alter LeiosFetch at all yet, so I'm not sure what it'd actually need.)
$ cabal bench leios-txcache-bench --benchmark-options=pure
Configuration is affected by the following files:
- cabal.project
Build profile: -w ghc-9.12.2 -O1
In order, the following will be built (use -v for more details):
- ouroboros-consensus-3.0.1.0 (bench:leios-txcache-bench) (file ouroboros-consensus/bench/leios-txcache-bench/Main.hs changed)
Created semaphore called cabal_semaphore_b with 5 slots.
Preprocessing benchmark 'leios-txcache-bench' for ouroboros-consensus-3.0.1.0...
Building benchmark 'leios-txcache-bench' for ouroboros-consensus-3.0.1.0...
[1 of 1] Compiling Main ( ouroboros-consensus/bench/leios-txcache-bench/Main.hs, /home/nfrisby/cardano-haskell/cardano-node/dist-newstyle/build/x86_64-linux/ghc-9.12.2/ouroboros-consensus-3.0.1.0/b/leios-txcache-bench/build/leios-txcache-bench/leios-txcache-bench-tmp/Main.o ) [Source file changed]
[2 of 2] Linking /home/nfrisby/cardano-haskell/cardano-node/dist-newstyle/build/x86_64-linux/ghc-9.12.2/ouroboros-consensus-3.0.1.0/b/leios-txcache-bench/build/leios-txcache-bench/leios-txcache-bench [Objects changed]
Running 1 benchmarks...
Benchmark leios-txcache-bench: RUNNING...
LeiosTxCache worst-case benchmark
EBs in index : 128
txs per EB : 3011
total txs : 385408
generating data... done
populating index... done
== pure-wrapped index ==
population:
time : 1.62 s
allocated : 953.6 MiB
alloc rate : 589.8 MiB/s
residency (post major GC):
live : 73.6 MiB
max live : 73.6 MiB
peak mem in use : 186.0 MiB
lookup — EB batch of 3011 (x20 each):
0% hits (0/3011): avg 2.57 ms, per-tx 852.6 ns, alloc 602.8 KiB
20% hits (603/3011): avg 2.25 ms, per-tx 746.9 ns, alloc 601.9 KiB
40% hits (1205/3011): avg 2.13 ms, per-tx 706.1 ns, alloc 602.2 KiB
60% hits (1807/3011): avg 2.04 ms, per-tx 676.9 ns, alloc 603.0 KiB
80% hits (2409/3011): avg 1.97 ms, per-tx 655.3 ns, alloc 755.2 KiB
100% hits (3011/3011): avg 1.93 ms, per-tx 641.6 ns, alloc 805.8 KiB
batch durations (ascending per column; 20 rows x 6 ratios):
0% 20% 40% 60% 80% 100%
1.87 ms 1.83 ms 1.64 ms 1.60 ms 1.51 ms 1.46 ms
1.92 ms 1.85 ms 1.66 ms 1.64 ms 1.52 ms 1.47 ms
1.95 ms 1.90 ms 1.66 ms 1.67 ms 1.60 ms 1.49 ms
1.99 ms 1.92 ms 1.74 ms 1.73 ms 1.63 ms 1.54 ms
2.09 ms 1.92 ms 1.87 ms 1.73 ms 1.71 ms 1.67 ms
2.18 ms 1.93 ms 1.91 ms 1.77 ms 1.76 ms 1.74 ms
2.31 ms 1.94 ms 1.92 ms 1.80 ms 1.78 ms 1.78 ms
2.35 ms 2.00 ms 1.96 ms 1.85 ms 1.84 ms 1.85 ms
2.38 ms 2.06 ms 2.02 ms 1.89 ms 1.85 ms 1.88 ms
2.51 ms 2.11 ms 2.13 ms 1.91 ms 1.88 ms 1.89 ms
2.54 ms 2.18 ms 2.14 ms 2.00 ms 1.89 ms 1.90 ms
2.57 ms 2.20 ms 2.15 ms 2.03 ms 1.96 ms 1.94 ms
2.59 ms 2.35 ms 2.16 ms 2.08 ms 1.97 ms 1.98 ms
2.72 ms 2.40 ms 2.16 ms 2.15 ms 2.02 ms 2.04 ms
2.78 ms 2.41 ms 2.20 ms 2.16 ms 2.05 ms 2.07 ms
2.79 ms 2.47 ms 2.34 ms 2.20 ms 2.07 ms 2.11 ms
3.05 ms 2.47 ms 2.35 ms 2.25 ms 2.07 ms 2.15 ms
3.15 ms 2.63 ms 2.37 ms 2.25 ms 2.11 ms 2.19 ms
3.18 ms 2.81 ms 2.67 ms 2.57 ms 2.41 ms 2.34 ms
4.43 ms 3.59 ms 3.47 ms 3.49 ms 3.83 ms 3.16 ms
Benchmark leios-txcache-bench: FINISH
Contributor
Author
There was a problem hiding this comment.
And:
$ (cd ../ouroboros-consensus/; git diff)
diff --git a/ouroboros-consensus/bench/leios-txcache-bench/Main.hs b/ouroboros-consensus/bench/leios-txcache-bench/Main.hs
index 5abb36f7f..c87b81415 100644
--- a/ouroboros-consensus/bench/leios-txcache-bench/Main.hs
+++ b/ouroboros-consensus/bench/leios-txcache-bench/Main.hs
@@ -49,7 +49,7 @@ numEbs = maxAnnouncementCount
-- hash + 2-byte size) in a compact, non-CBOR layout — the encoding-independent
-- ceiling. (The current CBOR-precise 'LeiosDemoTypes.maxTxsPerEb' is 13888.)
txsPerEb :: Int
-txsPerEb = 15_058
+txsPerEb = 15_058 `div` 5
-- | Timed repetitions of the batch lookup (plus one warmup).
numLookupRuns :: Int
@@ -103,10 +103,10 @@ main = do
runs :: [(String, IO BenchCache)]
runs = case args of
["pure"] -> [("pure-wrapped index", newPureLeiosTxCache)]
- ["ht"] -> [("hash-table (shift 22)", newHashTableLeiosTxCache 22 salt0 salt1)]
+ ["ht"] -> [("hash-table (shift 20)", newHashTableLeiosTxCache 20 salt0 salt1)]
_ ->
[ ("pure-wrapped index", newPureLeiosTxCache)
- , ("hash-table (shift 22)", newHashTableLeiosTxCache 22 salt0 salt1)
+ , ("hash-table (shift 20)", newHashTableLeiosTxCache 20 salt0 salt1)
]
mapM_ (uncurry runBench) runs
$ cabal bench leios-txcache-bench --benchmark-options=ht
Configuration is affected by the following files:
- cabal.project
Build profile: -w ghc-9.12.2 -O1
In order, the following will be built (use -v for more details):
- ouroboros-consensus-3.0.1.0 (bench:leios-txcache-bench) (first run)
Created semaphore called cabal_semaphore_c with 5 slots.
Preprocessing benchmark 'leios-txcache-bench' for ouroboros-consensus-3.0.1.0...
Building benchmark 'leios-txcache-bench' for ouroboros-consensus-3.0.1.0...
Running 1 benchmarks...
Benchmark leios-txcache-bench: RUNNING...
LeiosTxCache worst-case benchmark
EBs in index : 128
txs per EB : 3011
total txs : 385408
generating data... done
populating index... done
== hash-table (shift 20) ==
population:
time : 146.86 ms
allocated : 190.2 MiB
alloc rate : 1.26 GiB/s
residency (post major GC):
live : 52.5 MiB
max live : 81.1 MiB
peak mem in use : 111.0 MiB
lookup — EB batch of 3011 (x20 each):
0% hits (0/3011): avg 358.67 µs, per-tx 119.1 ns, alloc 602.8 KiB
20% hits (603/3011): avg 341.45 µs, per-tx 113.4 ns, alloc 602.3 KiB
40% hits (1205/3011): avg 361.91 µs, per-tx 120.2 ns, alloc 602.6 KiB
60% hits (1807/3011): avg 290.13 µs, per-tx 96.4 ns, alloc 602.9 KiB
80% hits (2409/3011): avg 401.07 µs, per-tx 133.2 ns, alloc 468.7 KiB
100% hits (3011/3011): avg 282.80 µs, per-tx 93.9 ns, alloc 448.8 KiB
batch durations (ascending per column; 20 rows x 6 ratios):
0% 20% 40% 60% 80% 100%
297.48 µs 277.78 µs 273.22 µs 274.18 µs 317.12 µs 274.50 µs
297.75 µs 280.25 µs 273.76 µs 274.65 µs 318.08 µs 274.63 µs
298.18 µs 285.64 µs 309.87 µs 274.67 µs 319.24 µs 275.14 µs
298.24 µs 301.16 µs 314.82 µs 275.57 µs 319.61 µs 275.24 µs
301.19 µs 308.46 µs 315.06 µs 276.55 µs 328.44 µs 275.45 µs
306.88 µs 314.30 µs 315.61 µs 277.53 µs 329.10 µs 275.56 µs
310.56 µs 314.90 µs 315.66 µs 278.09 µs 341.58 µs 276.29 µs
312.27 µs 315.50 µs 315.93 µs 279.55 µs 346.31 µs 276.71 µs
322.71 µs 316.06 µs 317.01 µs 281.43 µs 351.39 µs 276.98 µs
331.28 µs 319.27 µs 317.02 µs 285.00 µs 351.81 µs 277.36 µs
336.50 µs 324.67 µs 325.87 µs 285.41 µs 358.78 µs 278.16 µs
340.59 µs 334.38 µs 331.20 µs 285.86 µs 361.42 µs 278.45 µs
346.26 µs 336.06 µs 339.37 µs 287.26 µs 368.83 µs 279.72 µs
356.32 µs 341.74 µs 345.32 µs 289.20 µs 381.43 µs 281.43 µs
370.36 µs 355.89 µs 392.86 µs 293.70 µs 418.50 µs 282.09 µs
399.16 µs 362.99 µs 424.78 µs 297.65 µs 482.84 µs 286.69 µs
424.08 µs 396.84 µs 462.74 µs 297.68 µs 497.48 µs 288.40 µs
451.03 µs 434.97 µs 507.09 µs 300.26 µs 596.73 µs 300.49 µs
475.05 µs 436.00 µs 507.70 µs 323.20 µs 603.24 µs 302.11 µs
597.61 µs 472.21 µs 533.34 µs 365.24 µs 629.54 µs 320.61 µs
Benchmark leios-txcache-bench: FINISH
Contributor
Author
There was a problem hiding this comment.
The Reference numbers at 20% cardinality are eerily close to the Optimized numbers at 100% cardinality.
nfrisby
commented
Aug 6, 2026
| action `catch` \e -> traceWith tracer (toTrace e) >> throwIO e | ||
|
|
||
| {------------------------------------------------------------------------------- | ||
| Shadow LeiosTxCache wiring |
Contributor
Author
There was a problem hiding this comment.
Issue input-output-hk/ouroboros-leios#1021 includes
Show what is better with the transaction cache
This PR patently can't do that by observing a running node, since it doesn't alter the LeiosFetch decision logic to use LeiosTxCache instead of scanning querying the LeiosDb. (I'd like to defer that to a separate PR... but maybe it'll just end up being a small commit eventually added to this PR.)
But: perhaps I can add yet another variant to leios-txcache-bench/Main.hs that runs via the SQLite-backed LeiosDb?
Contributor
Author
There was a problem hiding this comment.
commit e7771e77f163b3d4bb7334f8965518a553a7b9b7
Author: Nicolas Frisby <nicolas.frisby@moduscreate.com>
Date: Thu Aug 6 20:46:06 2026 -0400
LeiosTxCache: also benchmark a SQLite impl
On my machine
$ lsblk -d -o NAME,MODEL,SIZE,ROTA,TRAN && findmnt -no SOURCE,FSTYPE -T .
NAME MODEL SIZE ROTA TRAN
nvme0n1 WD_BLACK SN770 2TB 1.8T 0 nvme
/dev/nvme0n1p2 ext4
I see the following:
┌──────────────────────┬───────────┬────────┐
│ │ no reopen │ reopen │
├──────────────────────┼───────────┼────────┤
│ cache small (512 kB) │ 292 ms │ 294 ms │
├──────────────────────┼───────────┼────────┤
│ cache big (256 MB) │ 17 ms │ 339 ms │
└──────────────────────┴───────────┴────────┘
This is for the 1.9 million txs case and using sudo so fadvise(DONT_NEED) works.
If I reduce the tx count to 20%, the averages are are reduced to 20%.
Contributor
Author
There was a problem hiding this comment.
Claue claims that those numbers match that SSD's expectations and that no (reasonable?) SSD would be better for SQLite
Contributor
Author
There was a problem hiding this comment.
The high-level summary: SQLite relies on two caches, its own page cache and the operating system's page cache (ie Virtual Memory).
If we neuter either of them (and it's not at all clear that the adversary can't send messages that would spoil either/both caches), then the latency for checking a single max-tx-count EB against a full SQLite-backed LeiosTxCache is intolerable.
Even if we neuter neither cache, the 17 ms time isn't great.
nfrisby
commented
Aug 8, 2026
| -- | ||
| -- Solution: continue to rely on there being hours of slack, but moreover | ||
| -- actively ensure that slack. In particular, evict EBs as they get "too old", | ||
| -- regardless of whether new EBs have been arriving. For example, the |
Contributor
Author
There was a problem hiding this comment.
TODO reconsider this
For example, the LeiosTxCacheIndex should evict any EBs that are older than the youngest X RBs on the current selection, for X≥128.
The assumption here is that it always takes longer the immutable tip to advance by k-X blocks than it does for the LeiosFetch logic to finish reading txs from the LeiosDb once it successfully looked them up in the LeiosTxCacheIndex.
I'm wavering on whether that's true.
- For a caught-up and well-connected
mainnetnode, lengthening the chain by k-X takes a very long time. - But, if the node were eclipsed for a while and then manages to escape it, then it'll suddenly see many blocks. If they're all empty, then maybe it could acquire and select k-X blocks "very quickly". ... But faster than the LeiosFetch logic could finish its LeiosDb reads? Seems very unlikely, as long as k-X is great enough.
- And on
mainnet, k-X is at least 1500 blocks, so that does seem feasible.
But what about testnets, where k-X might be smaller? And also what about a syncing mainnet node; is that qualitatively different than the node-escaping-eclipse above?
These questions make it tempting to reconsider the other solutions: either LeiosFetch detecting and recovering from hit-prune faults or else explicit pinning.
- Maybe the recovery isn't that complicated: do a first pass on the EbBody to find the txs that are not in the LeiosTxCacheIndex, and issue jobs for those. Also spawn a read of the backing store for the txs that were in the index. Once that read finishes, issue more jobs for any txs whose read failed. Still monotonic, just two-step instead of all at once.
- How could the adversary weaponize the extra delay for that second batch of LeiosFetch jobs being issued?
- Each individual pin should be very brief; only for the duration of reading some of a single EB's txs from the LeiosDb.
- The pinning has to prevent the LeiosDb from deleting the txs---it's not even a restriction on the LeiosTxCache. That coupling seems undesirable, and the only way to avoid it would be to have the LeiosTxCache's backing store be independent. That's doable, but more engineering.
- If those reads are pipelined and overlap for a sustained period, that might extend the duration significantly. And while something is pinned, we either tolerate that the maximum size of the backing store being temporarily exceeded, or else we block insertions? Neither sounds very appealing.
- Perhaps relevant observation: EbBodies can arrive in sustained bursts, but their average over longer time frames (hours) will be close to f=0.05 arrival per slot. So, on average, there will often be nothing pinned.
Contributor
Author
There was a problem hiding this comment.
Maybe the recovery isn't that complicated
In fact... maybe recovery = do nothing?
- This PR reuses the LeiosDb as the backing store. So: if the txs aren't there, then no volatile EB references them.
- The LeiosFetch logic should only be fetching EBs that this node considers to be volatile.
- So: if the txs aren't there, then LeiosFetch doesn't need them; the EB that lead to those reads can't still be volatile.
Edit: ah. The above argument only works if the LeiosDb is deduping txs and promptly, which it currently does. The point: if we're fetching txs for some EB, then that body is already in the LeiosDb, which means it's keeping all of its txs we already have alive. In that case, LeiosFetch doesn't even need to read the txs that the index claims are present, so there's no read that can fail. So there is no possibility of hit-prune fault: neither LeiosFetch nor LeiosVote ever read the bytes of the txs that the LeiosTxCacheIndex claims are present. The hit-prune fault still exists, it just doesn't manifest as promptly as I think. If the index is stale such that it still contains some EB that was already GC'd out of the LeiosDb, then a hit might prevent some txs from being fetched that need to be fetched. That either causes the EB closure to never be complete or it causes it to falsely claim to be complete---both are hit-prune faults.
Only if the LeiosDb weren't deduping txs would LeiosFetch need to read the txs the LeiosTxCacheIndex claims to be present, so that it can copy (ie "dup") them to the EB it's currently fetching. In that case, the source EB could be older than the destination EB, in which case the hit-prune fault could happen and would need to be recovered from (by refetching the txs that we just GC'd from some peers).
Edit: A meta-level point: I'll have more clarity here once I've actually implemented the LeiosFetch logic---the reasoning depends on when exactly various things happens, and I can't see that all yet.
Contributor
Author
There was a problem hiding this comment.
OK, here's my summary of the last couple "thinking out loud comments" (I'm writing these so I'll think about them on Monday, but I'm failing to stop thinking about them now, on Sunday :/.)
- LeiosTxCacheIndex hits should prevent LeiosFetch from immediately sending a request to peers for those txs.
- But, unless we trust the "hours of slack" argument or we add explicit synchronization, then we need a detection and recovery mechanism for the case were the cache index hits were stale: we actually do need to fetch those txs from peers.
- Suppose X was a tx that had a cache hit.
- LeiosFetch will immediately sends requests for the txs that weren't cache hits ✔️
- LeiosFetch should now double-check that X is in the LeiosDb
- If it is, then suppressing the requests for X was correct ✔️
- If it isn't, then now we do need to send those requests after all. They're slightly delayed compared to if the cache hadn't been stale but this case is a rarity (TODO confirm adversary can't induce it).
- The LeiosFetch bookkeeping can still be monotonic, it just has to be slightly more complicated.
- Both cache hits and cache misses lead to jobs being created, but they start as separate jobs: jobs for peer versus jobs for the "double-check" logic. Key point: the total number of outstanding jobs can already be set to the greatest value it'll ever have.
- The double-check logic processes its jobs by converting them into jobs-for-peers whenever the cache hit turned out to be stale. There can be at most one converted job per original job, so the double-check logic can cleverly decrement the outstanding-job count to keep it accurate.
- A simple version of the rule: initialize the outstanding-jobs counter with one double-check job per tx with a cache hit.
- On each confirmed-present tx, decrement the outstanding job count (can be batched).
- On each confirmed-absent tx, add it to an accumulating for-peers job, batched/flushed on some practical heuristics.
- On each flush, issue that for-peers jobs and decrement the outstanding-job count by one less than the number of txs in that new for-peers job.
- A simple version of the rule: initialize the outstanding-jobs counter with one double-check job per tx with a cache hit.
Contributor
Author
There was a problem hiding this comment.
Hrmm...
- Assume the LeiosDb is deduping.
- In particular, we're assuming that once the EbBody is inserted into the LeiosDb, any of its txs that are already in the database cannot be GC'd before this EB is GC'd.
- Assume that the LeiosDb pruning function's first step is to prune the LeiosTxCacheIndex.
- Thus, the cache never falsely claims some tx that was pruned out is present.
- Now, suppose the EbBody arrival handler does these steps in the following order.
- Insert EbBody into LeiosDb.
- Insert EbBody into LeiosTxCacheIndex.
- Issue fetch requests to peers for the EbBody's txs that are in the TxNotYetInserted state in the LeiosTxCacheIndex.
I think that's enough. I'm not seeing what could go wrong.
The key point is: the EB itself is now keeping those txs alive. So the only way reads of those txs could fail is if the EB itself was GC'd... but that means those fails no longer matter; the code that was expecting the read results should also abort, since it's processing an orphaned EB.
There's one downside: a deduping insert like that is going to involve non-trivial on-disk latency, so the requests to peers would be delayed by at least that much.
However, it does seem like a reasonable place to start: no locking and no recovery.
MOREOVER, the latency might not be that bad: the general expectation is that EbBody's will arrive before peers offer the corresponding EbClosure. So as long as this work finishesd before the offers arrive, the "injected latency" is hidden. (The "Issue fetch requests to peers" above is misleading: the EbBody handler could only do that when those peers had already offer that EB's closure, which should be a rare case: EbBodys should usually diffuse faster than their closures... unless the closures are already present, in which case the truly slow part of diffusion already doesn't exist.)
Edit: it'd require some more complexity, but it should be safe to offer the EbBody to our downstream peers as soon as we've confirmed its hash&size match. It's possible their request might arrive before we've finished inserting it into the LeiosDb---that's the complex case we'd have to support. For example, we could insert it into an in-memory store before we start the LeiosDb write and then delete it from that store when the write is over. Our LeiosFetch servers would handle a request by checking the in-memory store first and then falling through to the LeiosDb. Something like that 🤷 would be able to prevent the LeiosDb latency from slowing down the diffusion of EbBodies.
We switch the benchmark's body from Vector TxHash to ByteString in order to match the node's behavior. The Vector TxHash does improve the pure-wrapper's residency, since the Map's keys can be shared with the Vector's elements. Maybe, if the Map were preferred for the real node, then it might be worth the boxing overhead? But for now, we're simply storing the bytes of the body itself, and so the benchmark will do the same.
With these in place, the benchmark outputs are as follows; two separate runs.
In the worst-case scenario, the hash table eliminates ~80% of insertion
allocation, ~120 MiB heap footprint, 95% of ~20 ms per EB-sized batch of
lookups, and _all traversal_ during a major GC.
```
LeiosTxCache worst-case benchmark
EBs in index : 128
txs per EB : 15058
total txs : 1927424
generating data... done
populating index... done
== pure-wrapped index ==
population:
time : 14.00 s
allocated : 4.55 GiB
alloc rate : 332.5 MiB/s
residency (post major GC):
live : 367.8 MiB
max live : 367.8 MiB
peak mem in use : 878.0 MiB
lookup — EB batch of 15058 (x20 each):
0% hits (0/15058): avg 26.46 ms, per-tx 1757.4 ns, alloc 2.1 MiB
20% hits (3012/15058): avg 26.03 ms, per-tx 1728.4 ns, alloc 2.3 MiB
40% hits (6024/15058): avg 25.87 ms, per-tx 1718.1 ns, alloc 2.5 MiB
60% hits (9036/15058): avg 24.49 ms, per-tx 1626.3 ns, alloc 3.0 MiB
80% hits (12047/15058): avg 23.48 ms, per-tx 1559.1 ns, alloc 3.1 MiB
100% hits (15058/15058): avg 21.99 ms, per-tx 1460.2 ns, alloc 3.3 MiB
batch durations (ascending per column; 20 rows x 6 ratios):
0% 20% 40% 60% 80% 100%
25.55 ms 25.16 ms 24.36 ms 23.34 ms 22.58 ms 21.65 ms
25.70 ms 25.21 ms 24.37 ms 23.47 ms 22.59 ms 21.69 ms
25.71 ms 25.22 ms 24.70 ms 23.48 ms 22.60 ms 21.74 ms
25.72 ms 25.23 ms 24.71 ms 23.53 ms 22.61 ms 21.75 ms
25.78 ms 25.24 ms 24.78 ms 23.55 ms 22.64 ms 21.75 ms
25.78 ms 25.28 ms 24.78 ms 23.56 ms 22.83 ms 21.77 ms
25.82 ms 25.33 ms 25.08 ms 23.59 ms 22.92 ms 21.77 ms
25.88 ms 25.43 ms 25.12 ms 23.60 ms 23.05 ms 21.78 ms
25.93 ms 25.44 ms 25.19 ms 23.61 ms 23.20 ms 21.80 ms
26.02 ms 25.46 ms 25.30 ms 23.63 ms 23.32 ms 21.83 ms
26.16 ms 25.62 ms 25.34 ms 23.79 ms 23.36 ms 21.83 ms
26.20 ms 25.79 ms 25.42 ms 24.13 ms 23.45 ms 21.90 ms
26.29 ms 25.79 ms 25.89 ms 24.55 ms 23.56 ms 21.91 ms
26.38 ms 26.00 ms 26.15 ms 24.76 ms 23.64 ms 21.95 ms
26.40 ms 26.12 ms 27.42 ms 24.81 ms 23.67 ms 21.98 ms
26.47 ms 26.24 ms 27.48 ms 25.54 ms 23.68 ms 22.19 ms
27.15 ms 26.66 ms 27.63 ms 26.25 ms 23.73 ms 22.35 ms
28.58 ms 26.73 ms 27.79 ms 26.31 ms 23.76 ms 22.50 ms
28.64 ms 27.71 ms 27.84 ms 26.88 ms 25.82 ms 22.55 ms
29.09 ms 30.87 ms 28.06 ms 27.40 ms 26.52 ms 23.07 ms
```
```
LeiosTxCache worst-case benchmark
EBs in index : 128
txs per EB : 15058
total txs : 1927424
generating data... done
populating index... done
== hash-table (shift 22) ==
population:
time : 846.36 ms
allocated : 840.5 MiB
alloc rate : 993.1 MiB/s
residency (post major GC):
live : 219.9 MiB
max live : 321.1 MiB
peak mem in use : 429.0 MiB
lookup — EB batch of 15058 (x20 each):
0% hits (0/15058): avg 1.75 ms, per-tx 116.5 ns, alloc 2.5 MiB
20% hits (3012/15058): avg 1.81 ms, per-tx 120.2 ns, alloc 2.5 MiB
40% hits (6024/15058): avg 1.68 ms, per-tx 111.5 ns, alloc 2.7 MiB
60% hits (9036/15058): avg 1.73 ms, per-tx 115.0 ns, alloc 2.7 MiB
80% hits (12047/15058): avg 1.82 ms, per-tx 120.7 ns, alloc 2.9 MiB
100% hits (15058/15058): avg 1.82 ms, per-tx 120.9 ns, alloc 2.9 MiB
batch durations (ascending per column; 20 rows x 6 ratios):
0% 20% 40% 60% 80% 100%
1.44 ms 1.53 ms 1.54 ms 1.56 ms 1.62 ms 1.57 ms
1.46 ms 1.53 ms 1.55 ms 1.58 ms 1.63 ms 1.57 ms
1.46 ms 1.54 ms 1.55 ms 1.58 ms 1.64 ms 1.58 ms
1.46 ms 1.54 ms 1.56 ms 1.59 ms 1.65 ms 1.58 ms
1.47 ms 1.55 ms 1.57 ms 1.59 ms 1.65 ms 1.61 ms
1.48 ms 1.55 ms 1.57 ms 1.59 ms 1.67 ms 1.61 ms
1.49 ms 1.55 ms 1.58 ms 1.60 ms 1.68 ms 1.62 ms
1.50 ms 1.56 ms 1.59 ms 1.61 ms 1.69 ms 1.62 ms
1.51 ms 1.56 ms 1.59 ms 1.61 ms 1.70 ms 1.63 ms
1.51 ms 1.58 ms 1.60 ms 1.61 ms 1.71 ms 1.63 ms
1.53 ms 1.59 ms 1.60 ms 1.62 ms 1.72 ms 1.65 ms
1.57 ms 1.59 ms 1.60 ms 1.63 ms 1.75 ms 1.65 ms
1.57 ms 1.60 ms 1.61 ms 1.63 ms 1.78 ms 1.71 ms
1.63 ms 1.64 ms 1.62 ms 1.64 ms 1.80 ms 1.72 ms
1.65 ms 1.67 ms 1.62 ms 1.69 ms 1.81 ms 1.75 ms
2.06 ms 1.70 ms 1.65 ms 1.70 ms 1.86 ms 1.93 ms
2.12 ms 1.98 ms 1.68 ms 1.74 ms 1.88 ms 1.96 ms
2.41 ms 2.25 ms 1.68 ms 1.98 ms 1.89 ms 2.08 ms
2.86 ms 2.43 ms 2.00 ms 2.23 ms 2.23 ms 2.54 ms
2.91 ms 4.25 ms 2.82 ms 2.85 ms 3.00 ms 3.38 ms
```
The LeiosTxCache assumes the announcement has been inserted before its body is inserted. Similarly, the design of EB diffusion assumes upstream peers will send the announcement of a body before they offer that body. The implementation _currently_ doesn't enforce that. And so PR #2132 didn't notice that offers were being sent _before_ announcements. Part of the reason offers were being sent before announcements is because the forge loop itself was writing the EB body to disk. This commit instead has the forge loop innards return the EB so that the outer logic in one place can explicitly send the LeiosNotify announcement _before_ the body is written to the store, since writing the body to the store _triggers_ sending the offer via LeiosNotify. That's the order that the EB diffusion design requires. And it's _also_ the order that TxCache insertion requires, so that same code also does that now.
On my machine ``` $ lsblk -d -o NAME,MODEL,SIZE,ROTA,TRAN && findmnt -no SOURCE,FSTYPE -T . NAME MODEL SIZE ROTA TRAN nvme0n1 WD_BLACK SN770 2TB 1.8T 0 nvme /dev/nvme0n1p2 ext4 ``` I see the following: ┌──────────────────────┬───────────┬────────┐ │ │ no reopen │ reopen │ ├──────────────────────┼───────────┼────────┤ │ cache small (512 kB) │ 292 ms │ 294 ms │ ├──────────────────────┼───────────┼────────┤ │ cache big (256 MB) │ 17 ms │ 339 ms │ └──────────────────────┴───────────┴────────┘ This is for the 1.9 million txs case and using sudo so fadvise(DONT_NEED) works. If I reduce the tx count to 20%, the averages are are reduced to 20%.
Also, float the initialization of LeiosTxCache out to before ChainDB, so we can pass a callback into ChainDB.
…EbBodies If we already have the EB body _and it's still marked as `BodyAlreadyInserted` in the LeiosTxCache_, don't add it to `missingEbBodies`.
…EbTxs If we already have the tx _and it's still marked as `TxAlreadyInserted` in the LeiosTxCache_, don't add it to `missingEbTxs`.
The LeiosTxCache supplants the need for this functions. The corresponding checks now has a (tunable) false miss rate, but it also now has in-memory latency instead of requiring trips to the SQLite backing store. Prior to this commit, the protototype's LeiosFetch decision loop reconciled its outstanding set against the on-disk LeiosDb once per iteration (the function was named `filterMissingWork`) — a synchronous by-hash membership probe sitting directly on the path from "a peer offers an EB body/closure" to "we issue the fetch request(s)." The `leios-txcache-bench` microbenchmark quantifies that probe: for just a single (full) EB's transactions, the SQLite lookup runs about 10× slower than the in-memory index when its pages are warm and about 190× slower when they're cold — hundreds of milliseconds versus on the order of a millisecond. Because that probe sat on the decision path, its execution time was added to per-hop fetch latency on every hop of an EB body/closure: a steady latency tax in the warm case, and — if an adversary can force the page cache cold, which we cannot prove they cannot — a hundreds-of-milliseconds latency spike in the cold case (multiple by the number of (full) in-flight EBs in whole outstanding set). This patch replaces the probe with the in-memory LeiosTxCache, so the decision consults an always-ready in-memory index (about a millisecond, independent of DB size and page-cache state) instead of waiting on the database. That shaving of per-hop latency directly benefits Leios diffusion, which the security argument depends on — and because the DB probe was the one per-iteration cost that could spike ~unboundedly with DB size and cache state, removing it is a necessary step toward running the fetch loop more often than its current meager 2 Hz.
The msgLeiosBlock handler always looks up the txs immediately after inserting the body, so fuse those lookups with the body's ref count bumping pass (when it's not a no-op).
`packedRequest` was the function that could violate it, but now the `LeiosFetchDecisions` data type that carries the fetch decisions to `packedRequest` no longer provides that degree of freedom: it's provides one SlotNo-EbHash pair per tx, and a new test requires its real pair. The LeiosFetch code was inherited from the exploratory burst demo in October 2025. One of the things that demo tried out was issuing requests for EBs where the EbSlot accompanying the EbHash was not necessarily the slot of an announcement of that hash, but instead the _potentially greater_ slot of a _potentially different_ EB that shared that tx. In order to get a higher-priority EB's tx, we could fetch it from a peer who hadn't even (yet) offered that EB but had offered an EB that we know shares txs with it. Subsequent LeiosFetch design ruled this optimization out; it's a DoS vector. If the higher-priority EB is adversarial and withheld, then the victim might never fetch the lower-priority honest EB's txs from its peers that would actually serve them. If the peer had also offered the higher-priority EB, then the request could simply list that hash instead. Thus, it's not an actual loss to eliminate this optimization. This commit does so because this complexity was muddling a bug hunt (next couple commits fix that bug; `filterMissingWork` was masking it). When the reply arrives, the priority was being interpreted as the EbHash's actual slot, which was arming the bug. As of the bugfix, that wouldn't actually cause any problems anymore, but it's still preferable to just remove the ultimately undesired feature's complexity.
Prior to this commit, the msgLeiosBlockTxs handler was only removing its arrived
tx hashs from `missingEbTxs` _for the slotNo_ carried by the
MsgLeiosBlockTxsRequest that incurred this reply. That has two consequences.
- First, it was causing a crash, since it resulted in `missingEbTxs` and
`reverseEbIndexByTx` falling out of sync. (The tx was _completely_ deleted
from `reverseEbIndexByTx`.)
- Second, the persistence in `missingEbTxs` would lead to additional
requests. (Were it not for `filterMissingWork`, see below.)
But the Leios prototype's current LeiosDb and current LeiosFetch logic dedup EB
closures, so the arrival of a tx for one EB discharges it for all other EBs as
well. Additional requests for that tx shouldn't be redundantly sent on behalf of
those EBs.
If the LeiosDb didn't dedup EB closures, then this augmentation of the deletion
logic would still be correct, but something (eg the msgLeiosBlockTxs, the next
decision logic iteration, or etc) would need to _also_ store the arrived tx to
the closure of other EBs that reference it.
In the imminent LeiosFetch logic rewrite, we'll no longer be tracking the
overlap between EB closures, and this bugfix will be trampled. But until that
rewrite, this is a correct bugfix for the Leios prototype's current LeiosFetch
and its current LeiosDb.
Notes:
- This bug (both the crash and the redundant tx fetches) surfaced now because
the recently removed `filterMissingWork` was masking it.
- For EB bodies, the LeiosFetch decisions are simple: they don't try to dedup.
So, announcements of the same EB from multiple slots will arise in redundant
fetches of the body. We consider that harmless for bodies.
- Honest nodes will very rarely announce the same EB body, so this is not a
worthwhile optimization---it's not part of the work-preservation argument.
- An announcement is entirely free to pick which EB it announces, so this
would be nothing more than an optimization, one the adversary could always
trivially choose to avoid.
- In some sense, this commit supplants `95bff62b5c0ab0937dc03e964c9e24bc731b1399
LeiosFetch: bugfix, only emit MsgLeiosBlockTxsRequest with a real LeiosPoint`,
but in another sense, that other commit wasn't merely avoiding the crash.
I'll be shocked if this isn't the correct fix; see the upstream definitions here https://github.com/IntersectMBO/cardano-base/blob/12168e4b32b44d30dd401010ccd969accaf2add7/cardano-crypto-class/src/Cardano/Crypto/DSIGN/Class.hs#L285-L287
I erroneously removed acquiredEbBodies alongside filterMissingWork, replacing _both_ with the LeiosTxCache. That was a mistake for at least these reasons: - acquiredEbBodies had important relationships to data under the outstanding lock, and the LeiosTxCache is not under that lock, so using it instead requires multi-lock coordination. Unsurprisingly, the first attempt at that coordination was wrong/under-appreciated the challenge, and so had a bug: the missingEbBodies and LeiosTxCache could get out of sync in a way that resulted in the node constantly fetching an EB body over and over. - acquiredEbBodies merely contains the points, so it's OK for it to include _all_ volatile EBs---we don't _need_ to accept the false negatives that the cache would bring. However, replacing acquiredEbBodies with LeiosTxCache provided a couple of improvements. So this commit not only reintroduces acquiredEbBodies but also improves it in those same ways: - It's now pruned as the immutable tip advances. - It's directly updated by the forge (the filterMissingWork sledgehammer had been compensating for the old acquiredEbBodies not doing that). Now that it's being pruned, some of its uses needed to be enriched to correctly handle events (eg late message arrivals) related to data that has already been pruned out; hence the addition of acquiredEbBodiesPrunedSlot. ----- There were also some minor improvements done in passing.
With this commit, the relevant regression test fails.
```
a concurrent offer and body arrival never leave a held EB body listed (IOSimPOR): FAIL (0.04s)
*** Failed! Falsified (after 1 test):
Schedule control: ControlAwait [ScheduleMod (Thread {2,2}.14) ControlDefault [Thread {2}.7,Thread {2}.8,Thread {1}.0,Thread {1}.1]]
Thread {2,2} delayed at time Time 0s
until after:
Thread {1}
Thread {2}
held EB body still listed for fetching: [47f6c6404a56ea658d0b40e31c47eba73beaad3d3d71a5e61fc0e90ea48d6d99]
Use --quickcheck-replay="(SMGen 17010102127566570440 17528764287311168251,0)" to reproduce.
Use -p '/LeiosDemoLogic.Invariants/&&/a concurrent offer and body arrival never leave a held EB body listed (IOSimPOR)/' to rerun this test only.
```
That's because this commit (temorarily!) re-introduces the bug that was by
reintroducing acquireEbBodies.
…est" This reverts commit 60345472159e4ac0e0569ac4368539a7a634fd50.
The majority of each handler is now shared among the two sources of data: upstream peers and our own forge. This avoids code duplication and drift--we only have to get _most_ of the logic (eg management of the LeiosFetch bookkeeping state) correct once and in one place, since the forge is, in many ways, "just another source" for announcements/bodies/closures.
nfrisby
force-pushed
the
nfrisby/leios-first-txcache-increment
branch
2 times, most recently
from
August 14, 2026 16:32
cbadcca to
9b740fe
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is half of the implementation of Issue input-output-hk/ouroboros-leios#1021. The second half will be for LeiosFetch (and LeiosVoting, once it exists) to actually use the LeiosTxCache.
Suggestions:
LeiosTxCacheheader for a summary of the new component.cardano-crypto. But it still maybe deserves review by one of our crypto experts, since if it's attackable then our hash table's performance is unpredictable.