Repository navigation
perf(kernel): copy a folio before its move window - #20
Merged
Merged
Conversation
A move held a folio's faults for the whole page-by-page copy of 2 MiB from one node to the other, so a block window cost at least one cross-node copy and grew with memory bandwidth and run-queue pressure on the host. Each folio is now copied while it stays mapped. Under the folio lock its mappings are made clean and read-only and its dirty flag cleared first; a write meanwhile maps it writable and dirty again or, through a pin's release, marks the folio dirty. In the window the mappings are cleaned again before the zap: a folio found written is left mapped for a later pass, when its writer has moved on, and the last of three passes copies it again within its window. A PMD mapping needs the folio lock to become writable again; a PTE does not, so a dirty flag left by the zap of a dirty PTE also sends the copy into the window. The window of a folio not written meanwhile is a zap and a page cache swap. The window histogram keeps one entry per folio moved. movebench now bounds the median window by one 2 MiB copy timed on the same machine.
…copy A mapping can turn writable between the window's check for writes and the zap without the folio lock: a write fault on a PTE, mprotect, a NUMA hinting fault. The zap of a PTE leaves its dirty bit in the folio, but the zap of a whole file PMD drops it, so a store through a PMD made writable that way was lost to the copy. Splitting a file PMD keeps the bit, so a PMD is split by zapping one page of it first, and a folio the zap leaves dirty is copied again within its window. This corrects the previous commit's claim that a PMD needs the folio lock to become writable again. A folio pinned when it is about to be copied is refused, as a pin may write without a fault and be released without marking the folio dirty.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A move held faults on a folio for the whole page-by-page copy of 2 MiB across nodes, so every block window cost at least one cross-node copy and grew with memory bandwidth pressure and run-queue waits on a loaded host.
Each folio is now copied before its window, while it stays mapped:
folio_mkclean) and its dirty flag cleared, then the copy runs with the lock dropped.The second commit corrects the first commit's claim that a PMD needs the folio lock to become writable again.
The window of a folio not written meanwhile is a zap and a page cache swap. The window histogram keeps one entry per folio moved.
movebenchnow bounds the median window by one 2 MiB copy timed on the same machine (or the smallest bucket, 64 us), replacing the bound of 16 copies.Measured in the dev VM (two NUMA nodes, debug kernel with KASAN and lockdep),
tests/move.sh:The control run of main's module fails the new median bound; this change passes it.
tests/numa.shandtests/huge.shpass unchanged; no stale or torn grains, PMDs kept, no THP split.🤖 Generated with Claude Code