Kernel lock monitor scripts - #29
Open
shakeelb wants to merge 3 commits into
Open
Conversation
The pagefaultlatency tool was added in commit 1327ee7 ("Add new tool pagefaultlatency") but no corresponding entry was added to TOOLS.md, so the tool does not show up in the tools list. Add it, keeping the list alphabetized. Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
A task can acquire a sleeping lock and then, while still holding it, allocate memory. If that allocation cannot be satisfied immediately the kernel enters direct or memcg reclaim inline in the allocation path, which can run for hundreds of milliseconds. The lock stays held for the whole episode and every waiter is blocked behind it. This is hard to attribute with ordinary lock tooling: a contention profiler reports the lock and its waiters but not that the holder was stuck in reclaim, so the stall looks like lock contention when it is really memory pressure leaking into a critical section. Add a tool that connects the two. It tracks mutex, rw_semaphore and percpu_rw_semaphore (write side) acquisitions system-wide against the vmscan direct and memcg reclaim tracepoints, and on release reports any lock that was held across a reclaim episode: total hold time, how much of it was reclaim, whether a waiter was present, the acquire-site stack and the reclaim-entry stack of the longest episode. Locks still held when tracing stops are reported by END and tagged [INFLIGHT]. --waiters_only=true restricts output to locks that had a waiter at release, which is the subset that actually stalled another task. Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
When a sleeping lock is held for a long time, the useful question is often not what the holder was computing but whether it was running at all. A holder stops running for two very different reasons: it is still on the run queue and simply cannot get a CPU (preemption, CPU contention, cpu.max throttling), or it left the run queue blocked on an event (nested lock, RCU, reclaim, I/O). The two call for opposite fixes. The first is a scheduling or capacity problem and the critical section is fine; the second means something blocking is being done while the lock is held. Lock contention tooling reports neither, only that waiters waited. Add a tool that separates them. It tracks mutex, rw_semaphore and percpu_rw_semaphore (write side) acquisitions system-wide and uses sched_switch/sched_wakeup to split each lock's off-CPU time into a RUNNABLE and a SLEEPING band, reporting for each the accumulated time, the largest single episode, and that episode's stack -- the preempt point for RUNNABLE, the block site for SLEEPING -- alongside the acquire stack, waiter status and the holder's CPU affinity size. A RUNNABLE episode that began while the holder's CFS hierarchy was throttled is tagged [THROTTLED]. Kernels that defer cpu.max throttling to exit-to-userspace never throttle a holder mid-critical-section, so the presence of throttle_cfs_rq_work suppresses the tag. Locks still held when tracing stops are reported by END and tagged [INFLIGHT]. Reports are gated on per-band thresholds (--min_runnable_us, --min_sleep_us, both defaulting to 5000 us); --waiters_only=true further restricts output to locks that had a waiter at release. Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related tools for attributing long sleeping-lock hold times to what the holder was
actually doing while it held the lock. Both instrument
mutex,rw_semaphoreandpercpu_rw_semaphore(write side) acquisitions and releases system-wide, but answerdifferent questions:
reclaimlockmonitorreports locks held across a memory reclaimepisode (how much of the hold was reclaim, global or memcg, plus the acquire stack and the
reclaim-entry stack), and
runnablelockmonitorusessched_switch/sched_wakeupto splita lock's off-CPU time into RUNNABLE (on the run queue but not executing — preemption, CPU
contention,
cpu.maxthrottling) and SLEEPING (blocked) bands, each with its own stack.Ordinary lock-contention tooling shows the waiters but not why the holder was slow, which
is the part that determines the fix — memory pressure leaking into a critical section, a
capacity problem where the code is fine, or something blocking being done under the lock.
Both were run against bpftrace v0.26.0 on a 6.13 kernel, compile warning-free under
--dry-run, and every example in their READMEs is real captured output with the workloadthat produced it included.