fix: disable loguru variable inspection so runner exceptions surface - #2290
Open
abendrothj wants to merge 1 commit into
Open
fix: disable loguru variable inspection so runner exceptions surface#2290abendrothj wants to merge 1 commit into
abendrothj wants to merge 1 commit into
Conversation
loguru's diagnose mode calls repr() on every local in a traceback; repr of MLX CUDA objects can segfault mid-log, replacing the actual runner exception with an opaque signal 11. Observed masking a missing-CUDA_HOME error as a crash during heterogeneous ring validation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
loguru's
diagnose=True(the default) annotates tracebacks by callingrepr()on every local variable in every frame. On the MLX CUDA backend,repr()of some device objects can segfault the interpreter while the exception is being logged.The result is that the real error never reaches the log — the runner dies with an opaque
signal 11instead. I hit this while validating a heterogeneous Metal + CUDA cluster: a straightforward missing-CUDA_HOMEerror surfaced as an unexplained SIGSEGV, and the actual message was only recoverable by disabling diagnose.This is a diagnosability trap: it corrupts exactly the information you need at exactly the moment you need it, and the more unusual the failure, the more likely the locals are to be unreprable.
Fix
Pass
diagnose=Falseon all three sinks inlogger_setup.Variable inspection is a debugging nicety; tracebacks (
backtrace=True) are unaffected and still show the full call chain. Nothing else changes.