Repository navigation
Conversation
…eting it zmc's SIGHUP reload ran logTerm(); logInit(), which deletes the Logger and creates a new one. The dbQueue thread and the stream socket threads keep running through the reload, and zmDbDo() holds the Logger pointer across its query to restore the database log level afterwards, so the reload freed the object under them (heap use-after-free under ASan). Add logReload(), which re-initialises the existing instance under log_mutex: syslog is closed and reopened and the log file closed, as with a new instance, but the pointer stays valid. Only the reload calls it; logInit() and logTerm() are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zm_rtsp_server reloads with logTerm(); zmLoadDBConfig(); logInit() while its RTSP server threads keep logging, the same pattern as zmc. Use logReload() so the Logger they hold stays valid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Four threads log the way zmDbDo() does (fetch the Logger, change its database level, log, restore the level) while the main thread reloads the logger 300 times. With the reload as logTerm(); logInit(), AddressSanitizer reports a heap-use-after-free; with logReload() the instance stays valid. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
connortechnology
left a comment
There was a problem hiding this comment.
Reviewed and verified with a separate AddressSanitizer build of the test suite.
- Reproduced: with master's logger and the test changed back to
logTerm(); logInit()(what zmc's reload does on master), ASan reports a heap-use-after-free. A worker'sLogger::logPrint()(zm_logger.cpp:487) touches the instancelogTerm()freed. - Fixed: with this PR the same test passes with no reports.
- On current master: merged onto master after #5197, it merges cleanly and the full unit suite passes under ASan (250 cases).
- Locking:
reinitialise()holds the recursivelog_mutexwhileinitialise()runs.initialise()only logs a Debug line, which takeslog_mutexand then the queue's mutex, and neverdb_mutex, so the lock order stays the same as everywhere else. - Other call sites: the only other runtime
logTerm()calls are in the forked children that run EventStartCommand/EventEndCommand. Those are single-threaded, so they aren't affected.
One small thing in the test. Its comment and worker loop describe what zmDbDo() used to do: hold the Logger pointer across a query and set the database level to NOLOG around it. #5197 removed that. The test still exercises the bug, because the workers hold Logger::fetch() and log while the main thread reloads. But the worker could simply fetch and log, and the comment could name the threads that still log during a reload (the dbQueue thread and the stream socket threads).
release-1.38 has the same logTerm(); logInit() pair in zmc.cpp and zm_rtsp_server.cpp, so I'll backport this after merge.
Agreed on the SIGUSR1/2 handler follow-up; that's separate.
TL;DR
logInit(),logTerm(), the level fields and the SIGUSR1/2 handler are untouched. The reload still closes and reopens the log file and syslog, so its output is the same as before.Details: change, behaviour changes, checks
Before
Upstream master
821030e17, ASan build,kill -HUPto each zmc every 0.5-3 s with database debug logging:Change
src/zm_logger.cpp,src/zm_logger.h:Logger::reinitialise()(underlog_mutex: log path from the static config, syslog closed, theninitialise()), andlogReload():logInit()when there is no logger yet, otherwisereinitialise().src/zmc.cpp,src/zm_rtsp_server.cpp: reload withlogReload()instead oflogTerm(); logInit(). zm_rtsp_server has the same pattern; it is its own commit.tests/zm_logger_reload.cpp: reloads while four threads log; use-after-free under ASan on master, passes after.Behaviour changes
logInit()reinitialise in place (wider change, see follow-ups)After
Branch at
47ac523ed(base821030e17).Not run: clients (no API change), multi-server, release-1.38 (backport prepared, not run). Touches
zm_logger.*like #5197; the two merge cleanly together.Possible follow-ups
Offered for discussion, not part of this change:
logPrintreads without the lock, and logs from inside the signal handler, which takeslog_mutexand the database queue's mutex. If the signal lands on a thread already holding one, that thread could deadlock. Read in the code, not reproduced. Candidate: atomic level fields, and a handler that only sets a flag, with the next log line reporting the change. Benefit: removes a data race and a possible hang. Risk: the "Level changed" line appears with the next log line instead of at once.logInit()re-initialises in place, not just reloads. Benefit: one code path. Risk: touches every daemon's startup.refs #5202
🤖 Generated with Claude Code