Regression: mark_process_dead dropped in 1.3.0 — prometheus multiprocess files accumulate forever

[jorge-medina-infinitusai]

Summary

PR #3565 (commit 82d5a78cb, 2025-10-23) added a prometheus_client.multiprocess.mark_process_dead(self._pid) call in SupervisedProc teardown, so dead job processes' multiprocess metric files get cleaned up. That call was silently dropped three weeks later by the 1.3.0 release commit 8c2b967e7 (PR #3956, 2025-11-16), which wholesale-rewrote livekit/agents/ipc/supervised_proc.py (~350 lines replaced) — apparently from a branch base that predated #3565. No commit in history explicitly removes the call (git log -S mark_process_dead shows only the add), so this looks like an accidental regression rather than a deliberate revert.

It is still absent today on main (41553e109, 2026-08-27) and in the latest release we run, 1.6.10.

Evidence

$ git grep -n mark_process_dead 82d5a78cb -- '*.py'
82d5a78cb:livekit-agents/livekit/agents/ipc/supervised_proc.py:289:
    multiprocess.mark_process_dead(self._pid)

$ git grep -n mark_process_dead livekit-agents@1.6.10 -- '*.py'
(no matches)

$ git grep -n mark_process_dead main -- '*.py'
(no matches)

First-parent blob walk of ipc/supervised_proc.py shows the block present from 82d5a78cb until 8c2b967e7 (1.3.0), gone from then on. The rest of #3565's wiring survived — worker.py still sets up/wipes PROMETHEUS_MULTIPROC_DIR at startup and telemetry/http_server.py still serves a MultiProcessCollector — only the per-job teardown call was lost.

Impact

On long-lived workers, every completed job process leaves its *_<pid>.db files in PROMETHEUS_MULTIPROC_DIR forever (we observed 175 files / ~40 dead PIDs on a pod). Every /metrics render builds a fresh MultiProcessCollector that globs, mmaps, and parses all of them, so render time grows monotonically until Prometheus scrapes exceed the scrape timeout and the worker's series go absent. Pods stay Ready throughout, so the failure is invisible to Kubernetes health checks. We hit this twice in production (fleet-wide Absent - lk_agents_active_job_count alerts), each time "fixed" by a rolling restart.

Two asks

1. Re-add the mark_process_dead call in SupervisedProc teardown (restore what Configure Prometheus in multi process mode #3565 had).
2. Note that mark_process_dead only removes gauge_live*_<pid>.db files — counter_, histogram_, and summary_ files still accumulate even with the call restored. It may be worth extending teardown to unlink all *_<pid>.db suffixes for the dead PID, since the SDK owns the directory contract.

Happy to open a PR for either/both if that helps.

[Ricky-7-Yan]

I reproduced the missing teardown behavior on current \main\ and have a focused restoration ready. The change reintroduces the exact fail-open \mark_process_dead(pid)\ lifecycle hook after the child join and before \Process.close(), with a regression assertion in the existing supervised-process teardown test. I am intentionally not adding the broader wildcard deletion from ask 2 because that changes prometheus_client's storage semantics. Focused pytest and Ruff checks pass; I will open the PR shortly.