fix(runtime): reclaim credential-home locks held by a reused PID - #611
Merged
mldangelo-oai merged 3 commits intoAug 22, 2026
Merged
Conversation
recoverStaleCredentialHomeLock decided staleness from process.kill alone, so a reused PID was indistinguishable from the original owner and the lock was never reclaimed. The 30 s stale-age escape hatch only applied in the else branch of the PID check, which exempted a lock naming any live PID from expiry no matter how old it was. Acquisition then polled every 25 ms with no timeout. PR openai#448 fixed the malformed-PID half of this and left reused PIDs as separate work. Ownership is now proven by freshness rather than by PID liveness. A held lock refreshes its directory mtime on an unreferenced 5 s heartbeat, and one 30 s horizon now applies on every path, so a lock whose heartbeat has lapsed is reclaimed whoever owns that PID now. A definitively exited owner still recovers immediately. Because a suspended machine wakes with a stale-looking mtime while its owner is still alive, recovery of a lock whose PID looks alive waits one heartbeat interval and abandons the reclaim if the mtime advances. Owners that cannot be identified at all skip that wait, so exited and malformed owners recover as quickly as before.
rohanpoudel2
force-pushed
the
fix/credential-lock-heartbeat
branch
from
August 22, 2026 00:35
e7ca060 to
a0b3673
Compare
The credential-lock heartbeat tests replaced the global timer functions and asserted inside the mock bodies. Bun runs every test file in one process, so while those spies were installed any unrelated setInterval call received the mock: one test failed an assertion attributed to whichever test happened to be running, and the other aborted its own lock wait from the mock's else branch. Neither could be triggered by the code under test. Capture only the heartbeat registration and delegate every other timer call to the real implementation, then assert the interval, its unreferenced timer, and release-time clearing from the test body. The two near-identical mock setups become one helper. No production change and no change in what the tests cover.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Severity: P1. Fixes #228.
recoverStaleCredentialHomeLockdecided whether a credential-home lock was stale fromprocess.kill(owner.pid, 0)alone, so a reused process ID was indistinguishable from the originalowner and the lock was never reclaimed. The 30 s stale-age escape hatch applied only in the
elsebranch of the process-ID check, which exempted a lock naming any accepted live process ID from
expiry no matter how old it was. Acquisition then polled every 25 ms with no timeout.
Merged PR #448 fixed the malformed-process-ID half of #228 and stated that recovery of a genuinely
reused live process ID remained separate work. This is that work.
The lock is held for a whole scan and outlives a killed process, so a leftover owner record can name
an unrelated live process on the next run. A persisted state directory combined with process IDs that
restart from a low number makes the collision easy to hit, and every later scan then waits forever.
Changes
mtime on an unreferenced 5 s heartbeat, cleared when the lock is released. Heartbeat failures are
swallowed so they cannot interrupt a scan.
second expiry. A lock whose heartbeat has lapsed is reclaimed regardless of who owns that process ID
now. This replaces the age check that previously applied only to unidentifiable owners.
ESRCHresult still reclaims at once.its owner is still alive. Recovery of a lock whose process ID looks alive waits one heartbeat
interval and abandons the reclaim if the mtime advanced. Owners that cannot be identified at all
skip that wait, so exited and malformed owners recover as quickly as before. The wait honors the
abort signal.
lock error, credential-home device and inode pinning, and cancellation.
No public CLI surface changes: no new commands, flags, accepted values, environment variables, or
defaults. The remaining half of #228, that the wait itself is silent, is left for a separate change to
keep this diff reviewable.
A second, test-only commit stops the heartbeat tests from replacing the global timer functions for
every caller. They now capture just the heartbeat registration, delegate all other timer calls to the
real implementation, and assert from the test body rather than from inside a mock. Previously an
unrelated
setIntervalcall arriving while those spies were installed could fail an assertionattributed to another test, or abort the lock wait under test.
Testing
From
sdk/typescript/, against this branch onmainatf222faf:pnpm run test— four unseeded whole-suite runs, each 0 failures.pnpm run test --seed 12345— 1,576 passed, 29 skipped, 0 failed across 1,605 tests in 87 files.pnpm run types— passed.pnpm run format— passed, all matched files use Prettier style.git diff --check— passed.New focused coverage in the existing credential-lock suite: a stale lock naming a live process is
reclaimed, a stale lock whose process inspection is denied is reclaimed, a fresh live owner still
blocks and still cancels on abort, a heartbeat keeps a held lock across the stale horizon, and
recovery is abandoned when the owner heartbeat advances. The heartbeat and race tests drive the
captured interval callback and
utimesexplicitly rather than sleeping, and the race test isolatesits module mock in a subprocess. The existing exited-process and malformed-owner tests still pass.
Behavior was also measured directly against the reproduction in #228, on this base before and after
the change:
0and-1, 24 h old1, permission denied, 24 h oldTests use synthetic fixtures and temporary directories. No live scan, network, or credentials were
required. Post-push GitHub Actions checks remain pending.
Two local check failures are unrelated to this branch and reproduce identically on unmodified
mainin the same environment, which was verified by running them against a clean checkout:
audit:prodreports one pre-existing high advisory.package.jsonandpnpm-lock.yamlarebyte-identical to
mainon this branch, so no dependency changed here.check:packagefails in its installed-package smoke because the packed tarball resolves@openai/codexat a prerelease version that the public registry does not serve.Whole-suite runs on this machine also produce occasional 30-second timeouts in I/O-heavy plugin and
publication tests. Unmodified
mainproduced the same class of timeout at a comparable rate, intests that do not touch credential locking, so it is a local resource limit rather than a regression
here.
Risk and rollout
A lock whose owner has not heartbeated for 30 s is now reclaimable even when its recorded process ID
is live. That is the intended fix, and it is the only behavior change for a lock that is still being
heartbeated: a healthy owner is never displaced, because its heartbeat keeps the lock fresh and the
suspend/wake re-check backs off when the mtime advances. If two processes ever do overlap, release
still verifies the owner token and refuses to remove a lock it no longer owns.
Recovery of an unidentifiable or exited owner keeps its previous latency. Only the reused or
suspended live-owner case pays one heartbeat interval, replacing a wait that never ended. No
migration, configuration, schema, or public CLI change is required.
Public disclosure review