Skip to content

fix(runtime): reclaim credential-home locks held by a reused PID - #611

Merged
mldangelo-oai merged 3 commits into
openai:mainfrom
rohanpoudel2:fix/credential-lock-heartbeat
Aug 22, 2026
Merged

fix(runtime): reclaim credential-home locks held by a reused PID#611
mldangelo-oai merged 3 commits into
openai:mainfrom
rohanpoudel2:fix/credential-lock-heartbeat

Conversation

@rohanpoudel2

@rohanpoudel2 rohanpoudel2 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Severity: P1. Fixes #228.

recoverStaleCredentialHomeLock decided whether a credential-home lock was stale from
process.kill(owner.pid, 0) alone, so a reused process ID was indistinguishable from the original
owner and the lock was never reclaimed. The 30 s stale-age escape hatch applied only in the else
branch of the process-ID check, which exempted a lock naming any accepted live process ID from
expiry no matter how old it was. Acquisition then polled every 25 ms with no timeout.

Merged PR #448 fixed the malformed-process-ID half of #228 and stated that recovery of a genuinely
reused live process ID remained separate work. This is that work.

The lock is held for a whole scan and outlives a killed process, so a leftover owner record can name
an unrelated live process on the next run. A persisted state directory combined with process IDs that
restart from a low number makes the collision easy to hit, and every later scan then waits forever.

Changes

  • Prove ownership by freshness instead of by process-ID liveness. A held lock refreshes its directory
    mtime on an unreferenced 5 s heartbeat, cleared when the lock is released. Heartbeat failures are
    swallowed so they cannot interrupt a scan.
  • Apply one 30 s staleness horizon on every path, reusing the existing threshold rather than adding a
    second expiry. A lock whose heartbeat has lapsed is reclaimed regardless of who owns that process ID
    now. This replaces the age check that previously applied only to unidentifiable owners.
  • Keep immediate recovery for a definitively exited owner, so an ESRCH result still reclaims at once.
  • Keep waiting when a live owner is still heartbeating, so scans stay serialized.
  • Guard the suspend/wake race: a machine that sleeps mid-scan wakes with a stale-looking mtime while
    its owner is still alive. Recovery of a lock whose process ID looks alive waits one heartbeat
    interval and abandons the reclaim if the mtime advanced. Owners that cannot be identified at all
    skip that wait, so exited and malformed owners recover as quickly as before. The wait honors the
    abort signal.
  • Preserve the owner token check on release, the quarantine-then-remove sequence, the non-directory
    lock error, credential-home device and inode pinning, and cancellation.

No public CLI surface changes: no new commands, flags, accepted values, environment variables, or
defaults. The remaining half of #228, that the wait itself is silent, is left for a separate change to
keep this diff reviewable.

A second, test-only commit stops the heartbeat tests from replacing the global timer functions for
every caller. They now capture just the heartbeat registration, delegate all other timer calls to the
real implementation, and assert from the test body rather than from inside a mock. Previously an
unrelated setInterval call arriving while those spies were installed could fail an assertion
attributed to another test, or abort the lock wait under test.

Testing

From sdk/typescript/, against this branch on main at f222faf:

  • pnpm run test — four unseeded whole-suite runs, each 0 failures.
  • pnpm run test --seed 12345 — 1,576 passed, 29 skipped, 0 failed across 1,605 tests in 87 files.
  • pnpm run types — passed.
  • pnpm run format — passed, all matched files use Prettier style.
  • git diff --check — passed.

New focused coverage in the existing credential-lock suite: a stale lock naming a live process is
reclaimed, a stale lock whose process inspection is denied is reclaimed, a fresh live owner still
blocks and still cancels on abort, a heartbeat keeps a held lock across the stale horizon, and
recovery is abandoned when the owner heartbeat advances. The heartbeat and race tests drive the
captured interval callback and utimes explicitly rather than sleeping, and the race test isolates
its module mock in a subprocess. The existing exited-process and malformed-owner tests still pass.

Behavior was also measured directly against the reproduction in #228, on this base before and after
the change:

planted lock before after
exited process ID, 24 h old reclaimed in 3 ms reclaimed in 2 ms
process ID 0 and -1, 24 h old reclaimed in 2-4 ms reclaimed in 2-3 ms
live process ID, 24 h old never reclaimed reclaimed in 5005 ms
live process ID, 30 days old never reclaimed reclaimed in 5007 ms
process ID 1, permission denied, 24 h old never reclaimed reclaimed in 5004 ms

Tests use synthetic fixtures and temporary directories. No live scan, network, or credentials were
required. Post-push GitHub Actions checks remain pending.

Two local check failures are unrelated to this branch and reproduce identically on unmodified main
in the same environment, which was verified by running them against a clean checkout:

  • audit:prod reports one pre-existing high advisory. package.json and pnpm-lock.yaml are
    byte-identical to main on this branch, so no dependency changed here.
  • check:package fails in its installed-package smoke because the packed tarball resolves
    @openai/codex at a prerelease version that the public registry does not serve.

Whole-suite runs on this machine also produce occasional 30-second timeouts in I/O-heavy plugin and
publication tests. Unmodified main produced the same class of timeout at a comparable rate, in
tests that do not touch credential locking, so it is a local resource limit rather than a regression
here.

Risk and rollout

A lock whose owner has not heartbeated for 30 s is now reclaimable even when its recorded process ID
is live. That is the intended fix, and it is the only behavior change for a lock that is still being
heartbeated: a healthy owner is never displaced, because its heartbeat keeps the lock fresh and the
suspend/wake re-check backs off when the mtime advances. If two processes ever do overlap, release
still verifies the owner token and refuses to remove a lock it no longer owns.

Recovery of an unidentifiable or exited owner keeps its previous latency. Only the reused or
suspended live-owner case pays one heartbeat interval, replacing a wait that never ended. No
migration, configuration, schema, or public CLI change is required.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

@github-actions github-actions Bot added the bug Something isn't working label Aug 21, 2026
recoverStaleCredentialHomeLock decided staleness from process.kill alone,
so a reused PID was indistinguishable from the original owner and the
lock was never reclaimed. The 30 s stale-age escape hatch only applied in
the else branch of the PID check, which exempted a lock naming any live
PID from expiry no matter how old it was. Acquisition then polled every
25 ms with no timeout. PR openai#448 fixed the malformed-PID half of this and
left reused PIDs as separate work.

Ownership is now proven by freshness rather than by PID liveness. A held
lock refreshes its directory mtime on an unreferenced 5 s heartbeat, and
one 30 s horizon now applies on every path, so a lock whose heartbeat has
lapsed is reclaimed whoever owns that PID now. A definitively exited
owner still recovers immediately.

Because a suspended machine wakes with a stale-looking mtime while its
owner is still alive, recovery of a lock whose PID looks alive waits one
heartbeat interval and abandons the reclaim if the mtime advances. Owners
that cannot be identified at all skip that wait, so exited and malformed
owners recover as quickly as before.
@rohanpoudel2
rohanpoudel2 force-pushed the fix/credential-lock-heartbeat branch from e7ca060 to a0b3673 Compare August 22, 2026 00:35
The credential-lock heartbeat tests replaced the global timer functions and
asserted inside the mock bodies. Bun runs every test file in one process, so
while those spies were installed any unrelated setInterval call received the
mock: one test failed an assertion attributed to whichever test happened to
be running, and the other aborted its own lock wait from the mock's else
branch. Neither could be triggered by the code under test.

Capture only the heartbeat registration and delegate every other timer call
to the real implementation, then assert the interval, its unreferenced timer,
and release-time clearing from the test body. The two near-identical mock
setups become one helper.

No production change and no change in what the tests cover.

@mldangelo-oai mldangelo-oai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@mldangelo-oai
mldangelo-oai merged commit 79734fd into openai:main Aug 22, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Stale credential-home lock is never reclaimed when owner.json names a live or unusable pid, and the wait is silent and unbounded

2 participants