feat: metrics for the tasks of background workers - #2637
Open
nicolas-grekas wants to merge 4 commits into
Open
Conversation
Member
|
Didn't GitHub introduce support for stacked PRs lately? I think that it would greatly help reviewing those |
Contributor
Author
|
They do exist since July, but a stack needs all its branches in the same repository, cross-fork stacks aren't supported, and I can't push branches to php/frankenphp. If a maintainer pushes the four branches here (or gives me push rights), I'll recreate these as a proper stack, one diff per layer. Until then each PR says which commit to review, and the Commits tab shows that one alone. |
nicolas-grekas
force-pushed
the
bgworker-metrics
branch
4 times, most recently
from
September 7, 2026 13:54
fb19b27 to
c2efd1b
Compare
Background workers run a script in a loop outside the HTTP request cycle, sharing the PHP runtime with the request threads. Rebuilt on Server from php#2499: a background worker attaches to a php_server through WithWorkerServerScope() like any other worker. Declared with "background" in a worker block (php_server or global) or WithWorkerBackground() in Go. name is required, match is rejected, num >= 1. The lifecycle mirrors HTTP workers: re-run on a cooperative exit, restart with quadratic backoff on a crash, max_consecutive_failures fails Init() during startup only. drain() runs on shutdown, reboot and handler transitions so a parked script wakes up instead of waiting out the force-kill grace period. Every worker sees its declared name in $_SERVER['FRANKENPHP_WORKER'], HTTP workers included: the documented contract is to test its presence, not its value. Background workers also get $_SERVER['FRANKENPHP_WORKER_BACKGROUND'], so a script serving both roles can tell them apart with isset(). The script gets one handle, frankenphp_get_worker_handle(), a stream that reaches EOF when the worker is drained, meant to carry control messages later. It is backed by a socket pair, not a pipe: on Windows PHP's php_select() only waits properly on sockets before 8.5. Streams do not own the socket (php_sockop_close() would shutdown() it on Windows), so every call returns a fresh stream and closing one never affects another; the read timeout is infinite so a blocking read parks as well as stream_select() does. Both ends are non-inheritable. A worker counts as ready on its first wait on the handle (select cast or read), the background analog of frankenphp_handle_request(): Init() waits for it, ready_workers counts from it, and an exit before it is a boot failure. The handle's stream ops, copied from the socket ops at MINIT, report it once per run. A run gets one stream: every call returns the same resource until the script closes it, so fetching the handle in a loop does not grow the resource list of a request that never ends. Worker names are scoped like paths: unique within a php_server or among global workers. The script sees the declared name; metrics and logs report a scoped worker as "<server name>:<name>", with a numeric suffix on server names when two blocks resolve to the same one, never a name another block configured. The collision-driven renaming in the Caddy module is gone, and WithWorkerName() resolves within the request's server first. FRANKENPHP_WORKER held "1" in HTTP workers before, and workers of a php_server block were reported under their bare name unless it collided: both changes are called out in the docs. Supersedes php#2543 and php#2398.
The shared-state half of php#2287, on top of the background workers: a worker publishes a snapshot with frankenphp_set_vars(), requests and other workers read it with frankenphp_get_vars(). The persistent-zval toolkit from php#2366 does the cross-thread copies; this adds the two functions and a per-worker slot. set_vars() validates the tree, persists it and swaps it into the slot under a write lock; readers copy it into request memory under the read lock, so the previous table is only freed once no reader is on it. The slot belongs to the worker rather than a thread: it survives script restarts, serving the last snapshot meanwhile, and several threads of one worker simply publish last-writer-wins. The tables are freed in drainPHPThreads() once every PHP thread is gone and before the engine is, since freeing walks string headers. get_vars() resolves the name the way requests do, within the caller's server then among global workers. It blocks until the worker reached its ready point once: activateServers() runs after initWorkers(), so requests never wait, and a blocked caller is another background worker still booting. Those waits form a graph and a cycle is refused with an exception instead of deadlocking Init(); the wait also aborts on shutdown. A ready worker that never published throws. Publishing before the first wait on the handle therefore guarantees the snapshot exists before the server accepts requests. Being the first consumer keeping persistent trees across requests and exposing them repeatedly, this also fixes two fast paths of the toolkit: opcache-immutable arrays were exposed through refcounted zvals, and opcache only keeps their refcount at 2, so the second reader's release destroyed shared memory; and every interned string was shared by pointer, while only permanent ones (opcache, startup) outlive the request that interned them, so trees built from request-interned literals dangled once that request ended (the Windows job runs the embed without opcache). Immutable arrays now go through zvals without type flags, as php-src does for literals, and sharing a string requires IS_STR_PERMANENT. Left out on purpose, see php#2287: the per-request cache with === identity, the unchanged-data skip in set_vars(), ensure_background_worker() and lazy or catch-all workers, CLI hiding of the functions.
The task half of php#2319, on top of the background workers and their shared vars: a request, an HTTP worker or another background worker hands work to a named background worker with frankenphp_send_task(), which returns a stream carrying the updates the worker sends back with frankenphp_update_task(). The worker dequeues tasks with frankenphp_receive_task() after reading a "task\n" line on its handle: the one handle of php#2617 carries both the drain EOF and the wake-ups, so a script keeps a single stream_select() loop. The line is a wake-up, not a count: every thread of a pool gets one per task, the first one back in its loop takes the task and the others get null. send_task() blocks until a thread of the worker picks the task up and throws on timeout, so a busy worker pushes back on its senders instead of queueing without bounds; tasks queued while a thread restarts are signaled again on its next run. The wait also ends when the sender's own thread is drained for a restart or the shutdown, since the target's threads are drained too. Names resolve like frankenphp_get_vars() does. Each task gets a socket pair. The sender's stream is a socket stream over one end, one byte per update and EOF at completion, so stream_select() bounds the wait or multiplexes tasks, and a blocking read parks as well; closing it abandons the task. The receiver's stream is a socket stream over the other end: updates go through update_task(), the stream itself reports the sender's close as EOF to stream_select() and feof(), so a long task learns that nobody waits for its result, and update_task() throws. Closing it completes the task, unless the close is the resource cleanup of request shutdown, which means the script ended with the task open: the sender's next read throws instead of returning null. Sixteen updates are buffered per task, past that update_task() waits for the sender to read. Waking threads is what a task costs, so wake-ups are kept to a minimum. A send wakes one parked thread of the worker, round-robin, with the line on its handle; a thread that reads its handle while tasks are queued gets the line from the read op itself, so no wake-up is lost whichever loop shape the script uses, and after 10ms without pickup every thread is woken as a fallback. The sender waits for the pickup in the kernel, on its end of the task's pair, rather than in a Go select: waking a PHP thread parked inside a cgo callback costs Go a P hand-off, a byte on a socket does not. The thread taking the task writes that byte, a watcher goroutine does when the wait must end without a pickup. The queue mutex is never held across a syscall and taken once per wake-up, as a thread inside a cgo callback that loses it parks the same expensive way. In the Docker builder image this takes a task from 549 to 285us with one thread, a pool of 8 from 1745 to 320us, and 8 senders on 8 threads from 2k to 32k tasks/s. Each side of a task waits on its own descriptor of the task's channel, an eventfd on Linux, one end of a socket pair elsewhere for Windows's php_select(): the streams carry no data, they are what stream_select() waits on and what fclose() ends, the Go side holds the state the functions report. The descriptors belong to the task until both sides closed, then the pair is drained and pooled, so a task costs no socketpair, fcntl or close: about 12 syscalls instead of 18, 13% off the latency and up to a third more throughput under load in the same measurement. Payloads and updates follow the set_vars() whitelist and travel as persistent tables through the Go side, which owns them until they are copied into request memory. The streams reference their task through a cgo handle; the task is freed once both sides closed, or by the sender when no thread picked it up. The stop sockets of a worker's threads are now guarded by its task queue mutex, since senders write to them. Compared to php#2319: no queue ahead of pickup and no cancellation before it, no dedicated signaling stream, no global task table.
The existing worker metrics apply to background workers as they are:
busy_workers counts a thread holding a task, from pickup to the close of
the task's stream, and worker_queue_depth counts the tasks waiting for a
thread, which is the only queue a background worker has. Two new ones
break tasks down: worker_task_count{worker,outcome} with completed,
aborted (the script ended with the task open), abandoned (the sender
closed its stream first) or timeout (no thread picked the task up in
time), settled by whichever side closes first so every task counts once,
and worker_task_time, the seconds spent on tasks from pickup to close.
The threads endpoint follows: a background thread is busy while it holds
a task and waiting otherwise, counted per thread since a script may hold
several.
nicolas-grekas
force-pushed
the
bgworker-metrics
branch
from
September 7, 2026 14:11
c2efd1b to
6414164
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #2617, #2635 and #2636: the first three commits are those PRs', review the fourth one (
6414164).Metrics for the tasks of #2636, reusing what exists where the meaning carries over.
frankenphp_busy_workersnow counts a background worker thread holding a task, from pickup to the close of the task's stream, andfrankenphp_worker_queue_depthcounts the tasks waiting for a thread, the only queue a background worker has; their help texts say so. Two counters break tasks down:frankenphp_worker_task_count{worker,outcome}withcompleted,aborted(the script ended with the task open),abandoned(the sender closed its stream first) andtimeout(no thread picked the task up in time), settled by whichever side closes first so every task counts exactly once, andfrankenphp_worker_task_time{worker}, the seconds spent on tasks. The/frankenphp/threadsendpoint follows: a background thread is busy while it holds a task, waiting otherwise.The
Metricsinterface gainsStartWorkerTask,StopWorkerTaskandWorkerTaskOutcome. Tested through a Prometheus registry passed withWithMetrics(): one task of each outcome, then the gauges back to zero.