Sweep a killed request's tree from a supervisor-forked sweeper

CHANGELOG.md

@@ -13,6 +13,16 @@ gem but are what an operator runs against their own image.
 
 ## next / unreleased
 
+### HotCell::Server
+
+#### Added
+
+* The supervisor forks a sweeper every `sweep_interval` seconds (default 10) to delete the directories killed requests left behind. Before, only the next worker to answer on the same slot deleted them, so a slot whose every request was killed filled the scratch.
+
+#### Fixed
+
+* A worker no longer logs `slot.unswept` when the sweeper deleted the tree first.
+
 ## v0.4.1 / 2026-09-08
 
 ### Upgrading

docs/DEPLOYMENT.md

@@ -316,6 +316,7 @@ HotCell.limits concurrency: 4, queue_size: 8, queue_wait: 10, deadline: 30,
 | `queue_size` | `8` | Connections that may wait for a worker. When `running + queued` reaches `concurrency + queue_size`, the cell answers `capacity`. Use `0` to refuse instead of queueing. |
 | `queue_wait` | `10` | Seconds a queued connection may wait before the cell answers `capacity`. This makes a saturated cell answer with a verdict instead of holding the caller until its own timeout. |
 | `control_deadline` | `5` | Seconds a control connection may take to send its request. |
+| `sweep_interval` | `10` | How often, in seconds, the supervisor looks for the directories that killed requests left behind. When it finds one, it forks a sweeper process to delete them. The sweeper runs under `deadline` like a worker, so the supervisor and the requests in flight never wait on the deletion. |
 | `max_requests_per_worker` | `1` | Requests one worker serves before the cell discards it. `1` forks per request. `:unlimited` keeps a worker for the life of the cell. See "Settings that trade one for the other". |
 
 ### Security

docs/LOGS.md

@@ -41,6 +41,7 @@ Everything else is ours and sits under `hotcell.*`:
 | `hotcell.cause` | string | Why a worker was killed (`"deadline"`, `"memory"`, `"fsize"`, ...). |
 | `hotcell.signal` | string | Signal name (`"SIGKILL"`, `"SIGSEGV"`, ...). ECS has no field for signals. |
 | `hotcell.served` | integer | Requests a worker served before it was reaped. |
+| `hotcell.swept` | integer | Discarded trees a sweeper found cleared, on `scratch.swept`: unlinked by it, or already gone when it reached them. |
 | `hotcell.home` | string | The scratch directory a cleanup could not clear: a request's `$HOME` from a worker, the slot directory from the supervisor. |
 | `hotcell.directory` | string | The cell's working directory, on `cell.boot`. |
 | `hotcell.operations` | array | Registered operation names, on `cell.boot`. |
@@ -70,11 +71,17 @@ Everything else is ours and sits under `hotcell.*`:
 | `worker.unforkable` | ERROR | `hotcell.slot`, `error.type`, `error.message` |
 | `worker.undispatchable` | ERROR | `hotcell.slot`, `hotcell.op`, `error.type` |
 | `worker.unreadable_report` | ERROR | `message` |
+| `sweeper.forked` | INFO | — |
+| `sweeper.deadline` | WARN | `hotcell.deadline_s` |
+| `sweeper.died` | WARN | `hotcell.signal`, `process.exit_code`; a sweeper that ended abnormally by anything but the deadline kill |
+| `sweeper.unforkable` | ERROR | `error.type`, `error.message` |
+| `sweeper.crashed` | ERROR | `error.type`, `error.message` |
+| `scratch.swept` | INFO | `hotcell.swept`, `event.duration.ms` |
 | `control.abandoned` | WARN | `hotcell.waited_s` |
 | `control.unanswerable` | WARN | `error.type`, `error.message` |
 | `slot.uncleaned` | WARN | `hotcell.slot`, `hotcell.home`, `message` (boot sweep only) |
 | `slot.undiscarded` | WARN | `hotcell.slot`, `hotcell.home` |
-| `slot.unswept` | WARN | `hotcell.slot`, `hotcell.home` |
+| `slot.unswept` | WARN | `hotcell.slot`, `hotcell.home`; from the worker that answered on the slot or from the sweeper |
 | `scratch.unswept` | WARN | `hotcell.path`; `error.type` and `error.message` when the scratch itself could not be listed |
 
 ## What a worker wrote to fd 2

hotcell-server/lib/hot_cell/configuration.rb

@@ -23,6 +23,7 @@ class Configuration
       queue_wait:       10,  # seconds a queued connection may wait before it is answered `capacity`
       max_requests_per_worker:            1,   # requests a worker serves before it is discarded
       control_deadline: 5,   # seconds a control connection may take to send its request
+      sweep_interval:   10,  # seconds between the supervisor's checks for a killed request's tree to unlink
     }.freeze
 
     LIMITS = {
@@ -51,6 +52,7 @@ def initialize(**options)
       # in describe's JSON, and they may arrive as Active Support durations.
       @queue_wait = @queue_wait.to_f
       @control_deadline = @control_deadline.to_f
+      @sweep_interval = @sweep_interval.to_f
 
       # A nil is not "use the default" here, it is a missing number. A cell whose deadline is nil accepts
       # every request and then dies on the first arithmetic the supervisor does with it, so an explicit nil
@@ -103,6 +105,7 @@ def verify!
         positive! :concurrency, integer: true
         positive! :queue_wait
         positive! :control_deadline
+        positive! :sweep_interval
 
         unless queue_size.is_a?(Integer) && !queue_size.negative?
           raise ConfigurationError, "queue_size: #{queue_size} must not be negative"

hotcell-server/lib/hot_cell/filesystem.rb

@@ -18,21 +18,39 @@ module Filesystem
     #
     # `Dir.exist?` is not the guard, because it follows symlinks and answers false for a dangling one, and
     # an entry a tool left in a directory's place is exactly what this has to remove.
+    #
+    # A tree that is gone by the time the removal fails is the outcome this wants, however it went. Two
+    # sweepers can meet on one discarded tree — the worker that answered on the slot and the supervisor's
+    # own — and the loser's walk fails on an entry the winner unlinked first.
     def self.remove_tree(path)
-      return true unless File.exist?(path) || File.symlink?(path)
+      return true if gone?(path)
 
       FileUtils.remove_entry path
       true
     rescue SystemCallError
       repair_and_remove path
     end
 
+    # `lstat` rather than `File.exist?`, which answers false for a path it cannot stat as well as for one that
+    # is gone. Only ENOENT means gone; a tree behind a directory a tool made unsearchable is still there.
+    def self.gone?(path)
+      File.lstat path
+      false
+    rescue Errno::ENOENT
+      true
+    rescue SystemCallError
+      false
+    end
+    private_class_method :gone?
+
     def self.repair_and_remove(path)
+      return true if gone?(path)
+
       FileUtils.chmod_R 0o700, path, force: true
       FileUtils.remove_entry path
       true
     rescue SystemCallError
-      false
+      gone?(path)
     end
     private_class_method :repair_and_remove
   end

hotcell-server/lib/hot_cell/log.rb

@@ -35,11 +35,18 @@ class Log
       "worker.unforkable" => "ERROR",
       "worker.undispatchable" => "ERROR",
       "worker.unreadable_report" => "ERROR",
+      "sweeper.forked" => "INFO",
+      "sweeper.deadline" => "WARN",
+      "sweeper.unforkable" => "ERROR",
+      "sweeper.crashed" => "ERROR",
+      "scratch.swept" => "INFO",
       "control.abandoned" => "WARN",
       "control.unanswerable" => "WARN",
       "slot.uncleaned" => "WARN",
       "scratch.unswept" => "WARN",
       "slot.undiscarded" => "WARN",
+      "slot.unswept" => "WARN",
+      "sweeper.died" => "WARN",
     }.freeze
 
     def self.null

hotcell-server/lib/hot_cell/server.rb

@@ -15,6 +15,7 @@
 require "hot_cell/counters"
 require "hot_cell/control"
 require "hot_cell/worker"
+require "hot_cell/sweeper"
 require "hot_cell/supervisor"
 
 module HotCell

hotcell-server/lib/hot_cell/slot.rb

@@ -95,7 +95,8 @@ def remove_home
     #
     # A rename within one filesystem is O(1) and takes the tree out of the way. A worker sweeps it later,
     # after it has answered and before it reports itself idle — see Worker#serve, which is the one window
-    # where the unlinking costs nobody's latency.
+    # where the unlinking costs nobody's latency. A worker killed at its deadline never reaches that window,
+    # so the supervisor also forks a Sweeper on a timer, which unlinks in a process of its own.
     #
     # The destination carries a random suffix rather than a counter, because the tool that filled the
     # directory runs as this user and can write to the slot's directory. A predictable name lets it
@@ -133,12 +134,25 @@ def prepare
     # Unlinks whatever discard_home renamed out of the way. Partial progress is fine: a sweep killed
     # part-way leaves fewer entries for the next one, so this converges rather than repeating.
     def sweep
-      Dir.glob(File.join(directory, "discarded-*")).map { |path| Filesystem.remove_tree(path) }.all?
+      discarded.map { |path| Filesystem.remove_tree(path) }.all?
     rescue SystemCallError
       # The glob itself can fail, because the slot directory is a name a tool can replace — a symlink loop
       # in its place answers ELOOP here rather than for any one entry. This runs from the worker's ensure,
       # where a raise would replace the caller's response with a crash.
       false
     end
+
+    # What discard_home has renamed aside and nobody has unlinked yet.
+    def discarded
+      Dir.glob(File.join(directory, "discarded-*"))
+    end
+
+    # Streams the directory and stops at the first match, because the supervisor asks this in its loop and
+    # a tool can put as many entries beside the discarded ones as it likes; a glob would list and sort them all.
+    def discarded?
+      Dir.each_child(directory).any? { |name| name.start_with?("discarded-") }
+    rescue Errno::ENOENT
+      false
+    end
   end
 end