fix: remove orphaned runtimes when the executor starts

[ChiragAgg5k]

What

When the executor starts, remove any runtime containers it doesn't track, before it does anything else.

The issue

Self-hosted users report function containers stuck in a restart loop:

[08:00:21] [open-runtimes] Runtime started.
HTTP server successfully started!
[19:22:33] [open-runtimes] Code path /tmp/code.tar.gz not found.
[19:22:33] [open-runtimes] Code path /tmp/code.tar.gz not found.
[19:22:34] [open-runtimes] Code path /tmp/code.tar.gz not found.
[19:22:35] [open-runtimes] Code path /tmp/code.tar.gz not found.
[19:22:37] [open-runtimes] Code path /tmp/code.tar.gz not found.
...
[19:29:17] [open-runtimes] Code path /tmp/code.tar.gz not found.
[19:30:17] [open-runtimes] Code path /tmp/code.tar.gz not found.

The gaps grow 1s, 2s, 3s, 6s, 13s, 26s, 51s, then hold at 60s. That's Docker's restart backoff for a container that exits right after starting, and it never ends.

1. Runtimes restart forever, and their code lives in a host directory

Appwrite creates every function runtime with restartPolicy: 'always':

// appwrite/appwrite src/Executor/Executor.php
'restartPolicy' => 'always'

The runtime's /tmp, which holds code.tar.gz, is a bind mount of a directory on the executor host:

// src/Executor/Runner/Docker.php
$tmpFolder = sprintf('tmp/%s/', $runtimeName);           // /tmp/exc1-<projectId>-<deploymentId>/
$volumes = [
    \dirname($tmpSource) . ':/tmp:rw',                    // .../src  ->  /tmp  (code.tar.gz)
    \dirname($tmpBuild) . ':' . $codeMountPath . ':rw',
];
// ...
$this->orchestration->run(..., restart: $restartPolicy);

2. The executor forgets its runtimes when it restarts

Runtimes are tracked only in an in-memory Runtimes table. When the executor restarts (an upgrade, docker compose up, a crash or a reboot), the table starts empty, but the runtime containers keep running because of restart: always.

Nothing reconciles them on startup. onStart sets up networks, pulls images and starts maintenance. The SIGTERM cleanup was removed in c206e60 (it noted This doesn't actually work), and Docker::cleanup() has had no callers since.

3. Maintenance deletes the code of untracked runtimes that are still running

// src/Executor/Runner/Maintenance.php
private function cleanupTmp(): void
{
    foreach (\glob('/tmp/' . System::getHostname() . '-*') ?: [] as $entry) {
        $runtimeName = substr($entry, \strlen('/tmp/'));
        if ($this->runtimes->exists($runtimeName)) {   // empty after a restart
            continue;
        }

        $localDevice->deletePath($entry);              // deletes a live runtime's code
    }
}

The orphan keeps serving, because its code was already extracted into the container. The damage stays hidden until the container next exits on its own (a crash, an OOM kill, a Docker daemon restart or a host reboot). Docker then starts it again with an empty /tmp, and it loops forever.

Timeline

 executor (exc1)                         runtime exc1-proj-dep          host /tmp/exc1-proj-dep/src
 ───────────────                         ─────────────────────          ───────────────────────────
 createRuntime ─────────────────────────▶ run, restart=always   ◀─────── code.tar.gz  (mounted at /tmp)
 Runtimes table: { exc1-proj-dep }        serving ✔

 ── executor restarts ──
 Runtimes table: { }                      still running (orphan) ✔       code.tar.gz
 onStart: no reconciliation

 ── maintenance tick ──
 cleanupTmp(): not in table ──────────────────────────────────────────▶ deletePath()  ✘ gone
                                          still serving ✔ (code already extracted)

 ── runtime exits (crash / OOM / dockerd restart / reboot) ──
                                          docker restarts it ──────────▶ dir recreated empty
                                          "Code path /tmp/code.tar.gz not found." → exit
                                          restart → exit → restart …   ⟲ forever (1s … 60s backoff)

Reproduced end to end

Appwrite 2.4.0 with executor 0.29.0 (OPR_EXECUTOR_MAINTENANCE_INTERVAL=15):

1. Create and execute a node-22 function. The runtime is up with RestartPolicy=always and the mount /tmp/exc1-repro-…/src -> /tmp.
2. Control: while the executor tracks the runtime, maintenance leaves its directory alone.
3. docker restart exc1. The runtime keeps running.
4. The next tick logs [Maintenance] Removed /tmp/exc1-repro-6accabeeaebb5b66e83c., while the container is still Up.
5. Killing the runtime's server process from inside the container (to simulate a crash) starts the loop: Code path /tmp/code.tar.gz not found at 09:47:21, :21, :21, :22, :24, :27, :34, :47 …, and RestartCount=9 after 50s.

(A docker kill doesn't trigger the loop, because Docker treats it as a manual stop and skips the restart policy.)

Second bug, found by the test: the executor looks up the wrong container as itself

// app/http.php
$selfContainer = $orchestration->list(['name' => $hostname])[0] ?? ...;

Docker's name filter matches substrings. With hostname executor and an orphan named executor-test-orphan, [0] was the orphan, so Network::setup() tried to attach the orphan to the runtimes network:

[Network] Failed to connect container executor-test-orphan to network executor_runtimes ...

Production has the same naming pattern (executor exc1, runtimes exc1-…). Removing orphans before this lookup means only the executor's own container can match on startup.

Changes

// app/http.php — first thing in onStart
/* Remove runtimes left behind by a previous executor process */
$maintenance->removeOrphans();

// src/Executor/Runner/Maintenance.php
public function removeOrphans(): void
{
    $containers = $this->orchestration->list(['label' => 'openruntimes-executor=' . System::getHostname()]);
    $orphans = array_filter(
        $containers,
        fn (Container $container): bool => !$this->runtimes->exists($container->getName())
    );
    // force-remove each orphan in parallel with batch(), logging failures without throwing
}

• Deleted Docker::cleanup(), which nothing called and which this replaces.
• Orphans are removed rather than adopted. A runtime's secret lives only in the lost in-memory table and its code directory gets cleaned up, so it can't be reused safely. The cost is one cold start for that function.

Tests

testRestartRemovesOrphanedRuntimes (e2e) creates a runtime with restartPolicy: always, runs docker restart on the executor container, waits for /v1/health, and asserts the runtime container is gone.

• On main, it fails: the orphan container 97d2b59b0b38 still exists.
• On this branch, it passes. The executor logs [Maintenance] Removed 1/1 orphaned runtimes. and attaches its own container to the network.

composer format:check, analyze, refactor:check and the unit tests (37) pass locally.