Skip to content

server: no SIGTERM/graceful-shutdown handler drops in-flight saves on GAE recycle #814

Description

@bpowers

Problem

The Express server has no graceful-shutdown path. src/server/index.ts registers only an unhandledRejection handler:

process.on('unhandledRejection', (reason, p) => { ... });

async function main(): Promise<void> {
  const app = await createApp();
  app.listen();
}
setImmediate(main);

There is no SIGTERM listener, no server.close(), and no in-flight request drain.

Google App Engine sends SIGTERM to an instance on every scale-down and on every deploy. With no handler, the Node process is terminated abruptly, severing any in-flight request. The most damaging case is POST /api/projects/:u/:p (a model save, src/server/api.ts): a save that is mid-flight when the instance is recycled fails, and the user loses their changes.

A second casualty is the fire-and-forget setTimeout(async () => { await app.db.preview.deleteOne(...) }) preview-regeneration at src/server/api.ts:291. It is scheduled but not awaited as part of the response, so on an abrupt exit it never runs and the stale preview survives.

Why it matters

  • Reliability / correctness: model saves are the core write path; dropping them on routine GAE lifecycle events (scale-down, deploy) causes silent data loss from the user's perspective.
  • GAE recycles instances frequently, so this is not a rare edge case -- it happens on every deploy and under normal autoscaling.

Component(s) affected

  • src/server/index.ts (process lifecycle / no signal handling)
  • src/server/api.ts (~line 291, fire-and-forget preview deletion; the POST /api/projects/:u/:p save handler is the request most affected)

Possible approach

Add a SIGTERM (and ideally SIGINT) handler that:

  1. Stops accepting new connections (server.close() on the http.Server returned by app.listen() -- this requires plumbing the server handle out of createApp/app.listen()).
  2. Waits for in-flight requests to drain, with a bounded timeout before forcing exit (GAE allows a short grace period after SIGTERM).
  3. Flushes/awaits any deferred work that must complete (the preview-deletion setTimeout is a candidate to either await or make idempotent/durable rather than fire-and-forget).

Context

Identified during a pre-deploy audit of the Simlin Express server. This reliability gap is NOT addressed by PR #810. Related but distinct from tech-debt item #41 (POST /api/projects/:u/:p is not transactional + unconditional preview invalidation), which concerns Firestore transactionality rather than process lifecycle.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions