Skip to content

fix(render): cancel on SIGINT/SIGTERM so deferred cleanup runs - #406

Draft
jcogilvie wants to merge 5 commits into
crossplane:mainfrom
jcogilvie:jco/render-interrupt-cleanup
Draft

jcogilvie wants to merge 5 commits into
crossplane:mainfrom
jcogilvie:jco/render-interrupt-cleanup

Conversation

@jcogilvie

@jcogilvie jcogilvie commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Description of your changes

Stacked on #402 (→ #404 → #408); only the top two commits are new in this PR.

Interrupting crossplane render (Ctrl+C / SIGTERM) mid-render leaked every Function container, the crossplane internal render container and the temporary network. All of that cleanup is defer-based, but nothing converted SIGINT/SIGTERM into context cancellation, and the spinner's only signal handler called os.Exit(130), which skips defers.

Signal handling (fix(render): cancel on SIGINT/SIGTERM so deferred cleanup runs)

  • cmd/crossplane/main.go: derive a context with signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM) and bind it into kong as context.Context. If the run was interrupted, the process prints any error and exits 130.
  • render xr / render op: Run now takes the bound context.Context and bases its timeout context on it, so an interrupt cancels in-flight work and the deferred cleanup runs. The deferred closure from fix(render): warn on stderr when function runtimes can't be cleaned up #402 passes that ctx to StopFunctionRuntimes, which (as of fix(render): stop every function runtime gracefully and always remove Remove-policy containers #404) detaches cancellation internally with context.WithoutCancel, so cleanup still works after an interrupt. No //nolint:contextcheck is needed.
  • internal/terminal/spinner.go: the spinner no longer installs a signal handler or calls os.Exit. Signals are handled once for all commands in main. Spinner lifetime is handled once by the SpinnerPrinter wrappers, which start the spinner, run the wrapped func, then stop it, so a spinner stops when cancellation makes the wrapped func return.

Attach fix (fix(docker): unblock container attach read on context cancellation)

RunContainer blocked in stdcopy.StdCopy on the attach stream, and that call never checks ctx. So even after cancellation, the run waited until the render container exited by itself. RunContainer now closes the attach connection via context.AfterFunc(ctx, ...), and on cancellation it returns a wrapped ctx.Err() (context canceled) instead of the raw read error. The existing deferred force-remove then tears down the container.

Measured with a Function that hangs: before, cleanup finished ~115s after SIGINT. Re-verified on this stacked branch: the process exits 130 ~0.3s after SIGINT. The two fnlc-sigint-* Function containers, the crossplane/crossplane:stable render container and the crossplane-render-* network were all present mid-run and all gone afterwards.

Second signal

I checked what happens on a second Ctrl+C while a pretty spinner is running. Setup: crossplane dependency add http://127.0.0.1:18765/crds.yaml against a local server that accepts connections and never responds, run under a pseudo-TTY with script. This command still uses context.Background(), so the first SIGINT doesn't stop it and the spinner keeps running. Then I sent a second SIGINT.

  • With the default handler restored after the first signal (the previous revision of this PR), the second SIGINT killed the process. The captured output contained \x1b[?25l (hide cursor) and \x1b[?2004h (bracketed paste on) with no matching \x1b[?25h or \x1b[?2004l. The terminal was left with a hidden cursor and bracketed paste enabled. As a control, a normal spinner stop emits \x1b[?25h\x1b[?2004l.
  • So main now handles the second signal itself, once for all commands. After the first signal it listens for another SIGINT/SIGTERM. When one arrives, it writes ansi.ShowCursor + ansi.ResetModeBracketedPaste to stderr (only if stderr is a terminal) and calls os.Exit(130). Re-tested: the process exits 0.06s after the second SIGINT, and the output ends in \x1b[?25h\x1b[?2004l. With stderr redirected to a file, nothing extra is written.

A second signal still skips the remaining cleanup by design. That's the escape hatch if cleanup itself hangs.

Other commands

The root context is only injected into commands that ask for a context.Context in Run. Today that's just the two render commands. Other commands still build their own context.Background(), so they don't react to cancellation. Because NotifyContext now catches the first SIGINT for every command, an interrupted non-render command keeps running until it finishes and then exits 130. Before this change it died immediately. A second Ctrl+C exits immediately, as described above. Wiring the remaining commands to the bound context can be a follow-up.

Testing

I updated the existing render tests for the new Run signature. I didn't add a new automated test: this behaviour is signal and process-exit wiring, and testing it needs a subprocess harness. I verified it manually with Docker and a pseudo-TTY, as described above.

Fixes #400

I have:

  • Read and followed Crossplane's contribution process.
  • Run ./nix.sh flake check to ensure this PR is ready for review.
  • Added or updated unit tests. I only updated existing tests for the signature change (see above).
  • Linked a PR or a docs tracking issue to document this change.
  • Added backport release-x.y labels to auto-backport this PR.

Need help with this checklist? See the cheat sheet.

🤖 Generated with Claude Code

@jcogilvie
jcogilvie force-pushed the jco/render-interrupt-cleanup branch 2 times, most recently from 290dd58 to 68ce410 Compare October 2, 2026 18:08
jcogilvie and others added 5 commits October 2, 2026 15:59
…etwork operations

RuntimeDocker.Start and the render network helpers each construct their
own Docker client from the environment, so the container lifecycle and
network setup/teardown can only be exercised against a live Docker
daemon. That blocks unit-testing the follow-up fixes for crossplane#397, crossplane#398 and
crossplane#401.

Introduce narrow unexported interfaces covering exactly the moby client
methods each site uses: containerClient (image pull, container
inspect/create/start, plus containerCleanupClient for stop/remove) for
RuntimeDocker, and networkClient (network create/remove) for the render
network helpers. *client.Client satisfies both, enforced by compile-time
assertions.

RuntimeDocker gains an unexported dockerClient field; when nil, Start
builds the real client from the environment exactly as before.
createRenderNetwork and removeRenderNetwork now take the client as a
parameter, and dockerRenderEngine gains an unexported networks field;
when nil, Setup builds the real client only on the create-network
branch, with the same error wrapping as before. No exported signature,
the Engine interface, or the cleanup policy semantics change.

Add function-field mocks for both interfaces: mockContainerClient next
to the existing mockPullClient in runtime_docker_test.go, and
mockNetworkClient in network_test.go. Add table tests that assert on
results and sentinel errors, proving each seam is wired: network
create/remove, Setup's create-network branch and its cleanup,
RuntimeDocker.Start's create/start/inspect and image pull path, and its
stop closure for the Stop, Remove and Orphan cleanup policies.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
… Remove-policy containers

FunctionAddresses.Stop returned on the first runtime Stop error, leaving
the remaining runtimes running. It now attempts every runtime and returns
all failures joined.

StopFunctionRuntimes shared a single 5s deadline across all runtimes, and
ContainerStop used the daemon's default 10s grace period. A container slow
to exit on SIGTERM made ContainerStop fail, and the ContainerRemove that
should follow was skipped.

The Stop and Remove cleanup policies now stop with an explicit 3s grace
period. Remove then force removes the container whether or not the stop
succeeded, so a slow SIGTERM can no longer skip removal. Removal success
means success; if removal fails, the removal and stop errors are joined.

StopFunctionRuntimes gives each runtime its own timeout, sized as the
grace period plus a margin, derived from the caller's context without its
cancellation. Cleanup therefore still runs after the render context is
cancelled, but stays bounded.

This changes the exported signature of StopFunctionRuntimes from
(logging.Logger, *FunctionAddresses) to (context.Context,
*FunctionAddresses) error, matching other cleanup functions in the repo.
The xr and op render commands log the returned error as before.

Also correct the doc comments: Remove, not Stop, is the default cleanup
policy.

Fixes crossplane#397

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
StopFunctionRuntimes returns its error, but `crossplane render xr` and
`crossplane render op` only reported it via log.Info, and the CLI's logger
is a no-op unless --verbose is set. As a result both commands silently
left Function containers behind when stopping them failed.

Add WarnCleanupFailure, which writes a short warning (including the error)
to a writer and does nothing for a nil error, and call it with kong's
stderr from both commands' deferred cleanup alongside the existing log
line. The exit code is unchanged.

Fixes crossplane#399

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Interrupting `crossplane render` leaked every Function container, the
render engine container and the temporary network. All of that cleanup
is defer-based, but nothing turned SIGINT/SIGTERM into context
cancellation, and the spinner's signal handler called os.Exit(130),
which skips defers.

Derive a signal-aware context in main with signal.NotifyContext and
bind it into kong; render xr and render op now base their timeout
context on it. The spinner no longer handles signals; the SpinnerPrinter
wrappers stop it when the wrapped func returns. An interrupted run exits
130.

A second signal exits 130 immediately without waiting for cleanup. A
spinner may still be running then, and being killed by the default
handler left the cursor hidden and bracketed paste enabled, so main
first restores both when stderr is a terminal.

Fixes crossplane#400

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
RunContainer blocked in stdcopy.StdCopy, which doesn't observe ctx, so
an interrupted render waited until the render container exited by
itself (~115s with a hung Function) before cleanup could run. Close the
attach connection when ctx is done, and report ctx.Err() rather than
the resulting read error.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
@jcogilvie
jcogilvie force-pushed the jco/render-interrupt-cleanup branch from 68ce410 to 6c34ce5 Compare October 2, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

render: interrupting with Ctrl+C (SIGINT/SIGTERM) leaks Function containers, the render container, and the network

1 participant