Conversation
jcogilvie
force-pushed
the
jco/render-interrupt-cleanup
branch
2 times, most recently
from
October 2, 2026 18:08
290dd58 to
68ce410
Compare
…etwork operations RuntimeDocker.Start and the render network helpers each construct their own Docker client from the environment, so the container lifecycle and network setup/teardown can only be exercised against a live Docker daemon. That blocks unit-testing the follow-up fixes for crossplane#397, crossplane#398 and crossplane#401. Introduce narrow unexported interfaces covering exactly the moby client methods each site uses: containerClient (image pull, container inspect/create/start, plus containerCleanupClient for stop/remove) for RuntimeDocker, and networkClient (network create/remove) for the render network helpers. *client.Client satisfies both, enforced by compile-time assertions. RuntimeDocker gains an unexported dockerClient field; when nil, Start builds the real client from the environment exactly as before. createRenderNetwork and removeRenderNetwork now take the client as a parameter, and dockerRenderEngine gains an unexported networks field; when nil, Setup builds the real client only on the create-network branch, with the same error wrapping as before. No exported signature, the Engine interface, or the cleanup policy semantics change. Add function-field mocks for both interfaces: mockContainerClient next to the existing mockPullClient in runtime_docker_test.go, and mockNetworkClient in network_test.go. Add table tests that assert on results and sentinel errors, proving each seam is wired: network create/remove, Setup's create-network branch and its cleanup, RuntimeDocker.Start's create/start/inspect and image pull path, and its stop closure for the Stop, Remove and Orphan cleanup policies. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
… Remove-policy containers FunctionAddresses.Stop returned on the first runtime Stop error, leaving the remaining runtimes running. It now attempts every runtime and returns all failures joined. StopFunctionRuntimes shared a single 5s deadline across all runtimes, and ContainerStop used the daemon's default 10s grace period. A container slow to exit on SIGTERM made ContainerStop fail, and the ContainerRemove that should follow was skipped. The Stop and Remove cleanup policies now stop with an explicit 3s grace period. Remove then force removes the container whether or not the stop succeeded, so a slow SIGTERM can no longer skip removal. Removal success means success; if removal fails, the removal and stop errors are joined. StopFunctionRuntimes gives each runtime its own timeout, sized as the grace period plus a margin, derived from the caller's context without its cancellation. Cleanup therefore still runs after the render context is cancelled, but stays bounded. This changes the exported signature of StopFunctionRuntimes from (logging.Logger, *FunctionAddresses) to (context.Context, *FunctionAddresses) error, matching other cleanup functions in the repo. The xr and op render commands log the returned error as before. Also correct the doc comments: Remove, not Stop, is the default cleanup policy. Fixes crossplane#397 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
StopFunctionRuntimes returns its error, but `crossplane render xr` and `crossplane render op` only reported it via log.Info, and the CLI's logger is a no-op unless --verbose is set. As a result both commands silently left Function containers behind when stopping them failed. Add WarnCleanupFailure, which writes a short warning (including the error) to a writer and does nothing for a nil error, and call it with kong's stderr from both commands' deferred cleanup alongside the existing log line. The exit code is unchanged. Fixes crossplane#399 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
Interrupting `crossplane render` leaked every Function container, the render engine container and the temporary network. All of that cleanup is defer-based, but nothing turned SIGINT/SIGTERM into context cancellation, and the spinner's signal handler called os.Exit(130), which skips defers. Derive a signal-aware context in main with signal.NotifyContext and bind it into kong; render xr and render op now base their timeout context on it. The spinner no longer handles signals; the SpinnerPrinter wrappers stop it when the wrapped func returns. An interrupted run exits 130. A second signal exits 130 immediately without waiting for cleanup. A spinner may still be running then, and being killed by the default handler left the cursor hidden and bracketed paste enabled, so main first restores both when stderr is a terminal. Fixes crossplane#400 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
RunContainer blocked in stdcopy.StdCopy, which doesn't observe ctx, so an interrupted render waited until the render container exited by itself (~115s with a hung Function) before cleanup could run. Close the attach connection when ctx is done, and report ctx.Err() rather than the resulting read error. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Jonathan Ogilvie <jonathan.ogilvie@sumologic.com>
jcogilvie
force-pushed
the
jco/render-interrupt-cleanup
branch
from
October 2, 2026 20:23
68ce410 to
6c34ce5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
Stacked on #402 (→ #404 → #408); only the top two commits are new in this PR.
Interrupting
crossplane render(Ctrl+C / SIGTERM) mid-render leaked every Function container, thecrossplane internal rendercontainer and the temporary network. All of that cleanup isdefer-based, but nothing converted SIGINT/SIGTERM into context cancellation, and the spinner's only signal handler calledos.Exit(130), which skips defers.Signal handling (
fix(render): cancel on SIGINT/SIGTERM so deferred cleanup runs)cmd/crossplane/main.go: derive a context withsignal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM)and bind it into kong ascontext.Context. If the run was interrupted, the process prints any error and exits130.render xr/render op:Runnow takes the boundcontext.Contextand bases its timeout context on it, so an interrupt cancels in-flight work and the deferred cleanup runs. The deferred closure from fix(render): warn on stderr when function runtimes can't be cleaned up #402 passes that ctx toStopFunctionRuntimes, which (as of fix(render): stop every function runtime gracefully and always remove Remove-policy containers #404) detaches cancellation internally withcontext.WithoutCancel, so cleanup still works after an interrupt. No//nolint:contextcheckis needed.internal/terminal/spinner.go: the spinner no longer installs a signal handler or callsos.Exit. Signals are handled once for all commands in main. Spinner lifetime is handled once by theSpinnerPrinterwrappers, which start the spinner, run the wrapped func, then stop it, so a spinner stops when cancellation makes the wrapped func return.Attach fix (
fix(docker): unblock container attach read on context cancellation)RunContainerblocked instdcopy.StdCopyon the attach stream, and that call never checks ctx. So even after cancellation, the run waited until the render container exited by itself.RunContainernow closes the attach connection viacontext.AfterFunc(ctx, ...), and on cancellation it returns a wrappedctx.Err()(context canceled) instead of the raw read error. The existing deferred force-remove then tears down the container.Measured with a Function that hangs: before, cleanup finished ~115s after SIGINT. Re-verified on this stacked branch: the process exits 130 ~0.3s after SIGINT. The two
fnlc-sigint-*Function containers, thecrossplane/crossplane:stablerender container and thecrossplane-render-*network were all present mid-run and all gone afterwards.Second signal
I checked what happens on a second Ctrl+C while a pretty spinner is running. Setup:
crossplane dependency add http://127.0.0.1:18765/crds.yamlagainst a local server that accepts connections and never responds, run under a pseudo-TTY withscript. This command still usescontext.Background(), so the first SIGINT doesn't stop it and the spinner keeps running. Then I sent a second SIGINT.\x1b[?25l(hide cursor) and\x1b[?2004h(bracketed paste on) with no matching\x1b[?25hor\x1b[?2004l. The terminal was left with a hidden cursor and bracketed paste enabled. As a control, a normal spinner stop emits\x1b[?25h\x1b[?2004l.ansi.ShowCursor + ansi.ResetModeBracketedPasteto stderr (only if stderr is a terminal) and callsos.Exit(130). Re-tested: the process exits 0.06s after the second SIGINT, and the output ends in\x1b[?25h\x1b[?2004l. With stderr redirected to a file, nothing extra is written.A second signal still skips the remaining cleanup by design. That's the escape hatch if cleanup itself hangs.
Other commands
The root context is only injected into commands that ask for a
context.ContextinRun. Today that's just the two render commands. Other commands still build their owncontext.Background(), so they don't react to cancellation. BecauseNotifyContextnow catches the first SIGINT for every command, an interrupted non-render command keeps running until it finishes and then exits 130. Before this change it died immediately. A second Ctrl+C exits immediately, as described above. Wiring the remaining commands to the bound context can be a follow-up.Testing
I updated the existing render tests for the new
Runsignature. I didn't add a new automated test: this behaviour is signal and process-exit wiring, and testing it needs a subprocess harness. I verified it manually with Docker and a pseudo-TTY, as described above.Fixes #400
I have:
./nix.sh flake checkto ensure this PR is ready for review.Added or updated unit tests.I only updated existing tests for the signature change (see above).Linked a PR or a docs tracking issue to document this change.Addedbackport release-x.ylabels to auto-backport this PR.Need help with this checklist? See the cheat sheet.
🤖 Generated with Claude Code