CLI loses track of submitted work after a transient connection interruption #220

Background

I use an AI agent to submit work through Colab CLI, follow its output, and collect the result. Sometimes the CLI reports a disconnection and the agent can no longer follow the work it already submitted. The runtime may still be alive, but the agent cannot retrieve the remaining output or determine how the task finished.

Remote execution continuing after a client disconnect is expected. The problem is that a temporary connection interruption can leave the CLI unable to recover the original execution's output and result.

Reproduction

I reproduced one failure mode on a live CPU runtime:

1. Submit code that increments an execution counter, prints a marker, waits, and writes a completion receipt containing the counter.

2. In a diagnostic harness, close only the kernel WebSocket's TCP connection from the output hook after receiving the marker:

   import socket
   
   client = runtime.kernel_client._manager.client
   client.kernel_socket.sock.sock.shutdown(socket.SHUT_RDWR)

3. Check the CLI result and server-side assignment, then read the receipt through an independent CLI invocation on the same kernel.

Actual behavior

colab exec exits with Connection was lost., even with --timeout 0. The assignment remains active, and the receipt confirms that the original code finished once on the same kernel. The CLI does not recover its remaining output or final result.

Reconnection experiments also showed that the client could miss completion when a task finished during the interruption, or receive completion while missing intermediate output. Reestablishing the socket alone does not establish that the execution's output and result were recovered.

For colab run, automatic teardown after a monitoring error can also stop work before its result has been established.

Expected behavior

After a temporary connection interruption, continue following the original execution and retrieve its missing output and final result, including an exception produced during the interruption. Do not resubmit the code or automatically replace the runtime.

If recovery cannot establish the result within a bounded period, report that uncertainty and preserve the existing session for inspection. A monitoring failure should not itself cancel potentially running work or be treated as proof that the execution failed.