[hhoikoo]

resolves #NNN (BA-XXXX)

AbstractAgent.create_kernel rejects a creation that is still in progress (track_create), but accepts create_kernel for a kernel id that already finished creating and sits in kernel_registry. A duplicate request for a running kernel therefore re-allocates its slots and re-runs scratch and network setup for it. Now create_kernel raises KernelAlreadyExistsError for that id before any of those steps.

Change (src/ai/backend/agent/agent.py, errors/agent.py, errors/__init__.py)

• create_kernel: when not restarting and kernel_id in self.kernel_registry, raise KernelAlreadyExistsError with the kernel id in the message (409, ErrorDomain.KERNEL / CREATE / ALREADY_EXISTS). The check runs inside track_create, before the throttle semaphore, the KernelPreparing events, init_kernel_context, resource allocation, scratch and network.
• server.py: _create_kernel logs the error at trace and re-raises; the rpc_function wrapper skips error_monitor.capture_exception() for it (as for KernelNotFoundError). collect_error is unchanged and still emits one AgentErrorEvent (as it does for ResourceError and KernelNotFoundError); no lifecycle event is injected, so the registered kernel is not destroyed or cleaned. The manager sees the failure of that request only.
• Callers of create_kernel: server.py create_kernels (non-restart, now guarded) and restart_kernel (restarting=True, unaffected; its kernel is still in kernel_registry).

Rig evidence (agent log of a containerd-backend rig; not main)

Log line	What it shows
82, 186	first create_kernels for the kernel: "kernel creating" ... "kernel creation done" (~60 s)
187	a second "kernel creating" for the same kernel id 0.4 s after the first finished; track_create lets it through
196-202	second slot allocation and scratch preparation for that kernel
231	"kernel resource info read failed" while re-reading the resource spec
232	duplicate fails at ip netns add ("File exists")

After the failure the live kernel's netns, DNAT rules and scratch dir were gone (containerd cleanup is keyed by kernel id, #15588/#15613, not on main).

Shown vs inferred

• Shown: the duplicate passes the guard on main (track_create only); the lines above; the live kernel's host state gone afterwards.
• Inferred: the live kernel's slots were released, from the "kernel resource info read failed" warning at line 231 (no direct allocation dump).
• Not shown on main: the id-keyed cleanup is containerd-only; on main the duplicate's cost is the second allocation and the events.

Related: the duplicate's trigger is fixed separately in the manager (advisory lock PR, link to follow). This is the agent-side defence for any duplicate request.

Known gap: a container present but missing from kernel_registry (for example after recovery) is not covered.
A stale entry is the reverse case, pre-existing and not changed here: the base agent registers kernel_registry[kernel_id] after prepare_container and before start_container, so a crash mid-start can pickle a container-less entry that _clean_kernel_registry_loop (RUNNING kernels only) never cleans, and a re-sent create for that id then gets the 409.
When several kernels in one create_kernels request fail, the errors are wrapped in TaskGroupError and still reach capture_exception, the same as KernelNotFoundError.

Tests: tests/unit/agent/test_create_kernel_duplicate.py (conditions are fixtures)

• registered kernel, restarting=False: raises KernelAlreadyExistsError; init_kernel_context and the KernelPreparing events are not awaited.
• registered kernel, restarting=True: proceeds to init_kernel_context.
• unregistered kernel: proceeds.
• create_kernels with create_kernel raising the error: no ERROR-level log, capture_exception not awaited; one AgentErrorEvent produced.
• create_kernels, no lifecycle event injected: a regression guard that also passes on main.
• Red: on main with only the error class, guard absent, 2 of 5 fail (guard test, expected-outcome test). With the guard but without the server.py change, 1 of 5 fails (expected-outcome test). Green with both. Outputs in the evidence report.

Not run on a live agent.

Checklist: (if applicable)

• Backport targets: a fix: PR goes to every version in .github/maintained-versions.yml automatically.
  A Backport: line here names the targets instead — some of them, other ones, or none.
• Mention to the original issue
• Installer updates including:
  • Fixtures for db schema changes
  • New mandatory config options
• Update of end-to-end CLI integration tests in ai.backend.test
• API server-client counterparts (e.g., manager API -> client SDK)
• Test case(s) to:
  • Demonstrate the difference of before/after
  • Demonstrate the flow of abstract/conceptual models with a concrete implementation
• Documentation
  • Contents in the docs directory
  • docstrings in public interfaces and type annotations

[hhoikoo]

Closed: superseded by #15978 (start claim); not needed for the CC MVP.