[hhoikoo]

resolves #NNN (BA-XXXX)

ExtendedAsyncSAEngine.advisory_lock() now takes its session-level pg_advisory_lock on an AUTOCOMMIT connection, so the lock holder is never "idle in transaction" and PostgreSQL's idle_in_transaction_session_timeout can no longer drop a held lock.

Problem

• The engine runs at SERIALIZABLE, so the first statement on the pooled lock connection autobegins a transaction that stays open for the whole locked body.
• pgsql_connect_opts sets idle_in_transaction_session_timeout=60000 on every connection (models/base.py). A locked body longer than 60 s gets its connection terminated by PostgreSQL, which releases the session-level lock while the holder is still running.
• A queued waiter then takes the lock. On unlock the holder logs lock connection already closed, unlock skipped.

Change (src/ai/backend/manager/models/utils.py)

• lock_conn = await lock_conn.execution_options(isolation_level="AUTOCOMMIT") before the first statement. The same idiom is already used on pooled connections for READ COMMITTED in begin_session_read_committed() and begin_readonly_session_read_committed().

Does AUTOCOMMIT leak back into the pool? No. Shown from SQLAlchemy 2.0.54 source:

• DefaultDialect.set_connection_execution_options (engine/default.py ~L694) appends _reset_characteristics to _connection_record.finalize_callback when an execution option is applied.
• _ConnectionRecord.checkin (pool/base.py ~L765) pops and runs every finalize_callback before the connection returns to the pool.
• IsolationLevelCharacteristic.reset_characteristic (engine/characteristics.py ~L100) calls dialect.reset_isolation_level, which restores the engine-level isolation (SERIALIZABLE here) (engine/default.py ~L1004).
• test_lock_connection_isolation_level_is_restored checks that the next pooled connection reports SERIALIZABLE.

Rig evidence (a TDX-guest session start on a CoCo test rig; create_kernels RPC in flight ~60 s)

Log line	What it shows
2442	START pass A logs lifecycle schedule processing and takes the lock
2601-2624	nine periodic passes log lifecycle schedule processing ~13 s later; the START one blocks on the lock
3258	~60.3 s after the lock was taken, A writes PREPARED to CREATING
3262	A's unlock warns lock connection already closed, unlock skipped
3263	6 ms later a START pass logs session start attempted for the same session, with no lifecycle schedule processing of its own, i.e. the waiter from lines 2601-2624
3270	a second launch spec is sealed for the same session, and create_kernels is dispatched a second time

Shown vs inferred:

• Shown: the dead lock connection (line 3262), the waiter proceeding without a new pass (line 3263), the settings in code.
• Inferred: the instant PostgreSQL killed the connection (~60.0 s spacing; no PostgreSQL server log was captured).

Test (tests/unit/manager/models/test_advisory_lock.py, real PostgreSQL)

• test_lock_survives_idle_in_transaction_timeout: dedicated engine with idle_in_transaction_session_timeout=1000, holds advisory_lock for 2 s, probes from another connection with pg_try_advisory_xact_lock. Before the change: assert True is False (lock was gone). After: passes.
• test_lock_connection_isolation_level_is_restored: regression guard for the pool reset; it also passes on main.
• Existing test_lock, test_lock_timeout, test_dbutils.py, test_extended_async_sa_engine.py pass.

Side effect: a waiter now hits lock_timeout. A pass queued longer than lock_timeout=60000 for a lock gets 55P03, which advisory_lock converts to asyncio.CancelledError (models/utils.py L310-312). Before this change the holder's lock was dropped at 60 s, so such a waiter ran concurrently; now it is skipped. What happens to the skipped pass:

• It is a DoSokovanProcessIfNeededEvent / DoSokovanProcessScheduleEvent consumer task (event_dispatcher/dispatch.py L430-441 -> handlers/schedule.py L80-91 -> coordinator.py process_if_needed L1516-1528 -> process_lifecycle_schedule -> _process_lifecycle_handler_schedule L293, lock at L304-312).
• CancelledError is a BaseException: _handle_decoded re-raises it (common/events/dispatcher.py L589-596), _handle only catches Exception (L511). The task ends as cancelled; PersistentTaskGroup._on_task_done returns early for a cancelled task (aiotools taskgroup/persistent.py L222-224), so the consumer group is not shut down. Checked with a script: after a child raising CancelledError, task.cancelled() is True, the next create_task runs, _base_error is None.
• The periodic loop is not the failing task. GlobalTimer._generate_tick (common/distributed.py L50-66) only produces the event; the handler runs in a separate consumer task.
• The mark is deleted before the lock is taken: process_if_needed calls load_and_delete_schedule_mark (coordinator.py L1525), the lock comes at L304-312. So the 2 s process_if_needed tick does not pick up a pass skipped by 55P03. The session is recovered by the unconditional 60 s process_schedule tick (START spec, long_interval=60.0, coordinator.py L1658-1662, sokovan.py L68-87) or by a new START mark (a transition to PREPARED marks START: post_processors/schedule_marking.py L28). A session still PREPARED is then selected by the START pass (handlers/lifecycle/start_sessions.py L66-76).
• The un-acked stream message is pre-existing: the Exception and BaseException branches of _handle_decoded (common/events/dispatcher.py L577-596) return or raise before post_callback.done(), so a failed or cancelled handler never acks. main can already reach 55P03 by the same path; this PR only makes a wait longer than 60 s more likely, because PostgreSQL no longer kills the holder.
• It cannot cause a duplicate dispatch: auto-claim redelivers idle messages after 300 s (message_queue/redis_queue/consumer.py L29, _auto_claim L357-400). The replay is one more START pass serialised behind a lock that is now really held. A replayed process_if_needed finds no mark and returns False.
• Pre-existing, out of scope: asyncio.timeout(lock_conn_timeout) wraps the whole locked body, and the 55P03 mapping also catches a 55P03 raised inside the body.

Not in this change

• App Proxy coordinator: latent. It has the same pattern (advisory_lock in appproxy/coordinator/models/utils.py L205-225, idle_in_transaction_session_timeout=60000 in appproxy/coordinator/models/base.py L67, SERIALIZABLE by default) but no current caller of the lock on main b0e8cd0 (the AUTOCOMMIT change should be applied there if the lock gets a caller).
• Lock-independent alternative: the START pass claims sessions with a conditional PREPARED to claimed update before the RPC, which also survives a lock lost to a network blip or a PostgreSQL failover.

Checklist: (if applicable)

• Backport targets: a fix: PR goes to every version in .github/maintained-versions.yml automatically.
  A Backport: line here names the targets instead — some of them, other ones, or none.
• Mention to the original issue
• Installer updates including:
  • Fixtures for db schema changes
  • New mandatory config options
• Update of end-to-end CLI integration tests in ai.backend.test
• API server-client counterparts (e.g., manager API -> client SDK)
• Test case(s) to:
  • Demonstrate the difference of before/after
  • Demonstrate the flow of abstract/conceptual models with a concrete implementation
• Documentation
  • Contents in the docs directory
  • docstrings in public interfaces and type annotations

[hhoikoo]

Closed: superseded by #15978 (start claim); not needed for the CC MVP.