Skip to content

Dynamic workflow deadline leaks an httpx pool per LLM call and raises "Event loop is closed" #519

Description

@FacaHD

Summary

Long scans run with a workflow deadline (SKILLSPECTOR_MAX_WORKFLOW_SECONDS) emit repeated unretrieved-task errors and steadily leak HTTP connection pools:

Task exception was never retrieved
future: <Task finished coro=<AsyncClient.aclose() done, defined at ...> exception=RuntimeError('Event loop is closed')>
RuntimeError: Event loop is closed

Steps to reproduce

  1. Set SKILLSPECTOR_MAX_WORKFLOW_SECONDS so a dynamic deadline is active.
  2. Run a scan large enough to issue more than 128 LLM calls.
  3. Watch stderr: Event loop is closed appears repeatedly once the scan passes the cache limit.

Root cause

Under a workflow deadline every batch is sent with whatever time is left, so each call carries a slightly different timeout float. _model_for_call() expressed that by building a fresh chat model per call.

LangChain caches the underlying httpx client in an lru_cache keyed on (base_url, timeout, socket_options). A deadline that shrinks by a few milliseconds per call is therefore a new cache key every call — a long scan opens one connection pool per LLM request and starts evicting the oldest once it passes 128.

Eviction is what turns the leak into a crash. Each analyzer node is synchronous and reaches run_async()asyncio.run(), so it owns its event loop. A pool evicted while a later analyzer is running belongs to an earlier, already-closed loop, and the OpenAI SDK's client finaliser reacts to garbage collection with asyncio.get_running_loop().create_task(self.aclose()) — closing the earlier analyzer's sockets from the current analyzer's loop. Nothing awaits that task, so the RuntimeError surfaces only as "Task exception was never retrieved", once per eviction.

Impact

  • Noisy, misleading stderr output on every large scan run under a deadline.
  • One httpx connection pool per LLM request instead of one per analyzer.

A 200-call reproduction against the openai 3.11 / httpx 2 combination that reported the failure produced 75 unretrieved Event loop is closed tasks and 220 httpx clients.

Expected behaviour

A bounded scan should reuse one connection pool per analyzer and complete without unretrieved-task errors, while still honouring the dynamic deadline.

Environment

  • openai 3.11 with httpx 2.x
  • Scan run with SKILLSPECTOR_MAX_WORKFLOW_SECONDS set

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions