Repository navigation
Best practice for persisting large accumulated LangGraph state without hitting ScheduleActivityTask payload size limits #1894
Description
We're running LangGraph-based agents through temporalio.contrib.langgraph.LangGraphPlugin, where each graph node executes as a Temporal Activity. The workflows accumulate state across many sequential (and some parallel) node executions — investigation-style agents with 20-30+ nodes producing tool-call transcripts, messages, etc.
Before wrapping these graphs with Temporal, we ran them directly against LangGraph's own Postgres-backed checkpointer, which persists automatically after every node (every "superstep"). Once we moved execution behind Temporal Activities, that per-node persistence went away — the nodes now run as separate Activities without a shared in-process checkpointer, so nothing writes to Postgres until we explicitly do so ourselves.
Our first attempt reintroduced persistence as a single workflow.execute_activity(persist_result_activity, args=[...full accumulated final state...]) call after ainvoke() returns, so it survives even if no client ever reconnects to read handle.result(). That worked for small/medium runs, but a larger run hit:
BadScheduleActivityAttributes: ScheduleActivityTaskCommandAttributes.Input exceeds size limit.
...which terminated the entire workflow server-side (WORKFLOW_EXECUTION_TERMINATED) — a total loss, worse than the gap we were trying to close. Individual per-node Activity payloads never approached the limit (29/29 node-Activities completed fine); only the one bulk end-of-run payload did.
Our leading fix: restore the original per-node persistence behavior — write each node's own delta to Postgres from inside that node's own Activity execution (no extra execute_activity hop needed, since the node body already runs as an Activity) — rather than shipping the whole accumulated state through Temporal in one call at the end. This keeps every persistence write proportional to a single node's output, matching what the pre-Temporal execution already did natively.
Questions for the maintainers:
- Is per-node/per-Activity persistence the idiomatic pattern here, or is there Temporal-native support for this (e.g. something LangGraphPlugin already offers) that we're missing?
- We also considered gzip-compressing the final payload before passing it as Activity args — it only raises the ceiling rather than removing it, and doesn't restore true mid-run durability (a crash just before that one activity still loses the whole run's state). Are there better-supported options for cases where a single large Activity payload is genuinely unavoidable (custom
PayloadCodec, chunking, streaming activity results)? - Any general guidance on structuring Activities around something like LangGraph, where per-node output sizes vary widely and the framework's own native checkpointing gets bypassed by the Activity boundary?
Activity
Poojan6216 commentedon Sep 26, 2026
Hi, I spent some time digging into this. A few things that might help:
1. Per node persistence inside the node's Activity is the right place. Workflow code can't do I/O, and a separate persist Activity would carry the payload through history again. Two things I'd add:
2. Heads up on size. The plugin sends each node's full state as its Activity input (
ActivityInput(args=(state,), ...)incontrib/langgraph/_activity.py), and every Activity input is stored in history. So history grows roughly with the square of the transcript, even when every single node is under 2 MB. In an offline test with Temporal's default DataConverter and 60 KiB of new tool output per node:With smaller outputs the 50 MB history limit comes first instead. Either way, per node persistence alone hits a wall later.
Side note: depending on SDK version and the worker's
disable_payload_error_limitsetting, an oversized payload can also show up as a workflow task failure (PAYLOADS_TOO_LARGE) that keeps retrying instead of a termination, so it's worth alerting on both.3. The Temporal native answer for size is External Storage (Public Preview):
DataConverter(external_storage=ExternalStorage(drivers=[...])). It swaps large payloads for small references in history. Two caveats:If you can change the graph, keeping big tool outputs out of state (store them somewhere and keep an ID in state) fixes this at the root.
I'm building a small open source plugin that does the idempotent, fenced per node ledger plus a dedup storage driver for Postgres, with crash tests checked against Temporal's own history. Happy to share it here once the numbers are measured on a live server, if that's useful.
Poojan6216 commentedon Sep 28, 2026
Following up on my comment above: I turned those suggestions into a working plugin so they could be tested instead of argued. It's called Stepledger (Apache 2.0,
pip install stepledger): https://github.com/Poojan6216/stepledgerIt sits next to
LangGraphPluginwithout changing it or any graph code:stepledger reconcilechecks every row against the workflow's own history.What I measured on the Temporal dev server, all reproducible from
bench/:Known gaps, listed in the README:
execute_in="workflow"nodes and task-cache hits run no Activity, so they get no row.materialize()marks those positions as GAP rather than guessing.One question for the maintainers: Stepledger currently reads five private symbols from the LangGraph plugin, because there is no public hook for "this node's result was accepted by the workflow". Would you be open to a small public hook for that, such as a per-node completion callback? I'd be happy to open a PR if that direction makes sense.
brianstrauch commentedon Sep 29, 2026
Hi @dorelad-upwind, thanks for the issue! I would recommend using Temporal's External Storage feature to resolve the size limit error, and then introduce an Activity to persist that data from S3 to Postgres. To prevent double-storing data, you could look into the feasibility of writing a custom External Storage driver for your Postgres use-case. Feel free to re-open the issue if you have any follow-up questions!
brianstrauch commentedon Sep 29, 2026
For extra context, note that this issue is a follow-up of #1803