feat(AIC-3408): add Vercel AI SDK messages and agents adapters - #103
andrewklatzke wants to merge 7 commits into
Conversation
Route LaunchDarkly AI Configs through the official Python ai runtime and AI Gateway, including native graph execution and a direct-model opt-out. Co-authored-by: Cursor <cursoragent@cursor.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Use the shared span helpers in production, trace tool execution, and emit provider-compatible strict schemas for messages output.
…cel-adapter # Conflicts: # .release-please-manifest.json
CancelledError is not an Exception, so those spans were ending unmarked or as abandoned. Handoff tools now ignore extra model arguments instead of raising TypeError. Co-authored-by: Cursor <cursoragent@cursor.com>
Tool loops were collapsing every inference into one span and forcing the final schema onto intermediate tool calls, and the native graph span omitted the token totals the other adapters write.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 89c52d5. Configure here.
| results = [await _tool_result(call, executors, parent) for call in calls] | ||
| messages = [*messages, message, ai.tool_message(*results)] | ||
| raise RuntimeError( | ||
| f"Vercel messages run did not reach a final response within {MAX_STEPS} steps" |
There was a problem hiding this comment.
Structured follow-up can exhaust step limit
Medium Severity
When outputFormat and tools are both set, a no-tool-call turn now continues so a later constrained request can apply the schema. That extra iteration still counts against MAX_STEPS. If the unconstrained final answer arrives on the last allowed step, the loop ends and raises RuntimeError instead of performing the structured turn, so a completed tool loop with structured output fails.
Reviewed by Cursor Bugbot for commit 89c52d5. Configure here.
jeffdupont
left a comment
jeffdupont
left a comment
There was a problem hiding this comment.
Reviewed with the 1.0 freeze in mind. Despite the name, this isn't a wire format or a JS bridge. It wraps Vercel's own Python runtime, the ai package on PyPI (vercel-labs/ai-python, 0.7.0, classifier "Development Status :: 4 - Beta"), and it's the Python half of launchdarkly/js-ai-sdk#83, which merged on 29 Sep. The public names match JS one for one (vercel_messages, create_vercel_messages_handler, vercel_agents, create_vercel_agents_handler, vercel_graph, to_vercel_agents), and model_id.py is a line-for-line port of JS model-id.ts. Tests pass at 89c52d5: make test 1523 passed, 15 skipped; make typecheck and make lint clean, all exit 0. The branch conflicts with main only in .release-please-manifest.json, because main has since released 0.2.4.
Three things I'd like settled before GA:
- An AI Config can replace the Gateway API key and the model.
_request_paramsforwardsextra_headersandextra_queryfrommodel.parametersand puts every other unknown key intoextra_body. The Gateway client applies those headers after its own (ai/providers/ai_gateway/client/_client.py,request_headers.update(headers)). I reproduced it at 89c52d5 with the HTTP call stubbed. A config withextra_headers: {Authorization: "Bearer attacker-key", ai-language-model-id: "openai/gpt-5-pro"}sends thatAuthorizationheader in place of the customer'sAI_GATEWAY_API_KEY, and theai-language-model-idheader switches the model even thoughmodelis on the owned list.providerOptions.gateway.onlyalso reaches the request body, so a config can change which upstream provider serves the call. This is the same class of problem as #107: anyone who can edit an AI Config can redirect the customer's traffic. The agents handler (build_request_params) and the native graph share the code. Inline comment below. - The native graph doesn't send the per-node events the spec now requires. Since monorepo #26 (AIC-3211, merged into this branch through #114),
TESTING.md§2.2 has every native adapter send$ld:ai:graph:nodewithnodeKey/indexwhen it enters a node. The other three Python adapters do.to_vercel_agentsdoesn't, and it sends onlygeneration:successper node. JStoVercelAgentssends per-nodeduration:totalandtokens:*, plus$ld:ai:graph:path, which §2.2 now says not to send. So one graph run gives Monitoring three different event sets: Python Vercel, JS Vercel, and the other adapters. Event names are a data contract, so this should be fixed in both languages before 1.0. Inline comment below. - Should these packages ship as 1.0? The
aidependency is a 0.x public beta, and the bound isai>=0.7with no upper limit, so any 0.8 release can break it. The experimental lifecycle draft (TESTING.md §0) covers experimental names insidelaunchdarkly_ai_server. It doesn't say what to do with a whole provider package. I'd suggest keeping both Vercel packages on 0.x in both languages when the rest go to 1.0, writing that rule into §0 (a provider package built on a pre-1.0 runtime stays 0.x), and cappingai<0.8until then. The GA plan's package count doesn't include these two Python packages yet.
Smaller notes:
- The spec PR, launchdarkly/ai-sdks-monorepo#19, has no reviews, hasn't been updated since 22 Sep, and still points the python submodule at e83fa11. Its A.14 says
experimental_evaluateis "Not available in officialai0.7", and it skipsvercel_evaluatefor Python on that basis. Butai0.7.0 does exportai.ops.experimental_evaluate(model, state, questions, *, params=None). So either Python gets avercel_evaluate, or the spec should give a different reason for leaving it out. Since it wraps an upstreamexperimental_API, JSvercelEvaluateprobably belongs on an experimental entry point and not the package root. #19 also needs the$ld:ai:graph:nodeline in §2.x.3. - The last Bugbot comment (step limit,
handler.py:422) has no reply yet. With tools andoutputFormatboth set, a final answer that arrives on step 10 hitscontinueand then theRuntimeError. The structured follow-up also costs an extra model call and discards the first answer. - Graph-level events use
make_track_data(definition.root, ...), the same as the other adapters. Whatever #121 decides about whichconfigKeygraph events carry applies here too. - JS #83 spreads the whole
model.parametersbag intogenerateText/streamText(modelSettings), soheadersandproviderOptionsprobably reach the request there as well. I haven't checked this at runtime in JS. Since #83 has merged, it likely needs a follow-up there.
| reasoning_effort = raw.pop("reasoning_effort", raw.pop("reasoningEffort", None)) | ||
| if reasoning_effort is not None: | ||
| kwargs["reasoning"] = ai.ReasoningParams(effort=reasoning_effort) | ||
| for direct in ("metadata", "safety_identifier", "extra_headers", "extra_query"): |
There was a problem hiding this comment.
These keys come from the AI Config, and the Gateway client merges extra_headers over its own headers. I reproduced it at 89c52d5 with the HTTP call stubbed. With model.parameters.extra_headers = {"Authorization": "Bearer attacker-key", "ai-language-model-id": "openai/gpt-5-pro"}, the request to https://ai-gateway.vercel.sh/v4/ai/language-model carries the attacker's key in place of AI_GATEWAY_API_KEY, and the model id header is replaced even though model is owned. Everything left over goes into extra_body on line 106, so providerOptions.gateway.only (provider routing) gets through too.
I'd suggest an allowlist, like the one #107 uses for the other handlers: the sampling, max_tokens and reasoning_effort mappings above, and nothing that sets headers, query strings or providerOptions. Drop extra_headers and extra_query from this tuple and don't forward the rest as extra_body. build_request_params in vercel-agents (line 100) has the same code. A test that sets these keys and checks they don't reach the request would keep it fixed.
| final_text = "" | ||
| try: | ||
| while current is not None and current.key not in path: | ||
| path.append(current.key) |
There was a problem hiding this comment.
This is where a node is entered, but no $ld:ai:graph:node event is sent. TESTING.md §2.2 (monorepo #26) requires one per visited node, with nodeKey, a 0-based index and value 1, and openai-agents, claude-agents and langchain-agents all send it on main. Per node, this adapter sends only $ld:ai:generation:success (line 167). JS toVercelAgents also sends duration:total, tokens:input/output/total and $ld:ai:graph:path. Could this send graph:node here, and could the two languages agree on the per-node events? Then a Vercel graph run would look the same to Monitoring whichever SDK ran it.
| dependencies = [ | ||
| "launchdarkly-ai-server", | ||
| "opentelemetry-api>=1.25", | ||
| "ai>=0.7", |
There was a problem hiding this comment.
ai is 0.7.0 and marked Beta on PyPI, so 0.8 can make breaking changes, and this range accepts it. I'd cap it at ai>=0.7,<0.8 (same in vercel-messages) and keep this package on 0.x when the rest of the SDK goes to 1.0, until the upstream runtime is stable. The release-please entry already has bump-minor-pre-major, so it stays below 1.0 unless someone moves it on purpose.
3 participants
Summary
launchdarkly-ai-vercel-messagesandlaunchdarkly-ai-vercel-agentspackages that map LaunchDarkly AI Configs onto the official Pythonairuntime and AI Gatewaycreator/modelids.Test plan
make format-check && make lint && make typecheck && make testinpython/uv run python main.py vercel-agents launch-darkly-documentation-summarizer "..."and confirm a non-empty response, non-zero usage, and a judge scorevercel-messagesandvercel-directsuccess/failure keys fromintegration-config.jsonnative-graph-vercelfails cleanly if the Gateway free tier still blocks the graph modelMade with Cursor