Skip to content

fix(ai): only record a service tier the provider actually served - #4818

Merged
bernatixer merged 10 commits into
mainfrom
fix/ai-served-tier-provenance
Sep 10, 2026
Merged

bernatixer merged 10 commits into
mainfrom
fix/ai-served-tier-provenance

Conversation

@bernatixer

@bernatixer bernatixer commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem

  • The service_tier inside $ai_model_parameters is what PostHog cost processing prices from (feat(aio): price ai generations by their served service tier posthog#94200), so it must carry the tier the provider served — a requested tier can be refused.
  • getModelParams captured the requested tier from the call's own params, and the served value only overwrote it on the OpenAI success paths. Two surfaces leaked the requested tier with billable usage attached: OpenAI error captures (partial usage from an errored stream, no response value to overwrite), and every LangChain-emitted generation (the callback captured invocation params at run start and never merged the response's tier).

Changes

  • service_tier leaves getModelParams' request-side allowlist: the key now appears only when a response supplied it, so its presence is its provenance. This closes both leaks at once and matches the Python SDK (feat(ai): capture the served service tier into model parameters posthog-python#920). It also stops Anthropic events recording the requested 'auto'/'standard_only' — Anthropic capture passes no served value, so those events now carry no tier at all.
  • The LangChain callback merges the served tier from wherever langchain surfaces it: the message's response_metadata (the Responses adapter, streaming and non-streaming — also the only carrier on the middleware path) and generationInfo (the Completions adapter, streamed). Those are the only two places langchain-js reports it; llmOutput there carries token usage only.
  • Known limitation: non-streaming LangChain calls through the Completions adapter carry no tier — langchain does not surface it there — so they price at standard, the conservative direction, until langchain exposes it.
  • Ingestion independently guards the error-path population for already-released SDK versions (error events never tier-price), so this fix and that guard overlap on purpose.

Release info Sub-libraries affected

Libraries affected

  • @posthog/ai

Checklist

  • Tests for new code
  • Accounted for the impact of any changes across different platforms
  • Accounted for backwards compatibility of any changes (no breaking changes!)
  • Took care not to unnecessarily increase the bundle size

The requested tier leaked into $ai_model_parameters wherever no response
value overwrote it: openai error paths and the whole LangChain callback.
Cost processing prices from that key, so tokens could price at a tier the
provider never confirmed. service_tier leaves the request-side allowlist,
and the LangChain callback merges the served tier from generationInfo.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bernatixer bernatixer self-assigned this Sep 7, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bernatixer
bernatixer marked this pull request as ready for review September 7, 2026 10:39
@bernatixer
bernatixer requested a review from a team as a code owner September 7, 2026 10:40
@bernatixer
bernatixer requested a review from a team September 7, 2026 10:40
The Responses adapter stores it there for streaming and non-streaming
results, and the middleware path carries no generationInfo at all.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
arnohillen
arnohillen previously approved these changes Sep 7, 2026

@arnohillen arnohillen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agent drafted, arno reviewed

Verified at 5d776db9. The allowlist removal in packages/ai/src/utils.ts and the served-tier merge at utils.ts:145-146 do what the title says; the three new served-tier tests in packages/ai/tests/callbacks.test.ts pin generationInfo, response_metadata, and the none-served case.

One minor note, not blocking: the same allowlist removal also stops the requested service_tier ('auto' / 'standard_only') reaching $ai_model_parameters on Anthropic events (packages/ai/src/anthropic/index.ts:248, 268, 306, 331 call getModelParams(body) with no served tier to merge). That is consistent with the PR intent, but the changeset and body name only the OpenAI error paths and LangChain, so an Anthropic integrator reading the changelog will not expect it.

The llmOutput.service_tier fallback at callbacks.ts:725 has no test; every new test passes llmOutput: {}.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@arnohillen
arnohillen dismissed their stale review September 7, 2026 15:06

agent drafted, arno reviewed

Withdrawn; leaving review to the client-libraries team.

Cost processing prices only from this property: its writers assert
served values, unlike $ai_model_parameters.service_tier, which released
SDKs populated from the request.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread packages/ai/src/langchain/callbacks.ts Outdated
bernatixer and others added 4 commits September 10, 2026 10:16
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
langchain-js reports the served tier in response_metadata or
generationInfo; its llmOutput carries token usage only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@bernatixer
bernatixer merged commit 475743c into main Sep 10, 2026
58 checks passed
@bernatixer
bernatixer deleted the fix/ai-served-tier-provenance branch September 10, 2026 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants