Skip to content

feat(tracing): report cached input token usage - #1148

Merged
yaozheng-fang merged 1 commit into
volcengine:mainfrom
evanlowe:feat/cache-read-token-metrics
Sep 23, 2026
Merged

yaozheng-fang merged 1 commit into
volcengine:mainfrom
evanlowe:feat/cache-read-token-metrics

Conversation

@evanlowe

Copy link
Copy Markdown
Contributor

LLM token metrics currently expose input and output usage but omit cache hits, even when model response metadata includes cached input tokens

This adds gen_ai_token_type=cache_read to the existing gen_ai.client.token.usage metric and stops reporting cache reads as cache creation in traces

  • Keep input and output totals unchanged: 1,000 input tokens with 600 cached tokens report input=1000 and cache_read=600
  • Record an explicit zero when provided; omit cache-read samples when the normalized cache count is absent
  • Cache reads are a subset of input usage, so total-consumption queries must continue selecting input and output rather than summing every token type

Validation: pre-commit passed and 57 related tests passed, including Volcengine and BytePlus endpoint labels, streaming and non-streaming recording, zero or missing cache counts, and trace attributes

Validation uses local response fixtures and an in-memory OpenTelemetry reader; no live cloud calls were made

@yaozheng-fang
yaozheng-fang merged commit 587d9fe into volcengine:main Sep 23, 2026
13 checks passed
PolarisCzh pushed a commit to PolarisCzh/veadk-python that referenced this pull request Sep 29, 2026
Co-authored-by: TRAE CLI <traecli@bytedance.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants