gateway: tolerate non-string values in Bifrost log metadata (fixes 502'd rollups) - #1697
Merged
Merged
Conversation
…2'd rollups) Bifrost's logging plugin stamps bool markers into a row's metadata: `realtime: true` on realtime turns and `isAsyncRequest: true` on x-bf-async jobs (plugins/logging/main.go). The plugin decoded metadata straight into map[string]string, so a single such row in the window failed the whole /api/logs page and every rollup built on it (spend.by_agent, by_user, by_agent_user, histogram.cost) returned 502 — the dashboard went blank on 2026-09-15 after a realtime turn landed in the 24h window. Decode metadata through a metadataMap type instead: strings verbatim, bool/number values as their JSON text, nested and null values dropped. Applied to both the list row and the detail row.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What broke
Every spend/histogram rollup in the gateway UI started returning 502 on 2026-09-15 (`docker logs bifrost.sphinx`):
Bifrost's log `metadata` is a `map[string]interface{}` on the wire (`framework/logstore/tables.go`). The dim headers we stamp are strings, but Bifrost's own logging plugin writes bool markers on some rows (`plugins/logging/main.go`):
The plugin decoded metadata straight into `map[string]string`, so one such row on a page failed the whole `/api/logs` decode, and every rollup built on `windowedLogs` (by_agent, by_user, by_agent_user, histogram.cost, sessions, users) 502'd for as long as that row sat inside the window. Nothing in Bifrost changed (still pinned at `transports/v1.6.2`); a realtime turn simply landed in the logstore for the first time (the realtime WS path was being exercised earlier today, before #1695 shipped the wrapper block). Retention is 365d, so the row is not going away on its own.
Fix
Decode metadata through a `metadataMap` type with a tolerant `UnmarshalJSON`:
Applied to both the list row (`logstoreLog`) and the detail row (`logstoreLogDetail`). All existing indexing (`l.Metadata["run-id"]`) and the `map[string]string` output structs are untouched (named map → unnamed identical underlying type is assignable). No tygo drift: both types are unexported.
Verify
`go test ./internal/adminapi/` passes, including the new `TestMetadataMap_ToleratesNonStringValues`. `make tygo-check` clean.
To see the offending row on the swarm (from inside the network, with the same creds the plugin uses):
Release
Needs a new image (v0.4.264): publish a GitHub release once merged, same as #1696.