Skip to content

server: add /v1/audio/transcriptions/details for transcript detail fields - #401

Open
CryptVenture wants to merge 1 commit into
0xShug0:mainfrom
CryptVenture:pr/asr-detail-fields
Open

server: add /v1/audio/transcriptions/details for transcript detail fields#401
CryptVenture wants to merge 1 commit into
0xShug0:mainfrom
CryptVenture:pr/asr-detail-fields

Conversation

@CryptVenture

@CryptVenture CryptVenture commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Reworked per review: /v1/audio/transcriptions keeps its current response schema, and the richer response moves to an opt-in POST /v1/audio/transcriptions/details.

The problem

ASR models that align words, segment speech or separate speakers report that through TaskResult::word_timestamps, speech_segments and speaker_turns. /v1/audio/transcriptions serialises text and timing only, so for those models the alignment is computed and then dropped on the way out of the server. /v1/tasks/run already serialises the arrays, so the data reaches one route and not the other.

The route

Same request as /v1/audio/transcriptions — JSON with a server-local path, or a multipart upload. The response is a superset of the plain one: text first, timing last, with language, segments, speaker_turns and words in between where the model produced them.

Spans are sample offsets because that is what the models report, so sample_rate travels with them, emitted only when at least one array is present.

stream=true returns 400 on this route. The SSE response carries transcript deltas only and has nowhere to put the arrays, so accepting the request would return none of what the route exists to return.

The serialisation /v1/tasks/run already performed is factored into write_transcript_detail_fields and shared rather than duplicated.

Compatibility

/v1/audio/transcriptions and /v1/tasks/run produce byte-identical responses to main. Nothing existing changes shape.

Verification

macOS, Metal backend, Parakeet TDT 0.6B v3 q8_0, same 24 kHz input to every request:

POST /v1/audio/transcriptions           200  ["text","timing"]
POST /v1/audio/transcriptions/details   200  ["text","words","sample_rate","timing"]
                                             253 words, sample_rate 24000, text identical to the plain route
POST /v1/audio/transcriptions/details   200  same result via the JSON body form
     (Content-Type: application/json)
POST /v1/audio/transcriptions/details   400  streaming is not supported on /v1/audio/transcriptions/details
     (stream=true)
POST /v1/tasks/run                      200  ["text","timing","words"]  — unchanged

253 word timings that the plain route discards. ctest passes 40/40.

Docs

app/server/README.md gains a section for the route: request forms, the response example, a table of which field appears when, the sample-offset explanation, and the streaming restriction. README.md gains the route in its endpoint list.

@CryptVenture
CryptVenture force-pushed the pr/asr-detail-fields branch 2 times, most recently from 255eac1 to c287387 Compare September 2, 2026 23:30
@CryptVenture
CryptVenture marked this pull request as ready for review September 3, 2026 02:12
@0xShug0

0xShug0 commented Sep 3, 2026

Copy link
Copy Markdown
Owner

@CryptVenture This is caused by legacy design, and we never touched it because no user requested it before. My concern is that existing pipelines may already depend on the current /v1/audio/transcriptions response schema, so adding extra fields by default could break them.

A better solution is to add a separate opt-in endpoint, like POST /v1/audio/transcriptions/details, for the richer timestamp/detail response, and document the new endpoint clearly.

@CryptVenture

Copy link
Copy Markdown
Contributor Author

Agreed on the compatibility risk — a caller that iterates the response object or asserts on its keys would break, and that is not worth it for a field nobody asked for. I will rework this as you describe: /v1/audio/transcriptions goes back to its current schema untouched, and the richer response moves to an opt-in POST /v1/audio/transcriptions/details, documented alongside the existing route.

One question before I push, because the same argument cuts both ways.

This PR also added sample_rate to /v1/tasks/run. That route already returns the detail arrays today — words, and the segment and speaker arrays where the model produces them — but their offsets are in samples, and without the rate there is nothing to convert them with, so the WebUI cannot render them at all. Two ways to resolve that, and I would rather you pick:

  1. Leave /v1/tasks/run untouched as well, and have the WebUI call /v1/audio/transcriptions/details for ASR instead. Nothing existing changes shape, and the detail response has one home. This is my preference, and it follows your reasoning to its conclusion.
  2. Keep the single additive sample_rate key on /v1/tasks/run. Smaller diff, but it is still a default-schema change, which is the thing you objected to.

I will go with 1 unless you say otherwise. Either way the existing /v1/audio/transcriptions response is left exactly as it is.

…elds

ASR models that align words, segment speech or separate speakers report that
work through TaskResult::word_timestamps, speech_segments and speaker_turns.
/v1/audio/transcriptions serialises text and timing only, so for those models
the alignment is computed and then discarded on the way out of the server.

Rather than widen the existing response, which callers already build against,
this adds an opt-in route with the same request shape. /v1/audio/transcriptions
and /v1/tasks/run are byte-identical to before.

The detail response is a superset of the plain one: text first, timing last,
with language, segments, speaker_turns and words in between where the model
produced them. Spans are sample offsets because that is what the models report,
so sample_rate travels with them and is emitted only when at least one of the
arrays is present.

stream=true is rejected with a 400 on the detail route. The SSE response carries
transcript deltas only and has nowhere to put the arrays, so accepting the
request would return none of what the route exists to return.

The serialisation the generic task route already performed is factored into
write_transcript_detail_fields and shared, rather than duplicated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ATa5YkLUPMDPRL7w1gCo9p
@CryptVenture CryptVenture changed the title server: return ASR detail fields from /v1/audio/transcriptions and /v1/tasks/run server: add /v1/audio/transcriptions/details for transcript detail fields Sep 3, 2026
@CryptVenture

Copy link
Copy Markdown
Contributor Author

Pushed the rework, and I went with option 1 as flagged above — nothing existing changes shape.

  • /v1/audio/transcriptions: reverted to its current schema, byte-identical to main.
  • /v1/tasks/run: also reverted, byte-identical to main. The WebUI will call the new route instead.
  • New POST /v1/audio/transcriptions/details: same request forms (JSON path or multipart upload), response is a superset — text first, timing last, with language, segments, speaker_turns and words in between where the model produced them, plus sample_rate when any of those arrays is present.
  • stream=true returns 400 there, since the SSE response has nowhere to put the arrays.
  • Documented in app/server/README.md with the response example and a table of which field appears when; the route is added to the endpoint list in README.md.

Verified on macOS/Metal with Parakeet TDT 0.6B v3, same 24 kHz input throughout:

POST /v1/audio/transcriptions           200  ["text","timing"]
POST /v1/audio/transcriptions/details   200  ["text","words","sample_rate","timing"]  253 words, sample_rate 24000, same text
POST /v1/audio/transcriptions/details   400  stream=true
POST /v1/tasks/run                      200  ["text","timing","words"]  unchanged

ctest passes 40/40. Rebased on current main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants