Skip to content

feat(transcription): transcribe with any OpenAI-compatible server - #546

Open
chhoumann wants to merge 126 commits into
masterfrom
feat/transcription-openai-compatible
Open

chhoumann wants to merge 126 commits into
masterfrom
feat/transcription-openai-compatible

Conversation

@chhoumann

@chhoumann chhoumann commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Why

Freedom of choice among transcription providers, in a codebase where adding a provider stays cheap and each task has one way to do it. This PR and #544 were built in parallel on #542 and overlapped, so this branch now merges #544 and stacks on it, with one way to read an OpenAI-shaped answer and one json fallback per job.

Summary

Freedom of choice for transcription. A user can now transcribe with any server that speaks OpenAI's /v1/audio/transcriptions API: a local whisper.cpp, faster-whisper or speaches server, so the audio stays on their own machine, or a hosted one such as Groq or Together. They are no longer tied to the four built-in providers. Stacked on #544, which builds on #542.

It follows the recipe that added ElevenLabs and OpenRouter: one CREDENTIALS row, one provider file (providers/openaiCompatible.ts), one registry line, plus the settings this provider needs.

  • Settings. Choosing OpenAI-compatible server shows Transcription provider, Server address (placeholder http://localhost:8000/v1), OpenAI-compatible server API key, Model as a text field, then the shared rows. An address without http:// or https:// says so under the field. Speaker labels behaves as for any model without them: it reads OpenAI-compatible server does not label speakers. and can only be turned off.
  • Address. https://api.groq.com/openai/v1 and .../v1/audio/transcriptions reach the same endpoint, after spaces and trailing slashes are trimmed. Only http and https, and no user name or password in the address: the key has its own field.
  • Private hosts. requestWithTimeout and assertFetchableUrl take a privateHosts option, and only this provider's sender passes it, because the user typed the address. Feeds, transcripts, chapters and the hosted providers keep the old policy, which still blocks localhost and private networks. Artwork is unchanged: the media element loads a feed's own artwork address, private or not, as it does in the released version.
  • Key and gate. The key is optional. With none selected the request has no Authorization header. A selected key that is not on this device still stops the job. An empty model or a non-http(s) address stops it with a notice that says what to enter.
  • Request. Multipart model, file, response_format=verbose_json, timestamp_granularities[]=word and segment, in parts of 24 MB with no length cap. A 400 whose body names response_format, verbose_json or timestamp_granularities, as OpenAI's gpt-transcribe answers, is sent once more with response_format=json and no timestamp option, and every later part of the job goes straight to json. Any other 400 is not resent.
  • Answer. words[] become timed paragraphs, punctuated from text when the words carry none, else segments[] become timed paragraphs through feat(transcription): offer Voxtral Mini Transcribe 3 and Qwen3 ASR on OpenRouter #544's { segments } transcript, else text becomes untimed paragraphs.
  • Errors stay PodNotes' own sentences: The server at <host> rejected the API key., The server at <host> has no <model> model at that address. for a 404, PodNotes could not reach the server at <host>. for no answer. A refused json resend reads OpenAI-compatible server did not answer in a format PodNotes reads.
  • Privacy. The profile says url: false, so the audio is always uploaded and no episode address, public or private, reaches the server.
  • Panel. The player's Transcript panel reads Transcribe with your server without the Paid tag, because PodNotes cannot tell whether the user's server charges, and Transcribe with OpenAI-compatible server Paid overflowed a 300 px sidebar. The failed-published-transcript notice offers your OpenAI-compatible server without the API credit clause.

Merge with #544

feat/transcription-openrouter-models (#544, 92b5a19) is merged into this branch, and the base is now #544's branch.

  • Conflicts. providers/types.ts keeps feat(transcription): offer Voxtral Mini Transcribe 3 and Qwen3 ASR on OpenRouter #544's Segment type and { segments } PartTranscript variant. The verify skill's transcripts recipe keeps both providers' sections, with the OpenAI notice now naming GPT Transcribe as feat(transcription): offer Voxtral Mini Transcribe 3 and Qwen3 ASR on OpenRouter #544 changed it. The OpenAI-compatible e2e journey is renamed from J-TR-11 to J-TR-13, because feat(transcription): offer Voxtral Mini Transcribe 3 and Qwen3 ASR on OpenRouter #544's Voxtral and Qwen3 ASR journeys took J-TR-11 and J-TR-12.
  • One segments path. The OpenAI-compatible provider returns { segments } for a segments-only answer, which cues.ts times across parts as it does for Voxtral. Its segments-as-words reader is deleted.
  • One verbose_json reader. OpenRouter and OpenAI-compatible servers answer in OpenAI's verbose_json shape, so both now read it through providers/verboseJson.ts: words when the answer lists any, else segments, else nothing. feat(transcription): offer Voxtral Mini Transcribe 3 and Qwen3 ASR on OpenRouter #544's punctuated(words, text) moved there from openrouter.ts, so an OpenAI-compatible server that lists bare words, as a local Qwen or Whisper server can, gets its punctuation from text too. OpenRouter keeps "no speech" for an answer with neither, and the OpenAI-compatible provider keeps text for it.
  • json fallback once per job. OpenAI's API answers verbose_json for gpt-transcribe with 400 "response_format 'verbose_json' is not compatible with model ...". The fallback used to live in the provider, per part, so every part was uploaded twice. It now lives in the job (transcribe.ts). After the first such 400 the job asks for json for that part and every later one, through a textOnly field on PartInput.

Verification

After the merge

All with Node 22.23.2 through fnm exec, on head 3fe2dab:

  • npm run lint, format:check, typecheck, build, size, test (124 files, 2,932 tests), test:tooling (48 tests) and docs:build exit 0.
  • npm run test:e2e -- -t "J-TR" passes 13 tests, J-TR-11 (Voxtral), J-TR-12 (Qwen3 ASR) and J-TR-13 (OpenAI-compatible) included. npm run test:e2e -- tests/e2e/settings-tab.test.ts passes 11.
  • npm run size: the merge commit alone, built from a git archive export, measures 331,775 raw and 108,094 gzip bytes. With the shared reader and the job-level fallback the branch measures 331,542 and 108,060. The budgets are those plus the headroom the stack keeps, 2,554 raw and 97 gzip bytes: 334,096 and 108,157. scripts/size-baseline.md has an entry for the merge, and AGENTS.md names both budgets.
  • The two new tests fail on the merge commit's own source, and pass on the branch.

Plants, each through npm run plant. Every one failed its named test and passed once restored:

Plant Caught by
Fallback per part again (textOnly reset after each part) after a server refuses verbose_json, the episode's next part is sent once, as json
The OpenAI-compatible provider skips the punctuation repair (text withheld from the reader) words a server lists without punctuation take their punctuation from its text
Keep the bare word on a match (#544) Qwen's words, which it lists without punctuation, take their punctuation from its text
One token of lookahead (#544) the same test
OpenRouter saves an answer with neither words nor segments as untimed text (#544) an answer with neither words nor segments writes no note
Part offset dropped for segments (#544) Voxtral's segments from two parts of at most 900 s become paragraphs timed from the episode's start
Ignore segments an answer with segments and no words is saved as its note
Read words under the segment key an answer with words is saved as its note
Ignore json text an answer with text alone is saved as its note
Prefer segments over words sends one multipart request for word and segment times with the key, and times each paragraph
No json fallback (job rethrows the format 400) a server that refuses verbose_json is asked once more for json, and the note is untimed
The fallback asks for verbose_json again the same test
No json fallback (textOnly never set) the same test
Fall back twice (a third send) a server that refuses json too is not asked a third time
Fall back on any 400 a 400 about anything else is not sent again

One branch went instead of getting a test. Planting "fall back twice" as the removal of a textOnly || guard was not caught, because that guard only stopped a resend when a later part's json request was refused too. That is implausible, and it costs one bounded duplicate send, so the guard is deleted.

Runtime, Obsidian 1.14.4 on Linux, this worktree's isolated instance, fresh vault. fixtures.mjs server printed http://podnotes.test:26951/v1. I wrote Fixtures/Big Fixture.mp3 with ffmpeg -f lavfi -i sine=frequency=440:duration=1900 -ac 1 -b:a 128k, 1,900 s and 30,401,140 bytes, so it goes in two parts under the 24 MB cap. In Settings > PodNotes I chose OpenAI-compatible server with the keyboard, typed the address and the model json-only, which the fixture refuses verbose_json for with gpt-transcribe's 400. data.json read provider: "openai-compatible", model: "json-only", serverUrl: "http://podnotes.test:26951/v1". Then I picked Play with PodNotes from the file menu and ran Transcribe current episode from audio from the palette. fixtures.mjs transcriptions showed three requests:

{"authorization":null,"model":"json-only","responseFormat":"verbose_json","timestampGranularities":["word","segment"],"fileType":"audio/mp3","fileBytes":15200800}
{"authorization":null,"model":"json-only","responseFormat":"json","timestampGranularities":[],"fileType":"audio/mp3","fileBytes":15200800}
{"authorization":null,"model":"json-only","responseFormat":"json","timestampGranularities":[],"fileType":"audio/mp3","fileBytes":15200340}

The second part went once, as json, and the two parts sum to the file's bytes. transcripts/local file/Big Fixture.md was saved and opened with both parts' text, untimed. The notice read "Saved without speaker labels, because OpenAI-compatible server does not label speakers. These providers do: ElevenLabs, Deepgram, OpenRouter.", since speaker labels were on as on a new install. dev:errors reported none. The instance and the fixture server are stopped.

Untimed note from a two-part MP3 after one json fallback

Before the merge

  • npm run test passes 2,923 unit tests, 35 new. lint, format:check, typecheck, build, test:tooling (48 passed) and docs:build pass.
  • npm run test:e2e -- tests/e2e/journey-resilience.test.ts -t "J-TR" passes 10 tests, the new J-TR-11 (now J-TR-13) included. The journey fixture server now also answers POST /v1/audio/transcriptions. J-TR-11 seeds the provider with the server's address plus a trailing slash and no key, runs Transcribe current episode, and checks the timed note and the one request: no Authorization, the model, verbose_json, both granularities and the episode's audio byte for byte. The request goes through Obsidian's requestUrl to 127.0.0.1. npm run test:e2e -- tests/e2e/settings-tab.test.ts tests/e2e/journey-transcript.test.ts tests/e2e/focused-settings-save.test.ts passes 27 tests.
  • npm run size: the base built from an export of feat(transcription): one speaker label per voice across the parts of an episode #542's head measures 327,446 raw and 106,873 gzip bytes, this branch 330,739 and 107,768. Both budgets rise by the measured delta, 3,293 raw and 895 gzip, to 333,293 and 107,865, which keeps the base's 2,554 and 97 bytes of headroom. This is the first rise of the raw budget. AGENTS.md now names both.
  • Every plant below failed its named test and passed once restored, through npm run plant, each run against the test's own file (37 of 37):
Plant Caught by
Always send Authorization with no key selected, runs and sends no Authorization header
A selected key missing on this device is ignored a selected key that is not on this device stops the job...
Append the path to a full URL too the address " https://api.groq.com/openai/v1/audio/transcriptions/ " sends to the server's transcriptions endpoint
Trim one trailing slash only the address "https://api.groq.com/openai/v1//" ...
Do not trim spaces the address " https://.../audio/transcriptions/ " ...
Accept any scheme asks for an http(s) address in place of "ftp://files.example.com/v1"
No address check before the job asks for an http(s) address in place of "localhost:8000/v1"
No json fallback (format 400 read as rejected) a server that refuses verbose_json is asked once more for json...
The fallback asks for verbose_json again the same test
Fall back on any 400 a 400 about anything else is not sent again
Fall back twice a server that refuses json too is not asked a third time
Ask for word times only sends one multipart request for word and segment times with the key...
Prefer segments over words the same test
Read words under the segment key an answer with words is saved as its note
Ignore segments an answer with segments and no words is saved as its note
Ignore json text an answer with text alone is saved as its note
Claim speaker labels with speaker labels on, saves the note without them and says so...
No model check Transcribe current episode from audio asks for a model when none is entered...
Model not trimmed Transcribe current episode asks for a model when none is entered...
The sender keeps private hosts blocked a server on this machine or a private network at http://192.168.1.20:9000/v1 is reached
The request boundary drops privateHosts reaches this device or a private network only when asked, under every other rule
The request boundary allows private hosts for every caller the same test
Private hosts skip the other target rules the same test
A rejected key names the provider, not the server a rejected key says what to check, in PodNotes' words
No answer names the provider no answer says what to check...
A 404 names the provider no such address or model says what to check...
The profile takes addresses a public feed's episode is uploaded to a local server, and no request holds its address
The profile takes addresses and the private-feed check is gone a private feed's episode is uploaded to a local server...
The model dropdown shows for a server choosing it shows its address, optional key and model text rows, and saves what is typed
The Server address row is hidden its rows come in order: provider, address, key, model, then the shared rows
No address error an address without http:// or https:// says so under the field
A server's speaker row names the model ID Speaker labels can only be turned off, as for any model without them
Migration drops serverUrl carries an OpenAI-compatible server's address and model to the library that imports it
The same plant keeps a stored provider, model, server address and speaker settings as they are
The offer charges a server the offer names the user's own server, without a charge
The panel names the provider for a server names the user's own server without calling it paid
The panel calls a server paid the same test

Obsidian runtime

Obsidian 1.14.4 on Linux, this worktree's isolated instance, fresh vault. The verify skill's fixture server gains a transcription route, fixtures.mjs server printed http://podnotes.test:34900/v1, which the instance reaches at 127.0.0.1 over the real network (net.mjs logged the POST and did not answer it). The local file is Fixtures/Interview.mp3, 30 s of a 440 Hz tone at 32 kbps, 120,608 bytes, made with ffmpeg.

  1. Settings > PodNotes > Transcripts, Transcription provider chosen with the keyboard. The rows came in the order above. localhost:34900/v1 showed the error under the field. Then http://podnotes.test:34900/v1/ and whisper-large-v3-turbo, typed into the fields. data.json read provider: "openai-compatible", model: "whisper-large-v3-turbo", serverUrl: "http://podnotes.test:34900/v1/", openAICompatibleSecretId: "".
  2. Play with PodNotes from the file menu, then Transcribe current episode from audio from the palette, with no key. transcripts/local file/Interview.md opened with [00:00:00] and [00:00:12] paragraphs. The server recorded one request: {"authorization":null,"model":"whisper-large-v3-turbo","responseFormat":"verbose_json","timestampGranularities":["word","segment"],"fileType":"audio/mp3","fileBytes":120608}. Speaker labels were on, as on a new install, so the notice said the server does not label speakers.
  3. Model json-only, which the fixture refuses verbose_json for with gpt-transcribe's 400: two requests, verbose_json then json with no granularities, and an untimed note.
  4. An empty Model: Enter a model for your OpenAI-compatible server in the transcript settings., no request.
  5. Address http://podnotes.test:1/v1: three POSTs, then Transcription failed: PodNotes could not reach the server at podnotes.test:1. Check the address in the transcript settings.
  6. The player's Transcript panel read Transcribe with your server, and its button transcribed the episode.

dev:errors reported none. The instance and the fixture server are stopped.

Settings with the provider's rows

Server address without a scheme

Timed note from a local MP3

Untimed note after the json fallback

Unreachable server notice

Transcript panel button

Not checked here: real servers. The lead verifies against OpenRouter's and OpenAI's own endpoints with real keys. Pre-existing and not changed: a settings row with an error message narrows its text field, as Download path does.

Real servers after the merge (lead)

Obsidian 1.14.4, palette command, 60 s local clip of E231, a pass-through request log, build of the head after the fix below:

  • OpenAI's API (https://api.openai.com/v1, gpt-transcribe): verbose_json refused with 400, resent once as json, untimed 150-word note.
  • OpenRouter's endpoint given as the full .../audio/transcriptions/ URL (openai/whisper-large-v3-turbo): one verbose_json request, timed note.

The first rerun after the merge saved "only see sci -fi as cultural criticism" and "science -fictional": the merged reader prefers words, and Whisper lists each word with its leading space and splits "sci-fi" into " sci" and "-fi" (captured live). A piece without a leading space now continues the word before it; bare word lists (Grok, Scribe v2, MAI, Qwen) are unchanged. After the fix the real note reads "only see sci-fi as cultural criticism. They try to turn all these science-fictional stories". Plant: skip the join (caught by joins a Whisper word list, whose words carry their own leading space, back into its words). Checks: lint, format, typecheck, size (108,114 of 108,157), 2,933 unit tests, 48 tooling tests.

Release impact

Additive settings, schemaVersion stays 2: transcript.serverUrl (the address as typed) and the root openAICompatibleSecretId (SecretStorage podnotes-openai-compatible-api-key). A settings export carries the address and, when keys are included, secrets.openAICompatible. A device on 2.24.0 keeps both fields on its saves. An unknown provider string is still kept and refused as before. The provider is unreleased, so no released behavior changes. Docs follow after release, as for the other providers.

Note

Add OpenAI-compatible server as a transcription provider

  • Adds a new transcription provider in openaiCompatible.ts that sends multipart uploads to a user-entered server address, with optional bearer auth, verbose-JSON or text-only responses, and a 24,000,000-byte upload limit
  • Settings, persistence, migration, and import/export handle the server address and its own API-key credential; the UI shows Server address, API key, and an editable model text row, and hides the curated model dropdown
  • Local and private-network server addresses are allowed via a new privateHosts option in assertFetchableUrl.ts and networkRequest.ts; the transcript panel labels the choice "your server", drops the Paid tag, and stays enabled offline
  • Moves OpenRouter's verbose-JSON response parsing into a shared verboseJson.ts, including a fix that combines whitespace-marked word fragments into the preceding word
  • Behavioral change: a format failure kind (HTTP 400 mentioning response_format, verbose_json, or timestamp_granularities) ends the job but triggers one retry in text-only mode; raw bundle budget rises to 334,091 bytes and gzip to 109,234 bytes

Macroscope summarized b717d81.


Note

Medium Risk
Touches transcription credentials, user-entered network targets with a deliberate private-host exception, and shared verbose-JSON parsing used by OpenRouter; mitigated by extensive tests and unchanged policy for feed-driven fetches.

Overview
Adds OpenAI-compatible server as a fifth transcription provider so users can send episode audio to any HTTP(S) endpoint that implements OpenAI’s POST …/audio/transcriptions API (local Whisper stacks, Groq, Together, etc.), with optional API key, a free-text model, and transcript.serverUrl persisted on save/import.

Transcription behavior: uploads multipart audio (24 MB parts), requests verbose_json with word/segment granularities when supported, and on a single format refusal per job switches that part and all later parts to json only. Responses are parsed through new shared verboseJson.ts (also used by OpenRouter), including Whisper-style split tokens joined back into words and punctuation restored from text when needed.

Safety and UX: only this provider enables privateHosts on fetch/request so localhost and LAN URLs the user typed are allowed; episode URLs are never sent (url: false). Settings validate address scheme and block credentials in the URL; the Transcript panel labels the choice “your server”, omits Paid, and stays enabled offline.

Supporting changes: new e2e/fixture transcription routes and verify docs, credential/export/migration fields, failure copy that names the server host, and bundle budgets raised to 334,091 raw / 109,234 gzip.

Reviewed by Cursor Bugbot for commit b717d81. Bugbot is set up for automated code reviews on this repo. Configure here.

gpt-4o-transcribe-diarize rejects requests with more than 1400 s of audio,
but MP3 chunks were split by size alone. Any MP3 under about 120 kbps put
more than 1400 s in a 20 MB chunk, so every chunk failed.

For diarization, split MP3 at frame boundaries into the fewest equal parts
that stay under both 20 MB and 1400 s. Whisper keeps its size-only chunks.

Fixes #532
A chunk that OpenAI rejected with a status such as 400 or 401 was sent
three times, although each send fails the same way. A long diarized
episode went up three times before the error showed.

Retry only the statuses openai-node retries: 408, 409, 429 and 5xx, plus
network errors.

Refs #532
gpt-4o-transcribe-diarize takes up to 0.58 s per second of audio, measured
over 81 requests, so the 1,072 s parts a 54 min episode got would take about
10 min each, at the request timeout. Parts are now at most 600 s, which the
model answers in under 6 min.

Refs #532
Each test passes on the hand-written code. Together they catch wrong
derivations that no existing test sees: the -1 SecretStorage ID suffix
that released versions accept, the provider an export error names, the
kind a settings key row adopts a secret for, and the key migration
notices.
CREDENTIALS in src/types/Credentials.ts holds each kind's settings
field, export key and label. The credential repository, settings
decoder, settings transfer, settings tab and key migration notice loop
over it. IPodNotesSettings takes its key fields from it, so a kind with
no default in DEFAULT_SETTINGS fails typecheck. The plaintext key map
from before 2.19.0 stays fixed.
The published transcript renderer is the reader of the format that stays
when the diarization folder goes. The default template now lives only in
DEFAULT_SETTINGS, so src/constants.ts imports no service code.
src/services/transcription/ turns an episode into a transcript with any
provider. A provider is one file: its models, its request, and how to read
its answers. OpenAI (gpt-transcribe) and Deepgram (nova-3) are the first two.

The engine owns what providers share:

- audioSource.ts decides whether a provider may be given the episode's
  address. Only an address that a saved public feed lists, that carries no
  credential, and that names a public host passes. All other audio is
  uploaded.
- split.ts is audioChunker.ts with the caps as arguments and a start time
  on each part.
- http.ts sends a request at most three times, and only after a 429, a 5xx,
  or a lost connection. Its failure sentences never include the provider's
  own text, which can quote a private address.
- cues.ts turns words or text into the shapes of a published transcript.

Nothing calls the engine yet. Lint rejects an import of the folder's files
from outside it, except its index.
TranscriptionService keeps its queue, the published-transcript path, the
progress notice, and note saving. It calls the engine for audio.

- OpenAI transcribes with gpt-transcribe. whisper-1 and
  gpt-4o-transcribe-diarize, which OpenAI shuts down on 2027-02-26, are
  gone. A tooling test searches the built main.js for them.
- With speaker labels on and OpenAI selected, the transcript is saved
  without labels. A notice that stays until clicked names the providers
  that label speakers.
- Deepgram transcripts go through renderTranscriptBody, so each paragraph
  has a timestamp. Speakers are numbered in the order they first speak.
- Deepgram gets the address of a public episode and downloads it itself.
- A failure notice stays until clicked. Parts go out one at a time, and a
  401, 402, 404, or 400 stops the job before the next part.

Deletes src/services/diarization/, openaiTranscription.ts, the Whisper
worker pool, and createBinaryChunkFiles. getEpisodeAudioBuffer no longer
returns a basename, because uploads are named for their format only.
net.mjs answers OpenAI with { text } and Deepgram with the words of two
speakers, and logs the content type of each request. fixtures.mjs serves a
feed at priv3.supercast.com, an address that PodNotes keeps private. The
transcripts recipe covers OpenAI in parts, and Deepgram by address and by
upload.
The marketplace review test called TranscriptionService.waitForRetry, which
the engine replaced. An aborted run now rejects through waitForLifecycle, so
the test aborts a pending wait there and expects the same Error.
An aborted job no longer downloads the episode or decodes it into parts.
Until now only its requests were refused.

A 429 or a 5xx that asks to wait over a minute is sent once. A sooner send
would only upload the audio to be refused again.
A provider's name was written twice: once in its credential row and once in
its provider file. The row now owns it, so a provider file holds only its
models and its request. Each model gains the name the settings will show.
The provider was derived from the speaker settings: Deepgram with speaker
labels on, OpenAI otherwise. It is now stored as transcript.provider, with
transcript.model beside it, where "" follows the provider's recommended model.
schemaVersion stays 2 and both fields are additions, so a device on 2.24.0
keeps loading the library and keeps the fields on its own saves.

A library that has no transcript.provider gets the provider it ran with:
Deepgram for speaker labels with Deepgram, with or without its key, and OpenAI
otherwise. The migration reads only the stored document, adds no decode
warning, and gives the same answer on every load until a save stores it.

transcript.diarization.provider belongs to versions up to 2.24.0. This version
reads it for that migration alone, never writes it, keeps it in data.json as
found, and leaves it out of settings exports. Importing a 2.24.0 export that
labels speakers with Deepgram selects Deepgram.

A provider or model ID this version does not have is kept as stored and shown
under that name. An unknown provider refuses to transcribe with a notice. An
unknown model runs the recommended model and the notice says so.

The Transcripts settings now read: Transcription provider, the chosen
provider's API key, Model, Speaker labels, Speaker label format, then the path
and template. Speaker labels is disabled for a model that cannot label
speakers. The Diarization provider row is gone.
ElevenLabs is a credential row, one provider file and a registry line. Its key
reference is the new root field elevenLabsSecretId, stored in SecretStorage as
podnotes-elevenlabs-api-key, and a settings export carries the key as
secrets.elevenLabs. The export envelope stays at version 2.

A public episode goes to ElevenLabs by its address, so nothing is downloaded.
Any other episode is uploaded in one request. Word times and speakers reach
the note, and turning speaker labels off asks for none.

ElevenLabs answers 401 for a used-up quota as for a rejected key, and 400 for
a model the key may not use as for an address it could not download, so the
provider reads detail.code. Only the last of these is followed by an upload.
A request waits 10 minutes for its answer.

A new install starts on ElevenLabs with speaker labels on. A library saved up
to 2.24.0 that holds no OpenAI key and has speaker labels off moves to
ElevenLabs too, because it never set transcription up. A library that never
stored the speaker labels choice keeps them off.

The notice for a model without speaker labels now names ElevenLabs first.
A journey now seeds its transcription provider with its key, because a new
vault is on ElevenLabs. The OpenAI journeys choose OpenAI. A new journey runs
the default: a public episode goes to ElevenLabs by its address, no audio is
fetched, and the note has timestamps and speaker labels. The ElevenLabs stub
answers with the words ElevenLabs returns and records what each request named.

The settings tab test walks the new Transcripts rows, and the key recovery
test links the ElevenLabs key, the one a new vault shows.
The fixture provider answers api.elevenlabs.io with ElevenLabs' words, two
speakers with the second 12 s in, and logs the address a request named in
place of a file. The transcript guide follows the new settings rows and adds
the ElevenLabs runs and a 2.24.0 library check.
…choice

The provider row decided for itself whether the stored provider is one this
version has, and the Speaker label format row read the stored speaker setting.
Both now ask transcriptionChoice, as the key rows, the model row and the
Speaker labels row do. The engine no longer exports a type nothing imports.
The Transcripts tests looked each control up by a CSS selector inside its row
and asserted on a hidden row's inline style. They now query by role and by the
row's name, as a user or a screen reader finds a control, so a hidden row is
one the query does not find.
…ings measure

The transcription engine left main.js at 103,795 gzip bytes. The provider and
model settings, their migration and the ElevenLabs provider add 866, which
takes it to 104,661 and over the 104,000 budget. The budget rises by those 866
bytes to 104,866, so the 205 bytes of headroom stay as they were.
…m library

2.24.0 imports the diarization object whole and fills a missing provider
with OpenAI, so omitting the field moved Deepgram libraries to OpenAI.
Imports here keep the provider this library stored.
A library migrated with OpenAI speaker labels on could not turn them off,
so every transcription repeated the notice.
… one

The tests for a provider and a model a newer PodNotes stored used OpenRouter
and Grok STT 2.0, which this version is about to run. They now use AssemblyAI
and universal-3-6-pro.
OpenRouter is a credential row, one provider file and a registry line. Its
key reference is the new root field openRouterSecretId, stored in
SecretStorage as podnotes-openrouter-api-key, and a settings export carries
the key as secrets.openRouter.

It offers three models, each with the request measured to give word times
and speakers:

- Grok STT 2.0, recommended: base64 JSON with verbose_json and word and
  segment times, routed to xAI with provider.only, which takes diarize from
  provider.options.xai. A request routed elsewhere is answered with no
  speakers.
- ElevenLabs Scribe v2: multipart with diarize beside the model, or JSON with
  input_audio.url for an episode a saved public feed lists.
- MAI-Transcribe 2: multipart with diarize beside the model.

With speaker labels off, no request carries diarize. Sounds Scribe v2 lists
as audio_event items are left out.

OpenRouter takes 25 MB a request and gives the model 60 s, so an upload goes
in parts of at most 18 MB and 1,800 s. Parts of 24 MB were answered in 20 to
31 s.

A 402 says the account is out of credit, a 401 that the key was rejected,
and a 404 that the model is not offered. A 404 whose message names the
account's data policy or zero data retention says the model is not offered
under the account's OpenRouter privacy settings. Each ends the job at the
first part, and none quotes OpenRouter.
…t each part

A note saved with speaker labels asked for and none returned now says so:
"Saved without speaker labels, because Grok STT 2.0 returned none." OpenRouter
answers 200 with no speakers when a request reaches a route that ignores the
speaker option, so this is checked for every provider.

Labels from two parts already name different speakers. An episode sent in
parts now also says that its labels restart in each part, so one person can
have several labels, until labels are merged across parts.
…ll lists

PodNotes reads OpenRouter's public catalog of transcription models, without
the key, when the Model row draws and when an OpenRouter job starts. It keeps
the read in memory for 24 hours. A read that fails is not kept, so the next
one asks again, and the tested list is offered as it is.

The catalog only takes models away. A model it does not list is not offered,
and one whose expiration_date has come is skipped. A model with a later end
date reads "Grok STT 2.0, ends <date>" in the Model row. A catalog that would
leave no model is not followed.

A chosen model the catalog no longer lists stays selected in the Model row as
"<label>, no longer offered". A job runs the first offered model instead and
says "OpenRouter no longer offers ElevenLabs Scribe v2, so PodNotes used Grok
STT 2.0." The row redraws when a read lands while it is open.

A provider gets its catalog through an optional catalog() beside failure(),
so the registry owns the cache and the filter for any provider that has one.
A stub answers OpenRouter's transcriptions with two speakers, the second 12 s
in, and its model catalog with the three models PodNotes offers. It records
what each request named and a hash of the audio it carried, decoded from
base64 for JSON.

J-TR-09 plays a public episode on Scribe v2: one JSON request names the
episode's address, and the episode's audio is never fetched. J-TR-10 plays it
on the recommended Grok STT 2.0: one JSON request carries the audio as base64,
the same bytes the fixture server holds. Both notes have timestamps and
numbered speakers, and the catalog is read once, with no Authorization.

A journey seeds any provider by its credential row, and a model with it.
…eads them

The fixture provider answers openrouter.ai transcriptions with verbose_json
words from two speakers, the second 12 s in, and the model catalog with the
three models PodNotes offers. The log keeps openrouter.ai paths and reads the
address a JSON request names in input_audio.url.

The transcript guide adds the OpenRouter rows, its runs (Scribe v2 by
address, Grok by base64, a long MP3 in parts) and the notices that now name
OpenRouter.
The shared verbose_json reader adds no punctuation a word already has, as #544
now does.
…and tell contractions apart

Closing punctuation took a Latin set only, so Chinese sentences and trailing
dashes lost their marks; the sets now hold the CJK marks and dashes, and still
none of % or #. Keys dropped every apostrophe, so we'll matched well and could
take its comma; they keep apostrophes inside a word again and drop only the
quoting ones at its ends.
…carries it

An answer that listed a mark as an entry of its own, or a word with a mark
before an emoji, slipped past the check for punctuated answers, and its marks
were added a second time. Any entry without a letter or digit, or a word with
.,?!;: after its last one, now leaves the answer as the model gave it.
The shared verbose_json reader takes #544's latest punctuation rules: CJK marks
and dashes, keys that keep a word's inner apostrophes, and no restoring for an
answer that already carries punctuation.
…de a sentence run

Text without spaces between sentences was one token, so only its last mark was
placed. A CJK closing mark now ends a token and an opening one starts one.
The shared verbose_json reader ends a token at a CJK mark, as #544 now does.
A space went after each closing mark, so the second of 。」 or !! became a token
of its own that no word matched, and was lost. A run of marks now ends a token
together.
The shared verbose_json reader keeps a run of CJK marks on one token, as #544
now does.
…igit

A mark hidden behind an emoji, as Hello…🙂 or 你好。🙂, slipped past the
already-punctuated checks, and the word gained it twice. It also reached Grok,
Scribe v2 and MAI, which share the reader. A word now gains an opening mark
only if it starts with a letter or digit, and closing punctuation only if it
ends with one, so a word with any mark, symbol or emoji at that end keeps it
as given.
The shared verbose_json reader adds marks only where a word ends in a letter
or digit, as #544 now does.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3998cb1. Configure here.

Emoji such as 1️⃣ and ℹ️ start with a digit or letter, so a word like 1️⃣"Hello
passed the edge check and gained a second quote, with Grok, Scribe v2 and MAI
too. Only a word of letters and digits, with apostrophes and hyphens between
them, now gains marks; any other word keeps its text as given.
The shared verbose_json reader adds marks only to plain words, as #544 now does.
Plain words took letters and digits only, so 3.5, $5, 3:30, 5% and 1,000 kept
no comma or period from the text. A word now may also hold a number's .,:/
between digits, a currency sign before and % after, none of which is a mark
it can gain twice.
The shared verbose_json reader gives numbers, amounts, times and percentages
their punctuation, as #544 now does.
…r end

A model that attached a mark to the start of the next word, as Hello ,world,
slipped past the check for punctuated answers, and the plain word before it took
the same mark from the text. Any word with a mark before its first letter or
digit or after its last now leaves the whole answer as given. Apostrophes do
not count, so dogs' still gets its sentence's punctuation. The opening and
closing sets are now written once.
The shared verbose_json reader restores nothing when any word has a mark at
either end, as #544 now does.
Checking each word's edges kept missing shapes that hid a mark from the check,
such as a keycap emoji before a comma or a single quote on the word before, and
a plain neighbor then took the same mark from the text. Marks now come from the
text only when every word is plain: letters and digits, with apostrophes,
hyphens or a number's .,:/ between them. Then no word has a mark of its own,
and each token's marks go to one word, so none can be added twice. An answer
with any other word is kept as the model gave it.
The shared verbose_json reader restores punctuation only when no word has any,
as #544 now does.
…igits

The plain pattern let a comma, period, colon or slash sit between letters, so
world,again counted as plain, kept its comma, and the word before it took the
same comma from the text. Those separators now belong to numbers alone.
The shared verbose_json reader counts .,:/ as part of a plain word only between
digits, as #544 now does.
A word with a combining mark, as a decomposed é or any Hindi word, failed the
plain pattern, so its whole answer got no punctuation from the text. Letters
now include combining marks, in the pattern and the matching keys, and the
Devanagari danda joins the closing marks.
The shared verbose_json reader counts combining marks as part of a plain word,
as #544 now does.
Base automatically changed from feat/transcription-openrouter-models to master November 16, 2026 16:14
@chhoumann

Copy link
Copy Markdown
Owner Author

Retargeted to master after #544 merged (264ab15), and brought up to date (2a1bb11). This branch already contains #544's last head, c69d365, and git diff c69d365a origin/master is empty, so master is recorded as merged with -s ours.

Since #544's review this branch has merged each of #544's fixes in turn. The shared verbose_json reader's punctuated is now byte-for-byte #544's final version, and #544's tests run against this shared reader:

  • Punctuation comes from text only when every word is plain: letters, combining marks and digits with apostrophes or hyphens between them, or a number with .,:/ between digits and an optional currency sign or %. Otherwise the answer is kept as given.
  • Letter-by-letter matching with a 200-token resync window.
  • CJK tokens and runs of marks, and the Devanagari danda.
  • A word keeps its own text and gains only the opening and closing marks of its token.

Checks: lint, format:check, typecheck, build, size, test (2,988), test:tooling (56). Budgets are 333,881 raw and 109,138 gzip: 3,335 raw and 958 gzip bytes over master's 330,546 and 108,180, measured.

Obsidian 1.14.4, from earlier in this stack: a 70 min local MP3 against the fixture OpenAI-compatible server, chosen through Settings. It went as one request with the typed model, verbose_json, word and segment granularities, and no Authorization header, and saved a timed note. No runtime errors.

A server on this machine transcribes a local file with no internet, and the
palette command already allowed it, but the Transcript panel turned the button
off whenever Obsidian was offline. It now stays on for the user's own server.
The hosted providers still turn it off.
@chhoumann

Copy link
Copy Markdown
Owner Author

FAIL
Verified head a5fa68f by an independent verifier (GPT 6 Astra).

  • P2: credential-bearing server URLs fail and expose their password in the notice. Settings accepts http://review:FAKE@127.0.0.1:32993/v1. Transcribing a local MP3 sends nothing and displays Failed to construct 'Request' ... http://review:FAKE@127.0.0.1:32993/v1/audio/transcriptions. Confirmed the userinfo finding. Reject unsupported userinfo during address validation or serialize multipart independently of that URL and sanitize the failure.
  • P2: a silent part removes every timestamp from the episode. A 30,401,140-byte MP3 sends two parts. Returning {text:"",words:[],segments:[]} for the first and timed words for the second saves only Hello, world! Next sentence., without links. The same responses through OpenRouter retain links at 950.047 and 962.047 seconds on both revisions. Confirmed the silent-part finding; preserve an empty timed result instead of {text:""}.
  • Compared master 264ab15b and head in isolated Obsidian 1.14.4 on Linux, with fake keys only. All 40 OpenRouter source cases and 49 runtime replay cases per revision match byte-for-byte, as do request fields and 201 paired upload hashes. All 402 uploads decode within their model caps. A separate 110-answer comparison across all five models also matches in all 550 cases.
  • Exercised Settings with mouse/keyboard, palette and panel transcription for public/private feeds, a local MP3 and a 40-minute M4A, with/without a key. Verified localhost/LAN access, address normalization and queries, empty models, missing keys, word/segment/text answers, punctuation, Whisper pieces, HTTP failures and unreachable servers. No episode URL enters the captured forms. Multipart fallback sends verbose_json,json,json; ordinary 400s do not trigger fallback.
  • The offline-panel fix works. At 300 px, Transcribe with your server fits, has no Paid tag, stays enabled offline and saves a local MP3 transcript. ElevenLabs is disabled offline and enabled online on both revisions.
  • Inherited privacy limitation: both revisions load a public feed's artwork from http://127.0.0.1:32993/artwork.svg, confirmed by server GETs and the rendered image. The PR's blanket artwork-blocking claim is inaccurate, but this is not a new regression. Feed-fetch and hosted-sender boundary probes retain their private-host restrictions.
  • All requested head checks pass through fnm exec --using 22.23.2: npm run lint, npm run format:check, npm run typecheck, npm run build, npm run size, npm run test (2,988 tests), and npm run test:tooling (56 tests). That installed Node entry reports v22.23.4. The bundle exactly meets both budgets, 333,896 raw and 109,144 gzip bytes.
  • Read the PR, review threads and feat(transcription): move transcription onto one engine and current models #538–feat(transcription): offer Voxtral Mini Transcribe 3 and Qwen3 ASR on OpenRouter #544's verdicts/replies. Inspected saved notes, requests, settings and screenshots; dev:errors is clear. Restored the pinned head and identical bundle, changed no tracked files, and stopped the fixture servers and isolated Obsidian. Evidence and repeatable drivers are under /home/christian/.t3/worktrees/PodNotes/b00d406e-248c27d3db030b5d/.obsidian-e2e-artifacts/review546/, starting with verification.md.

… silent parts timed

An address with a user name and password passed the checks, then failed in the
request, and the error showed the whole address, password included. It is now
refused like an address without http:// or https://, under the field and in the
notice, and nothing is sent. A timed answer with no speech came back as empty
text, which made every part of the episode untimed; it now stays timed, so the
other parts keep their links. A timed answer with text and no times still
keeps its text.
@chhoumann

Copy link
Copy Markdown
Owner Author

On the a5fa68f verdict: both defects are fixed in b717d81, as answered on their threads (userinfo refused before any request, silent parts kept timed). test (2,991) and test:tooling (56) pass. Budgets are 334,091 raw and 109,234 gzip, measured. The artwork note is right: a feed's own artwork address can be on a private network on master too. That limit is inherited, and the description's wording overstated it, so I'm correcting the description.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant