Skip to content

chore(tooling): review transcription models weekly - #541

Merged
chhoumann merged 64 commits into
masterfrom
chore/model-review
Nov 16, 2026
Merged

chhoumann merged 64 commits into
masterfrom
chore/model-review

Conversation

@chhoumann

@chhoumann chhoumann commented Nov 14, 2026 •

Copy link
Copy Markdown
Owner

Summary

Goal: no PodNotes user stays on an old transcription model. The curated list in src/services/transcription/providers/*.ts is the source of truth, so a maintainer needs a weekly signal when OpenRouter retires one or ships a newer one. This is tooling only and adds no plugin bytes.

  • scripts/model-review.mjs (npm run review:models) reads the curated IDs from the provider files, fetches the public GET https://openrouter.ai/api/v1/models?output_modalities=transcription (no key), and prints a short Markdown report: curated OpenRouter models missing from the catalog or carrying an expiration_date, catalog models newer than a curated model of the same vendor family (direct providers count, so deepgram/nova-4 is newer than nova-3), and uncurated models listed once as candidates. It exits 1 only when a curated OpenRouter model is gone or past its end date (same ends <= today rule the registry uses). --catalog <file> reads a saved catalog.
  • .github/workflows/model-review.yml runs it Mondays 06:00 UTC and by workflow_dispatch, never on pull requests. It uses the default GITHUB_TOKEN with contents: read and issues: write, creates or edits one open model-review issue with the report, and then fails the run when the script exited 1. It is not in protect-master.json, runs one at a time in the model-review concurrency group so two runs cannot each open an issue, and pins actions to the same SHAs as test.yml.
  • AGENTS.md lists the command in Common Commands.

Base is master, since #540 merged with the curated OpenRouter list.

Verification

Node 22.23.2 via fnm. All pass on this branch: npm run lint, npm run format:check, npm run typecheck, npm run test:tooling (9 files, 54 tests), npm run test (125 files, 2882 tests), npm run build. Not run: docs:build (no docs change) and Obsidian runtime checks (no plugin code change).

Tests:

  • tests/tooling/modelReview.test.ts runs the report on tests/fixtures/transcription/openrouter-catalog.json (trimmed live entries plus a deliberately retired x-ai/grok-stt-2.0 and a deliberately newer deepgram/nova-4) and runs the CLI for exit codes.
  • scripts/model-review.test.ts proves the IDs read from the provider files equal what PROVIDERS in the registry offers, so the reader cannot drift from the code.

Plants (npm run plant, each must type-check, fail the named test, and pass once restored):

Plant Caught by
else if (model.expiration_date) { to else if (model.expiration_date === "never") { (drops the expiration check) model review > reports a curated model past its end date
own.every((curatedModel) => compare(version, curatedModel.version) > 0) to own.every(() => false) (drops the family check) model review > reports a model newer than a curated model of its family

The workflow itself has not run: GitHub runs schedule only from the default branch, so the first real run needs a manual workflow_dispatch after merge to master.

Live run on 2026-11-14 (npm run --silent review:models, exit 0, 31 catalog models, every curated model present with no end date):

# Transcription model review

Checked 2026-11-14 against the OpenRouter catalog (31 models).

## Candidates, not curated

- `moondream/parakeet-ultra` (Moondream: Parakeet Ultra)
- `nvidia/nemotron-3.5-transcribe` (NVIDIA: Nemotron 3.5 Transcribe)
- `mistralai/voxtral-small-transcribe-3-0` (Mistral: Voxtral Small Transcribe 3)
- `mistralai/voxtral-mini-transcribe-3-0` (Mistral: Voxtral Mini Transcribe 3)
- `assemblyai/universal-3-6-pro` (AssemblyAI: Universal-3.6 Pro)
- `elevenlabs/scribe-v2-medical` (ElevenLabs: Scribe v2 Medical)
- `google/gemini-3.5-transcribe` (Google: Gemini 3.5 Transcribe)
- `fish-audio/transcribe-1-pro` (Fish Audio: Transcribe 1 Pro)
- `assemblyai/universal-3-5-pro` (AssemblyAI: Universal-3.5 Pro)
- `meta/muse-voice-transcribe-1.0` (Meta: Muse Voice Transcribe 1.0)
- `nvidia/nemotron-3.5-asr-streaming-multilingual-0.6b` (NVIDIA: Nemotron 3.5 ASR Streaming Multilingual 0.6B)
- `mistralai/voxtral-small-24b-2507-stt` (Mistral: Voxtral Small 24B 2507 STT)
- `mistralai/voxtral-mini-3b-2507` (Mistral: Voxtral Mini 3B 2507)
- `qwen/qwen3-asr-1.7b` (Qwen: Qwen3 ASR 1.7B)
- `qwen/qwen3-asr-0.6b` (Qwen: Qwen3 ASR 0.6B)
- `fish-audio/transcribe-1` (Fish Audio: Transcribe 1)
- `nvidia/parakeet-tdt-0.6b-v3` (NVIDIA: Parakeet TDT 0.6B v3)
- `mistralai/voxtral-mini-transcribe` (Mistral: Voxtral Mini Transcribe)
- `google/chirp-3` (Google: Chirp 3)
- `openai/gpt-4o-mini-transcribe` (OpenAI: GPT-4o Mini Transcribe)
- `openai/whisper-large-v3` (OpenAI: Whisper Large V3)
- `openai/whisper-large-v3-turbo` (OpenAI: Whisper Large V3 Turbo)
- `openai/whisper-1` (OpenAI: Whisper 1)
- `openai/gpt-4o-transcribe` (OpenAI: GPT-4o Transcribe)

Release impact

None. No plugin, settings, storage, API, or URI change. Conventional type chore does not trigger a release.

Note

Add weekly transcription model review workflow and CLI

  • Adds a model-review CLI (model-review.mjs) that reads curated model IDs from transcription provider sources, compares them to the OpenRouter catalog (fetched live or from a saved file), and reports missing curated models, models past their end date, newer models in curated families, and uncurated candidates
  • Adds a weekly Monday 06:00 UTC workflow (model-review.yml) that runs npm run review:models, writes the report to the step summary, and creates or updates a model-review issue when the report has a curated-model problem
  • Covers the CLI and comparison logic with tests, including real-process CLI runs against a catalog fixture
  • Behavioral Change: the workflow exits with failure when a curated model is gone or past its end date; Model review is documented in AGENTS.md as an exception to the PR-workflow concurrency rule

Macroscope summarized eb87b1f.


Note

Low Risk
Tooling-only changes with no plugin runtime impact; the workflow only reads the repo and writes/updates a labeled GitHub issue using the default token.

Overview
Adds weekly maintainer tooling to keep curated OpenRouter transcription models in sync with the live catalog—no plugin or runtime changes.

A new npm run review:models command (scripts/model-review.mjs) scrapes model IDs from src/services/transcription/providers/*.ts, loads the public OpenRouter transcription catalog (or --catalog for offline runs), and emits a Markdown report for gone/missing curated models, upcoming or past expiration dates, newer models in the same vendor family (including cross-mapping between OpenRouter and direct provider IDs), and uncurated catalog candidates. The process exits with code 1 only when a curated OpenRouter model is missing or already past its end date.

.github/workflows/model-review.yml runs that script on Mondays and via workflow_dispatch, appends the report to the job summary, and maintains a single open model-review GitHub issue (update in place, or create only when curated gaps or family upgrades need attention—candidate-only churn does not reopen a closed issue). The job also fails when the script signals a critical curated-model problem. A model-review concurrency group prevents duplicate issues.

Tests cover comparison logic, CLI exit codes, a catalog fixture, and a guard that file-scanned IDs stay aligned with PROVIDERS in the registry. AGENTS.md documents the command and workflow concurrency behavior.

Reviewed by Cursor Bugbot for commit eb87b1f. Bugbot is set up for automated code reviews on this repo. Configure here.

gpt-4o-transcribe-diarize rejects requests with more than 1400 s of audio,
but MP3 chunks were split by size alone. Any MP3 under about 120 kbps put
more than 1400 s in a 20 MB chunk, so every chunk failed.

For diarization, split MP3 at frame boundaries into the fewest equal parts
that stay under both 20 MB and 1400 s. Whisper keeps its size-only chunks.

Fixes #532
A chunk that OpenAI rejected with a status such as 400 or 401 was sent
three times, although each send fails the same way. A long diarized
episode went up three times before the error showed.

Retry only the statuses openai-node retries: 408, 409, 429 and 5xx, plus
network errors.

Refs #532
gpt-4o-transcribe-diarize takes up to 0.58 s per second of audio, measured
over 81 requests, so the 1,072 s parts a 54 min episode got would take about
10 min each, at the request timeout. Parts are now at most 600 s, which the
model answers in under 6 min.

Refs #532
Each test passes on the hand-written code. Together they catch wrong
derivations that no existing test sees: the -1 SecretStorage ID suffix
that released versions accept, the provider an export error names, the
kind a settings key row adopts a secret for, and the key migration
notices.
CREDENTIALS in src/types/Credentials.ts holds each kind's settings
field, export key and label. The credential repository, settings
decoder, settings transfer, settings tab and key migration notice loop
over it. IPodNotesSettings takes its key fields from it, so a kind with
no default in DEFAULT_SETTINGS fails typecheck. The plaintext key map
from before 2.19.0 stays fixed.
The published transcript renderer is the reader of the format that stays
when the diarization folder goes. The default template now lives only in
DEFAULT_SETTINGS, so src/constants.ts imports no service code.
src/services/transcription/ turns an episode into a transcript with any
provider. A provider is one file: its models, its request, and how to read
its answers. OpenAI (gpt-transcribe) and Deepgram (nova-3) are the first two.

The engine owns what providers share:

- audioSource.ts decides whether a provider may be given the episode's
  address. Only an address that a saved public feed lists, that carries no
  credential, and that names a public host passes. All other audio is
  uploaded.
- split.ts is audioChunker.ts with the caps as arguments and a start time
  on each part.
- http.ts sends a request at most three times, and only after a 429, a 5xx,
  or a lost connection. Its failure sentences never include the provider's
  own text, which can quote a private address.
- cues.ts turns words or text into the shapes of a published transcript.

Nothing calls the engine yet. Lint rejects an import of the folder's files
from outside it, except its index.
TranscriptionService keeps its queue, the published-transcript path, the
progress notice, and note saving. It calls the engine for audio.

- OpenAI transcribes with gpt-transcribe. whisper-1 and
  gpt-4o-transcribe-diarize, which OpenAI shuts down on 2027-02-26, are
  gone. A tooling test searches the built main.js for them.
- With speaker labels on and OpenAI selected, the transcript is saved
  without labels. A notice that stays until clicked names the providers
  that label speakers.
- Deepgram transcripts go through renderTranscriptBody, so each paragraph
  has a timestamp. Speakers are numbered in the order they first speak.
- Deepgram gets the address of a public episode and downloads it itself.
- A failure notice stays until clicked. Parts go out one at a time, and a
  401, 402, 404, or 400 stops the job before the next part.

Deletes src/services/diarization/, openaiTranscription.ts, the Whisper
worker pool, and createBinaryChunkFiles. getEpisodeAudioBuffer no longer
returns a basename, because uploads are named for their format only.
net.mjs answers OpenAI with { text } and Deepgram with the words of two
speakers, and logs the content type of each request. fixtures.mjs serves a
feed at priv3.supercast.com, an address that PodNotes keeps private. The
transcripts recipe covers OpenAI in parts, and Deepgram by address and by
upload.
The marketplace review test called TranscriptionService.waitForRetry, which
the engine replaced. An aborted run now rejects through waitForLifecycle, so
the test aborts a pending wait there and expects the same Error.
An aborted job no longer downloads the episode or decodes it into parts.
Until now only its requests were refused.

A 429 or a 5xx that asks to wait over a minute is sent once. A sooner send
would only upload the audio to be refused again.
A provider's name was written twice: once in its credential row and once in
its provider file. The row now owns it, so a provider file holds only its
models and its request. Each model gains the name the settings will show.
The provider was derived from the speaker settings: Deepgram with speaker
labels on, OpenAI otherwise. It is now stored as transcript.provider, with
transcript.model beside it, where "" follows the provider's recommended model.
schemaVersion stays 2 and both fields are additions, so a device on 2.24.0
keeps loading the library and keeps the fields on its own saves.

A library that has no transcript.provider gets the provider it ran with:
Deepgram for speaker labels with Deepgram, with or without its key, and OpenAI
otherwise. The migration reads only the stored document, adds no decode
warning, and gives the same answer on every load until a save stores it.

transcript.diarization.provider belongs to versions up to 2.24.0. This version
reads it for that migration alone, never writes it, keeps it in data.json as
found, and leaves it out of settings exports. Importing a 2.24.0 export that
labels speakers with Deepgram selects Deepgram.

A provider or model ID this version does not have is kept as stored and shown
under that name. An unknown provider refuses to transcribe with a notice. An
unknown model runs the recommended model and the notice says so.

The Transcripts settings now read: Transcription provider, the chosen
provider's API key, Model, Speaker labels, Speaker label format, then the path
and template. Speaker labels is disabled for a model that cannot label
speakers. The Diarization provider row is gone.
ElevenLabs is a credential row, one provider file and a registry line. Its key
reference is the new root field elevenLabsSecretId, stored in SecretStorage as
podnotes-elevenlabs-api-key, and a settings export carries the key as
secrets.elevenLabs. The export envelope stays at version 2.

A public episode goes to ElevenLabs by its address, so nothing is downloaded.
Any other episode is uploaded in one request. Word times and speakers reach
the note, and turning speaker labels off asks for none.

ElevenLabs answers 401 for a used-up quota as for a rejected key, and 400 for
a model the key may not use as for an address it could not download, so the
provider reads detail.code. Only the last of these is followed by an upload.
A request waits 10 minutes for its answer.

A new install starts on ElevenLabs with speaker labels on. A library saved up
to 2.24.0 that holds no OpenAI key and has speaker labels off moves to
ElevenLabs too, because it never set transcription up. A library that never
stored the speaker labels choice keeps them off.

The notice for a model without speaker labels now names ElevenLabs first.
A journey now seeds its transcription provider with its key, because a new
vault is on ElevenLabs. The OpenAI journeys choose OpenAI. A new journey runs
the default: a public episode goes to ElevenLabs by its address, no audio is
fetched, and the note has timestamps and speaker labels. The ElevenLabs stub
answers with the words ElevenLabs returns and records what each request named.

The settings tab test walks the new Transcripts rows, and the key recovery
test links the ElevenLabs key, the one a new vault shows.
The fixture provider answers api.elevenlabs.io with ElevenLabs' words, two
speakers with the second 12 s in, and logs the address a request named in
place of a file. The transcript guide follows the new settings rows and adds
the ElevenLabs runs and a 2.24.0 library check.
…choice

The provider row decided for itself whether the stored provider is one this
version has, and the Speaker label format row read the stored speaker setting.
Both now ask transcriptionChoice, as the key rows, the model row and the
Speaker labels row do. The engine no longer exports a type nothing imports.
The Transcripts tests looked each control up by a CSS selector inside its row
and asserted on a hidden row's inline style. They now query by role and by the
row's name, as a user or a screen reader finds a control, so a hidden row is
one the query does not find.
…ings measure

The transcription engine left main.js at 103,795 gzip bytes. The provider and
model settings, their migration and the ElevenLabs provider add 866, which
takes it to 104,661 and over the 104,000 budget. The budget rises by those 866
bytes to 104,866, so the 205 bytes of headroom stay as they were.
…m library

2.24.0 imports the diarization object whole and fills a missing provider
with OpenAI, so omitting the field moved Deepgram libraries to OpenAI.
Imports here keep the provider this library stored.
A library migrated with OpenAI speaker labels on could not turn them off,
so every transcription repeated the notice.
… one

The tests for a provider and a model a newer PodNotes stored used OpenRouter
and Grok STT 2.0, which this version is about to run. They now use AssemblyAI
and universal-3-6-pro.
OpenRouter is a credential row, one provider file and a registry line. Its
key reference is the new root field openRouterSecretId, stored in
SecretStorage as podnotes-openrouter-api-key, and a settings export carries
the key as secrets.openRouter.

It offers three models, each with the request measured to give word times
and speakers:

- Grok STT 2.0, recommended: base64 JSON with verbose_json and word and
  segment times, routed to xAI with provider.only, which takes diarize from
  provider.options.xai. A request routed elsewhere is answered with no
  speakers.
- ElevenLabs Scribe v2: multipart with diarize beside the model, or JSON with
  input_audio.url for an episode a saved public feed lists.
- MAI-Transcribe 2: multipart with diarize beside the model.

With speaker labels off, no request carries diarize. Sounds Scribe v2 lists
as audio_event items are left out.

OpenRouter takes 25 MB a request and gives the model 60 s, so an upload goes
in parts of at most 18 MB and 1,800 s. Parts of 24 MB were answered in 20 to
31 s.

A 402 says the account is out of credit, a 401 that the key was rejected,
and a 404 that the model is not offered. A 404 whose message names the
account's data policy or zero data retention says the model is not offered
under the account's OpenRouter privacy settings. Each ends the job at the
first part, and none quotes OpenRouter.
…t each part

A note saved with speaker labels asked for and none returned now says so:
"Saved without speaker labels, because Grok STT 2.0 returned none." OpenRouter
answers 200 with no speakers when a request reaches a route that ignores the
speaker option, so this is checked for every provider.

Labels from two parts already name different speakers. An episode sent in
parts now also says that its labels restart in each part, so one person can
have several labels, until labels are merged across parts.
…ll lists

PodNotes reads OpenRouter's public catalog of transcription models, without
the key, when the Model row draws and when an OpenRouter job starts. It keeps
the read in memory for 24 hours. A read that fails is not kept, so the next
one asks again, and the tested list is offered as it is.

The catalog only takes models away. A model it does not list is not offered,
and one whose expiration_date has come is skipped. A model with a later end
date reads "Grok STT 2.0, ends <date>" in the Model row. A catalog that would
leave no model is not followed.

A chosen model the catalog no longer lists stays selected in the Model row as
"<label>, no longer offered". A job runs the first offered model instead and
says "OpenRouter no longer offers ElevenLabs Scribe v2, so PodNotes used Grok
STT 2.0." The row redraws when a read lands while it is open.

A provider gets its catalog through an optional catalog() beside failure(),
so the registry owns the cache and the filter for any provider that has one.
A stub answers OpenRouter's transcriptions with two speakers, the second 12 s
in, and its model catalog with the three models PodNotes offers. It records
what each request named and a hash of the audio it carried, decoded from
base64 for JSON.

J-TR-09 plays a public episode on Scribe v2: one JSON request names the
episode's address, and the episode's audio is never fetched. J-TR-10 plays it
on the recommended Grok STT 2.0: one JSON request carries the audio as base64,
the same bytes the fixture server holds. Both notes have timestamps and
numbered speakers, and the catalog is read once, with no Authorization.

A journey seeds any provider by its credential row, and a model with it.
…eads them

The fixture provider answers openrouter.ai transcriptions with verbose_json
words from two speakers, the second 12 s in, and the model catalog with the
three models PodNotes offers. The log keeps openrouter.ai paths and reads the
address a JSON request names in input_audio.url.

The transcript guide adds the OpenRouter rows, its runs (Scribe v2 by
address, Grok by base64, a long MP3 in parts) and the notices that now name
OpenRouter.
Audio frames inside a tag's declared size were counted and could be cut
through, sending an undecodable first part, and frame-like bytes in a
trailing tag added a part.
…engine

# Conflicts:
#	src/services/TranscriptionService.test.ts
#	src/services/TranscriptionService.ts
#	src/services/audioChunker.test.ts
#	src/services/audioChunker.ts
#	src/services/diarization/openaiProvider.ts
#	src/services/diarization/types.ts
#	src/services/openaiTranscription.test.ts
#	src/services/openaiTranscription.ts
#	tests/mocks/openaiFailures.ts
…h cap

A file in a format other than MP3 under the size cap went whole whatever its
length, so a long .m4a passed a model's maxSeconds. Its length now comes from
its metadata through a media element, without decoding it, and audio over the
cap is decoded into WAV parts that fit both caps. Audio of an unknown length
still goes whole.
A public feed's enclosure with AWS or Google Cloud signing parameters passed the
feed credential check, so Deepgram got the whole signed address. A query
parameter whose name holds signature, credential or token, or Akamai's hdnts,
now uploads the audio instead.
…nload

A WebView may load no metadata for media it does not play, so the length probe
could wait forever, and an unload could not stop it before a decode. The probe
now reads the length as unknown after 5 s or when the job is aborted, and an
aborted job decodes nothing.
…nscription-providers

# Conflicts:
#	AGENTS.md
#	scripts/size-baseline.md
#	scripts/size.mjs
#	scripts/size.test.ts
#	src/persistence/settingsCodec.ts
#	src/settingsTransfer.ts
#	src/types/Credentials.ts
#	src/ui/settings/PodNotesSettingsTab.ts
…nscription-openrouter

# Conflicts:
#	AGENTS.md
#	scripts/size-baseline.md
#	scripts/size.mjs
#	scripts/size.test.ts
…address

An address goes in one request, so Scribe v2 on OpenRouter got a public episode
of any length by address, past the 1,800 s its parts are capped at for
OpenRouter's 60 s limit. An episode the feed says is longer than the model's
cap is now uploaded in parts. One of unknown length still goes by address.
Base automatically changed from feat/transcription-openrouter to master November 16, 2026 09:58
@chhoumann

Copy link
Copy Markdown
Owner Author

Retargeted to master after #540 merged (a152aac), and brought up to date (0f8a7c7). I merged #540's last head, df7e41c, which merged cleanly, then recorded master as merged with -s ours. git diff df7e41c3 origin/master is empty. The PR now differs from master by 7 tooling files only. The bundle is unchanged at 106,562 gzip bytes.

  • Checks: lint, format:check, typecheck, build, size, test (2,911), test:tooling (55).
  • Live run: npm run review:models against OpenRouter's public catalog (30 models) exited 0. Every curated model is listed, and it reported 24 candidates that PodNotes does not curate.

@chhoumann

Copy link
Copy Markdown
Owner Author

FAIL
Verified head 0f8a7c7 by an independent verifier (GPT 6 Astra).

  • P2: overlapping runs create duplicate issues. In model-review.yml lines 41–46, listing and creating are separate operations with no serialization. I ran the exact shell twice against a local gh mock, synchronized both empty list responses, and observed two creates. A subsequent run edited one and left both open. Serialize this workflow's issue upsert to meet the one-open-issue claim.
  • Live npm run review:models returned 30 models, 24 candidates, and exit 0. Every curated OpenRouter model was present.
  • Constructed saved catalogs passed all assertions. Removal, past expiration, expiration today, and an empty catalog exit 1 with reports. Healthy, future-expiring, newer-sibling, and direct-provider overlap cases exit 0. Malformed JSON and an invalid response shape exit 1 without a report. The resolved scribe_v3 regression is fixed.
  • Workflow permissions are limited to contents: read and issues: write, checkout credentials are not persisted, actions match the pinned test-workflow SHAs, and only GITHUB_TOKEN is needed. The report survives exit 1 in the extracted review shell; malformed input stops before issue updates. No code-push permission or command is present.
  • Sequential issue creation/update and suppression of candidate-only creation worked locally. An existing issue receives “Nothing to review.” but stays open; closure is manual. I dispatched no workflow and changed no GitHub issues or labels.
  • All requested checks passed after npm ci: npm run lint, npm run format:check, npm run typecheck, npm run build, npm run size, npm run test (2,911 tests), and npm run test:tooling (55 tests). Bundle size is 326,389 raw bytes and 106,562 gzip bytes, within both budgets.
  • I used fnm exec --using 22.23.2 throughout; this host's selected installation reports Node v22.23.4. Read all seven changed files, the PR description, comments, and review thread. Local and remote heads remain the specified SHA, and the tracked worktree is unchanged.

Two overlapping runs could each find no open model-review issue and each
open one.
@chhoumann

Copy link
Copy Markdown
Owner Author

Fixed in 644c92b. Model review now has concurrency: { group: model-review, cancel-in-progress: false }. Runs go one at a time, so a second run sees the issue the first one opened and edits it. A newer run can replace a queued one, which is harmless here because it writes the same report. AGENTS.md says why this workflow has a concurrency block when the PR-check workflows must not: it is no PR check.

No test was added. GitHub's queueing cannot run locally, and a test that only asserts the block exists would be a tautology. The YAML parses, and lint, format:check and test:tooling (55) pass.

The note that an existing issue gets "Nothing to review." but stays open is intended. The maintainer closes it after acting on the review.

@chhoumann

Copy link
Copy Markdown
Owner Author

PASS+NOTES
Verified head 644c92b by an independent verifier (GPT 6 Astra).

  • The previous duplicate-issue finding is addressed. The workflow-wide model-review group with cancel-in-progress: false permits one running run under GitHub's documented concurrency rules. New arrivals can replace a pending run. This scheduled/manual workflow is not a PR check, and the PR-check workflows remain unchanged, as AGENTS.md requires. GitHub queue execution was not tested locally or dispatched.
  • Live npm run review:models exited 0 with 30 models and 24 candidates. Every curated OpenRouter model was present.
  • Eleven constructed saved catalogs passed report and exit-code assertions. Removed, expired, expiring-today, and empty catalogs exited 1 with reports. Healthy, future-expiring, newer-sibling, direct-provider overlap, and candidate-only catalogs exited 0. Malformed JSON and an invalid response shape exited 1 without reports. elevenlabs/scribe-v3 correctly appeared once as newer than OpenRouter's scribe-v2 despite direct scribe_v3 being curated.
  • Ran the exact review/upsert shell locally with a mock gh. Exit-1 reports reached the summary and retained their status. Sequential runs created once and edited twice; candidates alone created nothing. A healthy report updated the existing issue without closing it. Malformed input stopped before upsert. No GitHub issues or labels were changed.
  • Workflow permissions are limited to contents: read and issues: write; it uses only GITHUB_TOKEN, disables persisted checkout credentials, and pins actions to the same SHAs as test.yml. No code write, push, or release dispatch is present.
  • After npm ci, all requested checks passed: npm run lint, npm run format:check, npm run typecheck, npm run build, npm run size, npm run test (2,911 tests), and npm run test:tooling (55 tests). Bundle size is 326,389 raw bytes and 106,562 gzip bytes, within both budgets.
  • Read all seven changed files, the PR description, every comment, and the review thread. Used fnm exec --using 22.23.2 throughout; that installation reports Node v22.23.4 on this host. Local and remote heads match the verified SHA, and the worktree is clean. No Obsidian runtime check was needed for this tooling-only diff.
  • Nonblocking note: the PR description still says the workflow has no concurrency block and names feat/transcription-openrouter as its base. Update those sentences to describe the final implementation and master base.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 644c92b. Configure here.

Comment thread scripts/model-review.mjs Resolved
The review dropped every catalog model with an end date, so a successor in a
curated family that OpenRouter still offers went unreported. It is now reported
until its end date, as the plugin offers it. A model with an end date is still
no candidate to start using.
@chhoumann

Copy link
Copy Markdown
Owner Author

PASS
Verified head eb87b1f by an independent verifier (GPT 6 Astra).

  • Live npm run review:models exited 0 with 30 catalog models and 24 candidates. Every curated OpenRouter model was present.
  • All 18 constructed saved-catalog cases passed report and exit-code assertions. Removed models, past/today curated end dates, and an empty catalog produced reports and exit 1. Malformed JSON and an invalid response shape exited 1 without a report.
  • Bugbot's expiring-successor finding is fixed. Newer siblings with no end date or a future date appear; those ending today or earlier do not. Older siblings without dates are candidates; dated older siblings are excluded. Direct scribe_v3 does not suppress the OpenRouter scribe-v3 upgrade, and direct-provider models are not falsely reported missing.
  • Ran the exact workflow review/upsert shell locally with a mock gh. Exit-1 reports reached the summary before final failure. Sequential runs created once and edited twice. Candidates alone created nothing; a future-expiring successor created an issue; a healthy report updated the existing issue without closing it. Malformed input stopped before upsert.
  • Workflow permissions are only contents: read and issues: write, with only GITHUB_TOKEN and no persisted checkout credentials or code-push command. Actions match the pinned Test workflow SHAs. The workflow-wide model-review concurrency group serializes running runs without cancelling the active run. GitHub queue execution was not dispatched or tested remotely; no GitHub issues or labels were changed.
  • After npm ci, all requested checks passed: npm run lint, npm run format:check, npm run typecheck, npm run build, npm run size, npm run test (2,911 tests), and npm run test:tooling (56 tests). Bundle size is 326,389 raw bytes and 106,562 gzip bytes, within both budgets.
  • Read all seven changed files, the PR description, both earlier verdicts, and every review thread and author reply. Used fnm exec --using 22.23.2 throughout; this host's selected installation reports Node v22.23.4. Local and remote heads match the verified SHA, and the worktree is clean. No Obsidian runtime check was needed for this tooling-only diff.

@chhoumann
chhoumann merged commit a0c85e6 into master Nov 16, 2026
18 checks passed
@chhoumann
chhoumann deleted the chore/model-review branch November 16, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant