Repository navigation
chore(tooling): review transcription models weekly #541
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
64 commits
Select commit
Hold shift + click to select a range
aca94c4
test(transcription): send OpenAI diarization no part over 1400 s
chhoumann 217b040
fix(transcription): cap OpenAI diarization chunks at 1400 s
chhoumann 61c8c52
test(transcription): send a rejected OpenAI request once
chhoumann 44947b5
fix(transcription): send a rejected OpenAI request once
chhoumann a9a3b07
fix(transcription): keep OpenAI diarization requests inside the timeout
chhoumann 6958e09
test(credentials): pin key handling a credentials table will derive
chhoumann ff81dbd
refactor(credentials): derive API key handling from one table
chhoumann f128d4c
refactor(transcription): keep the speaker label format in the renderer
chhoumann 6887b5a
feat(transcription): add one engine for transcription providers
chhoumann 42b55a3
feat(transcription): transcribe through the engine with current models
chhoumann 2f9cd67
test(verify): answer OpenAI and Deepgram as the engine reads them
chhoumann 989185f
test(e2e): read the abort rejection through waitForLifecycle
chhoumann 7d46678
fix(transcription): stop sooner after an abort or a long Retry-After
chhoumann d34a410
Merge remote-tracking branch 'origin/refactor/credentials-table' into…
chhoumann fcb830e
refactor(transcription): name providers from the credentials table
chhoumann f79a09a
feat(transcription): choose the provider and model in settings
chhoumann e1e9b29
feat(transcription): add ElevenLabs Scribe v3 and make it the default
chhoumann 0b796cf
test(e2e): transcribe through the provider the settings choose
chhoumann b31e544
test(verify): answer ElevenLabs as the engine reads it
chhoumann 8cae654
refactor(settings): read the provider and speaker labels through the …
chhoumann 77201f7
test(settings): find the Transcripts rows by role and name
chhoumann 19a96c5
build(size): raise the gzip budget by the 866 bytes the provider sett…
chhoumann 4c09afa
fix(settings): export Deepgram to 2.24.0 importers only for a Deepgra…
chhoumann 1eef3e6
fix(settings): let speaker labels be turned off for a model without them
chhoumann ccc2439
test(transcription): name a provider no PodNotes plans as the unknown…
chhoumann 4b18cee
feat(transcription): transcribe with OpenRouter
chhoumann cb3b63d
feat(transcription): say when speaker labels are missing or restart a…
chhoumann d9886c9
feat(transcription): offer only the OpenRouter models its catalog sti…
chhoumann bfe28e5
test(e2e): transcribe through OpenRouter's Scribe v2 and Grok STT 2.0
chhoumann 9764051
test(verify): answer OpenRouter and its model catalog as the engine r…
chhoumann a844ca7
build(size): raise the gzip budget by the 1,296 bytes OpenRouter meas…
chhoumann c46f215
fix(transcription): time words in a later part to the millisecond
chhoumann 23a4d86
fix(transcription): ask OpenRouter for word times and upload when it …
chhoumann dd00fa5
chore(tooling): report retired and newer transcription models
chhoumann 3b1d1fa
chore(tooling): review transcription models weekly
chhoumann f4a2f74
docs: document npm run review:models
chhoumann 321cb52
fix(tooling): open a model review issue only when a curated model nee…
chhoumann df5ea09
fix(transcription): cut MP3 parts without a one-frame tail or an unco…
chhoumann 9f091c1
fix(tooling): report a newer OpenRouter model even when a direct prov…
chhoumann 3a14c53
fix(transcription): drop the summed-seconds cap that split off one-fr…
chhoumann 27d8958
fix(transcription): split MP3 parts equally even when the size cap binds
chhoumann da3baf9
fix(transcription): fill MP3 parts up to the size cap before adding one
chhoumann 25470c3
fix(transcription): cut MP3 parts as few as fit, moving cuts either way
chhoumann 520f2dd
fix(transcription): sum each MP3 frame's duration when cutting parts
chhoumann aea5d1a
fix(transcription): skip ID3v2 tags before, between and after MP3 frames
chhoumann 7370a71
Merge remote-tracking branch 'origin/master' into feat/transcription-…
chhoumann d544390
fix(transcription): split long audio in other formats under the lengt…
chhoumann cc142ba
build(size): raise the gzip budget by the 118 bytes the length check …
chhoumann a8c8417
fix(transcription): upload episodes whose address is a signed URL
chhoumann 7b64684
build(size): set the gzip budget to the 104,235 bytes the engine fixe…
chhoumann 1f0b4c2
fix(transcription): upload episodes signed with an Akamai hdnea token
chhoumann 37fdc08
build(size): set the gzip budget to the 104,238 bytes the engine fixe…
chhoumann 79addc3
fix(transcription): stop waiting for audio metadata after 5 s or on u…
chhoumann 3641fb8
build(size): set the gzip budget to the 104,290 bytes the engine fixe…
chhoumann 73b5460
Merge commit '3641fb8b5aae2158b4c7250bdba5de92c6ac3fa5' into feat/tra…
chhoumann c6adf83
Merge remote-tracking branch 'origin/master' into feat/transcription-…
chhoumann 53546f4
Merge commit 'c6adf836d8f442f32e31f366b09cd0e674e39718' into feat/tra…
chhoumann f60f448
Merge remote-tracking branch 'origin/master' into feat/transcription-…
chhoumann 34d90f9
fix(transcription): upload an episode longer than the model takes by …
chhoumann df7e41c
build(size): set the gzip budget to the 106,562 bytes OpenRouter meas…
chhoumann 691f8a9
Merge commit 'df7e41c3bf760072c67851db05d575ce3327c3ae' into chore/mo…
chhoumann 0f8a7c7
Merge remote-tracking branch 'origin/master' into chore/model-review
chhoumann 644c92b
fix(tooling): run one model review at a time
chhoumann eb87b1f
fix(tooling): report a newer model whose end date is still to come
chhoumann File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| name: Model review | ||
| on: | ||
| workflow_dispatch: | ||
| schedule: | ||
| - cron: "0 6 * * 1" | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| # One run at a time, so two runs cannot both find no open issue and each open one. A newer | ||
| # run replaces a queued one, which is harmless here: it writes the same report. | ||
| concurrency: | ||
| group: model-review | ||
| cancel-in-progress: false | ||
|
|
||
| jobs: | ||
| review: | ||
| name: Review models | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| contents: read | ||
| issues: write | ||
| env: | ||
| GH_TOKEN: ${{ github.token }} | ||
| steps: | ||
| - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 | ||
| with: | ||
| node-version-file: ".nvmrc" | ||
|
|
||
| - name: Review models | ||
| id: review | ||
| run: | | ||
| status=0 | ||
| node scripts/model-review.mjs > "$RUNNER_TEMP/report.md" || status=$? | ||
| # A catalog that could not be read leaves no report, and fails the job here. | ||
| test -s "$RUNNER_TEMP/report.md" | ||
| cat "$RUNNER_TEMP/report.md" >> "$GITHUB_STEP_SUMMARY" | ||
| echo "status=$status" >> "$GITHUB_OUTPUT" | ||
|
|
||
| - name: Update the review issue | ||
| run: | | ||
| gh label create model-review --description "Weekly transcription model review" --force | ||
| number=$(gh issue list --label model-review --state open --limit 1 --json number --jq '.[0].number // empty') | ||
| if [ -n "$number" ]; then | ||
| gh issue edit "$number" --body-file "$RUNNER_TEMP/report.md" | ||
| elif grep -qE '^## (Curated models gone or ending|Newer models in a curated family)$' "$RUNNER_TEMP/report.md"; then | ||
| # Uncurated candidates alone do not reopen a review the maintainer closed. | ||
| gh issue create --title "Review transcription models" --label model-review --body-file "$RUNNER_TEMP/report.md" | ||
| fi | ||
| env: | ||
| GH_REPO: ${{ github.repository }} | ||
|
|
||
| - name: Fail when a curated model is gone or past its end date | ||
| if: steps.review.outputs.status != '0' | ||
| run: exit 1 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,194 @@ | ||
| // Reports whether the transcription models PodNotes curates are still current. The curated | ||
| // IDs are read from the provider files, and the models OpenRouter's public catalog lists | ||
| // are compared with them. Exits 1 when a curated OpenRouter model is gone or past its end date. | ||
| // | ||
| // npm run review:models [-- --catalog <saved catalog.json>] | ||
| import { readdirSync, readFileSync } from "node:fs"; | ||
| import { fileURLToPath, pathToFileURL } from "node:url"; | ||
|
|
||
| const PROVIDERS = fileURLToPath( | ||
| new URL("../src/services/transcription/providers/", import.meta.url), | ||
| ); | ||
| const CATALOG_URL = "https://openrouter.ai/api/v1/models?output_modalities=transcription"; | ||
|
|
||
| /** | ||
| * @typedef {{ provider: string, id: string }} Curated | ||
| * @typedef {{ id: string, name?: string, created?: number, expiration_date?: string | null }} CatalogModel | ||
| */ | ||
|
|
||
| /** | ||
| * Every model ID in the provider files, with the provider whose file lists it. | ||
| * @param {string} [directory] | ||
| * @returns {Curated[]} | ||
| */ | ||
| export function curatedModels(directory = PROVIDERS) { | ||
| return readdirSync(directory) | ||
| .filter((file) => file.endsWith(".ts") && !file.endsWith(".test.ts")) | ||
| .sort() | ||
| .flatMap((file) => { | ||
| const source = readFileSync(new URL(file, pathToFileURL(`${directory}/`)), "utf8"); | ||
| return [...source.matchAll(/^\s+id: "([^"]+)",$/gm)].map(([, id]) => ({ | ||
| provider: file.slice(0, -".ts".length), | ||
| id, | ||
| })); | ||
| }); | ||
| } | ||
|
|
||
| /** | ||
| * A model's family is its vendor and the words of its name that are not a version, so | ||
| * `elevenlabs/scribe-v2` and the direct `scribe_v3` belong to one family. | ||
| * @param {string} provider the vendor of an ID that names none | ||
| * @param {string} id | ||
| */ | ||
| function parse(provider, id) { | ||
| const slash = id.indexOf("/"); | ||
| const vendor = slash < 0 ? provider : id.slice(0, slash); | ||
| const words = id | ||
| .slice(slash + 1) | ||
| .toLowerCase() | ||
| .split(/[-_]/); | ||
| const isVersion = (/** @type {string} */ word) => /^v?\d+(\.\d+)*$/.test(word); | ||
| return { | ||
| family: [vendor, ...words.filter((word) => !isVersion(word))].join("-"), | ||
| version: words | ||
| .filter(isVersion) | ||
| .flatMap((word) => word.replace(/^v/, "").split(".").map(Number)), | ||
| }; | ||
| } | ||
|
|
||
| /** | ||
| * @param {number[]} a | ||
| * @param {number[]} b | ||
| */ | ||
| function compare(a, b) { | ||
| for (let i = 0; i < Math.max(a.length, b.length); i++) { | ||
| const difference = (a[i] ?? 0) - (b[i] ?? 0); | ||
| if (difference !== 0) return difference; | ||
| } | ||
| return 0; | ||
| } | ||
|
|
||
| /** | ||
| * @param {{ curated: Curated[], catalog: CatalogModel[], today: string }} input | ||
| * @returns {{ | ||
| * gone: string[], | ||
| * ending: { id: string, ends: string, past: boolean }[], | ||
| * newer: { model: CatalogModel, than: string }[], | ||
| * candidates: CatalogModel[], | ||
| * }} | ||
| */ | ||
| export function review({ curated, catalog, today }) { | ||
| const listed = new Map(catalog.map((model) => [model.id, model])); | ||
| const gone = []; | ||
| const ending = []; | ||
| for (const { provider, id } of curated) { | ||
| if (provider !== "openrouter") continue; | ||
| const model = listed.get(id); | ||
| if (!model) gone.push(id); | ||
| else if (model.expiration_date) { | ||
| const ends = model.expiration_date.slice(0, 10); | ||
| ending.push({ id, ends, past: ends <= today }); | ||
| } | ||
| } | ||
|
|
||
| /** @type {Map<string, { provider: string, id: string, version: number[] }[]>} */ | ||
| const families = new Map(); | ||
| for (const { provider, id } of curated) { | ||
| const { family, version } = parse(provider, id); | ||
| families.set(family, [...(families.get(family) ?? []), { provider, id, version }]); | ||
| } | ||
|
|
||
| /** | ||
| * The newest curated model a catalog model is measured against: the OpenRouter one when | ||
| * the family has any, since a direct provider's version says nothing about it. | ||
| * @param {{ provider: string, id: string, version: number[] }[]} own | ||
| */ | ||
| const latest = (own) => { | ||
| const viaOpenRouter = own.filter(({ provider }) => provider === "openrouter"); | ||
| return (viaOpenRouter.length > 0 ? viaOpenRouter : own).reduce((a, b) => | ||
| compare(b.version, a.version) > 0 ? b : a, | ||
| ); | ||
| }; | ||
|
|
||
| const newer = []; | ||
| const candidates = []; | ||
| // A model is offered until its end date, as the plugin's registry has it, so a newer one | ||
| // is reported while offered. A model with an end date is no candidate to start using. | ||
| const offered = catalog | ||
| .filter((model) => !model.expiration_date || model.expiration_date.slice(0, 10) > today) | ||
| .sort((a, b) => (b.created ?? 0) - (a.created ?? 0)); | ||
| for (const model of offered) { | ||
| const { family, version } = parse("", model.id); | ||
| const own = families.get(family); | ||
| if (own && compare(version, latest(own).version) > 0) { | ||
| newer.push({ model, than: latest(own).id }); | ||
| } else if ( | ||
| !model.expiration_date && | ||
| !own?.some((curatedModel) => compare(version, curatedModel.version) === 0) | ||
| ) { | ||
| candidates.push(model); | ||
| } | ||
| } | ||
| return { gone, ending, newer, candidates }; | ||
| } | ||
|
|
||
| /** | ||
| * @param {ReturnType<typeof review>} result | ||
| * @param {{ catalogSize: number, today: string }} context | ||
| */ | ||
| export function renderReport({ gone, ending, newer, candidates }, { catalogSize, today }) { | ||
| /** @type {string[]} */ | ||
| const lines = [ | ||
| "# Transcription model review", | ||
| "", | ||
| `Checked ${today} against the OpenRouter catalog (${catalogSize} models).`, | ||
| ]; | ||
| /** | ||
| * @param {string} heading | ||
| * @param {string[]} items | ||
| */ | ||
| const section = (heading, items) => { | ||
| if (items.length > 0) lines.push("", `## ${heading}`, "", ...items.map((i) => `- ${i}`)); | ||
| }; | ||
| section("Curated models gone or ending", [ | ||
| ...gone.map((id) => `\`${id}\` is not in the catalog.`), | ||
| ...ending.map(({ id, ends, past }) => `\`${id}\` ${past ? "ended" : "ends"} ${ends}.`), | ||
| ]); | ||
| section( | ||
| "Newer models in a curated family", | ||
| newer.map(({ model, than }) => `\`${model.id}\` is newer than \`${than}\`.`), | ||
| ); | ||
| section( | ||
| "Candidates, not curated", | ||
| candidates.map(({ id, name }) => `\`${id}\`${name ? ` (${name})` : ""}`), | ||
| ); | ||
| if (gone.length + ending.length + newer.length + candidates.length === 0) { | ||
| lines.push("", "Nothing to review."); | ||
| } | ||
| return `${lines.join("\n")}\n`; | ||
| } | ||
|
|
||
| /** @param {string[]} args */ | ||
| async function main(args) { | ||
| const file = args[args.indexOf("--catalog") + 1]; | ||
| const body = | ||
| args.includes("--catalog") && file | ||
| ? JSON.parse(readFileSync(file, "utf8")) | ||
| : await fetch(CATALOG_URL, { signal: AbortSignal.timeout(30_000) }).then((response) => { | ||
| if (!response.ok) throw new Error(`The catalog answered ${response.status}.`); | ||
| return response.json(); | ||
| }); | ||
| /** @type {CatalogModel[]} */ | ||
| const catalog = body.data; | ||
| const today = new Date().toISOString().slice(0, 10); | ||
| const result = review({ curated: curatedModels(), catalog, today }); | ||
| process.stdout.write(renderReport(result, { catalogSize: catalog.length, today })); | ||
| if (result.gone.length > 0 || result.ending.some(({ past }) => past)) process.exitCode = 1; | ||
| } | ||
|
|
||
| if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { | ||
| main(process.argv.slice(2)).catch((error) => { | ||
| console.error(`Model review failed: ${error instanceof Error ? error.message : error}`); | ||
| process.exitCode = 1; | ||
| }); | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| import { expect, it } from "vitest"; | ||
| import { PROVIDERS } from "src/services/transcription/registry"; | ||
| import { curatedModels } from "./model-review.mjs"; | ||
|
|
||
| it("reads the model IDs the providers offer", () => { | ||
| const offered = Object.entries(PROVIDERS).flatMap(([provider, { models }]) => | ||
| models.map(({ id }) => ({ provider, id })), | ||
| ); | ||
| const byProvider = (a: { provider: string }, b: { provider: string }) => | ||
| a.provider.localeCompare(b.provider); | ||
| expect(curatedModels().sort(byProvider)).toEqual(offered.sort(byProvider)); | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| { | ||
| "data": [ | ||
| { | ||
| "id": "deepgram/nova-4", | ||
| "name": "Deepgram: Nova 4", | ||
| "created": 1794500000, | ||
| "expiration_date": null | ||
| }, | ||
| { | ||
| "id": "x-ai/grok-stt-2.0", | ||
| "name": "SpaceXAI: Grok STT 2.0", | ||
| "created": 1794329167, | ||
| "expiration_date": "2026-11-01" | ||
| }, | ||
| { | ||
| "id": "elevenlabs/scribe-v2", | ||
| "name": "ElevenLabs: Scribe v2", | ||
| "created": 1791386190, | ||
| "expiration_date": null | ||
| }, | ||
| { | ||
| "id": "elevenlabs/scribe-v2-medical", | ||
| "name": "ElevenLabs: Scribe v2 Medical", | ||
| "created": 1791385619, | ||
| "expiration_date": null | ||
| }, | ||
| { | ||
| "id": "google/chirp-3", | ||
| "name": "Google: Chirp 3", | ||
| "created": 1777997783, | ||
| "expiration_date": null | ||
| }, | ||
| { | ||
| "id": "openai/gpt-transcribe", | ||
| "name": "OpenAI: GPT Transcribe", | ||
| "created": 1785973897, | ||
| "expiration_date": null | ||
| }, | ||
| { | ||
| "id": "x-ai/grok-stt-1.0", | ||
| "name": "SpaceXAI: Grok STT 1.0", | ||
| "created": 1784817014, | ||
| "expiration_date": "2026-11-15" | ||
| }, | ||
| { | ||
| "id": "deepgram/nova-3", | ||
| "name": "Deepgram: Nova 3", | ||
| "created": 1784075761, | ||
| "expiration_date": null | ||
| } | ||
| ] | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.