Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
aca94c4
test(transcription): send OpenAI diarization no part over 1400 s
chhoumann Nov 13, 2026
217b040
fix(transcription): cap OpenAI diarization chunks at 1400 s
chhoumann Nov 13, 2026
61c8c52
test(transcription): send a rejected OpenAI request once
chhoumann Nov 13, 2026
44947b5
fix(transcription): send a rejected OpenAI request once
chhoumann Nov 13, 2026
a9a3b07
fix(transcription): keep OpenAI diarization requests inside the timeout
chhoumann Nov 13, 2026
6958e09
test(credentials): pin key handling a credentials table will derive
chhoumann Nov 13, 2026
ff81dbd
refactor(credentials): derive API key handling from one table
chhoumann Nov 13, 2026
f128d4c
refactor(transcription): keep the speaker label format in the renderer
chhoumann Nov 13, 2026
6887b5a
feat(transcription): add one engine for transcription providers
chhoumann Nov 13, 2026
42b55a3
feat(transcription): transcribe through the engine with current models
chhoumann Nov 13, 2026
2f9cd67
test(verify): answer OpenAI and Deepgram as the engine reads them
chhoumann Nov 13, 2026
989185f
test(e2e): read the abort rejection through waitForLifecycle
chhoumann Nov 13, 2026
7d46678
fix(transcription): stop sooner after an abort or a long Retry-After
chhoumann Nov 13, 2026
d34a410
Merge remote-tracking branch 'origin/refactor/credentials-table' into…
chhoumann Nov 13, 2026
fcb830e
refactor(transcription): name providers from the credentials table
chhoumann Nov 13, 2026
f79a09a
feat(transcription): choose the provider and model in settings
chhoumann Nov 13, 2026
e1e9b29
feat(transcription): add ElevenLabs Scribe v3 and make it the default
chhoumann Nov 13, 2026
0b796cf
test(e2e): transcribe through the provider the settings choose
chhoumann Nov 13, 2026
b31e544
test(verify): answer ElevenLabs as the engine reads it
chhoumann Nov 13, 2026
8cae654
refactor(settings): read the provider and speaker labels through the …
chhoumann Nov 13, 2026
77201f7
test(settings): find the Transcripts rows by role and name
chhoumann Nov 13, 2026
19a96c5
build(size): raise the gzip budget by the 866 bytes the provider sett…
chhoumann Nov 13, 2026
4c09afa
fix(settings): export Deepgram to 2.24.0 importers only for a Deepgra…
chhoumann Nov 14, 2026
1eef3e6
fix(settings): let speaker labels be turned off for a model without them
chhoumann Nov 14, 2026
ccc2439
test(transcription): name a provider no PodNotes plans as the unknown…
chhoumann Nov 14, 2026
4b18cee
feat(transcription): transcribe with OpenRouter
chhoumann Nov 14, 2026
cb3b63d
feat(transcription): say when speaker labels are missing or restart a…
chhoumann Nov 14, 2026
d9886c9
feat(transcription): offer only the OpenRouter models its catalog sti…
chhoumann Nov 14, 2026
bfe28e5
test(e2e): transcribe through OpenRouter's Scribe v2 and Grok STT 2.0
chhoumann Nov 14, 2026
9764051
test(verify): answer OpenRouter and its model catalog as the engine r…
chhoumann Nov 14, 2026
a844ca7
build(size): raise the gzip budget by the 1,296 bytes OpenRouter meas…
chhoumann Nov 14, 2026
c46f215
fix(transcription): time words in a later part to the millisecond
chhoumann Nov 14, 2026
23a4d86
fix(transcription): ask OpenRouter for word times and upload when it …
chhoumann Nov 14, 2026
dd00fa5
chore(tooling): report retired and newer transcription models
chhoumann Nov 14, 2026
3b1d1fa
chore(tooling): review transcription models weekly
chhoumann Nov 14, 2026
f4a2f74
docs: document npm run review:models
chhoumann Nov 14, 2026
321cb52
fix(tooling): open a model review issue only when a curated model nee…
chhoumann Nov 14, 2026
df5ea09
fix(transcription): cut MP3 parts without a one-frame tail or an unco…
chhoumann Nov 14, 2026
9f091c1
fix(tooling): report a newer OpenRouter model even when a direct prov…
chhoumann Nov 14, 2026
3a14c53
fix(transcription): drop the summed-seconds cap that split off one-fr…
chhoumann Nov 14, 2026
27d8958
fix(transcription): split MP3 parts equally even when the size cap binds
chhoumann Nov 14, 2026
da3baf9
fix(transcription): fill MP3 parts up to the size cap before adding one
chhoumann Nov 14, 2026
25470c3
fix(transcription): cut MP3 parts as few as fit, moving cuts either way
chhoumann Nov 14, 2026
520f2dd
fix(transcription): sum each MP3 frame's duration when cutting parts
chhoumann Nov 14, 2026
aea5d1a
fix(transcription): skip ID3v2 tags before, between and after MP3 frames
chhoumann Nov 14, 2026
7370a71
Merge remote-tracking branch 'origin/master' into feat/transcription-…
chhoumann Nov 14, 2026
d544390
fix(transcription): split long audio in other formats under the lengt…
chhoumann Nov 14, 2026
cc142ba
build(size): raise the gzip budget by the 118 bytes the length check …
chhoumann Nov 14, 2026
a8c8417
fix(transcription): upload episodes whose address is a signed URL
chhoumann Nov 14, 2026
7b64684
build(size): set the gzip budget to the 104,235 bytes the engine fixe…
chhoumann Nov 14, 2026
1f0b4c2
fix(transcription): upload episodes signed with an Akamai hdnea token
chhoumann Nov 14, 2026
37fdc08
build(size): set the gzip budget to the 104,238 bytes the engine fixe…
chhoumann Nov 14, 2026
79addc3
fix(transcription): stop waiting for audio metadata after 5 s or on u…
chhoumann Nov 14, 2026
3641fb8
build(size): set the gzip budget to the 104,290 bytes the engine fixe…
chhoumann Nov 14, 2026
73b5460
Merge commit '3641fb8b5aae2158b4c7250bdba5de92c6ac3fa5' into feat/tra…
chhoumann Nov 16, 2026
c6adf83
Merge remote-tracking branch 'origin/master' into feat/transcription-…
chhoumann Nov 16, 2026
53546f4
Merge commit 'c6adf836d8f442f32e31f366b09cd0e674e39718' into feat/tra…
chhoumann Nov 16, 2026
f60f448
Merge remote-tracking branch 'origin/master' into feat/transcription-…
chhoumann Nov 16, 2026
34d90f9
fix(transcription): upload an episode longer than the model takes by …
chhoumann Nov 16, 2026
df7e41c
build(size): set the gzip budget to the 106,562 bytes OpenRouter meas…
chhoumann Nov 16, 2026
691f8a9
Merge commit 'df7e41c3bf760072c67851db05d575ce3327c3ae' into chore/mo…
chhoumann Nov 16, 2026
0f8a7c7
Merge remote-tracking branch 'origin/master' into chore/model-review
chhoumann Nov 16, 2026
644c92b
fix(tooling): run one model review at a time
chhoumann Nov 16, 2026
eb87b1f
fix(tooling): report a newer model whose end date is still to come
chhoumann Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .github/workflows/model-review.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: Model review
on:
workflow_dispatch:
schedule:
- cron: "0 6 * * 1"

permissions:
contents: read

# One run at a time, so two runs cannot both find no open issue and each open one. A newer
# run replaces a queued one, which is harmless here: it writes the same report.
concurrency:
group: model-review
cancel-in-progress: false

jobs:
review:
name: Review models
runs-on: ubuntu-latest
permissions:
contents: read
issues: write
env:
GH_TOKEN: ${{ github.token }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
with:
node-version-file: ".nvmrc"

- name: Review models
id: review
run: |
status=0
node scripts/model-review.mjs > "$RUNNER_TEMP/report.md" || status=$?
# A catalog that could not be read leaves no report, and fails the job here.
test -s "$RUNNER_TEMP/report.md"
cat "$RUNNER_TEMP/report.md" >> "$GITHUB_STEP_SUMMARY"
echo "status=$status" >> "$GITHUB_OUTPUT"

- name: Update the review issue
run: |
gh label create model-review --description "Weekly transcription model review" --force
number=$(gh issue list --label model-review --state open --limit 1 --json number --jq '.[0].number // empty')
if [ -n "$number" ]; then
gh issue edit "$number" --body-file "$RUNNER_TEMP/report.md"
elif grep -qE '^## (Curated models gone or ending|Newer models in a curated family)$' "$RUNNER_TEMP/report.md"; then
# Uncurated candidates alone do not reopen a review the maintainer closed.
gh issue create --title "Review transcription models" --label model-review --body-file "$RUNNER_TEMP/report.md"
fi
env:
GH_REPO: ${{ github.repository }}

- name: Fail when a curated model is gone or past its end date
if: steps.review.outputs.status != '0'
run: exit 1
4 changes: 3 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ root symlinks for local Obsidian loading.
- `npm run test:tooling`: run the real-process Svelte checker failure tests and the build's syntax floor test, including rejection of tagged templates that disable Obsidian's eval cache. CI runs these separately on every PR.
- `npm run build`: type-check and produce the production plugin bundle.
- `npm run plant -- <test name> <file> <search> <replace> [<file> <search> <replace>...] [-- <command...>]`: plant a defect and pass only when the exactly named test fails on it and the command passes again once restored. Each search must match exactly once, and the plant must type-check. The files are restored on exit or interrupt. The command defaults to the unit suite filtered to the test name.
- `npm run review:models`: compare the curated transcription models in `src/services/transcription/providers/` with OpenRouter's public catalog, report models that are gone, ending, newer in a curated family, or not curated, and exit 1 when a curated OpenRouter model is gone or past its end date. The `Model review` workflow runs it weekly and keeps one `model-review` issue current. It is not a required check.
- `npm run docs:build`: build the Starlight documentation and check its internal links.

`@typescript/native` aliases TypeScript 7. `typecheck` and `lint:tsconfig` call
Expand Down Expand Up @@ -402,7 +403,8 @@ The `Test`, `Documentation`, `PR Title`, `Dependency Review`, and `Platforms`
workflows have no `concurrency` block. Do not add one: a group cancels a
pending run when a newer run joins it, even with `cancel-in-progress: false`,
and a cancelled run on a PR head turns its check rollup to `FAILURE`. Duplicate
runs on one head, such as after a reopen, must all finish.
runs on one head, such as after a reopen, must all finish. `Model review` is no
PR check and has one, so two runs cannot each open a `model-review` issue.

Recovery is for the owner. Agents never dispatch release workflows. The release
stage accepts only the refs below, re-derives and re-verifies the release commit
Expand Down
1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
"size:data": "node scripts/persistence-size.mjs",
"count:writes": "node scripts/write-counts.mjs",
"plant": "node scripts/plant.mjs",
"review:models": "node scripts/model-review.mjs",
"typecheck": "node node_modules/@typescript/native/bin/tsc --project tsconfig.json --noEmit && node node_modules/@typescript/native/bin/tsc --project tsconfig.test.json --noEmit",
"lint": "oxlint --deny-warnings src tests/e2e tests/tooling scripts .claude/skills/verify && svelte-policy",
"lint:tsconfig": "node node_modules/@typescript/native/bin/tsc --project tsconfig.json --noEmit --noCheck && node node_modules/@typescript/native/bin/tsc --project tsconfig.test.json --noEmit --noCheck",
Expand Down
194 changes: 194 additions & 0 deletions scripts/model-review.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,194 @@
// Reports whether the transcription models PodNotes curates are still current. The curated
// IDs are read from the provider files, and the models OpenRouter's public catalog lists
// are compared with them. Exits 1 when a curated OpenRouter model is gone or past its end date.
//
// npm run review:models [-- --catalog <saved catalog.json>]
import { readdirSync, readFileSync } from "node:fs";
import { fileURLToPath, pathToFileURL } from "node:url";

const PROVIDERS = fileURLToPath(
new URL("../src/services/transcription/providers/", import.meta.url),
);
const CATALOG_URL = "https://openrouter.ai/api/v1/models?output_modalities=transcription";

/**
* @typedef {{ provider: string, id: string }} Curated
* @typedef {{ id: string, name?: string, created?: number, expiration_date?: string | null }} CatalogModel
*/

/**
* Every model ID in the provider files, with the provider whose file lists it.
* @param {string} [directory]
* @returns {Curated[]}
*/
export function curatedModels(directory = PROVIDERS) {
return readdirSync(directory)
.filter((file) => file.endsWith(".ts") && !file.endsWith(".test.ts"))
.sort()
.flatMap((file) => {
const source = readFileSync(new URL(file, pathToFileURL(`${directory}/`)), "utf8");
return [...source.matchAll(/^\s+id: "([^"]+)",$/gm)].map(([, id]) => ({
provider: file.slice(0, -".ts".length),
id,
}));
});
}

/**
* A model's family is its vendor and the words of its name that are not a version, so
* `elevenlabs/scribe-v2` and the direct `scribe_v3` belong to one family.
* @param {string} provider the vendor of an ID that names none
* @param {string} id
*/
function parse(provider, id) {
const slash = id.indexOf("/");
const vendor = slash < 0 ? provider : id.slice(0, slash);
const words = id
.slice(slash + 1)
.toLowerCase()
.split(/[-_]/);
const isVersion = (/** @type {string} */ word) => /^v?\d+(\.\d+)*$/.test(word);
return {
family: [vendor, ...words.filter((word) => !isVersion(word))].join("-"),
version: words
.filter(isVersion)
.flatMap((word) => word.replace(/^v/, "").split(".").map(Number)),
};
}

/**
* @param {number[]} a
* @param {number[]} b
*/
function compare(a, b) {
for (let i = 0; i < Math.max(a.length, b.length); i++) {
const difference = (a[i] ?? 0) - (b[i] ?? 0);
if (difference !== 0) return difference;
}
return 0;
}

/**
* @param {{ curated: Curated[], catalog: CatalogModel[], today: string }} input
* @returns {{
* gone: string[],
* ending: { id: string, ends: string, past: boolean }[],
* newer: { model: CatalogModel, than: string }[],
* candidates: CatalogModel[],
* }}
*/
export function review({ curated, catalog, today }) {
const listed = new Map(catalog.map((model) => [model.id, model]));
const gone = [];
const ending = [];
for (const { provider, id } of curated) {
if (provider !== "openrouter") continue;
const model = listed.get(id);
if (!model) gone.push(id);
else if (model.expiration_date) {
const ends = model.expiration_date.slice(0, 10);
ending.push({ id, ends, past: ends <= today });
}
}

/** @type {Map<string, { provider: string, id: string, version: number[] }[]>} */
const families = new Map();
for (const { provider, id } of curated) {
const { family, version } = parse(provider, id);
families.set(family, [...(families.get(family) ?? []), { provider, id, version }]);
}

/**
* The newest curated model a catalog model is measured against: the OpenRouter one when
* the family has any, since a direct provider's version says nothing about it.
* @param {{ provider: string, id: string, version: number[] }[]} own
*/
const latest = (own) => {
const viaOpenRouter = own.filter(({ provider }) => provider === "openrouter");
return (viaOpenRouter.length > 0 ? viaOpenRouter : own).reduce((a, b) =>
compare(b.version, a.version) > 0 ? b : a,
);
};

const newer = [];
const candidates = [];
// A model is offered until its end date, as the plugin's registry has it, so a newer one
// is reported while offered. A model with an end date is no candidate to start using.
const offered = catalog
.filter((model) => !model.expiration_date || model.expiration_date.slice(0, 10) > today)
.sort((a, b) => (b.created ?? 0) - (a.created ?? 0));
Comment thread
chhoumann marked this conversation as resolved.
Resolved
for (const model of offered) {
const { family, version } = parse("", model.id);
const own = families.get(family);
if (own && compare(version, latest(own).version) > 0) {
newer.push({ model, than: latest(own).id });
} else if (
!model.expiration_date &&
!own?.some((curatedModel) => compare(version, curatedModel.version) === 0)
) {
candidates.push(model);
}
}
return { gone, ending, newer, candidates };
}

/**
* @param {ReturnType<typeof review>} result
* @param {{ catalogSize: number, today: string }} context
*/
export function renderReport({ gone, ending, newer, candidates }, { catalogSize, today }) {
/** @type {string[]} */
const lines = [
"# Transcription model review",
"",
`Checked ${today} against the OpenRouter catalog (${catalogSize} models).`,
];
/**
* @param {string} heading
* @param {string[]} items
*/
const section = (heading, items) => {
if (items.length > 0) lines.push("", `## ${heading}`, "", ...items.map((i) => `- ${i}`));
};
section("Curated models gone or ending", [
...gone.map((id) => `\`${id}\` is not in the catalog.`),
...ending.map(({ id, ends, past }) => `\`${id}\` ${past ? "ended" : "ends"} ${ends}.`),
]);
section(
"Newer models in a curated family",
newer.map(({ model, than }) => `\`${model.id}\` is newer than \`${than}\`.`),
);
section(
"Candidates, not curated",
candidates.map(({ id, name }) => `\`${id}\`${name ? ` (${name})` : ""}`),
);
if (gone.length + ending.length + newer.length + candidates.length === 0) {
lines.push("", "Nothing to review.");
}
return `${lines.join("\n")}\n`;
}

/** @param {string[]} args */
async function main(args) {
const file = args[args.indexOf("--catalog") + 1];
const body =
args.includes("--catalog") && file
? JSON.parse(readFileSync(file, "utf8"))
: await fetch(CATALOG_URL, { signal: AbortSignal.timeout(30_000) }).then((response) => {
if (!response.ok) throw new Error(`The catalog answered ${response.status}.`);
return response.json();
});
/** @type {CatalogModel[]} */
const catalog = body.data;
const today = new Date().toISOString().slice(0, 10);
const result = review({ curated: curatedModels(), catalog, today });
process.stdout.write(renderReport(result, { catalogSize: catalog.length, today }));
if (result.gone.length > 0 || result.ending.some(({ past }) => past)) process.exitCode = 1;
}

if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
main(process.argv.slice(2)).catch((error) => {
console.error(`Model review failed: ${error instanceof Error ? error.message : error}`);
process.exitCode = 1;
});
}
12 changes: 12 additions & 0 deletions scripts/model-review.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import { expect, it } from "vitest";
import { PROVIDERS } from "src/services/transcription/registry";
import { curatedModels } from "./model-review.mjs";

it("reads the model IDs the providers offer", () => {
const offered = Object.entries(PROVIDERS).flatMap(([provider, { models }]) =>
models.map(({ id }) => ({ provider, id })),
);
const byProvider = (a: { provider: string }, b: { provider: string }) =>
a.provider.localeCompare(b.provider);
expect(curatedModels().sort(byProvider)).toEqual(offered.sort(byProvider));
});
52 changes: 52 additions & 0 deletions tests/fixtures/transcription/openrouter-catalog.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
{
"data": [
{
"id": "deepgram/nova-4",
"name": "Deepgram: Nova 4",
"created": 1794500000,
"expiration_date": null
},
{
"id": "x-ai/grok-stt-2.0",
"name": "SpaceXAI: Grok STT 2.0",
"created": 1794329167,
"expiration_date": "2026-11-01"
},
{
"id": "elevenlabs/scribe-v2",
"name": "ElevenLabs: Scribe v2",
"created": 1791386190,
"expiration_date": null
},
{
"id": "elevenlabs/scribe-v2-medical",
"name": "ElevenLabs: Scribe v2 Medical",
"created": 1791385619,
"expiration_date": null
},
{
"id": "google/chirp-3",
"name": "Google: Chirp 3",
"created": 1777997783,
"expiration_date": null
},
{
"id": "openai/gpt-transcribe",
"name": "OpenAI: GPT Transcribe",
"created": 1785973897,
"expiration_date": null
},
{
"id": "x-ai/grok-stt-1.0",
"name": "SpaceXAI: Grok STT 1.0",
"created": 1784817014,
"expiration_date": "2026-11-15"
},
{
"id": "deepgram/nova-3",
"name": "Deepgram: Nova 3",
"created": 1784075761,
"expiration_date": null
}
]
}
Loading
Loading