Mirror from veed-llm-editor

.gitignore

@@ -34,3 +34,6 @@ veed/.veed-client.json
 # Internal build docs / working files (not published)
 DESIGN.md
 plans/
+
+# transcription provider choice (owner-only, never committed)
+.open-edit-prefs.json

AGENTS.md

@@ -28,8 +28,8 @@ the work calls for edits, motion graphics, or other compositions.
 When the user says "run the open-edit skill on `<video.mp4>`", follow
 `.claude/skills/open-edit/SKILL.md` exactly (the FAST PATH):
 **preflight** (automatically prepare the workspace-local runtime; ask before global installs or updates) →
-**prep** (VEED transcript via `veed/go.ts`, then `prep/prep.ts`: `meta.json` + `word-timings.json` from
-VEED's real per-word times + base frames) → **sample ONE style** (`sample-style.ts`, facet-scored, seeded,
+**prep** (transcript from the recorded provider — `veed/go.ts` or `prep/transcribe.ts` — then `prep/prep.ts`: `meta.json` + `word-timings.json` from
+the transcript's real per-word times + base frames) → **sample ONE style** (`sample-style.ts`, facet-scored, seeded,
 zero tokens → `style.json`; the pool is the recipes-only runtime index `refs/tags.json`, so the draw is
 always recipe-backed) → **design + render** (`pipeline/scripts/generate-recipe.ts --run runs/<key>
 --record`, a SCRIPT driving the full gate chain: the compiled recipe emits the .wv document → lint → `--verify`
@@ -61,7 +61,7 @@ fast-path/refinement switch.
   into the final .wv document deterministically. Authored + validated offline (derived from the sheet);
   `hasRecipe` keys on the module existing.
 - `pipeline/scripts/` — `sample-style.ts` (facet-scored seeded style pick) · `generate-recipe.ts` (runs the compiled recipe: generate → lint → `--verify` fix loop → `--record` → probe) · `lint-template.ts` (mechanical engine-limit gate) · `probe-qa.ts` (frame QA vs the source) · `resolve-video.ts` (the video-arg resolution rule shared by both entry points) · `synth-word-timings.ts` (imported by prep) · `extract-beat-frames.ts` (imported by prep) · `mux-audio.sh`, `preflight.sh`, `install-veed-engine.sh` (deterministic).
-- `veed/` — VEED-native transcription + login (owns `transcript.json`; real per-word timings). `prep/prep.ts` — `meta.json` + `word-timings.json` + base frames (needs the VEED transcript). `refs/` — `html/` refs + `tags.json` (v3, the RUNTIME INDEX — recipes only, facet taxonomy, `fit` = aspect SOT) + per-ref `recipe.md` (the prose recipe sheet a compiled recipe is derived from — the fast path runs the compiled module, never the sheet; the creative pass reads sheets as craft substrate and REMIX donors).
+- `veed/` — VEED-native transcription + login (one writer of `transcript.json`; real per-word timings). `prep/prep.ts` — `meta.json` + `word-timings.json` + base frames (needs a transcript from any provider). `refs/` — `html/` refs + `tags.json` (v3, the RUNTIME INDEX — recipes only, facet taxonomy, `fit` = aspect SOT) + per-ref `recipe.md` (the prose recipe sheet a compiled recipe is derived from — the fast path runs the compiled module, never the sheet; the creative pass reads sheets as craft substrate and REMIX donors).
 - `config.ts` — all machine paths (ffmpeg / ffprobe / veed-engine). `docs/` — FLOW (orchestration) · recipe-format (the recipe law). Engine support matrix = the `feature-support.md` asset downloaded with the engine release into `.veed-engine/` (not vendored here).
 
 ## Hard rules (do not drift — these protect output quality)
@@ -109,6 +109,64 @@ workspace. There is no project-scoped path: the no-project route is the only one
 - `transcript-mapper.ts` converts VEED captions to the pipeline's timestamped chunk shape.
 - `readiness.ts` reports live-run prerequisites.
 
+## Transcription providers
+
+`runs/<key>/transcript.json` is the only seam. Anything that writes that shape is a valid provider and
+nothing downstream can tell which one ran. **There is no default provider**: on a cold start the choice
+belongs to the user (SKILL.md step 1), and picking one for them is a defect, not a shortcut. VEED is
+listed first because it transcribes best, not because it wins ties.
+
+| Provider | What it is | Entry point |
+| --- | --- | --- |
+| `veed` | Premium quality, hosted. One-time browser sign-in; limits are the VEED account's. | `veed/go.ts` |
+| `whisperx` | Free, local, offline. Two tiers: `medium` (slower, better) and `small.en` (fastest, weaker on names). CPU-bound here — CTranslate2 has no GPU path on Apple Silicon. | `prep/transcribe.ts` |
+| `custom` | The user's own service or MCP. **We provide no support code**: you obtain a Whisper-family JSON however their tool works, then map it. No credential ever passes through OpenEdit. | `prep/whisper.ts <json> <video>` |
+
+The choice is recorded in `$OPEN_EDIT_ROOT/.open-edit-prefs.json` — the runtime root, which is not the
+user's project root when the runtime is a managed clone — as `{ transcription: { provider, model? } }`, and is not
+re-asked. Write it with `node --import tsx prep/transcribe.ts --record <provider> [--model <id>]` rather
+than by hand; `prep/transcribe.ts` also exports `readPrefs` / `writePrefs` / `recordedModel` for
+programmatic use. An absent, corrupt or unrecognised file reads as a cold start with a stated reason, and
+a recorded `model` becomes the default tier for later runs.
+
+Every provider entry point takes `<video.mp4> [...]`, like `prep/prep.ts`, and writes one `runs/<key>` per
+video — so a batch asks the provider question, signs in, and installs once. A failure stops the batch and
+leaves the transcripts already written in place.
+
+```sh
+# whisperx, installed on request only — isolated uv/pipx environment, ~2 GB with weights,
+# removed again with `uv tool uninstall whisperx`
+bash pipeline/scripts/install-whisperx.sh                       # uv/pipx, pinned interpreter
+node --import tsx prep/transcribe.ts <video.mp4> [...] [--model medium] [--language de]
+
+# custom: any Whisper-family JSON the user's service produced, one json per video
+node --import tsx prep/whisper.ts transcription.json <video.mp4> [<json> <video.mp4> ...]
+```
+
+`prep/whisper-mapper.ts` accepts the Python Whisper family (WhisperX, openai-whisper,
+whisper-timestamped, mlx-whisper), the OpenAI API's `verbose_json` with
+`timestamp_granularities=["word"]`, and whisper.cpp's `-oj -ml 1` millisecond offsets.
+
+**Per-word times are mandatory**, and a transcript with none at all is refused — `prep` would otherwise
+even-split every cue and the word reveals would drift. Individual words a provider leaves untimed
+(WhisperX does this whenever alignment cannot match a word) keep their text and get a window
+interpolated from their timed neighbours, across segment boundaries where necessary; the count is
+reported, because approximate timing is acceptable and missing words are not. The mapper asserts its own
+output word count against its input and throws if they differ, so no future edit can reintroduce a
+silent drop. A "word" containing whitespace — whisper.cpp without `-ml 1` emits whole sentences that way
+— is split into tokens and counted as inferred, rather than passing as word-timed and being even-split
+later. Provider order is not trusted: chunks are sorted, and words whose times run backwards are
+reordered so the caption text and the reveals agree.
+
+If a transcription MCP is available, prefer one that writes a file and returns its path. Routing
+hundreds of timestamps through model context invites drift in the numbers themselves.
+
+A failed VEED run is classified rather than retried blindly (SKILL.md step 1): out of credits returns to
+the provider question with VEED still offered — a free account covers about 2 minutes a month, so
+`veed/orchestrate.ts` names https://www.veed.io/pricing and the local alternative in the error itself; a
+login failure retries the login once; anything else retries the command once. The recorded provider is
+never rewritten on failure.
+
 Run the isolated VEED tests without network access:
 
 ```sh

README.md

@@ -41,7 +41,7 @@ rather than collapsing every ask onto captions.
 | Platform | Apple Silicon Mac, macOS Tahoe 26.0 |
 | Intel Macs, earlier macOS | Not supported |
 | Windows, Linux | Planned; prioritisation depends on demand |
-| Account | veed.io account for transcription — [sign up](https://www.veed.io/signup) or [login](https://www.veed.io/login) |
+| Transcription | needed only to caption speech — OpenEdit asks once and remembers: VEED ([sign up](https://www.veed.io/signup) / [login](https://www.veed.io/login)), WhisperX locally, or your own service |
 
 ## Installation
 Download this repo, or install via the command line with:
@@ -132,20 +132,40 @@ as the render backend instead.
 
 ## Transcription
 
-The default workflow provides access to VEED's transcription and requires a VEED account. Usage limits
-are those of your VEED account. The audio track is uploaded and stored in order to transcribe it; the
-video is not uploaded.
+OpenEdit asks once which provider to use and records the answer at the runtime root
+(`.open-edit-prefs.json`); it does not ask again. Every provider writes the same file —
+`runs/<key>/transcript.json`, with real per-word timings — and nothing downstream can tell which one ran.
 
-By default OpenEdit does not create a VEED project. The agent guides the one-time browser login and
-stores a refreshable token at `veed/.veed-token.json` inside the runtime.
-
-For manual use:
+**VEED** — best quality, and the default workflow. Requires a VEED account; usage limits are your
+account's, and a free account covers about 2 minutes of transcription a month, beyond which it needs a
+[plan](https://www.veed.io/pricing). The audio track is uploaded and stored in order to transcribe it; the
+video is not uploaded. By default OpenEdit does not create a VEED project. The agent guides the one-time
+browser login and stores a refreshable token at `veed/.veed-token.json` inside the runtime.
 
 ```sh
 node --import tsx veed/login.ts
 node --import tsx veed/go.ts /path/to/video.mp4
 ```
 
+**WhisperX** — free, local and offline; nothing leaves your machine. Installed on request, and the first
+run also downloads a model. CPU-bound on Apple Silicon, with two quality tiers.
+
+```sh
+bash pipeline/scripts/install-whisperx.sh                  # on request, once
+node --import tsx prep/transcribe.ts /path/to/video.mp4    # --model medium for the better tier
+```
+
+**Your own service** — produce a Whisper-family JSON however you like (WhisperX, openai-whisper,
+whisper-timestamped, mlx-whisper, the OpenAI API with `timestamp_granularities=["word"]`, or whisper.cpp
+`-oj`) and map it. No credentials pass through OpenEdit.
+
+```sh
+node --import tsx prep/whisper.ts transcription.json /path/to/video.mp4
+```
+
+Per-word timings are required whichever provider you use: without them the caption reveals drift out of
+sync with the audio, so a transcript that has none is refused rather than rendered badly.
+
 ## Scope and limitations
 
 V1 targets captions. Motion graphics, charts, and brandbook-matched styling render today, but are less

SETUP.md

@@ -24,8 +24,10 @@ Preflight has three modes: bare applies safe local setup, `--dry` reports withou
 4. **veed-engine-cli** — the render engine. Its first install is automatic and local; later updates require
    approval. `pipeline/scripts/install-veed-engine.sh` downloads and verifies the latest macOS-arm64 release into `.veed-engine/`.
    Set `VEED_ENGINE_BIN` to use an existing installation.
-5. **VEED login** — `node --import tsx veed/login.ts` (one-time OAuth, ~30-day refreshable token). Transcription
-   runs through VEED's transcription API. See `README.md`.
+5. **A transcription provider** — the skill asks once and remembers the answer. Either
+   **VEED** (`node --import tsx veed/login.ts`, one-time OAuth, ~30-day refreshable token; a free account
+   covers about 2 minutes a month) or **WhisperX** locally (`bash pipeline/scripts/install-whisperx.sh`,
+   free and offline). You can also point it at your own service. See `README.md`.
 6. **config.ts** — optional machine-path overrides. Videos live anywhere:
    invoke the skill with the video's path (absolute, or relative to your CWD).
 

config.ts

@@ -18,3 +18,10 @@ export const FFPROBE = process.env.VEED_ENGINE_FFPROBE
 // install if behind — no version is pinned here. Default: the repo-local .veed-engine/ install produced
 // by pipeline/scripts/install-veed-engine.sh.
 export const VEED_ENGINE_BIN = process.env.VEED_ENGINE_BIN ?? path.join(REPO_ROOT, '.veed-engine', 'veed-engine-cli');
+
+// WhisperX — the local, free transcription provider (prep/transcribe.ts), installed on request by
+// pipeline/scripts/install-whisperx.sh. Default: PATH.
+export const WHISPERX_BIN = process.env.WHISPERX_BIN ?? 'whisperx';
+
+// Fallback quality tier, used only until the user's choice is recorded in .open-edit-prefs.json.
+export const WHISPERX_MODEL = process.env.WHISPERX_MODEL ?? 'small.en';

docs/FLOW.md

@@ -9,15 +9,16 @@ is the conceptual map. One video → ONE captioned MP4. No per-shot intermediate
 | # | step | kind | what |
 |---|------|------|------|
 | 0 | preflight | SCRIPT | check `veed-engine-cli` vs the latest release; offer install if stale/missing |
-| 1 | prep | SCRIPT | VEED transcript (`veed/go.ts`; chunks = beats, real per-word timings) → `word-timings.json` (per-word absolute-ms reveal delays, synthesized by `prep/prep.ts`) + base frames; auto-detect aspect → write `meta.json` (canvas + duration + paths) |
+| 1 | prep | SCRIPT | transcript from the chosen provider (VEED `veed/go.ts`, local WhisperX `prep/transcribe.ts`, or the user's own via `prep/whisper.ts`; chunks = beats, real per-word timings) → `word-timings.json` (per-word absolute-ms reveal delays, synthesized by `prep/prep.ts`) + base frames; auto-detect aspect → write `meta.json` (canvas + duration + paths) |
 | 2 | analyse | AGENT (vision) | **OPT-IN, skipped by default** — run only when the user asks to refine the style; then ONE nameless bg subagent reads the frames + transcript → `analysis.json` (per-beat shot/bboxes/neg-space/brightness in CANVAS px), and step 4 re-runs with it. The **only** vision pass. `analysis.json` existing IS the fast-path/refinement switch. |
 | 3 | sample ONE style | SCRIPT | `pipeline/scripts/sample-style.ts` — facet-scored seeded draw of one aspect-matched ref from `refs/tags.json` (v3 RUNTIME INDEX, recipes only; `fit` = aspect SOT; transcript energy weights the draw) → `style.json` (facets, energy, coverage, alternates). Zero tokens. The CREATIVE PASS routes here too: face-1 (user brought materials) → step 4 from-scratch; face-2 REMIX (any creative iteration) / RE-ROLL (variants asks) / patch (defects). |
 | 4 | design + render | SCRIPT (recipe) · INLINE (creative face-1 · remix) | default → **compiled recipe** (`pipeline/scripts/generate-recipe.ts`: the generator module emits `final/template.wv` + `manifest.json` deterministically — word delays from `word-timings.json` by construction, zero tokens — then drives the gate chain: lint → `--verify` with the mechanical ladder fix loop (≤2 cycles) → `--record` → probe-qa); **face-1** (INLINE, orchestrator-authored: one design system from the USER'S materials, 1-2 recipe sheets as craft substrate, per `director-brief.md`; lint + `--verify` to exit 0, then `--record` → probe-qa); **REMIX** (inline donor blend per the brief's REMIX MODE in `runs/<key>-remix`, same gates) |
 | 5 | mux audio | SCRIPT | ffmpeg muxes the source audio onto the silent render → `final/out.mp4` (the deliverable) |
 | 5.5 | preview | SCRIPT (parallel) | `preview/server.ts runs/<key>` — localhost preview opened for the user (read-only): watch and scrub the footage, follow the transcript, preview the subtitles; auto-swaps to the new `final/out.mp4` when an amend re-render lands. Launched in the background; the pipeline never waits on it |
 
 ## Where quality lives (do not let these drift)
-- **Prep (1)** transcribes via VEED (real per-word timings), then extracts the base
+- **Prep (1)** transcribes with whichever provider the user chose — asked once, recorded in
+  `.open-edit-prefs.json` at the runtime root (real per-word timings either way) — then extracts the base
   frames and fixes the canvas from the source aspect; every downstream step reads `runs/<key>/meta.json`
   (never re-derives dims).
 - **Analysis (2)** is opt-in — the default run derives vibe from the transcript and places by safe margins.

pipeline/director-brief.md

@@ -1,11 +1,21 @@
 # DIRECTOR — the engine contract (read this + your execution contract)
 
-You are executing the DESIGN + RENDER pass for ONE short-form video. You author ONE continuous captioned
-composition over the real footage and render it to an MP4. Author it as a single `.wv`
+You are executing the DESIGN + RENDER pass for ONE short-form video. Usually that means ONE continuous
+captioned composition over the real footage, rendered to an MP4. Author it as a single `.wv`
 document (HTML/CSS, a Chrome subset rendered by `veed-engine-cli`) that plays across the WHOLE clip —
 each spoken caption shown only in its own cue window. No per-beat files: one timeline,
 one render.
 
+**A `.wv` is an extension of CSS and can be treated as such** — an HTML fragment plus a `<style>` block,
+standard CSS throughout, no JavaScript and no proprietary timeline format. The motion IS CSS motion:
+`@keyframes` + `animation` + `animation-delay` are the timeline. So everything you already know about CSS
+applies directly here; the only thing to learn is which parts of it this engine does not implement (ENGINE
+LIMITS below).
+
+Footage is a LAYER in that document, never a prerequisite for it. A run with no source video (motion
+graphics, stills, slides, generated imagery) authors the same single `.wv` with no footage layer, and every
+rule below applies to it unchanged.
+
 Commit ONE design system in a single pass — do NOT re-litigate the aesthetic once chosen. You verify with
 `veed-engine-cli --verify` (an analytic playback check — see RENDER + VERIFY), then render. You do NOT
 extract frames / eyeball the video / run any visual self-check unless the execution contract EXPLICITLY tells you
@@ -16,6 +26,9 @@ Your execution contract (the filled step-4B contract from the SKILL) gives: the
 authority) plus 1-2 recipe SHEETS as craft substrate, the video path, the DIRECTION, the ENGAGEMENT mode,
 and the ANIMATION LEVEL. `analysis.json` (per-beat composition facts) exists ONLY on style-refine runs — when
 present it is authoritative for placement; when absent, place by the safe margins below.
+With no source video there is no `meta.json`, `transcript.json` or video path to hand you: the contract
+carries the canvas, the duration and the content from the ASK instead, and there are no cue windows to
+honour. That is a complete contract, not a missing one — proceed on it.
 
 (Recipe-backed picks never reach this pass: recipes are compiled code — `refs/html/<id>/recipe.ts`, run
 by `pipeline/scripts/generate-recipe.ts` — so you execute this contract only for a CREATIVE run: face-1 (the user