Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
178 changes: 123 additions & 55 deletions .claude/skills/open-edit/SKILL.md

Large diffs are not rendered by default.

11 changes: 6 additions & 5 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ fast-path/refinement switch.
reads the FOOTAGE frames** — it writes per-beat facts in CANVAS px to `analysis.json`; the design pass
composes from those numbers when the file exists (safe margins otherwise) and never opens the frames.
(User-SUPPLIED materials on face-1 are the one exception to "no vision in design" — studying them is the point.)
- Engine = `veed-engine-cli` (the veed render engine, downloaded directly from the upstream `veedstudio/weave-renderer-public-releases` repo into `.veed-engine/` via `pipeline/scripts/install-veed-engine.sh`; **Step 0 of the SKILL self-checks the version** vs the latest GitHub release).
- Engine = `veed-engine-cli` (the veed render engine, downloaded directly from the upstream `veedstudio/weave-renderer-public-releases` repo into `.veed-engine/` via `pipeline/scripts/install-veed-engine.sh`; **the SKILL's PREFLIGHT step self-checks the version** vs the latest GitHub release).

## Conventions
- Canvas (probed by prep): the source's own width/height (rotation-corrected) and fps (nominal `r_frame_rate`); `aspect` is an orientation label — 9:16 when h ≥ w, else 16:9. Beat render frame = `round(beatMidSec * fps)`.
Expand All @@ -113,7 +113,7 @@ workspace. There is no project-scoped path: the no-project route is the only one

`runs/<key>/transcript.json` is the only seam. Anything that writes that shape is a valid provider and
nothing downstream can tell which one ran. **There is no default provider**: on a cold start the choice
belongs to the user (SKILL.md step 1), and picking one for them is a defect, not a shortcut. VEED is
belongs to the user (SKILL.md, the PREP step), and picking one for them is a defect, not a shortcut. VEED is
listed first because it transcribes best, not because it wins ties.

| Provider | What it is | Entry point |
Expand Down Expand Up @@ -161,7 +161,7 @@ reordered so the caption text and the reveals agree.
If a transcription MCP is available, prefer one that writes a file and returns its path. Routing
hundreds of timestamps through model context invites drift in the numbers themselves.

A failed VEED run is classified rather than retried blindly (SKILL.md step 1): out of credits returns to
A failed VEED run is classified rather than retried blindly (SKILL.md, the PREP step): out of credits returns to
the provider question with VEED still offered — a free account covers about 10 minutes a month, so
`veed/orchestrate.ts` names https://www.veed.io/pricing and the local alternative in the error itself; a
login failure retries the login once; anything else retries the command once. The recorded provider is
Expand All @@ -171,6 +171,7 @@ Run the isolated VEED tests without network access:

```sh
node --import tsx --test \
tests/cli-entry.test.ts tests/oauth.test.ts tests/orchestrate.test.ts tests/token-store.test.ts \
tests/transcript-mapper.test.ts
tests/cli-entry.test.ts tests/login-page.test.ts tests/oauth.test.ts tests/orchestrate.test.ts \
tests/token-store.test.ts \
tests/transcript-mapper.test.ts tests/workspace.test.ts
```
5 changes: 3 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,9 @@ rather than collapsing every ask onto captions.

| | |
| --- | --- |
| Platform | Apple Silicon Mac, macOS Tahoe 26.0 |
| Intel Macs, earlier macOS | Not supported |
| Platform | Apple Silicon Mac. Preflight requires macOS on arm64 and stops there |
| Intel Macs | Not supported — the renderer ships macOS-arm64 only |
| Older macOS | Built and tested on Tahoe 26.0. Nothing checks the version, so earlier releases may work — untested |
| Windows, Linux | Planned; prioritisation depends on demand |
| Transcription | needed only to caption speech — OpenEdit asks once and remembers: VEED ([sign up](https://www.veed.io/signup) / [login](https://www.veed.io/login)), WhisperX locally, or your own service |

Expand Down
20 changes: 19 additions & 1 deletion config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ export const FFMPEG = process.env.VEED_ENGINE_FFMPEG ?? 'ffmpeg';
export const FFPROBE = process.env.VEED_ENGINE_FFPROBE
?? (process.env.VEED_ENGINE_FFMPEG ? process.env.VEED_ENGINE_FFMPEG.replace(/ffmpeg([^/]*)$/, 'ffprobe$1') : 'ffprobe');

// veed-engine-cli render engine. Step 0 (pipeline/scripts/preflight.sh) checks your binary against the
// veed-engine-cli render engine. PREFLIGHT (pipeline/scripts/preflight.sh) checks your binary against the
// latest veedstudio/weave-renderer-public-releases GitHub release (upstream repo name) and offers an
// install if behind — no version is pinned here. Default: the repo-local .veed-engine/ install produced
// by pipeline/scripts/install-veed-engine.sh.
Expand All @@ -25,3 +25,21 @@ export const WHISPERX_BIN = process.env.WHISPERX_BIN ?? 'whisperx';

// Fallback quality tier, used only until the user's choice is recorded in .open-edit-prefs.json.
export const WHISPERX_MODEL = process.env.WHISPERX_MODEL ?? 'small.en';

// --- WCAG pass (creative runs run it by default; recipes via --wcag) -------
// The analyzer is not bundled with the release engine binary; default to a
// sibling weave-renderer checkout, overridable via the env vars.
export const WEAVE_RENDERER_ROOT =
process.env.WEAVE_RENDERER_ROOT ?? path.join(REPO_ROOT, '..', 'weave-renderer');

// wcag-contrast analyzer binary. Build once in the checkout:
// cd src/crates/wcag-contrast && cargo build
export const WCAG_CONTRAST_BIN =
process.env.WCAG_CONTRAST_BIN ??
path.join(WEAVE_RENDERER_ROOT, 'src', 'crates', 'wcag-contrast', 'target', 'debug', 'wcag-contrast');

// Remediation applier — lives in this repo (application policy belongs to the
// consumer). Requires plain `node` (native .ts type-stripping, >=22.18), NOT
// tsx; wcag-pass spawns it accordingly.
export const WCAG_REMEDIATE =
process.env.WCAG_REMEDIATE ?? path.join(REPO_ROOT, 'pipeline', 'scripts', 'wcag', 'remediate.ts');
18 changes: 9 additions & 9 deletions docs/FLOW.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,15 +6,15 @@ reference/brand/concept) and creative iteration on a delivered result are author
The whole thing is encoded executably in `.claude/skills/open-edit/SKILL.md` (the golden master) — this doc
is the conceptual map. One video → ONE captioned MP4. No per-shot intermediates, no user-approval gate.

| # | step | kind | what |
|---|------|------|------|
| 0 | preflight | SCRIPT | check `veed-engine-cli` vs the latest release; offer install if stale/missing |
| 1 | prep | SCRIPT | transcript from the chosen provider (VEED `veed/go.ts`, local WhisperX `prep/transcribe.ts`, or the user's own via `prep/whisper.ts`; chunks = beats, real per-word timings) → `word-timings.json` (per-word absolute-ms reveal delays, synthesized by `prep/prep.ts`) + base frames; auto-detect aspect → write `meta.json` (canvas + duration + paths) |
| 2 | analyse | AGENT (vision) | **OPT-IN, skipped by default** — run only when the user asks to refine the style; then ONE nameless bg subagent reads the frames + transcript → `analysis.json` (per-beat shot/bboxes/neg-space/brightness in CANVAS px), and step 4 re-runs with it. The **only** vision pass. `analysis.json` existing IS the fast-path/refinement switch. |
| 3 | sample ONE style | SCRIPT | `pipeline/scripts/sample-style.ts` — facet-scored seeded draw of one aspect-matched ref from `refs/tags.json` (v3 RUNTIME INDEX, recipes only; `fit` = aspect SOT; transcript energy weights the draw) → `style.json` (facets, energy, coverage, alternates). Zero tokens. The CREATIVE PASS routes here too: face-1 (user brought materials) → step 4 from-scratch; face-2 REMIX (any creative iteration) / RE-ROLL (variants asks) / patch (defects). |
| 4 | design + render | SCRIPT (recipe) · INLINE (creative face-1 · remix) | default → **compiled recipe** (`pipeline/scripts/generate-recipe.ts`: the generator module emits `final/template.wv` + `manifest.json` deterministically — word delays from `word-timings.json` by construction, zero tokens — then drives the gate chain: lint → `--verify` with the mechanical ladder fix loop (≤2 cycles) → `--record` → probe-qa); **face-1** (INLINE, orchestrator-authored: one design system from the USER'S materials, 1-2 recipe sheets as craft substrate, per `director-brief.md`; lint + `--verify` to exit 0, then `--record` → probe-qa); **REMIX** (inline donor blend per the brief's REMIX MODE in `runs/<key>-remix`, same gates) |
| 5 | mux audio | SCRIPT | ffmpeg muxes the source audio onto the silent render → `final/out.mp4` (the deliverable) |
| 5.5 | preview | SCRIPT (parallel) | `preview/server.ts runs/<key>` — localhost preview opened for the user (read-only): watch and scrub the footage, follow the transcript, preview the subtitles; auto-swaps to the new `final/out.mp4` when an amend re-render lands. Launched in the background; the pipeline never waits on it |
| step | kind | what |
|------|------|------|
| preflight | SCRIPT | check `veed-engine-cli` vs the latest release; offer install if stale/missing |
| prep | SCRIPT | transcript from the chosen provider (VEED `veed/go.ts`, local WhisperX `prep/transcribe.ts`, or the user's own via `prep/whisper.ts`; chunks = beats, real per-word timings) → `word-timings.json` (per-word absolute-ms reveal delays, synthesized by `prep/prep.ts`) + base frames; auto-detect aspect → write `meta.json` (canvas + duration + paths) |
| analyse | AGENT (vision) | **OPT-IN, skipped by default** — run only when the user asks to refine the style; then ONE nameless bg subagent reads the frames + transcript → `analysis.json` (per-beat shot/bboxes/neg-space/brightness in CANVAS px), and the DESIGN + RENDER step re-runs with it. The **only** vision pass. `analysis.json` existing IS the fast-path/refinement switch. |
| sample ONE style | SCRIPT | `pipeline/scripts/sample-style.ts` — facet-scored seeded draw of one aspect-matched ref from `refs/tags.json` (v3 RUNTIME INDEX, recipes only; `fit` = aspect SOT; transcript energy weights the draw) → `style.json` (facets, energy, coverage, alternates). Zero tokens. The CREATIVE PASS routes here too: face-1 (user brought materials) → the DESIGN + RENDER step, from-scratch; face-2 REMIX (any creative iteration) / RE-ROLL (variants asks) / patch (defects). |
| design + render | SCRIPT (recipe) · INLINE (creative face-1 · remix) | default → **compiled recipe** (`pipeline/scripts/generate-recipe.ts`: the generator module emits `final/template.wv` + `manifest.json` deterministically — word delays from `word-timings.json` by construction, zero tokens — then drives the gate chain: lint → `--verify` with the mechanical ladder fix loop (≤2 cycles) → `--record` → probe-qa); **face-1** (INLINE, orchestrator-authored: one design system from the USER'S materials, 1-2 recipe sheets as craft substrate, per `director-brief.md`; lint + `--verify` to exit 0, then `--record` → probe-qa); **REMIX** (inline donor blend per the brief's REMIX MODE in `runs/<key>-remix`, same gates) |
| mux audio | SCRIPT | ffmpeg muxes the source audio onto the silent render → `final/out.mp4` (the deliverable) |
| preview | SCRIPT (parallel) | `preview/server.ts runs/<key>` — localhost preview opened for the user (read-only): watch and scrub the footage, follow the transcript, preview the subtitles; auto-swaps to the new `final/out.mp4` when an amend re-render lands. Launched in the background; the pipeline never waits on it |

## Where quality lives (do not let these drift)
- **Prep (1)** transcribes with whichever provider the user chose — asked once, recorded in
Expand Down
4 changes: 4 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,12 @@
"typecheck": "tsc --noEmit"
},
"devDependencies": {
"@types/culori": "^4.0.1",
"@types/node": "^22.0.0",
"tsx": "4.21.0",
"typescript": "^7.0.2"
},
"dependencies": {
"culori": "^4.0.2"
}
}
6 changes: 3 additions & 3 deletions pipeline/director-brief.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Commit ONE design system in a single pass — do NOT re-litigate the aesthetic o
extract frames / eyeball the video / run any visual self-check unless the execution contract EXPLICITLY tells you
to. You do not redesign based on renders, run legibility passes, or chase timing.

Your execution contract (the filled step-4B contract from the SKILL) gives: the run dir, `meta.json` (canvas W/H/fps + `durationSec` + all paths),
Your execution contract (the filled DESIGN + RENDER variant B contract from the SKILL) gives: the run dir, `meta.json` (canvas W/H/fps + `durationSec` + all paths),
`transcript.json` (captions + windows), the USER'S MATERIALS (their reference/brand/concept — the design
authority) plus 1-2 recipe SHEETS as craft substrate, the video path, the DIRECTION, the ENGAGEMENT mode,
and the ANIMATION LEVEL. `analysis.json` (per-beat composition facts) exists ONLY on style-refine runs — when
Expand Down Expand Up @@ -191,14 +191,14 @@ The binary is `{run dir}/../../.veed-engine/veed-engine-cli` (the preflight-mana
Do NOT extract frames or run any ffmpeg/visual self-check unless the execution contract EXPLICITLY instructs it — --verify
is the self-check. Author the timeline correctly up front using the recipe + limits above so --verify passes on the
first pass; it is a safety net, not a design loop. (The last gate, `probe-qa` — mechanical frame QA vs the source —
runs right after the record (SKILL step 4, same turn); it is not part of this verify loop and never a licence to
runs right after the record (the SKILL's DESIGN + RENDER step, same turn); it is not part of this verify loop and never a licence to
eyeball frames.)

## OUTPUT
- `{run}/final/template.wv` — the single-timeline document.
- `{run}/final/manifest.json` — EXACT: `{"render":{"width":{W},"height":{H},"fps":{FPS},"duration":{durationSec}}}`
(duration covers the FULL clip).
- `{run}/final/out.silent.mp4` — the render (video only; audio is muxed in step 5).
- `{run}/final/out.silent.mp4` — the render (video only; audio is muxed in the MUX AUDIO step).

## REPORT (fold into your summary — after probe-qa + mux, same turn, no preamble)
- aesthetic name + fonts + hex palette + recurring device.
Expand Down
37 changes: 34 additions & 3 deletions pipeline/scripts/generate-recipe.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Recipe runner — the fast path's step 4 for recipe-backed picks. Loads the compiled recipe for the
// Recipe runner — the fast path's DESIGN + RENDER for recipe-backed picks. Loads the compiled recipe for the
// run's sampled ref (refs/html/<id>/recipe.ts), generates runs/<key>/final/{template.wv,
// manifest.json} deterministically (zero tokens), then optionally drives the full gate chain:
// lint → --verify (mechanical ladder fix loop) → --record → probe-qa. Run OUTSIDE any sandbox when
Expand Down Expand Up @@ -43,17 +43,33 @@ async function main(): Promise<number> {
module: { type: 'string' },
record: { type: 'boolean' },
verify: { type: 'boolean' },
wcag: { type: 'boolean' },
'wcag-apply': { type: 'boolean' },
'progress-output': { type: 'boolean' },
},
});
const runArg = values.run;
if (!runArg) {
console.error('usage: node --import tsx pipeline/scripts/generate-recipe.ts --run <runDir> [--verify] [--record] [--module <path>]');
console.error('usage: node --import tsx pipeline/scripts/generate-recipe.ts --run <runDir> [--verify] [--record] [--wcag] [--wcag-apply] [--module <path>]');
return 2;
}
const runDir = resolve(runArg);
const doRecord = values.record ?? false;
const doVerify = (values.verify ?? false) || doRecord; // record only ever happens on a clean verify
// --wcag: opt-in WCAG AA contrast pass between verify and record (DEFAULT on
// the creative faces, which run the gates by hand — see the skill; opt-in
// here so the recipe fast path stays byte-identical until asked). It runs
// DETECT-ONLY: it audits real rendered contrast + emits contrast-statistics.json
// and reports, but changes nothing. It only makes sense inside the gate
// chain — bare --wcag would run without a rendered final/ to audit.
// --wcag-apply implies --wcag and runs the pass in APPLY mode (remediate +
// promote a remediated final/template.wv on measured improvement).
const doWcagApply = values['wcag-apply'] ?? false;
const doWcag = (values.wcag ?? false) || doWcagApply;
if (doWcag && !doVerify) {
console.error('[generate-recipe] --wcag requires --verify or --record (the WCAG pass runs after a clean verify)');
return 2;
}

// --module needs no style.json (independent/standalone runs); the sampled-ref path requires it.
const moduleOverride = values.module;
Expand All @@ -73,7 +89,7 @@ async function main(): Promise<number> {
return 2;
}
if (!style.hasRecipe || !existsSync(modPath)) {
console.error(`[generate-recipe] no compiled recipe for "${style.refId}" (${modPath}) — run the from-scratch inline pass (SKILL step 4 variant B)`);
console.error(`[generate-recipe] no compiled recipe for "${style.refId}" (${modPath}) — run the from-scratch inline pass (SKILL: DESIGN + RENDER variant B)`);
return 3;
}
}
Expand Down Expand Up @@ -128,6 +144,21 @@ async function main(): Promise<number> {
return 1;
}

// Gate 3.5 — WCAG: audit real rendered contrast + emit contrast-statistics.json,
// then report. DETECT-ONLY: status 'attention' lists every low-contrast
// selector but changes nothing — record still consumes the verified final/.
if (doWcag) {
const { runWcagPass } = await import('./wcag-pass.ts');
try {
const d = runWcagPass(runDir, { apply: doWcagApply });
console.log(`[generate-recipe] wcag: ${d.status}${d.promoted ? ' (remediated template promoted into final/template.wv)' : ''}`);
for (const n of d.notes) console.log(`[generate-recipe] wcag: ${n}`);
} catch (e) {
console.error(String((e as Error).message ?? e));
return 1;
}
}

if (doRecord) {
const outMp4 = join(finalDir, 'out.silent.mp4');
const r = spawnSync(VEED_ENGINE_BIN, [finalDir, '--progress-output', '--record', outMp4], { stdio: 'inherit' });
Expand Down
2 changes: 1 addition & 1 deletion pipeline/scripts/mux-audio.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
# Step 5 — restore the soundtrack. veed-engine-cli renders video only; this muxes the original audio
# MUX AUDIO — restore the soundtrack. veed-engine-cli renders video only; this muxes the original audio
# onto the clean silent render. Deterministic; run OUTSIDE any sandbox is NOT required (ffmpeg only).
#
# bash pipeline/scripts/mux-audio.sh <run-dir>
Expand Down
13 changes: 13 additions & 0 deletions pipeline/scripts/preflight.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,19 @@
# Compatibility entrypoint. The installable skill owns the canonical preflight implementation.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"

# WCAG tooling (report-only, like everything here): the wcag pass is a DEFAULT gate on creative
# runs and hard-fails at run time when the analyzer is missing — surface that up front instead of
# mid-pipeline. Path mirrors the config.ts default (env-overridable the same way).
WEAVE_RENDERER_ROOT="${WEAVE_RENDERER_ROOT:-$REPO_ROOT/../weave-renderer}"
WCAG_CONTRAST="${WCAG_CONTRAST_BIN:-$WEAVE_RENDERER_ROOT/src/crates/wcag-contrast/target/debug/wcag-contrast}"
if [ ! -x "$WCAG_CONTRAST" ]; then
echo "wcag: analyzer NOT FOUND ('$WCAG_CONTRAST') — the wcag pass will fail. Build it:"
echo " cd $WEAVE_RENDERER_ROOT/src/crates/wcag-contrast && cargo build"
else
echo "wcag: analyzer ok"
fi

# Only default the workspace when the caller did not give one, otherwise it is passed twice.
for arg in "$@"; do
[ "$arg" = "--workspace" ] && exec bash "$REPO_ROOT/.claude/skills/open-edit/scripts/preflight.sh" "$@"
Expand Down
Loading