diff --git a/server/lib/README.md b/server/lib/README.md index a4a765d917..c788dddbf7 100644 --- a/server/lib/README.md +++ b/server/lib/README.md @@ -159,6 +159,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub | `mediaModels.js` | Single source of truth for image/video model metadata. Entry fields are documented in the module docblock — note `repoFiles[]`, which narrows a model's own repo to an explicit file list for an aggregate repo that holds far more than the runner loads (MiniMax H3 CUDA). | | `minimaxH3Memory.js` | The declared weight-placement table for the three MiniMax H3 entries (#5420) — H3 is the one video model family whose components fit nowhere unassisted, so "does this box have enough" is a render gate, not a UI fact. `MINIMAX_H3_MEMORY_PROFILES` maps entry id → `{ shippedRepo, shippedRevision, profiles }`, each profile carrying an honest `minMemoryGb` host floor and (CUDA only) a `minVramGb` device floor, ordered best-first. Every capacity number is HOISTED from what already existed — the CUDA tiers are `resolve_offload_profile()`'s own thresholds in `scripts/generate_minimax_h3_cuda.py`, the host floors are the entries' `memoryGb` — the sole new number being `MINIMAX_H3_HOST_RESERVE_GB`, a policy reserve held back for the OS. `applyMiniMaxH3MemoryProfiles(list)` is the load-time backfill (twin of migration 317) and guards BOTH `repo` and `revision`, like the speed-profile decorator. `selectMiniMaxH3MemoryProfile({ model, totalMemoryGb })` picks the best profile the HOST can hold (VRAM is the runner's call — the server has no synchronous device view); `miniMaxH3MemoryDeclineReason()` RETURNS the fail-closed reason so the render path can 400 it and a status surface can show it, and returns `null` on an UNMEASURED host — "not measured" is a deferral to the runner, never the same as zero. `validateMiniMaxH3MemoryProfileTable` / `sanitizeMiniMaxH3MemoryProfiles` warn + strip a hand-edited table (NaN floor, duplicate/reserved id, mis-ordered tiers) at load. | | `videoContinuity.js` | How chunk N+1 of a chained video render is conditioned on chunk N. `resolveContextFrames(requested)` normalizes the tail-window size (absent → `DEFAULT_CONTEXT_FRAMES` = 22 ≈ 1s @ 24fps; an explicit `0` is preserved as "last frame only", NOT collapsed into the default) and clamps to `MIN_CONTEXT_FRAMES`..`MAX_CONTEXT_FRAMES`. `resolveContinuityStrategy({model, contextFrames})` picks `'window'` (LTX-2 `extend_from_video` conditioned on the prior chunk's last N frames — motion, not just a pose) or `'frame'` (extract the last frame, run i2v), degrading to `'frame'` on any runtime outside `CONTEXT_WINDOW_RUNTIMES` rather than rejecting. `extendLatentFrames` / `extendedPixelFrames` convert across the VAE's `LATENT_FRAME_STRIDE` (8 pixel frames per latent), `contextPrefixFrames({totalFrames, extendLatents})` measures how much of an extend render is echoed context to trim back off before stitching (0 = leave it alone), and `tailWindowStartFrame` gives the cut index for the window itself. Pure — importable from `prepareParams.js` without dragging in `local.js`. Mirrored for the picker in `client/src/lib/videoGenParams.js`, pinned by `videoContinuity.parity.test.js`. | +| `videoPromptLinter.js` | Deterministic lint pass for continuous-video clip prompts (part of #6217, independent of `scriptVideoCompiler.js` #6225 by interface — lints already-built prompt strings plus caller-supplied framing/reference metadata rather than a compiler-shaped clip). `lintClipPrompt(clip, {bible, maxLength})` checks a single clip: a `cutType: 'continue'` clip must open with `"Hard cut to :"` and use a framing distinct from `previousFraming`; every `references` entry (`{kind: 'cast'|'locations', id}`) must resolve to a bible descriptor present VERBATIM in the prompt (never paraphrased); the prompt must not contain the banned cross-clip referents (same/still/again/continues/as before), negatives (no/without/never), or UI-overlay terms (text/caption/overlay), matched on word boundaries so substrings like "against" or "context" don't false-positive; and the prompt must be `<= MAX_CLIP_PROMPT_LENGTH` (800) characters. Returns `{pass, reasons[]}`, never throws. `lintClips(clips, {bible, maxLength})` lints an ordered chain, deriving each clip's `previousFraming` from the preceding array element's `framing` when not set explicitly. | | `videoDisclosure.js` | Video Gen provenance/licensing + backend policy-scope facts (#3674). `VIDEO_MODEL_DISCLOSURES` maps each shipped video model id to `{ shippedRepo, disclosure }` (model card URL, weights license, runtime license, pinned-snapshot download size in decimal GB, review date) — every value checked against a primary upstream source, and any fact that could not be established is OMITTED so the UI renders "Unknown" instead of guessing. `applyVideoDisclosures(list)` is the load-time backfill (twin of migration 237) with the same preservation guards: an existing `disclosure` key wins, a custom id is skipped, and a `repo` pointed at a fork keeps Unknown. `VIDEO_BACKEND_DISCLOSURES` / `videoBackendDisclosure(id)` state where inference runs (`execution: local or hosted`) and whose policy applies — execution facts only, never a restrictiveness ranking. `APACHE_2` / `GEMMA_TERMS` are the shared license descriptors `videoTextEncoders.js` reuses, so a license-text correction reaches both tables from one edit. | | `videoDraftDecoders.js` | Preview-fidelity ("draft") video decode (#5423). `VIDEO_DRAFT_DECODERS` declares each entry id → a separately downloaded decoder asset, pin-guarded on repo AND revision; `applyVideoDraftDecoders(list)` is the load-time backfill and `validateDraftDecoderTable` / `sanitizeDraftDecoders` warn + strip a hand-edited row (missing pin, a multi-shard or path-traversing file list, a runtime whose builder emits no draft flags) so a full decode can never report itself as a draft one. `DRAFT_DECODE_FULL` (`'full'`) is a deliberate NO-OP — `isFullDecode(id)` makes absence and that sentinel the same request, so a full-decode render builds byte-identical spawn args. `draftDecodeDeclineReason({ model, decodeId, models, runtimeRevision, assetCached })` RETURNS (never throws) the reason a draft decode does not apply — the model is a delivery target in the finish graph (`isDeliveryVideoModel`), it declares no decoder, the installed runner checkout is not the revision the asset was verified against, or the weights are not downloaded — because a knob that only makes a render cheaper must degrade rather than 400 a submitted job; `resolveVideoDraftDecoder()` returns the concrete asset or `null`. `publicVideoDraftDecodeOptions(model)` is the picker payload (empty for a model with no decoder, so the client renders no control) and `downloadableVideoDraftDecoders(list)` the download targets. The table ships NO entry, and as of 2026-08-30 that is a decision rather than a gap ([ADR](../../docs/decisions/2026-08-30-h3-draft-decoder-asset.md)): the shim substitutes one file into upstream’s checkpoint root under a STRICT key match, so a candidate must be a complete unquantized full-`VideoVAE` checkpoint — which every genuinely light decoder (TAE, quantized repack, decoder-only head) is not. | | `videoFinishProfiles.js` | Draft → delivery ("Finish") relationships between video models (#3696). `VIDEO_FINISH_PROFILES` declares each fast draft entry id → `{ shippedRepo, finishModelId }`, only for pairs that share a runtime, base repo and supported modes (the same weights at a different step budget), so re-rendering the draft's seed reproduces its composition instead of re-rolling it. `applyVideoFinishProfiles(list)` is the load-time backfill (twin of migration 238) with the usual preservation guards (existing key wins, custom id skipped, forked `repo` skipped); `validateFinishProfileGraph(list)` returns the graph problems (missing / self-referencing / chained target, runtime / repo / supportedModes mismatch) and `sanitizeFinishProfiles(list)` warns + strips them at load so a typo can never surface a Finish button targeting nothing. `finishTargetForModel(model, availableModels)` resolves the delivery entry scoped to what this install can run. | diff --git a/server/lib/index.js b/server/lib/index.js index 1393b06ae7..f4e83b1fcd 100644 --- a/server/lib/index.js +++ b/server/lib/index.js @@ -155,6 +155,7 @@ export * from './mediaModels.js'; export * from './minimaxH3Memory.js'; export * from './videoContinuity.js'; export * from './videoDisclosure.js'; +export * from './videoPromptLinter.js'; export * from './videoDraftDecoders.js'; export * from './videoFinishProfiles.js'; export * from './videoSpeedProfiles.js'; diff --git a/server/lib/videoPromptLinter.js b/server/lib/videoPromptLinter.js new file mode 100644 index 0000000000..fef36e38a6 --- /dev/null +++ b/server/lib/videoPromptLinter.js @@ -0,0 +1,116 @@ +/** + * Deterministic lint pass for continuous-video clip prompts (part of #6217's + * chained-generation feature). Catches the prompt patterns that cause visual + * smear/collapse when a video model chains one clip onto the next: a missing + * hard-cut opener, a re-used camera framing, a paraphrased bible descriptor, + * or language that reads fine in prose but breaks a video prompt — referring + * back to "the same" shot, negating something, or naming on-screen text. + * + * Pure text analysis — no I/O, no video-backend awareness. Independent of + * `scriptVideoCompiler.js` (#6225) by interface: it lints already-built clip + * prompt strings plus caller-supplied framing/reference metadata, not a + * compiler-shaped clip object. `continuousVideo.js` (#6227) composes this + * with the compiler's output. + * + * Returns structured per-clip results rather than throwing — callers decide + * how to surface a failing lint (block submission, warn, retry the prompt). + */ + +import { resolveBibleDescriptor } from './scriptVideoCompiler.js'; + +export const MAX_CLIP_PROMPT_LENGTH = 800; +const HARD_CUT_PREFIX = 'Hard cut to'; + +const BANNED_REFERENTS = ['same', 'still', 'again', 'continues', 'as before']; +const BANNED_NEGATIVES = ['no', 'without', 'never']; +const BANNED_OVERLAY_TERMS = ['text', 'caption', 'overlay']; + +const escapeRegExp = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); + +// \b (not a hand-rolled [^a-z0-9] boundary) so "same_text" isn't flagged as containing +// the standalone word "same" — \w already includes '_', matching how prose reads a word. +const bannedTermPattern = (term) => ({ term, regex: new RegExp(`\\b${escapeRegExp(term).replace(/\s+/g, '\\s+')}\\b`, 'i') }); +const BANNED_REFERENT_PATTERNS = BANNED_REFERENTS.map(bannedTermPattern); +const BANNED_NEGATIVE_PATTERNS = BANNED_NEGATIVES.map(bannedTermPattern); +const BANNED_OVERLAY_PATTERNS = BANNED_OVERLAY_TERMS.map(bannedTermPattern); + +const findBannedTerms = (prompt, patterns) => patterns.filter(({ regex }) => regex.test(prompt)).map(({ term }) => term); + +const asString = (value) => (typeof value === 'string' ? value : ''); + +/** + * Lint a single clip prompt against every rule. + * + * @param {object} clip + * @param {string} clip.prompt + * @param {'fresh'|'continue'} clip.cutType + * @param {string} [clip.framing] - this clip's camera framing/angle, required to check the hard-cut opener on a 'continue' clip + * @param {string} [clip.previousFraming] - the preceding chained clip's framing/angle, checked only when cutType === 'continue' + * @param {Array<{kind: 'cast'|'locations', id: string}>} [clip.references] - bible entries this clip's prompt must carry verbatim + * @param {object} [options] + * @param {object} [options.bible] + * @param {number} [options.maxLength] + * @returns {{pass: boolean, reasons: string[]}} + */ +export function lintClipPrompt(clip, { bible, maxLength = MAX_CLIP_PROMPT_LENGTH } = {}) { + const prompt = asString(clip?.prompt); + const { cutType } = clip || {}; + const framing = asString(clip?.framing).trim() || null; + const previousFraming = asString(clip?.previousFraming).trim() || null; + const references = Array.isArray(clip?.references) ? clip.references : []; + const reasons = []; + + if (cutType === 'continue') { + const opener = framing ? `${HARD_CUT_PREFIX} ${framing}:` : null; + if (!opener || !prompt.trimStart().startsWith(opener)) { + reasons.push(opener + ? `missing hard-cut opener "${opener}"` + : 'missing hard-cut opener: clip.framing was not provided'); + } + if (framing && previousFraming && framing.toLowerCase() === previousFraming.toLowerCase()) { + reasons.push(`framing "${framing}" repeats the preceding clip's framing — a continuing clip needs a distinct camera framing/angle`); + } + } + + for (const ref of references) { + const descriptor = resolveBibleDescriptor(bible, ref?.kind, ref?.id); + if (!descriptor) { + reasons.push(`no bible descriptor found for ${ref?.kind}/${ref?.id}`); + } else if (!prompt.includes(descriptor)) { + reasons.push(`prompt is missing the verbatim bible descriptor for ${ref.kind}/${ref.id}`); + } + } + + for (const term of findBannedTerms(prompt, BANNED_REFERENT_PATTERNS)) { + reasons.push(`banned cross-clip referent "${term}" — describe what is visible in THIS clip instead of referring back to a previous one`); + } + for (const term of findBannedTerms(prompt, BANNED_NEGATIVE_PATTERNS)) { + reasons.push(`banned negative construction "${term}" — video models tend to render negated content instead of omitting it`); + } + for (const term of findBannedTerms(prompt, BANNED_OVERLAY_PATTERNS)) { + reasons.push(`banned UI overlay language "${term}" — video models tend to render literal on-screen text/captions from this`); + } + + if (prompt.length > maxLength) { + reasons.push(`prompt is ${prompt.length} characters, over the ${maxLength} character limit`); + } + + return { pass: reasons.length === 0, reasons }; +} + +/** + * Lint an ordered array of clips (chain order, same as + * `scriptVideoCompiler.compileScriptToClips` emits them). Each clip's + * `previousFraming` defaults to the preceding array element's `framing` when + * not set explicitly. + * + * @returns {{pass: boolean, results: Array<{index: number, pass: boolean, reasons: string[]}>}} + */ +export function lintClips(clips, { bible, maxLength = MAX_CLIP_PROMPT_LENGTH } = {}) { + const clipList = Array.isArray(clips) ? clips : []; + const results = clipList.map((clip, index) => { + const previousFraming = clip?.previousFraming ?? clipList[index - 1]?.framing ?? null; + return { index, ...lintClipPrompt({ ...clip, previousFraming }, { bible, maxLength }) }; + }); + return { pass: results.every((r) => r.pass), results }; +} diff --git a/server/lib/videoPromptLinter.test.js b/server/lib/videoPromptLinter.test.js new file mode 100644 index 0000000000..106d683d92 --- /dev/null +++ b/server/lib/videoPromptLinter.test.js @@ -0,0 +1,162 @@ +import { describe, it, expect } from 'vitest'; +import { lintClipPrompt, lintClips, MAX_CLIP_PROMPT_LENGTH } from './videoPromptLinter.js'; + +const bible = { + cast: { hero: { descriptor: 'A weathered pilot in a scuffed orange flight suit.' } }, + locations: { hangar: { descriptor: 'A cavernous steel hangar lit by hanging sodium lamps.' } }, +}; + +const passingContinueClip = { + cutType: 'continue', + framing: 'a low tracking shot', + previousFraming: 'a wide establishing shot', + prompt: 'Hard cut to a low tracking shot: A weathered pilot in a scuffed orange flight suit. A cavernous steel hangar lit by hanging sodium lamps. She strides toward the ship.', + references: [{ kind: 'cast', id: 'hero' }, { kind: 'locations', id: 'hangar' }], +}; + +describe('lintClipPrompt', () => { + it('passes a clip that satisfies every rule', () => { + expect(lintClipPrompt(passingContinueClip, { bible })).toEqual({ pass: true, reasons: [] }); + }); + + it('passes a fresh-cut clip with no hard-cut opener requirement', () => { + const clip = { + cutType: 'fresh', + framing: 'a wide establishing shot', + prompt: `${bible.cast.hero.descriptor} She stands in the hangar doorway.`, + references: [{ kind: 'cast', id: 'hero' }], + }; + expect(lintClipPrompt(clip, { bible })).toEqual({ pass: true, reasons: [] }); + }); + + it('flags a continuing clip missing the hard-cut opener', () => { + const clip = { ...passingContinueClip, prompt: passingContinueClip.prompt.replace('Hard cut to a low tracking shot: ', '') }; + const { pass, reasons } = lintClipPrompt(clip, { bible }); + expect(pass).toBe(false); + expect(reasons.some((r) => r.includes('hard-cut opener'))).toBe(true); + }); + + it('flags a continuing clip whose opener framing does not match clip.framing', () => { + const clip = { ...passingContinueClip, prompt: passingContinueClip.prompt.replace('a low tracking shot', 'a close-up') }; + const { pass, reasons } = lintClipPrompt(clip, { bible }); + expect(pass).toBe(false); + expect(reasons.some((r) => r.includes('hard-cut opener'))).toBe(true); + }); + + it('flags a continuing clip whose framing repeats the preceding clip', () => { + const clip = { ...passingContinueClip, framing: 'a wide establishing shot', prompt: passingContinueClip.prompt.replace('a low tracking shot', 'a wide establishing shot') }; + const { pass, reasons } = lintClipPrompt(clip, { bible }); + expect(pass).toBe(false); + expect(reasons.some((r) => r.includes('repeats the preceding clip'))).toBe(true); + }); + + it('flags a missing bible descriptor entry', () => { + const clip = { ...passingContinueClip, references: [{ kind: 'cast', id: 'nobody' }] }; + const { pass, reasons } = lintClipPrompt(clip, { bible }); + expect(pass).toBe(false); + expect(reasons.some((r) => r.includes('no bible descriptor found'))).toBe(true); + }); + + it('flags a prompt that paraphrases the bible descriptor instead of quoting it verbatim', () => { + const clip = { ...passingContinueClip, prompt: passingContinueClip.prompt.replace('A weathered pilot in a scuffed orange flight suit.', 'A weary pilot in an orange flight suit.') }; + const { pass, reasons } = lintClipPrompt(clip, { bible }); + expect(pass).toBe(false); + expect(reasons.some((r) => r.includes('verbatim bible descriptor'))).toBe(true); + }); + + it.each(['same', 'still', 'again', 'continues', 'as before'])('flags the banned cross-clip referent "%s"', (term) => { + const clip = { ...passingContinueClip, prompt: `${passingContinueClip.prompt} It looks ${term} as it did.` }; + const { pass, reasons } = lintClipPrompt(clip, { bible }); + expect(pass).toBe(false); + expect(reasons.some((r) => r.includes(`banned cross-clip referent "${term}"`))).toBe(true); + }); + + it('does not flag a substring false-positive for a banned referent', () => { + const clip = { ...passingContinueClip, prompt: `${passingContinueClip.prompt} She is against the wall, staged in the design.` }; + expect(lintClipPrompt(clip, { bible }).pass).toBe(true); + }); + + it.each(['no', 'without', 'never'])('flags the banned negative construction "%s"', (term) => { + const clip = { ...passingContinueClip, prompt: `${passingContinueClip.prompt} There is ${term} sound here.` }; + const { pass, reasons } = lintClipPrompt(clip, { bible }); + expect(pass).toBe(false); + expect(reasons.some((r) => r.includes(`banned negative construction "${term}"`))).toBe(true); + }); + + it('does not flag a substring false-positive for a banned negative', () => { + const clip = { ...passingContinueClip, prompt: `${passingContinueClip.prompt} A canoe glides into the hangar.` }; + expect(lintClipPrompt(clip, { bible }).pass).toBe(true); + }); + + it.each(['text', 'caption', 'overlay'])('flags the banned UI overlay term "%s"', (term) => { + const clip = { ...passingContinueClip, prompt: `${passingContinueClip.prompt} A ${term} appears on screen.` }; + const { pass, reasons } = lintClipPrompt(clip, { bible }); + expect(pass).toBe(false); + expect(reasons.some((r) => r.includes(`banned UI overlay language "${term}"`))).toBe(true); + }); + + it('does not flag a substring false-positive for a banned overlay term', () => { + const clip = { ...passingContinueClip, prompt: `${passingContinueClip.prompt} The context of the scene is tense.` }; + expect(lintClipPrompt(clip, { bible }).pass).toBe(true); + }); + + it('flags a prompt over the character length bound', () => { + const clip = { ...passingContinueClip, prompt: `${passingContinueClip.prompt} ${'x'.repeat(MAX_CLIP_PROMPT_LENGTH)}` }; + const { pass, reasons } = lintClipPrompt(clip, { bible }); + expect(pass).toBe(false); + expect(reasons.some((r) => r.includes('over the'))).toBe(true); + }); + + it('honors a custom maxLength', () => { + const clip = { ...passingContinueClip }; + expect(lintClipPrompt(clip, { bible, maxLength: 10 }).pass).toBe(false); + }); + + it('never throws on malformed clip fields — returns a failing result instead', () => { + expect(() => lintClipPrompt({ prompt: null, cutType: 'continue', framing: 42, references: 'not-an-array' }, { bible })).not.toThrow(); + const { pass } = lintClipPrompt({ prompt: null, cutType: 'continue', framing: 42, references: 'not-an-array' }, { bible }); + expect(pass).toBe(false); + expect(() => lintClipPrompt(null, { bible })).not.toThrow(); + expect(() => lintClipPrompt(undefined, { bible })).not.toThrow(); + }); + + it('does not flag a banned term abutting an underscore as a standalone word', () => { + const clip = { ...passingContinueClip, prompt: `${passingContinueClip.prompt} same_frame_id logged.` }; + expect(lintClipPrompt(clip, { bible }).pass).toBe(true); + }); +}); + +describe('lintClips', () => { + it('passes an array of chained clips and derives previousFraming from the preceding clip', () => { + const fresh = { + cutType: 'fresh', + framing: 'a wide establishing shot', + prompt: `${bible.cast.hero.descriptor} She stands in the hangar doorway.`, + references: [{ kind: 'cast', id: 'hero' }], + }; + const continued = { ...passingContinueClip }; + delete continued.previousFraming; + const { pass, results } = lintClips([fresh, continued], { bible }); + expect(pass).toBe(true); + expect(results).toHaveLength(2); + expect(results[0].index).toBe(0); + expect(results[1].index).toBe(1); + }); + + it('reports a failing clip among passing ones', () => { + const bad = { ...passingContinueClip, prompt: passingContinueClip.prompt.replace('Hard cut to a low tracking shot: ', '') }; + const { pass, results } = lintClips([passingContinueClip, bad], { bible }); + expect(pass).toBe(false); + expect(results[0].pass).toBe(true); + expect(results[1].pass).toBe(false); + }); + + it('never throws on a non-array clips argument', () => { + expect(() => lintClips(null, { bible })).not.toThrow(); + expect(() => lintClips(undefined, { bible })).not.toThrow(); + expect(() => lintClips('not-an-array', { bible })).not.toThrow(); + expect(() => lintClips(42, { bible })).not.toThrow(); + expect(lintClips(null, { bible })).toEqual({ pass: true, results: [] }); + expect(lintClips('not-an-array', { bible })).toEqual({ pass: true, results: [] }); + }); +});