Skip to content

[feat] agent: add H3 cinematography compiler, storyboard audit, contact-sheet QC, and shot revision - #342

Open
BeatSeat wants to merge 2 commits into
ThinkFlowLab:mainfrom
BeatSeat:feat/agent-h3-shot-orchestration
Open

BeatSeat wants to merge 2 commits into
ThinkFlowLab:mainfrom
BeatSeat:feat/agent-h3-shot-orchestration

Conversation

@BeatSeat

Copy link
Copy Markdown

Summary

  • Add opendirector.domain.h3_cinematography for canonical HappyHorse-1.0 (H3) shot compilation and deterministic linting across 7 canonical shot sizes, 20 camera motions in 12 motion families, natural-English speed/amplitude phrasing, 17n+5 frame-grid helpers, dialogue + soundscape linting, <Picture N> / <Video N> / <Audio N> ordinal alignment (accounting for embedded video audio tracks as wired in happyhorse_local), and multi-shot anti-homogenization sequence checks.
  • Upgrade asset.preview (AssetPreviewQuery, CreationAssets.preview, /projects/{id}/assets/{asset_id}/preview, and AgentTools) with videoScope="contact_sheet" (columns, rows), rendering timestamped multi-frame JPEG contact sheets (work/previews/{asset_id}-sheet-{columns}x{rows}.jpg) plus objective measuredFacts (meanFrameDelta, maxFrameDelta, firstLastSimilarity, suspectedFrozen, blackFrameIndices, blackFrameRatio, matchesH3Grid, hasAudio) so Agents are no longer limited to Frame-0 video previews while keeping technical measurements strictly separate from creative acceptance.
  • Add shared query storyboard.audit (StoryboardAuditQuery + opendirector.application.creation_storyboard.audit_project_storyboard) exposed uniformly across HTTP /queries, AgentTools, and opendirector video audit-storyboard to check cross-shot Subject continuity gaps, unrendered upstream reference keyframes, video continuation depth (> 3), unlinked timeline visual clips, live H3 prompt/ordinal regressions, sequence diversity, and pilot-gate pending video duration.
  • Upgrade opendirector video CLI (create with --last-frame, --continue-from, --aspect-ratio, --resolution, --audio, --title; compile-h3-prompt; lint-h3-prompt; lint-shot-plan; audit-storyboard; inspect-shot for local files or --asset UUID; and lineage-aware revise-shot resolving target timeline clips automatically via TimelineDocument.inputConnections and returning non-destructive canvas.materialize commands).
  • Add the h3-shot-director skill (SKILL.md, h3-vocabulary.md, shot-orchestration.md), regenerate src/contracts/creation.schema.json and src/contracts/generated.ts, and document the new workflows bilingually in docs/guides/cli.md.

Linked issue

Fixes #338
Refs #107, #191, #261, #263

  • Fixes #N or Refs #N is included, or no issue applies

Labels

Suggested labels (external contributor cannot apply labels directly): type:feat, area:agent, area:assets, area:timeline, area:docs, modality:video.

  • Labels applied

Test plan

  • Local:
    • python3 scripts/generate_contracts.py --check (passed)
    • PYTHONPATH=packages/core/src:apps/api/src lint-imports (passed: 137 files, 4 contracts kept, 0 broken)
    • PYTHONPATH=packages/core/src:apps/api/src pytest tests/test_h3_shot_orchestration.py -q (10 passed, 1 skipped because ffmpeg/ffprobe binaries are not installed in this container)
    • PYTHONPATH=packages/core/src:apps/api/src pytest tests/test_creation_contracts.py tests/test_skill_snapshot.py tests/test_agent_metadata.py tests/test_agent_gateway.py tests/test_cli.py -q (141 passed)
    • npm run check:i18n (passed: 3014 keys, 0 problems)
  • Left to CI:
    • npm run build (node_modules / tsc not installed in local environment)
    • Real ffmpeg/ffprobe execution (test_real_ffmpeg_shot_inspection in tests/test_h3_shot_orchestration.py)
    • Full Python/API/frontend/browser matrix and bundled OpenCode CLI checks in GitHub Actions

Evidence that it works

  • Media attached, or N/A because: Backend domain/application query, agent tool gateway, skill, contract schema, and CLI feature with no React canvas/UI component layout change; verified end-to-end via FastAPI ASGI + ToolGateway + real PIL contact-sheet JPEG rendering + CliRunner in tests/test_h3_shot_orchestration.py.

Provider verification

  • Real-provider calls are described honestly, or none were made: Verified against real local project storage, real PIL JPEG contact-sheet rendering, and deterministic media/model doubles (FakeHappyHorseAdapter / deterministic frame probe & extractor); no live GPU happyhorse_local worker or external commercial video API calls were executed in this environment.

Bilingual UI

  • Both locales are updated, or no user-facing copy changed (docs/guides/cli.md updated in both zh-CN and en-US; npm run check:i18n passed with 0 problems).

…ct-sheet QC, and shot revision

## summary

- Add `opendirector.domain.h3_cinematography` encoding MiniMax-H3's canonical
  cinematography vocabulary (`SHOT_SIZES`, 20 `CAMERA_MOTIONS` across 12
  families, natural-English amplitude/speed phrasing, dialogue and
  `Overall Soundscape` / `Non-Diegetic Music` structure, `17n+5` latent frame
  grid math, and `<Picture N>` / `<Video N>` / `<Audio N>` ordinal alignment
  with `fasth3.native_prompt` embedded video soundtracks), plus
  `compile_h3_prompt`, `lint_h3_prompt`, and anti-homogenization `lint_shot_plan`.
- Extend `asset.preview` (`videoScope="contact_sheet"`) and
  `opendirector.media.shot_inspection` to render timestamped JPEG contact sheets
  for registered project video assets and local files while reporting
  deterministic `measuredFacts` (`meanFrameDelta`, `maxFrameDelta`,
  `suspectedFrozen`, `blackFrameIndices`, `matchesH3Grid`, `hasAudio`) separate
  from Agent creative review.
- Add `storyboard.audit` (`opendirector.application.creation_storyboard`) across
  HTTP `/queries`, Agent `query`, and `opendirector video audit-storyboard` to
  audit cross-shot `Subject` anchor continuity, upstream keyframe readiness,
  continuation chain depth, unlinked timeline clips, and pilot-gate budgets on
  live `ProjectSnapshot` state.
- Extend `opendirector video` (`create --last-frame/--continue-from`,
  `compile-h3-prompt`, `lint-h3-prompt`, `lint-shot-plan`, `audit-storyboard`,
  `inspect-shot --asset`, and lineage-aware `revise-shot`), ship the built-in
  `h3-shot-director` Agent skill, regenerate creation contracts, and update
  bilingual CLI documentation.

## validation

- `python scripts/generate_contracts.py --check`
- `lint-imports`
- `pytest tests/test_h3_shot_orchestration.py tests/test_creation_contracts.py tests/test_skill_snapshot.py tests/test_agent_metadata.py tests/test_agent_gateway.py tests/test_cli.py -q`
- `npm run check:i18n`

Fixes: ThinkFlowLab#338
Refs: ThinkFlowLab#107, ThinkFlowLab#191, ThinkFlowLab#261, ThinkFlowLab#263
@github-actions github-actions Bot added area:agent Agent runtime, assistant chat and tools area:docs Documentation / 文档 labels Nov 15, 2026
@github-actions
github-actions Bot requested review from FayeSpica and wuzhf9 November 15, 2026 09:27
…contracts

## summary

- Detect contradictory single-shot camera motions and CLI/SD pseudo-parameter
  tags outside `<d>...</d>` dialogue in `lint_h3_prompt`, extract compiled H3
  camera motion verb phrases in `extract_prompt_shot_features`, and publish
  `vllmOmniFrames` alongside `gridFrames` in `h3_frame_grid`.
- Unify H3 reference-source resolution across `storyboard.audit` and
  `opendirector video lint-h3-prompt` in `resolve_h3_spec_reference_sources`,
  preserving `promptContent` mention order, `first_frame` `context_only`
  filtering, and `assetId` deduplication.
- Respect group-inherited canvas visibility in `audit_project_storyboard`,
  enforce mutually exclusive scopes and unique `specIds` in
  `StoryboardAuditQuery`, and regenerate TypeScript and JSON contracts.
- Reuse `TIMELINE_DURATION_EPSILON`, preserve existing `EditorInputConnection`
  IDs on repeated slot revisions, standardize domain error messages, and gate
  real FFmpeg inspection with `@requires_ffmpeg()`.

## validation

- `.venv/bin/python3 scripts/generate_contracts.py --check`
- `PYTHONPATH=packages/core/src:apps/api/src .venv/bin/lint-imports`
- `PYTHONPATH=packages/core/src:apps/api/src .venv/bin/python3 -m pytest tests/test_h3_shot_orchestration.py tests/test_creation_contracts.py tests/test_skill_snapshot.py tests/test_agent_metadata.py tests/test_agent_gateway.py tests/test_cli.py -q`
- `npm run check:i18n`

Refs: ThinkFlowLab#338
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent runtime, assistant chat and tools area:docs Documentation / 文档

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feat] agent: 支持 H3 专业分镜编译、工程分镜连续性审计、视频多帧质检预览与单镜局部返工

1 participant