Skip to content

feat: support exact 17k+5 frame durations for vLLM-Omni - #334

Open
FayeSpica wants to merge 3 commits into
ThinkFlowLab:mainfrom
FayeSpica:feat/vllm-omni-exact-duration
Open

FayeSpica wants to merge 3 commits into
ThinkFlowLab:mainfrom
FayeSpica:feat/vllm-omni-exact-duration

Conversation

@FayeSpica

@FayeSpica FayeSpica commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Allow users and agents to request exact segment lengths through OpenDirector's vLLM-Omni video UI and CLI. For H3 at 24 fps, exactly 107 frames (107 = 17 × 6 + 5) requires 107 / 24 = 4.458333333333333 seconds; whole-second inputs cannot express this length. Preserve the supplied decimal duration without rounding it to whole seconds.

  • Preserve durationSeconds through validation, project storage and request serialization; expose support through inputRules.fractionalDuration.
  • Retain the MiniMax-H3 adapter's existing 17k+5 frame alignment: an already aligned request maps to exactly its target frame count (for example, 107 frames), while other durations map to the nearest supported length. Other video protocols keep their discrete duration choices.
  • Update CLI guidance and both language guides. No frame-count input mode is introduced.

Linked issue

No linked issue.

Labels

type:feat, area:provider, area:canvas, modality:video.

Test plan

  • python -m pytest tests/test_creation_parameters.py tests/test_rule_facts.py tests/test_vllm_omni_h3.py -q: 58 passed on 1acbdcca.
  • Duration-rule and resolution-choice frontend tests: 74 passed on the same implementation before the additional 107-frame test.
  • Build, contract generation and i18n checks passed. Isolated browser checks cover decimal entry and persistence in both UI languages.
  • Full CI remains pending.

Evidence that it works

Mock-provider tests verify that 107 / 24 seconds is serialized as exactly num_frames=107, satisfying 17k+5. They also preserve 5.25 and 6.583333 seconds through accepted jobs, request serialization and project reopening.

Captured from the running app on port 18000: 4.458333333333333 seconds (107 / 24) entered in both languages and confirmed in the persisted project. No generation was submitted for these screenshots.

中文 English
小数秒时长输入 Fractional duration input

Provider verification

Validation used a model double and real local media files. Successful real-provider generation and decoded output frame counts remain unverified, including FastH3's handling of extra_params.duration.

Bilingual UI

zh-CN and en-US hints, CLI guidance and both user guides updated. Both UI languages checked; npm run check:i18n passed.

Signed-off-by: Weiming Liao <liaowm5@gmail.com>
@FayeSpica FayeSpica added type:feat New capability or behaviour area:canvas Canvas, nodes and direct manipulation area:provider Model providers and generation protocols modality:video Video generation, editing or rendering labels Nov 15, 2026
Signed-off-by: Weiming Liao <liaowm5@gmail.com>
@FayeSpica
FayeSpica marked this pull request as ready for review November 15, 2026 06:01
Signed-off-by: Weiming Liao <liaowm5@gmail.com>
@github-actions github-actions Bot added the area:docs Documentation / 文档 label Nov 15, 2026
@github-actions
github-actions Bot requested a review from wuzhf9 November 15, 2026 06:14

@hsliuustc0106 hsliuustc0106 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: Approve with follow-ups

整体设计正确、CI 全绿(head 31ff8a1):duration_matches 单一判定前后端同步、InputRules.fractionalDuration 经 exclude_if 序列化、H3 走 h3_frames 出 num_frames。以下按 file:symbol 逐条标注,从「先确认」到 Nit 排列:

  • P1 先确认范围:fractionalDuration 对所有 vllm_omni 视频置真(非仅 H3)。
  • P2 需验证:FastH3 小数路径未端到端验证,form['seconds'] 对 fast 是死代码。
  • P3 边界:接受区间与 H3 可达帧不对齐。
  • P3/Nit:时长控件 UI 与全精度序列化小问题。

if rules['profile'] == 'kling-official':
validate_kling_outputs(capabilities, config['modelId'])
if config.get('kind') == 'video' and config.get('protocol') == 'vllm_omni':
rules['fractionalDuration'] = True

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

范围建议收紧到 H3。这里只要 kind=='video' 且 protocol=='vllm_omni' 就把 fractionalDuration 置真,于是 standard-video(通用 vLLM-Omni 视频)和 fasth3-omni 也会走区间判定。配合 rule_facts.duration_matches 的 min(durations) <= x <= max(durations),一个只声明离散时长的普通 vLLM-Omni 模型(例如 durations 为 [4,8])会接受并原样下发 4–8 之间任意小数,前端时长控件也会退化成自由滑杆;本 PR 的浏览器用例正是把稀疏时长 fixture 从 vllm_omni 改到 volcengine 才保住离散行为。若非确认所有受支持的 vllm_omni 视频模型都能处理任意秒数,建议改成 rules['profile'] in {'vllm-omni-h3','fasth3-omni'} 时才置真,或引入显式的每模型能力位。

parameters = task['parameters']
form = {'model': config['modelId'], 'prompt': prepared['prompt']}
if 'duration' in parameters: form['seconds'] = format(parameters['duration'], 'g')
if 'duration' in parameters:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FastH3 的小数时长实际不会走这里设置的 form['seconds']:本块的写入会紧接着被 fast 分支的 form.update({'seconds': str(parameters.get('seconds','5')), ...}) 覆盖,而视频 task 参数里没有 'seconds' 键(只有 'duration',见 creation_generation.py 的 _parameters),因此 FastH3 恒定提交 seconds=5,真实时长只经 extra_params.duration(float,见第 191 行)下发。它是否生效取决于服务端是否读 extra_params.duration,PR 描述也承认该路径未验证;而 UI 已对 FastH3 打开小数输入。建议补一条真实 FastH3 的 wire/集成断言或明确标注此限制;第 184 行对 standard-video 有效,但对 fast 分支是死代码,可考虑仅在非 fast 时执行以更清晰。


def duration_matches(seconds: float, durations: list[int] | tuple[int, ...], fractional: bool = False) -> bool:
if fractional:
return bool(durations) and min(durations) <= seconds <= max(durations)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

区间判定只取 min/max,和 H3 实际可达帧不一致:h3_frames(vllm_omni.py)把阶数 steps 夹在 [6,20],对应 107–345 帧、24fps 即 4.458–14.375s;但这里按配置 durations 的 [min,max](H3 预设为 4–15)放行,于是端点 4s/15s 会分别被静默输出为 4.458s/14.375s,用户选区间端点时实际时长偏差最大。建议要么把可接受区间与实际可达帧对齐,要么在 UI 回显 h3_frames 对齐后的实际时长(目前仅靠用户指南提示「拼接前核对实际时长」)。

if (event.buttons === 1) pointerHint(event);
}}
onKeyDown={(event) => {
if (fractional) return;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

小数时长下有几处小问题:(1) 第 114 行 if (fractional) return; 直接禁用了滑杆的 Home/End/方向键步进;(2) 第 103/155 行 disabled={disabled || values.length <= 1} 在单值配置下会连数字输入一起禁掉,用户无法录入小数;(3) 第 54–61 行 hint() 在取值恰为 min/max 时仍返回 limit 提示(此时其实合法)。建议:fractional 时保留键盘步进或统一交给数字输入;单值时仍允许数字输入(仅禁用滑杆);边界取值不再提示 limit。

if 'duration' in parameters: form['seconds'] = format(parameters['duration'], 'g')
if 'duration' in parameters:
seconds = parameters['duration']
form['seconds'] = str(int(seconds)) if float(seconds).is_integer() else str(seconds)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

form['seconds'] 从 format(seconds,'g')(6 位有效数字)改为 str(seconds) 后会输出 4.458333333333333 这类全精度串——这正是精确性所需(H3 的精确帧走 num_frames,不受影响),但标准 vLLM-Omni 服务端能否解析全精度小数尚未验证。建议确认后端对全精度浮点的接受度,必要时固定到某个足够精度。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:canvas Canvas, nodes and direct manipulation area:docs Documentation / 文档 area:provider Model providers and generation protocols modality:video Video generation, editing or rendering type:feat New capability or behaviour

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants