Skip to content

Commit e820b58

Browse files
myxamediyarclaudewaleedlatif1
authored
feat(video): accept a start frame image for fal.ai video models (#9761)
* feat(video): accept a start frame image for fal.ai video models Fal models with an image-to-video sibling endpoint now take an optional start frame. When one is attached, requests go to the image-to-video endpoint with the image as a data URI in the model's image field, and the block's settings dropdowns follow that endpoint's options. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(video): keep fal settings enabled without a start frame - Drop the optional start frame fields from `dependsOn`: an array there disables the field until every entry is filled, which greyed out the settings for text-only runs. Options and conditions already re-read all block values. - Pass the run's abort signal through to the start frame download. - The fal tool now offers `visualReference` to agents for models with an image-to-video endpoint; update the provider selection test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(video): send the model default for fal settings a frame change invalidates Attaching or removing a start frame switches a fal model between its text-to-video and image-to-video endpoints, and the settings picked for one can be invalid on the other (Seedance 2.5 takes 16:9 from a prompt but only auto from a frame). The V3 block now drops any stored setting the selected endpoint does not accept, so the run uses the model default instead of failing validation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(video): drop only settings a start frame change made stale Runway models also switch endpoints when a start frame is attached, and their stored settings can be valid on one and not the other (Kling 4 takes 1920:1080 from a prompt but only auto ratios from a frame). Both fal and direct providers now drop a setting only when the selected endpoint rejects it and the model's other endpoint accepts it. Values neither endpoint lists still reach the request builder, which maps legacy Runway ratios such as 16:9 and rejects anything else. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(video): check fal start frame provenance before sending it to the model A fal start frame now passes the same opaque model-input provenance check as a Runway reference before its bytes reach the model. * fix(video): pick fal settings from the active start frame input The Reference Image is a basic upload / advanced reference pair, and the serializer sends only the active member. The fal settings fields counted a file left in the inactive member too, so switching to an empty advanced input kept showing the start frame endpoint's settings while the run called the prompt endpoint. Resolve the active member with `resolveActiveCanonicalValue`, and pass `__canonicalModes` to dropdown option functions as the editor already does for conditions, so a field's visibility and its options agree. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(video): reject a reference file that is not an image before submitting An agent can pass any file as the start frame or Runway reference, and nothing checked its type before the paid request. Validation now returns a 400 for a non-image `visualReference` before the file is downloaded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Waleed Latif <walif6@gmail.com>
1 parent 07a8d0c commit e820b58

17 files changed

Lines changed: 1304 additions & 189 deletions

File tree

‎apps/sim/app/workspace/[workspaceId]/w/[workflowId]/components/panel/components/editor/components/sub-block/components/dropdown/dropdown.tsx‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -164,15 +164,19 @@ export const Dropdown = memo(function Dropdown({
164164
}, [multiSelect, value])
165165

166166
// Derived option lists read the block's own values (a model's valid reasoning efforts);
167-
// `dependsOn` already re-renders this control when one of those siblings changes.
167+
// `dependsOn` already re-renders this control when one of those siblings changes. They also get
168+
// `__canonicalModes`, as conditions do, so both read the same active member of a canonical pair.
168169
const activeWorkflowId = useWorkflowRegistry((state) => state.activeWorkflowId)
169170
const blockValues = useSubBlockStore((state) =>
170171
activeWorkflowId ? state.workflowValues[activeWorkflowId]?.[blockId] : undefined
171172
)
172173
const evaluatedOptions = useMemo(() => {
173-
if (typeof options === 'function') return options({ values: blockValues ?? {} })
174-
return options ?? EMPTY_OPTIONS
175-
}, [options, blockValues])
174+
if (typeof options !== 'function') return options ?? EMPTY_OPTIONS
175+
const values = blockValues ?? {}
176+
return options({
177+
values: canonicalModes ? { ...values, __canonicalModes: canonicalModes } : values,
178+
})
179+
}, [options, blockValues, canonicalModes])
176180

177181
const reactiveOptions = useMemo(
178182
() =>
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
import { describe, expect, it } from 'vitest'
2+
import { evaluateSubBlockCondition } from '@/lib/workflows/subblocks/visibility'
3+
import { VideoGeneratorV3Block } from '@/blocks/blocks/video_generator'
4+
5+
const frame = { id: 'f', name: 'frame.png', url: '', size: 3, type: 'image/png', key: 'k' }
6+
7+
function aspectRatioField() {
8+
const field = VideoGeneratorV3Block.subBlocks.find((subBlock) => subBlock.id === 'falAspectRatio')
9+
if (!field) throw new Error('Video Generator must expose a fal aspect ratio field')
10+
return field
11+
}
12+
13+
function optionIds(values: Record<string, unknown>) {
14+
const { options } = aspectRatioField()
15+
if (typeof options !== 'function') throw new Error('Fal aspect ratio options must be derived')
16+
return options({ values }).map((option) => (typeof option === 'string' ? option : option.id))
17+
}
18+
19+
describe('Video Generator fal start frame', () => {
20+
// Happy Horse takes an aspect ratio from a prompt but not from a start frame.
21+
const base = { provider: 'falai', model: 'alibaba/happy-horse/text-to-video' }
22+
23+
it('reads the prompt endpoint when the stored upload is in the inactive mode', () => {
24+
const values = {
25+
...base,
26+
visualReferenceUpload: frame,
27+
visualReferenceInput: '',
28+
__canonicalModes: { visualReference: 'advanced' },
29+
}
30+
31+
expect(evaluateSubBlockCondition(aspectRatioField().condition, values)).toBe(true)
32+
expect(optionIds(values)).toContain('16:9')
33+
})
34+
35+
it('reads the start frame endpoint when the upload is the active mode', () => {
36+
const values = { ...base, visualReferenceUpload: frame }
37+
38+
expect(evaluateSubBlockCondition(aspectRatioField().condition, values)).toBe(false)
39+
expect(optionIds(values)).not.toContain('16:9')
40+
})
41+
})

‎apps/sim/blocks/blocks/video_generator.ts‎

Lines changed: 162 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,21 @@
1+
import { isRecordLike } from '@sim/utils/object'
12
import { VideoIcon } from '@/components/icons'
23
import {
34
directVideoDefaults,
45
directVideoModels,
56
getDirectVideoModel,
67
} from '@/lib/media/direct-video-catalog'
7-
import { falVideoModels, getFalVideoModel } from '@/lib/media/fal-video-catalog'
8+
import {
9+
type FalVideoSettings,
10+
falVideoModels,
11+
getFalVideoModel,
12+
getFalVideoVariant,
13+
} from '@/lib/media/fal-video-catalog'
14+
import {
15+
type CanonicalGroup,
16+
type CanonicalModeOverrides,
17+
resolveActiveCanonicalValue,
18+
} from '@/lib/workflows/subblocks/visibility'
819
import { AuthMode, type BlockConfig, IntegrationType, type SubBlockConfig } from '@/blocks/types'
920
import { normalizeFileInput, parseOptionalBooleanInput } from '@/blocks/utils'
1021

@@ -42,40 +53,69 @@ const FALAI_AUDIO_DEFAULT_OFF_MODELS = ['kling-o3-pro', 'kling-o3-4k']
4253
const FAL_MODEL_DEFAULT = { id: 'default', label: 'Model default' } as const
4354
const endpointModels = falVideoModels.filter((model) => !model.legacyId)
4455

45-
function tuningField(
46-
id: string,
47-
title: string,
48-
key: 'durations' | 'aspectRatios' | 'resolutions' | 'audioField' | 'promptOptimizer'
49-
): SubBlockConfig {
56+
// Never list these optional fields in `dependsOn`: an array there disables the field until every
57+
// entry has a value. Options and conditions already re-read all block values on any change.
58+
const START_FRAME_FIELDS = ['visualReferenceInput', 'visualReferenceUpload']
59+
const HIDDEN = { field: 'provider', value: [] }
60+
61+
const START_FRAME_GROUP: CanonicalGroup = {
62+
canonicalId: 'visualReference',
63+
basicId: 'visualReferenceUpload',
64+
advancedIds: ['visualReferenceInput'],
65+
}
66+
67+
/**
68+
* Whether the run will send a start frame: the serialized `visualReference`, or else only the
69+
* upload/reference member in the active mode, since the serializer drops the inactive one.
70+
*/
71+
function hasStartFrame(values?: Record<string, unknown>) {
72+
if (!values) return false
73+
const overrides = isRecordLike(values.__canonicalModes)
74+
? (values.__canonicalModes as CanonicalModeOverrides)
75+
: undefined
76+
const reference =
77+
values.visualReference ?? resolveActiveCanonicalValue(START_FRAME_GROUP, values, overrides)
78+
return Array.isArray(reference) ? reference.length > 0 : Boolean(reference)
79+
}
80+
81+
/** Settings of the endpoint a fal run will call: image-to-video once a start frame is attached. */
82+
function selectedFalSettings(values?: Record<string, unknown>) {
83+
const model = getFalVideoModel(values?.model)
84+
return getFalVideoVariant(values?.model, Boolean(model?.image) && hasStartFrame(values))
85+
}
86+
87+
type FalSettingKey = 'durations' | 'aspectRatios' | 'resolutions' | 'audioField' | 'promptOptimizer'
88+
89+
function tuningField(id: string, title: string, key: FalSettingKey): SubBlockConfig {
5090
return {
5191
id,
5292
title,
5393
type: 'dropdown',
5494
value: () => FAL_MODEL_DEFAULT.id,
55-
condition: {
56-
field: 'provider',
57-
value: 'falai',
58-
and: {
59-
field: 'model',
60-
value: endpointModels.filter((model) => model[key]).map((model) => model.modelId),
61-
},
95+
condition: (values) => {
96+
const model = getFalVideoModel(values?.model)
97+
return model && !model.legacyId && selectedFalSettings(values)?.[key]
98+
? { field: 'provider', value: 'falai' }
99+
: HIDDEN
62100
},
63101
dependsOn: ['model'],
64102
options: (params) => {
65103
if (!params) {
66-
const values = endpointModels.flatMap((model) => {
67-
const choices = model[key]
68-
return Array.isArray(choices)
69-
? choices.map((value) =>
70-
key === 'durations' ? String(Number.parseFloat(String(value))) : String(value)
71-
)
72-
: choices
73-
? ['true', 'false']
74-
: []
75-
})
104+
const values = endpointModels.flatMap((model) =>
105+
[model, model.image].flatMap((settings) => {
106+
const choices = settings?.[key]
107+
return Array.isArray(choices)
108+
? choices.map((value) =>
109+
key === 'durations' ? String(Number.parseFloat(String(value))) : String(value)
110+
)
111+
: choices
112+
? ['true', 'false']
113+
: []
114+
})
115+
)
76116
return [FAL_MODEL_DEFAULT, ...[...new Set(values)].map((id) => ({ id, label: id }))]
77117
}
78-
const model = getFalVideoModel(params?.values.model)
118+
const model = selectedFalSettings(params?.values)
79119
if (!model?.[key]) return []
80120
if (key === 'audioField' || key === 'promptOptimizer') {
81121
return [FAL_MODEL_DEFAULT, { id: 'true', label: 'On' }, { id: 'false', label: 'Off' }]
@@ -109,6 +149,49 @@ const falVideoFields = [
109149
tuningField('falPromptOptimizer', 'Prompt Optimizer', 'promptOptimizer'),
110150
]
111151

152+
/** Settings fields that models with a legacy ID keep, and the catalog setting each one sets. */
153+
const LEGACY_FAL_SETTING_KEYS: Record<string, keyof FalVideoSettings> = {
154+
duration: 'durations',
155+
aspectRatio: 'aspectRatios',
156+
resolution: 'resolutions',
157+
generateAudio: 'audioField',
158+
promptOptimizer: 'promptOptimizer',
159+
}
160+
161+
/**
162+
* Drops a stored setting the endpoint a run calls rejects but the model's other endpoint accepts,
163+
* so the model default applies. Attaching or removing a start frame switches endpoints without
164+
* clearing settings picked for the other one: Seedance 2.5 takes `16:9` from a prompt but only
165+
* `auto` from a start frame. Any other value passes through to the request's own legacy mapping
166+
* and validation.
167+
*/
168+
function dropStaleSetting(
169+
value: unknown,
170+
allowed: readonly (number | string)[] | undefined,
171+
otherAllowed: readonly (number | string)[] | undefined,
172+
numeric = false
173+
) {
174+
const normalize = (choice: unknown) => (numeric ? Number.parseFloat(String(choice)) : choice)
175+
const accepts = (choices?: readonly (number | string)[]) =>
176+
choices?.some((choice) => normalize(choice) === normalize(value)) ?? false
177+
return value !== undefined && !accepts(allowed) && accepts(otherAllowed) ? undefined : value
178+
}
179+
180+
function keepFalSettingsAllowed(
181+
params: Record<string, unknown>,
182+
settings: FalVideoSettings | undefined,
183+
other: FalVideoSettings | undefined
184+
) {
185+
return {
186+
...params,
187+
duration: dropStaleSetting(params.duration, settings?.durations, other?.durations, true),
188+
aspectRatio: dropStaleSetting(params.aspectRatio, settings?.aspectRatios, other?.aspectRatios),
189+
resolution: dropStaleSetting(params.resolution, settings?.resolutions, other?.resolutions),
190+
generateAudio: settings?.audioField ? params.generateAudio : undefined,
191+
promptOptimizer: settings?.promptOptimizer ? params.promptOptimizer : undefined,
192+
}
193+
}
194+
112195
function withFalVideoCatalog(subBlocks: SubBlockConfig[]): SubBlockConfig[] {
113196
return subBlocks.flatMap((subBlock) => {
114197
const condition = subBlock.condition
@@ -141,17 +224,23 @@ function withFalVideoCatalog(subBlocks: SubBlockConfig[]): SubBlockConfig[] {
141224
]
142225
}
143226
if (subBlock.id === 'apiKey' && !condition.not) return [...falVideoFields, subBlock]
227+
const key = LEGACY_FAL_SETTING_KEYS[subBlock.id]
228+
if (key)
229+
return [
230+
{
231+
...subBlock,
232+
condition: (values) => {
233+
const image = getFalVideoModel(values?.model)?.image
234+
return image && !image[key] && hasStartFrame(values) ? HIDDEN : condition
235+
},
236+
},
237+
]
144238
return [subBlock]
145239
})
146240
}
147241

148242
function selectedDirectModel(values?: Record<string, unknown>) {
149-
const withImage = [
150-
values?.visualReference,
151-
values?.visualReferenceInput,
152-
values?.visualReferenceUpload,
153-
].some((reference) => (Array.isArray(reference) ? reference.length > 0 : Boolean(reference)))
154-
return getDirectVideoModel(values?.provider, values?.model, withImage)
243+
return getDirectVideoModel(values?.provider, values?.model, hasStartFrame(values))
155244
}
156245

157246
const directVideoFields = (
@@ -168,7 +257,7 @@ const directVideoFields = (
168257
title,
169258
type: key === 'promptOptimizer' ? 'switch' : 'dropdown',
170259
value: key === 'promptOptimizer' ? undefined : () => FAL_MODEL_DEFAULT.id,
171-
dependsOn: ['provider', 'model', 'visualReferenceInput', 'visualReferenceUpload'],
260+
dependsOn: ['provider', 'model'],
172261
condition: (values) => ({
173262
field: 'provider',
174263
value: selectedDirectModel(values)?.settings?.[key] ? [String(values?.provider)] : [],
@@ -302,9 +391,18 @@ function withDirectVideoCatalog(subBlocks: SubBlockConfig[]): SubBlockConfig[] {
302391
dependsOn: ['provider', 'model'],
303392
},
304393
]
305-
if (field.id === 'visualReferenceUpload' || field.id === 'visualReferenceInput')
394+
if (START_FRAME_FIELDS.includes(field.id))
306395
return [
307-
{ ...field, required: { field: 'model', value: ['gen4_turbo', 'gen-4', 'gen-4-turbo'] } },
396+
{
397+
...field,
398+
condition: (values) =>
399+
values?.provider === 'runway' ||
400+
(values?.provider === 'falai' && getFalVideoModel(values.model)?.image)
401+
? { field: 'provider', value: ['runway', 'falai'] }
402+
: HIDDEN,
403+
dependsOn: ['provider', 'model'],
404+
required: { field: 'model', value: ['gen4_turbo', 'gen-4', 'gen-4-turbo'] },
405+
},
308406
]
309407
return [field]
310408
})
@@ -1950,28 +2048,54 @@ export const VideoGeneratorV3Block: BlockConfig = {
19502048
? 'video_higgsfield'
19512049
: (VideoGeneratorV2Block.tools.config?.tool(params) ?? 'video_runway'),
19522050
params: (params) => {
1953-
if (params.provider === 'falai')
1954-
return VideoGeneratorV2Block.tools.config?.params?.(params) ?? params
2051+
if (params.provider === 'falai') {
2052+
const falParams = VideoGeneratorV2Block.tools.config?.params?.(params) ?? params
2053+
const model = getFalVideoModel(params.model)
2054+
if (!model) return { ...falParams, visualReference: undefined }
2055+
const withImage = Boolean(model.image && falParams.visualReference)
2056+
return {
2057+
...keepFalSettingsAllowed(
2058+
falParams,
2059+
getFalVideoVariant(params.model, withImage),
2060+
getFalVideoVariant(params.model, !withImage)
2061+
),
2062+
visualReference: withImage ? falParams.visualReference : undefined,
2063+
}
2064+
}
19552065
const visualReference =
19562066
params.provider === 'runway'
19572067
? normalizeFileInput(params.visualReference, { single: true })
19582068
: undefined
19592069
const model = getDirectVideoModel(params.provider, params.model, Boolean(visualReference))
2070+
const other = getDirectVideoModel(params.provider, params.model, !visualReference)?.settings
19602071
const setting = (key: string, supported: unknown, fallbackKey?: string) => {
19612072
if (!supported) return undefined
19622073
const value = params[key] ?? (fallbackKey ? params[fallbackKey] : undefined)
19632074
return value === FAL_MODEL_DEFAULT.id ? undefined : value
19642075
}
1965-
const duration = setting('duration', model?.settings?.durations)
2076+
const duration = dropStaleSetting(
2077+
setting('duration', model?.settings?.durations),
2078+
model?.settings?.durations,
2079+
other?.durations,
2080+
true
2081+
)
19662082
return {
19672083
provider: params.provider,
19682084
model: params.model,
19692085
apiKey: params.apiKey,
19702086
prompt: params.prompt,
19712087
visualReference,
19722088
duration: duration ? Number(duration) : undefined,
1973-
aspectRatio: setting('aspectRatio', model?.settings?.aspectRatios),
1974-
resolution: setting('resolution', model?.settings?.resolutions),
2089+
aspectRatio: dropStaleSetting(
2090+
setting('aspectRatio', model?.settings?.aspectRatios),
2091+
model?.settings?.aspectRatios,
2092+
other?.aspectRatios
2093+
),
2094+
resolution: dropStaleSetting(
2095+
setting('resolution', model?.settings?.resolutions),
2096+
model?.settings?.resolutions,
2097+
other?.resolutions
2098+
),
19752099
generateAudio: parseOptionalBooleanInput(
19762100
setting('directGenerateAudio', model?.settings?.audio, 'generateAudio')
19772101
),

0 commit comments

Comments
 (0)