tools: validate the WebUI catalog in the sync check - #376
Closed
CryptVenture wants to merge 5 commits into
Closed
Conversation
Sixteen specs described a request surface the engine does not implement, or
failed to describe one it does. Each change below was checked against the C++
that reads the option.
Declarations that were wrong
- voxcpm1: num_inference_steps 50 -> 10, max_tokens 1024 -> 4096, min_tokens
0 -> 2, retry_badcase_max_times 2 -> 3, per voxcpm1/types.h:13-19.
- fish_audio, outetts: text_chunk_mode default "word_budget" -> "default".
The shared parser in framework/text/chunking.cpp throws on "word_budget",
so the documented default was un-passable. Both move from the
text_chunk_mode_full preset to explicit values, because "default" is not
currently in that preset; a separate change corrects the preset itself.
- midashenglm_gen: seed min -1 -> 0. parse_u32_option rejects any leading
minus, so a spec-legal -1 was a runtime error, and the "-1 selects a random
seed" claim went with it.
- dramabox: five descriptions claimed their default came from config.json.
dramabox/assets.cpp reads config.json for architecture only.
- irodori_tts: text_chunk_mode values widened to the four the shared parser
accepts.
Declarations for options nothing reads
- granite5asr: language. The session hard-codes "en" and never reads it.
- firered_audio: session option firered_audio.helper_graph_arena_mb. Only
the fireredtts3-prefixed key exists, and FireRedAudio validates session
options strictly, so this one actively lied.
Options the engine reads and no spec declared
- minimax_music3: the five options added with the performance pack --
flow_uncond_interval, flow_uncond_warmup, ensemble_takes,
ensemble_prefix_frames, flow_chunk_hop_frames -- plus the
minimax_music3.pipeline_overlap session option. ensemble_takes returns
several named outputs and had no way to be requested.
- minimax_h3: first_block_cache_start_percent and _end_percent, the primary
window whose _sigma override was already declared.
- sense_asr: "vad" added to the audio_chunk_mode enum; the session
implements it.
- parakeet_tdt: language. Every other strict ASR family declares it, and it
is a field of the OpenAI-compatible transcription contract, so a per-model
rejection of it is a protocol violation. Documented as accepted and
ignored, since Parakeet reads only keep_language_tags.
- qwen3_forced_aligner, miocodec: options.session filled in with the seven
and six prefixed session options their sessions actually read.
Required flags that lie
- heartmula: lyrics and tags marked required. heartmula/session.cpp:437-442
throws when either is empty, but the specs said optional, so a client had
no way to know before the engine did.
Deliberately not changed, with reasons recorded in the descriptions:
fish_audio and outetts require reference_text only when reference audio is
present, and the schema's `required` is a plain bool with no conditional form,
so marking them required would break plain TTS with a built-in voice.
mms_forced_aligner keeps return_timestamps: it is never read, but the family
validates strictly and the CLI's --words-out injects the key, so deleting it
would break forced alignment from the command line.
The WebUI bundle is regenerated because catalog.ts inlines model_specs at
frontend build time, so a spec edit does not reach users until it is rebuilt.
Validation:
python3 tools/check_loader_catalog_sync.py # ok, in sync
cd webui/native && npm run build
ctest -R model_spec_system_test # passes
Backend tested: Metal (Apple M4 Max).
Known limitations: the corrected declarations were verified by reading the
option-parsing code, not by running each of the sixteen families -- most have
no package installed here. No spec gained schema_version: the five that
declare options without it would need dependencies and load blocks they do not
have, and switching them to full v1 validation is not verifiable without
running every affected model.
A third of the shipped packages could not be installed from the native WebUI,
four model families were invisible, and several controls sent values the
engine never asked for. Measured by executing catalog.ts against the real
specs: 128 of 188 packages were reachable before, 161 after, with none lost.
Package resolution (catalog.ts, types.ts)
- Group install choices by target_directory instead of a hardcoded
six-family allowlist. Packages that are precision variants of one model
share a directory; packages that are different models do not. The
allowlist had to be edited by hand for every new family and left every
other family with at most one q8 and one fp16 choice.
- Extend the precision-suffix list to q2_k..q6_k, q4_0, q5_0 and f32. It
covered only q8/f16/bf16/safetensors/orig, so an entry whose download_id
ended in an uncovered suffix narrowed to that single package and hid its
siblings -- which is why PersonaPlex offered only Q4_K while its own spec
marks Q8_0 default.
- Honour ui.recommended_package. Every spec has one, all 64 resolve to a
real package, and nothing read the field.
- Label packages so that no two buttons of one entry read alike, and so that
the text fits the button. The build alone ("Q8_0", "Q4_K ConvRot") is
enough for nearly every entry; where several models share a target
directory and collide on precision, the label keeps what the display names
do not share ("Turbo BF16", "XL SFT BF16"), and falls back to the full
display name if even that is ambiguous. The full name stays on the title
and the aria-label, and both button lines clamp inside the border.
- Match relatedness on target_directory instead of an id prefix. The prefix
test pulled the IndexTTS2.5 packages into the IndexTTS2 entry, whose fp16
slot was then decided by spec-file array order.
- An entry with no installable package is kept and flagged instead of
silently vanishing from the whole UI.
Coverage (models_catalog.json, 84 -> 98 entries)
- Add moss_voicegen, which has an Apache-2.0 licence, an active loader, a
published GGUF package and a matching UI tab, and no way to be selected.
- Add f5_tts (Habibi unified plus seven Arabic dialect checkpoints), with
its non-commercial licence named in the input hint.
- Add the four non-English PocketTTS checkpoints and DotTTS Edit, all
shipped and none reachable.
- Point personaplex and supertonic at the packages their own specs mark
default and recommended.
- Rewrite every download_id to an exact packages[].id. 46 named a value that
was not a package id and resolved only through a compatibility shim.
- Regenerate every path from the resolved package. 40 named pre-GGUF
directories no package installs; granite5asr was missing the models/
prefix entirely.
- Retask the five entries that throw without reference audio from tts to
clon, so the form requires the reference the engine demands.
Request wiring (+page.svelte, Arena.svelte)
- Stop sending a fixed max_tokens of 1024 on every request. Engine ceilings
range from 300 to 4096; this halved Qwen3-TTS, quartered MOSS-TTS-Local
and capped VibeVoice, whose default is no explicit cap. Blank now means
the model's own limit.
- Default the seed to -1, which the field's own label already documents as
random. A fixed 1234 made fourteen families whose engine default is a
fresh random seed return identical audio every run.
- Render segments, speaker turns and word timings as a table with SRT and
VTT export. This data was already produced and discarded.
- Offer built-in voices from the spec and per-model voices from the server;
both were plumbed and never read.
- Hide the lyrics and duration controls for models that reject them, require
a voice description for voice design and a transcript plus language for
forced alignment, and send a seed for the conversion and analysis tasks
that declare one.
- Fix a hero subtitle that rendered the raw lookup key on the Voice
conversion and Source separation tabs, and a double-encoded ellipsis in
the models-folder status line.
Validation:
cd webui/native && npx svelte-check --tsconfig ./tsconfig.json # 0 errors
cd webui/native && npm run build
Reachability measured by loading the built catalog.ts through Vite's SSR
loader and counting install choices, before and after.
Backend tested: Metal (Apple M4 Max). Verified end to end against a live
server: OmniVoice renders from a request shaped like the UI's, and Parakeet
transcribes it back verbatim with nine word timings.
Known limitations: min_vram_gb for the fourteen new entries is copied from
architecturally comparable existing entries, not measured -- no spec field
carries it and every existing value is hand-authored. f5_tts and moss_voicegen
have no package installed here, so their entries are verified by resolution
rather than by download and run. The 22 safetensors packages remain
uninstallable from the UI, unchanged and deliberate.
The parameter file covered 42 of the catalogued families and got several of
them wrong. It now covers 63 groups and 414 controls, each default pinned to
the C++ that reads it.
Controls that did nothing
- index_tts2 emitted "lang"; the engine reads only "language". This is the
multilingual selector on the model whose headline feature is multilingual
cloning, and it had never worked.
- miotts emitted "best_of_n"; the engine reads "miotts.best_of_n".
- firered-audio-tts exposed top_k, top_p and temperature, which that family
reads only on its understanding path, never on generation.
Defaults that overrode the model
Every default in this file is sent on every request, so a value here is not
a fallback, it is an override.
- vibevoice num_inference_steps was 10 while the shipped GGUF carries
ddpm_num_inference_steps 20, so the UI silently halved diffusion quality.
The default is now omitted rather than corrected: the value is read per
package, and any literal would override it again.
- firered-audio-asr max_new_tokens was 512 against an engine default of 300,
and sending it at all defeated the model's own bump to 1024 when
enable_thinking is set. Omitted for the same reason.
- vevo2 temperature and top_k are read from the checkpoint's own generation
config; the literals here overrode them.
- personaplex text_temperature and text_top_k are documented to follow
temperature and top_k when unset, which pinning them prevented.
- fireredtts3 voice design carried the instruct-path guidance_scale of 2.0
instead of its own tuned 1.2.
- ace_step shift was 3.0, a value the engine applies only on the extract
route.
- Deleted the seed entries: the server overwrites options["seed"] from the
top-level field, so they never had any effect.
Ranges that permitted hard errors
Nine minimums sat below the engine's guard, so a legal-looking slider
position produced a 500: dramabox durations, confucius4_tts temperature and
top_p, qwen3_tts temperature, midashenglm_gen min_stop_step, controlfoley
duration_sec and guidance_scale, vevo2 and personaplex temperature.
echo_tts num_inference_steps had a maximum equal to its own default, so the
control could only reduce quality.
Values that do not exist
ace_step offered a "remix" route. The engine defines seven routes and throws
on anything else; "remix" appears nowhere in the tree. Removed, along with
the five parameters that existed only to serve it, none of which is read.
Missing controls
22 families had no group at all, so their entire option surface was
reachable only by hand-writing JSON -- including silero_vad's threshold and
min-speech/min-silence knobs, which are the point of that model, kroko_asr's
beam search and hotword biasing, muscriptor's output format, and the whole
control surface of dots_tts, outetts, glm_tts and fish_audio. Added, along
with quality-critical options missing from existing groups: omnivoice's
reference_max_seconds, irodori_tts's seven guidance knobs, index_tts2's
sampling block, chatterbox's min_p (its actual truncation filter; top_p
defaults to a documented no-op), qwen3_tts's sub-talker block,
minimax_music3's ensemble takes, and seed_vc's pitch conditioning without
which the SVC entry cannot do what SVC is for.
Also renames seed_vc's three *_cfg_rate controls to the *_guidance_scale names
its spec declares; they worked only through a deprecated alias table that
throws if both spellings arrive.
The file's own comment claimed only user-modified values are sent, which has
not been true in this UI. It now documents the real behaviour, including that
omitting a default is the way to say "unset", because the option parser skips
empty values.
Validation:
python3 -m json.tool webui/configs/model_params.json
python3 tools/check_loader_catalog_sync.py # ok, in sync
cd webui/native && npm run build
Backend tested: Metal (Apple M4 Max). Verified against a live server that a
request carrying empty-string parameter sentinels renders normally.
Known limitations: defaults were read from the engine source rather than
observed per model, since most of these families have no package installed
here. Controls were deliberately left out where a value is a sentinel rather
than a setting (silero_vad neg_threshold and max_speech_duration_s), where the
option is session-scoped and would be rejected as a request option
(soprano_tts text_chunk_size), where the legal values are discovered from the
package (nemotron_asr lookahead_tokens), and for file-path and raw-tensor
inputs. The five families that read no request options keep no group.
Seven keys were missing from all four translation files, and two more were
missing from every language including English.
- file.clear and file.preview sit on every file input in the app, so their
absence left the localised UI visibly half-English.
- task.midi labelled MuScriptor in English for every non-English user.
- request.autoDuration, request.rewriteCaption, request.rewritingCaption and
voice.configured were likewise untranslated.
- studio.subtitle.conversion and studio.subtitle.separation did not exist at
all, so the hero rendered the literal lookup key on those two tabs. The
English entries are added alongside; these files carry the translations.
Also fills in param.* coverage, which stood at roughly ten keys per language
out of nearly three hundred. Chinese gains 104 keys back-filled from the
original Chinese label, info and placeholder text already in
model_params.json, which the build discards because catalog.ts strips Han
characters -- so that wording was present in the repo but unreachable even for
Chinese users. Italian, Polish and Russian gain 32 each, limited to entries
with an English source; entries whose only source text is Chinese are left for
a pass after those gain English variants.
Note for anyone adding param.* keys: they are namespaced by spec family, not
by the key used in model_params.json. Nine groups there are keyed by catalog
entry id, and the lookup resolves the family first, so a key written with the
group name never fires.
Removes arena.subtitle and models.reinstall from the four files; neither is
referenced by any component.
Validation:
python3 -m json.tool on each of the four language files
cd webui/native && npm run build
Key sets of the four files verified identical to each other outside param.*,
and every key present in the English source.
Known limitations: 33 entries across the three European languages remain the
English string. Each was checked and left deliberately -- loanwords, acronyms
and identical cognates such as Arena, Model, Backend, RTF and WER. The Russian
runtime.backend is the one arguable case, where a Cyrillic transliteration
would match the file's style; it is flagged rather than changed.
The sync check compared registered loaders, model_specs and model_manager_v2
and reported "in sync" while never opening webui/configs/models_catalog.json,
the file that decides which models the WebUI shows. That gap is why 46 catalog
entries could name a download_id that is not a package id, 40 could name a
path no package installs, one could point at the package its own spec
demoted, and four complete model families could be invisible -- with no CI
signal for any of it.
Adds a catalog pass:
- entry family exists in model_specs
- download_id is an exact packages[].id in that entry's own family
- path matches the resolved package's target directory or installed file
- every spec family with installable GGUF packages has an entry (warning)
- every distinct installable GGUF target directory is reachable (warning),
which is what catches shipped-but-unreachable checkpoints
- each spec's default and ui.recommended_package can actually be selected
- task and mode are in the vocabularies parsed out of session.cpp rather
than a hardcoded list
- duplicate entry ids
Deliberate exclusions are derived, not listed: a package counts as installable
only when its effective download.kind is huggingface_snapshot, so the two
families that carry kind "unsupported" for licence reasons drop out on their
own.
Also models the native package manager's kind filter, which the previous
summary line did not: it now reports manager_packages and server_packages
separately, exposing that the Python tool this check measures knows four more
packages than the C++ manager the server actually runs.
Finally, cross-checks model_params.json controls against the specs, scoped to
families that document their request surface -- an empty options.request means
undocumented, not unsupported, and checking those too produces roughly eighty
false reports. Bundled families with no spec are skipped, since there is no
declared surface to compare against.
Validation:
python3 tools/check_loader_catalog_sync.py # exit 0, 31 warnings, all
# pre-existing informational
python3 tools/check_loader_catalog_sync.py --self-test # 10 tests, OK
Each new error class was mutation-tested against a deliberately broken copy of
the catalog -- unknown family, family-name download_id, cross-family
download_id, wrong path, unknown task, unknown mode, bundled entry carrying a
download_id, duplicate id, and an entry repointed off its family default --
rather than trusted on a passing run.
Known limitations: the reachability model mirrors the front end's
target-directory grouping without parsing catalog.ts, so a future change to
the resolution rules must be reflected here by hand. The parameter cross-check
is a warning, not an error, because option strictness is per-loader and cannot
be proven from the specs alone. Self-tests extend the existing in-file
--self-test rather than adding a suite, matching how the four CI workflows
already invoke this script.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tools/check_loader_catalog_sync.pycompared registered loaders,model_specsandmodel_manager_v2and reported "in sync" while never openingwebui/configs/models_catalog.json— the file that decides which models the WebUI shows.That gap is why 46 catalog entries could name a
download_idthat is not a package id, 40 could name a path no package installs, one could point at the package its own spec demoted, and four complete model families could be invisible, with no CI signal for any of it.The catalog pass
model_specsdownload_idis an exactpackages[].idin that entry's own familypathmatches the resolved package's target directory or installed filedefaultandui.recommended_packagecan actually be selectedtaskandmodeare in the vocabularies parsed out ofsession.cpprather than a hardcoded listDeliberate exclusions are derived, not listed: a package counts as installable only when its effective
download.kindishuggingface_snapshot, so the two families that carry kindunsupportedfor licence reasons drop out on their own.Two further checks
The summary now models the native package manager's kind filter, reporting
manager_packagesandserver_packagesseparately — which exposes that the Python tool this check measures knows four more packages than the C++ manager the server actually runs.It also cross-checks
model_params.jsoncontrols against the specs, scoped to families that document their request surface: an emptyoptions.requestmeans undocumented, not unsupported, and checking those too produces roughly eighty false reports. Bundled families with no spec are skipped, since there is no declared surface to compare against.Scope
Tooling only. No engine, server or UI code. Affects CI signal, not runtime behaviour.
Validation
Each new error class was mutation-tested against a deliberately broken copy of the catalog — unknown family, family-name
download_id, cross-familydownload_id, wrong path, unknown task, unknown mode, bundled entry carrying adownload_id, duplicate id, and an entry repointed off its family default — rather than trusted on a passing run.Backend tested: Metal, Apple M4 Max (full suite green, 39/39).
Depends on #372, #373, #374 and #375; this branch contains them, and the checker is what proves the catalog they change is consistent.
Known limitations
The reachability model mirrors the front end's target-directory grouping without parsing
catalog.ts, so a future change to the resolution rules must be reflected here by hand. The parameter cross-check is a warning, not an error, because option strictness is per-loader and cannot be proven from the specs alone. Self-tests extend the existing in-file--self-testrather than adding a suite, matching how the four CI workflows already invoke this script.