Features and Improvements for AI Playground 3.2.0-beta - #576
Open
mschuettlerTNG wants to merge 23 commits into
Open
Features and Improvements for AI Playground 3.2.0-beta#576mschuettlerTNG wants to merge 23 commits into
mschuettlerTNG wants to merge 23 commits into
Conversation
Port Phison's aiDAPTIV+ Knowledge Manager RAG (Approach A) from #250 onto dev-leslie: fixed shared prefix + document context first for KV-cache reuse across presets/queries, with the preset system prompt appended after. Encapsulation (addresses review on #250): - Isolate KM logic in langchainPhisonKm.ts / phisonKmRag.ts / types/phisonKmRag.ts - Keep default RAG retrieval (global top-K + maxResults) for non-KM users - Persist boundary-only mergedGroups with a staleness fingerprint Follow-up fixes on top of the upstream PR: - Re-clamp contextSize when model max or KM floor enforcement changes - Point KV warmup at llama.cpp upstream when Home Agent is active - Keep Standard RAG selectable to clear a persisted KM preference - Share GROUP_SEPARATOR; add warmup consecutive-failure tests - Map supportsPhisonKmRag onto the merged Assistant preset Based on work by TingyinKuo-Phison / YuchengKuo-Phison in #250. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
PR #271 imports @/lib/ttsVoiceSeed from qwen3TextToSpeech, but the module was never merged to dev (only existed on a model-management WIP branch). That broke production installer builds (vue-tsc) for v3.2.0-alpha.phison-test1. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
KM returns a merged group Document with source but historically without loc, and formatRagSources dropped any source that lacked page/line info — so the Source Docs chip never appeared. Always register the filename, and forward the best-scoring chunk's loc when available. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
A merged group spans many chunks, so forwarding one chunk's loc produced a misleading Page/Lines citation. Keep the Source Docs chip (filename via source metadata) but omit loc for group retrieval results. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* docs: plan a model management view
Design plan for a dedicated Model Management view (LM Studio-style "My
Models", simplified for AI Playground) covering: show in folder, delete
from disk, hide from the model picker, favorite, edit capabilities, and
batch download, with filters for name, capability, backend and use case.
Covers the data-layer work it needs: a unified ModelEntry row type across
the three model universes that exist today (LLM/embedding, media, speech),
a modelPreferences store for the missing user layer, and three new
main-process capabilities (scan with size/mtime/absolute path, reveal in
file manager, path-guarded delete).
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* docs(model management): record the six design decisions
- Hard delete instead of trashing, with an explicit permanence warning in
the confirmation dialog; the path-containment guard is now the feature's
safety-critical code and gets its own test file.
- hidden/favorite stay per machine in localStorage, as proposed.
- hidden applies everywhere: enumerate every affected surface (chat picker,
embedding dropdown, ComfyUI model inputs, cloud models, and the
listHomeAgentModels/configureHomeAgent tools), and note what is
deliberately unaffected (the agentic ComfyUI tools and the Home Agent
/imgGen picker choose presets, not models). Crucially, hidden must NOT
filter textInference.llmModels, which also resolves activeModel and the
download params: it becomes a field on LlmModel, filtered at each
presentation site by a shared keep-selected predicate.
- The media catalog stays preset-derived, so drop the models.json media
entries follow-up.
- Capability editing is exposed in essentials too, since missing metadata
makes a model unusable.
- largeMoe is editable but needs a tooltip: it is a hardware gate, so
ticking it hides the model on machines without Phison aiDAPTIV+.
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* feat(models): unified model entry model, preferences and disk operations
Phase 1 of the model management plan: the data layer, with no new UI yet.
- assets/js/models/{types,library,overrides,visibility,downloadParams}.ts:
one ModelEntry row shape across the LLM/embedding, media and speech
catalogs, plus pure filter/sort/merge helpers.
- store/modelPreferences.ts: the user layer the app was missing (hidden,
favorite, capability overrides), persisted, with no store dependencies.
- store/models.ts: capability overrides are now the top merge layer, so
editing a models.json model's capabilities sticks instead of being
discarded on the next refresh. The previous in-memory entry no longer
contributes capabilities, which is what made a removed override linger.
Adds removeCustomModel for dropping a mistyped custom entry.
- pathsManager: scanModelLibrary reports each model's absolute path, size
and newest mtime; resolveModelPath refuses anything outside a configured
model directory (realpath, so symlinks cannot escape) and mirroredModelPaths
finds the ComfyUI copies of faceswap/facerestore weights.
- main/preload/env.d.ts: scanModelLibrary, showModelInFolder and
deleteModelPath. Deletion is permanent by design and path-guarded.
- hidden/favorite are carried as fields on Model and LlmModel and applied by
pickers (chat model selector, embedding dropdown, ComfyUI model inputs,
Home Agent list/configure tools) rather than filtered in llmModels, which
also resolves activeModel and the download params.
- errors: new 'model' category for model-file operations.
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* feat(models): add the Model Management view
Phase 2 and 3 of the plan: the UI on top of the Phase 1 data layer.
- views/ModelManager.vue: a full-screen overlay (the settings sidebars are far
too narrow for a nine-column table), with a use-case sidebar, toolbar and
table. Mounted before the dialog layer in App.vue so the download and delete
dialogs it opens still appear on top of it.
- store/modelLibrary.ts: entries, filters, selection and the actions. Media
models are derived from preset requiredModels, which also gives the "used
by" column and the delete warning for free.
- components/models/: toolbar (search, capability filters, backend/status
filters, show-hidden, batch buttons), table with per-row actions,
ModelCapabilityFields extracted so the Add Model dialog and the capability
editor cannot drift, and DeleteModelDialog.
- Delete is permanent, so its dialog names the paths, the space reclaimed, the
presets that need the model and whether it is currently selected, and its
confirm button says "Delete permanently".
- Batch download and batch delete reuse the multi-select; downloads go through
the existing (already multi-model) DownloadDialog.
- components/ui/table: shadcn-vue Table primitives, which the repo lacked.
- textInference.clearSelectionOfModel so deleting the selected model does not
leave the next chat turn pointing at missing weights.
- 67 new MODEL_MANAGER_* i18n keys across all 13 locales, plus a parity test
that keeps the key sets and placeholder tokens in sync from now on.
- Unit tests for the scan, the delete path guard, entry derivation, filters,
sorting, capability merge precedence and download-param derivation.
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* chore(models): track the source files the models/ ignore rule swallowed
`models/` in .gitignore matches any directory of that name, so
`WebUI/src/assets/js/models/` and `WebUI/src/components/models/` were
silently excluded and the two model-management commits landed without the
modules and components they describe: the ModelEntry types, the pure
library/overrides/visibility/downloadParams helpers and their unit tests,
the library table, toolbar, row actions and the delete/capability dialogs.
Nothing that imports them would build from a fresh clone.
The rule already had this problem once (the vendored Qwen3-TTS tree), so the
fix is the same: negate the source directories by name.
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* fix(models): resolve hidden/favorite live instead of from the catalog snapshot
Hiding a model in Model Management removed its row from the management table
but left it in the chat model picker until something else triggered a catalog
refresh. Runtime logging showed the model arriving at the picker with
`hidden: false` while the preference was set, and with it not being the current
selection, so the "never hide the selection" rule was not what kept it.
The flags were baked into the `Model` snapshot by `refreshModels()`, which only
re-runs on a catalog refresh — a preference write does not cause one, so every
consumer of `models.models` kept serving the flags from the moment of the last
refresh. Capability overrides get away with the same treatment because
`saveCapabilities` refreshes the catalog itself.
`hidden`/`favorite` now come off the `Model` type entirely and are resolved in
the `llmModels` and `llmEmbeddingModels` computeds, which read the preferences
ref and therefore re-evaluate on the write itself. They stay carried fields on
`LlmModel` and are still never filtered out of `llmModels`, which also resolves
`activeModel`, the capability computeds and the download params.
The lookup step is `withPreferenceFlags` in models/visibility.ts, next to the
picker predicates it feeds, with tests covering that it reads through the
lookup on every call — the regression being "flags came from a stale snapshot".
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* feat(models): prune the directories a model delete leaves empty
Deleting one quantization out of `owner---repo/` removed the file and left the
repo directory behind forever, so the model tree accumulated empty folders.
`pruneEmptyModelDirs` walks up from the deleted path removing empty
directories, stopping at the configured model directory and refusing to step
outside the model tree at all.
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* docs(model management): record the implementation outcome
Marks the plan as implemented and adds an implementation-notes section: the
reactivity bug the original design invited (preferences baked into the catalog
snapshot) and why capabilities and preferences belong in different layers, the
.gitignore rule that swallowed the new source directories, the smaller things
testing turned up (empty repo directories after a delete, ambiguous labels,
template-only `languages`, overlay opacity, dialog stacking order), and what is
still open.
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* feat(models): phase 4 — speech catalog, folder editor, e2e, dead code
- Speech models are a real use case now: the STT/TTS/Qwen3-TTS repo ids the
features load are listed in the library, so they can be seen, pre-fetched and
deleted. Neither feature has a model picker, so before this they were
invisible until first use. ModelEntry.requiredByPresets becomes requiredBy,
since what needs a model is a preset for media and a feature for speech; the
delete warning's wording follows in all 13 locales.
- Model folders dialog wires up models.applyPathsSettings and the previously
unused FolderSelector, grouped by use case.
- restorePathsSettings was not merely incomplete, it threw: updateModelPaths
read every configured key out of the map it was given, and the handler passed
only the three LLM keys, so it raised a TypeError on the first ComfyUI key.
updateModelPaths now keeps unlisted directories as they are, and defaults come
from a snapshot taken on first load — in every install mode except a shared
one the config the app writes is the file it shipped, so nothing else survives
a user's edit to restore to.
- FolderSelector's path field and browse button had no accessible name, and the
chat Model row is now a named group, since the picker's trigger is named after
whichever model is selected.
- e2e: ModelManagerPage plus a model-management spec covering listing and
filtering, hide propagating to the chat picker, favorite ordering, batch
download reaching the dialog, and the folders dialog. Validated against the
real app; it found two bugs in the page object (the header row counted as a
model, and visibleLabels collapsing to one entry).
- Retires what the feature supersedes: the models.download() stub,
models.downloadList, ModelDropDownItem.vue and App.vue's dead AddLLMDialog
wiring.
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* fix(models): make the folders dialog usable in a project checkout
Three problems only visible once the model-paths editor existed:
- Saving a folder rewrote all 19 directories from portable relative paths to
absolute ones, and re-indented the file, so moving one folder produced a
whole-file diff in a tracked dev config. Untouched directories now keep the
spelling, indentation and trailing newline they had.
- Restoring defaults re-serialised the snapshot instead of copying it, so a
restore was equivalent but not identical. It copies the file now.
- The config lives inside the Vite root, so writing it triggered a full page
reload that closed the view the user was editing — the dialog looked like it
dismissed itself. Nothing in the renderer imports the file (the main process
reads it with fs), so the dev server no longer watches it.
Verified against the running app: editing one folder is now a one-line diff, the
Models view survives the save, and Restore defaults leaves the config
byte-identical.
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* Clean-up model manager
Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
* Clean-up agent settings
Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
* Improve e2e test for model manager
Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
* Enforce font and spacing consistency in settings
Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
* Implement review suggestions
Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
---------
Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
Co-authored-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
Co-authored-by: Leslie Lazzarino <62446606+leslielazzarino@users.noreply.github.com>
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
feat: aiDAPTIV+ KM RAG shared-prefix KV-cache reuse (from #250)
feat: add Game Agent, Quick Coder, and the Pi agent harness Agent Mode runs coding-agent sessions in the main process. Game Agent plans, codes, play-tests, and files games; Quick Coder writes a one-shot game and can hand it over. Also Qwen3.8 sampling and MTP flags, Laminar tracing, a games library with arcade samples, LTX GGUF presets, add/edit model UX, installer cache fix, and Vitest + Pi smoke in CI. Signed-off-by: Markus Schuettler <markus.schuettler@tngtech.com> Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
* V3.1.2 beta hotfix1 rc1.1 (#256) * Remove Hardcoded ComfyUI Launch parameter on Linux remove hard coded startup comfyui parameters. allow user to adjust it via UI. * probe XPU before Linux spawn instead of forcing --cpu ComfyUI could start with --cpu when usableXpuConfirmed was still unset. await XPU probe on linux instead of pre-probe CPU fallback * Fix stale ComfyUI Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Keep pytorch <2.13 for linux compatibility Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve error handling of huggingface connection issues Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> --------- Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> Co-authored-by: Qiacheng Li <qiacheng.li@intel.com> � Conflicts: � WebUI/electron/subprocesses/processLifecycle.ts � comfyui-deps/pyproject-flexible-venv.toml � comfyui-deps/pyproject.toml � service/model_downloader.py * Last month of development (#262) * Remove Hardcoded ComfyUI Launch parameter on Linux remove hard coded startup comfyui parameters. allow user to adjust it via UI. * probe XPU before Linux spawn instead of forcing --cpu ComfyUI could start with --cpu when usableXpuConfirmed was still unset. await XPU probe on linux instead of pre-probe CPU fallback * Fix stale ComfyUI Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Keep pytorch <2.13 for linux compatibility Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve error handling of huggingface connection issues Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add pre-commit hooks, to enforce formatting and linting Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix OpenVINO version display and change Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> � Conflicts: � WebUI/electron/subprocesses/openVINOBackendService.ts * feat: ovms int4 kv cache * chore: update author name * Pre-commit hooks: do not fail when formatting Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Hybrid mode POC Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add hybrid mode BE Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Support multiple external providers Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve UI Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Refactor setup wizard rows Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Rename to cloud mode Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve cloud mode settings Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Use cloud mode in agentic Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve cloud mode Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * feat: add preset management to media creation tools * Update app version to 3.2.0-alpha Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix cloud mode bugs Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add basic Playwright start-app test Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add Playwright test for Agentic Also fix bugs found through the test. Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Remove release from build pipeline, as it caused mayhem on the non-fork version of the repo Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve tests Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * feat: port Qwen3-TTS agent tool and optional backend sidecar Ports qiacheng/AI-Playground@95e9641d and @bd215057 onto the current branch: the synthesizeTextToSpeech agent tool, the optional Qwen3-TTS Python sidecar (Intel XPU PyTorch stack), Documents/audio output, and tool-output sanitization so base64 WAV never inflates the LLM context or stored history. Wired into the existing home-agent-backend pattern (registry, backend services store, setup wizard, loopback auth, settings). Hardening applied on top of the upstream POC: - Two-slot model cache: custom_voice and voice_design models stay resident once loaded instead of reloading on every mode switch. - Dedicated inference mutex so concurrent /api/synthesize requests serialize on the shared accelerator (generate paths aren't thread-safe). - Input-length bound (QWEN3_TTS_MAX_CHARS, default 5000) rejecting oversized text before it can hang generation or exhaust device memory. Verified: qwen-tts 0.1.1 API, both HF model ids, and all speaker names match upstream; vue-tsc, eslint, and unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add TTS with Qwen 3 Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Report current preset and model Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Select preset from mode selector Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Remove preset-selector from settings Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix agentic: don't reset the chat stream when image tools free the GPU Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Expose model folder in all user installer Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Expose model folder in all user installer, share backends Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve all user installer Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Update and fix E2E tests Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix OpenVINO version display and change Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * feat: ovms int4 kv cache * Refactoring part 1 Clean up dead code, fix typo/version pin, route error paths through error sink Architecture cleanup pass: remove dead IPC surface, fix a few small inconsistencies, and migrate genuine error/catch paths onto the central errors.report() sink (per AGENTS.md convention) instead of bare toast.error. Dead code & small fixes: - Remove dead IPC commands refreshLLMModles / getDownloadedLLMs (both returned [] after the old ipexllm backend was removed). Dropped the handlers, preload bridge entries, and env.d.ts type defs. - Fix `singal` -> `signal` typo (all references) in service/model_download_adpater.py. - Pin OpenVINO backend to Python ==3.12.* (was >=3.10) to match the service/, home-agent/, and ComfyUI/ backends. Error-handling consistency (route through useErrors sink): - InstallationManagement.vue: 5 service start/stop/setup failures now go through errors.report(...) with category backend/setup and serviceName context; removed the now-unused toast import. - McpServerDialog.vue: add/update failure reported as surface:'inline' (dialog already renders the message inline, drops a redundant toast); remove failure reported as a normal toast. - PromptArea.vue: audio-recorder error, image-load catch, and RAG-add catch now go through errors.report(...). Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Refactoring part 2 #1 Backend process-lifecycle helpers (composition, not inheritance) - New WebUI/electron/subprocesses/processLifecycle.ts exporting terminateProcessTree() (SIGTERM → Windows taskkill /T /F tree / SIGKILL) and waitForServerReadyOrThrow() (health poll; supports llamacpp's getStartupError and openvino's stderr/exit-signal diagnostics). - service.ts (base stop()), openVINOBackendService.ts and llamaCppBackendService.ts now delegate to these instead of each reimplementing health-check / kill logic. - Fix: llamacpp LLM/embedding stops previously used SIGTERM→SIGKILL with no Windows tree-kill, leaking descendant processes and causing reinstall EPERM; they now tree-kill like the other backends. #2 Shared Python loopback-auth - New backend_shared/aipg_loopback_auth.py: stdlib-only, transport-agnostic evaluate_loopback_auth() + get_loopback_token(). - service/web_api.py and home-agent/web_api.py import it via a ../backend_shared sys.path bootstrap (resolves identically in dev and packaged layouts); each keeps its own response/CORS handling so behavior is preserved. - build-config.json: package backend_shared/ alongside the backends. #3 Split comfyUiPresets store (1600 → 1368 lines) - New comfyUiMessages.ts: ComfyUI WS message schema + execution-error summarizer. - New comfyUiWorkflowTransforms.ts: workflow-graph transforms (OVMS image url inject, model-path normalize, node bypass). - comfyUiPresets.ts imports both; reactive store surface unchanged. Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Use lighter agentic test for e2e smoke test Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve default device selection Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * chore: extract prompt status bar and adjust styling * chore: avoid issues with layout shift during interaction * Add user selection of default device Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix default device selection bugs Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Use device uuids when available, not just names Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Show if Llama or OpenVINO in banner Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add selected hardware to bar of selected options Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Toolbar insted of circle for used context Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * chore: adjust settings panel styling * Various UI fixes & improvements Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Reintroduce release in build pipeline Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix typing errors Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Pin same ruff version and config everywhere Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix pdf parsing using unpdf Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix most vulnerabilities Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add pre-commit hooks, to enforce formatting and linting Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix OpenVINO version display and change Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> � Conflicts: � WebUI/electron/subprocesses/openVINOBackendService.ts * feat: ovms int4 kv cache * chore: update author name * Pre-commit hooks: do not fail when formatting Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Hybrid mode POC Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add hybrid mode BE Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Support multiple external providers Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve UI Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Refactor setup wizard rows Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Rename to cloud mode Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve cloud mode settings Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Use cloud mode in agentic Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve cloud mode Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * feat: add preset management to media creation tools * Update app version to 3.2.0-alpha Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix cloud mode bugs Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add basic Playwright start-app test Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add Playwright test for Agentic Also fix bugs found through the test. Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Remove release from build pipeline, as it caused mayhem on the non-fork version of the repo Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve tests Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * feat: port Qwen3-TTS agent tool and optional backend sidecar Ports qiacheng/AI-Playground@95e9641d and @bd215057 onto the current branch: the synthesizeTextToSpeech agent tool, the optional Qwen3-TTS Python sidecar (Intel XPU PyTorch stack), Documents/audio output, and tool-output sanitization so base64 WAV never inflates the LLM context or stored history. Wired into the existing home-agent-backend pattern (registry, backend services store, setup wizard, loopback auth, settings). Hardening applied on top of the upstream POC: - Two-slot model cache: custom_voice and voice_design models stay resident once loaded instead of reloading on every mode switch. - Dedicated inference mutex so concurrent /api/synthesize requests serialize on the shared accelerator (generate paths aren't thread-safe). - Input-length bound (QWEN3_TTS_MAX_CHARS, default 5000) rejecting oversized text before it can hang generation or exhaust device memory. Verified: qwen-tts 0.1.1 API, both HF model ids, and all speaker names match upstream; vue-tsc, eslint, and unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add TTS with Qwen 3 Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Report current preset and model Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Select preset from mode selector Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Remove preset-selector from settings Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix agentic: don't reset the chat stream when image tools free the GPU Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Expose model folder in all user installer Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Expose model folder in all user installer, share backends Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve all user installer Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Update and fix E2E tests Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix OpenVINO version display and change Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * feat: ovms int4 kv cache * Refactoring part 1 Clean up dead code, fix typo/version pin, route error paths through error sink Architecture cleanup pass: remove dead IPC surface, fix a few small inconsistencies, and migrate genuine error/catch paths onto the central errors.report() sink (per AGENTS.md convention) instead of bare toast.error. Dead code & small fixes: - Remove dead IPC commands refreshLLMModles / getDownloadedLLMs (both returned [] after the old ipexllm backend was removed). Dropped the handlers, preload bridge entries, and env.d.ts type defs. - Fix `singal` -> `signal` typo (all references) in service/model_download_adpater.py. - Pin OpenVINO backend to Python ==3.12.* (was >=3.10) to match the service/, home-agent/, and ComfyUI/ backends. Error-handling consistency (route through useErrors sink): - InstallationManagement.vue: 5 service start/stop/setup failures now go through errors.report(...) with category backend/setup and serviceName context; removed the now-unused toast import. - McpServerDialog.vue: add/update failure reported as surface:'inline' (dialog already renders the message inline, drops a redundant toast); remove failure reported as a normal toast. - PromptArea.vue: audio-recorder error, image-load catch, and RAG-add catch now go through errors.report(...). Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Refactoring part 2 #1 Backend process-lifecycle helpers (composition, not inheritance) - New WebUI/electron/subprocesses/processLifecycle.ts exporting terminateProcessTree() (SIGTERM → Windows taskkill /T /F tree / SIGKILL) and waitForServerReadyOrThrow() (health poll; supports llamacpp's getStartupError and openvino's stderr/exit-signal diagnostics). - service.ts (base stop()), openVINOBackendService.ts and llamaCppBackendService.ts now delegate to these instead of each reimplementing health-check / kill logic. - Fix: llamacpp LLM/embedding stops previously used SIGTERM→SIGKILL with no Windows tree-kill, leaking descendant processes and causing reinstall EPERM; they now tree-kill like the other backends. #2 Shared Python loopback-auth - New backend_shared/aipg_loopback_auth.py: stdlib-only, transport-agnostic evaluate_loopback_auth() + get_loopback_token(). - service/web_api.py and home-agent/web_api.py import it via a ../backend_shared sys.path bootstrap (resolves identically in dev and packaged layouts); each keeps its own response/CORS handling so behavior is preserved. - build-config.json: package backend_shared/ alongside the backends. #3 Split comfyUiPresets store (1600 → 1368 lines) - New comfyUiMessages.ts: ComfyUI WS message schema + execution-error summarizer. - New comfyUiWorkflowTransforms.ts: workflow-graph transforms (OVMS image url inject, model-path normalize, node bypass). - comfyUiPresets.ts imports both; reactive store surface unchanged. Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Use lighter agentic test for e2e smoke test Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve default device selection Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * chore: extract prompt status bar and adjust styling * chore: avoid issues with layout shift during interaction * Add user selection of default device Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix default device selection bugs Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Use device uuids when available, not just names Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * chore: adjust settings panel styling * Show if Llama or OpenVINO in banner Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add selected hardware to bar of selected options Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Toolbar insted of circle for used context Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Various UI fixes & improvements Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Reintroduce release in build pipeline Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix typing errors Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Pin same ruff version and config everywhere Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix pdf/txt file handling in cloud mode Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Turn Qwen TTS into preset Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve default GPU selection Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * UI restyling of new features Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add e2e regression test for pdf Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Expand e2e to cover TTS and both Llama/OpenVINO Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix e2e tests on Linux Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Support device selection in TTS Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Merge chat presets Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Remove NPU capability filter Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Change install for all default folder, to fix permission issues Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix mcp issues and e2e test the tools Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add info to MCP Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Remove local servers from cloud mode Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Update translations Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve TTS UI Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * docs: evaluate replacing qwen-tts with vllm-omni vLLM-Omni supports Qwen3-TTS and its speech API is a good fit, but it runs on vLLM, which has no Windows support and no installable Intel XPU path (Docker or a oneAPI source build only, against torch 2.13 which we ceiling out). It also has no CPU platform, a 304 MB wheel, and pre-allocates 92% of VRAM per instance on a GPU we already share with the LLM and ComfyUI backends. Not adoptable. Audits where the advisories actually come from: 13 of 18 are pillow, pulled in only by gradio, pulled in only by qwen-tts for a demo CLI the sidecar never imports. Excluding gradio drops 35 packages, verified safe (the venv that synthesized audio had none of them). The remaining transformers==4.57.3 pin is load-bearing: qwen-tts 0.1.1 on transformers 5.x fails six times, ending in NaN logits rather than an API error. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * docs: scope the vendor-and-port option for qwen-tts Measures what "vendor the 12 Hz subset and port it" actually costs, and finds the two halves are separable with very different risk. Vendoring at the current pin is pure deletion: ~5.4k of 9.3k lines kept, and it sheds torchaudio, einops, sox, onnxruntime, protobuf and flatbuffers (114 -> 79 -> 72 packages across the three dependency sets, one lock harness) — but fixes none of the transformers advisories on its own. Porting to transformers 5.x is the expensive half. Diffed all 35 transformers symbols the kept files import: 18 unchanged, 17 changed of which ~9 are typing-only, leaving 6 behavioral ones. The NaN failure is not among them, and the vendored encoder subclasses transformers' MimiModel, whose config schema now silently drops the checkpoint's rope_theta. Also establishes that the A/B can be numeric rather than by ear: generation is bit-reproducible with a fixed seed (max_abs_diff 0.0, sampled and greedy), so a frozen corpus of codec token sequences can gate a port. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * feat(ui): add contextual help mode with ? toggle Let users enter help mode from the header, click annotated controls for topic popovers, and exit with Esc or the toggle again. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(ui): extend help mode with preset topics and banner UX Co-authored-by: Cursor <cursoragent@cursor.com> * fix(qwen3-tts): correct out-of-range default port in the sidecar argparse defaulted --port to 69001, which is above the TCP maximum of 65535 and silently wrapped to 3465, so running the sidecar by hand landed on a port that matched neither the flag nor the docstring. The Electron service always passes --port from the 57000-57999 range it allocates, so only manual runs were affected. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * refactor(qwen3-tts): vendor the 12 Hz subset instead of the qwen-tts package The qwen-tts package pinned transformers==4.57.3 and accelerate==1.12.0 by equality and pulled gradio for a demo CLI (qwen_tts/cli/demo.py) the sidecar never imports, which dragged pillow, fastapi, uvicorn, starlette, python-multipart and ~30 more packages into an install that already runs Flask. Its 25 Hz tokenizer, which our 12 Hz checkpoints never touch, was also the only importer of sox, onnxruntime, torchaudio and einops. So vendor the 12 Hz path from the wheel (Apache-2.0, provenance and the exact local edits in vendor/README.md) and declare the real dependencies directly. 8 of the 10 vendored files are byte-identical to the wheel; the two that differ do so by 2 and 51 lines, all of it removing the 25 Hz imports, Auto* registrations and decode() branch. The 2299-line modeling_qwen3_tts.py is untouched. The lock drops from 145 to 99 packages, which clears 13 of the 18 published advisories in it (all the pillow ones, reached only through gradio). The transformers advisories remain: the pin is load-bearing, since this code does not run on transformers 5.x. transformers is therefore constrained to <5 explicitly rather than pinned by equality. Output is unchanged, not just believed to be: codec token sequences and waveforms are bit-identical to the upstream package across four fixed-seed cases including a greedy decode (tests/vendor_parity.py). .gitignore's blanket models/ rule matched the vendored core/models/ directory, so it is negated for that path; ruff.toml excludes the tree so the formatter leaves it diffable against upstream. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * fix(help): keep controls inert and anchor the help panel correctly Follow-up fixes on the contextual help POC: - Help mode only swallowed `click`, so radix-vue selects (which open on `pointerdown`) and sliders still reacted while help mode was meant to be inert. Pointer events are now captured too; `click` still arrives and is what opens the panel. - The panel took a one-shot `getBoundingClientRect()` and never moved, so it drifted away from its anchor when a settings sidebar scrolled. It now re-anchors on scroll/resize and closes if the anchor leaves the DOM. - Closing a preset panel left the highlight ring behind: the cleanup watcher keyed on `panelTopicId`, which stays null for dynamic topics, so closing was a null -> null transition that never fired. It now keys on the resolved topic. - The above/below flip used a hardcoded 160px estimate; the panel is measured after render and clamped inside the viewport. - `resolveHelpTarget` had two unreachable mode-button branches and an unused export, and re-ran `closest()` at every ancestor level. It is now a single ancestor walk, and `getHelpTopic` no longer resolves inherited `Object` properties as topics. - Preset lookup by name could pick a chat preset while in an image mode; it now prefers the type matching the active mode. - `cursor: help` was applied to every button, promising a topic even where the answer is the "no help topic here" toast. It is scoped to annotated elements, and other controls read as `not-allowed` since they are inert. - Accessibility: dropped the inaccurate `aria-modal`, moved focus into the panel and back to the toggle on close, and gave the toggle `aria-label`/`aria-pressed`. Also reuses one `extendedDescriptionText` helper instead of duplicating the `extendedDescription` resolution in PresetSelector, and reformats the new files so `prettier --check` passes. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * refactor(help): give the demo tour and help mode one copy registry `HELP_TOPICS` was a second set of per-control help text keyed off the same DOM ids the demo tour already anchors to, so editing one surface silently left the other stale. The tour's copy moves verbatim into `assets/js/help/tourSteps.ts` next to the click-to-learn topics, and `TourStep.id` is typed as `#${HelpTopicId}` plus an explicit tour-only anchor. Renaming an anchor in one surface now fails type-check in the other instead of orphaning a step. The two surfaces keep their own wording on purpose: the tour narrates a guided walkthrough, help mode answers "what is this". No tour copy changes. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * test(qwen3-tts): guard the vendored tree and check parity with upstream tests/test_vendor.py is stdlib-only and runs in 0.1 s: it asserts the vendored file inventory, that no vendored module imports anything pyproject.toml does not declare, that the 25 Hz dependencies (sox, onnxruntime, torchaudio, einops) and gradio stay absent, and that upstream's Apache-2.0 notices survive. A re-vendor from a new wheel that drags the 25 Hz tree back fails here rather than silently re-adding four packages to the install. tests/vendor_parity.py is the slow check that makes vendoring trustworthy: it synthesizes a fixed case list under both the vendored tree and the upstream package and compares codec token sequences and waveforms bit-exactly. Codes are the primary signal since they are RNG-immune and localize a mismatch to the talker rather than the codec decoder. It is named so unittest discovery skips it, and it runs each implementation in a subprocess because both register qwen3_tts with transformers' Auto* registries. Importing upstream in this venv now fails on the removed 25 Hz imports, so those modules are stubbed in the upstream child (with real __spec__s, or torch._dynamo's module scan raises). Follows service/tests/: stdlib unittest, no new dependencies. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * docs: record that the qwen-tts vendoring landed Marks Step A of the vendor-and-port option as done, replaces the superseded uv override recommendation, and adds the parity and sidecar evidence. Also records how the lock was regenerated despite this VM's egress blocking the PyTorch CDN host, since the workaround is reusable and the clean fix is an allowlist entry. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * test(help): cover target resolution and preset help topics 27 tests over the pure parts of help mode: - `resolveHelpTarget` precedence (element id vs `data-aipg-help` vs preset and variant annotations), the ancestor walk, and the guard that stops help mode explaining its own toggle and panel. - `helpTopicFromPresetVariant` resolving by internal variant name, including a regression test for the display-label bug: a label misses the `extendedDescription` map and the fallback-to-first then describes a different variant. - Parity between the demo tour anchors and the click-to-learn topics, so a tour step can't be added without help copy. `resolveHelpTarget` reads only `id`, `getAttribute` and `parentElement`, so a small element stand-in covers it under vitest's `node` environment. Adding a DOM environment was rejected on purpose: npm re-resolves this lockfile from scratch on any install, so a dev dependency would drag a ~4000 line lockfile rewrite into an unrelated PR. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * docs: record the verified post-vendoring advisory count Re-ran the advisory audit against the new lock: 5 matches over 95 packages, down from 18 over 137. Notes why the two non-transformers residuals are deliberate — torch is held below 2.13 on purpose, and setuptools is an unbounded requirement of torch that does not move on --upgrade-package, for an sdist-build advisory a service that never builds an sdist cannot reach. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * feat(qwen3-tts): run the vendored engine on transformers 5 4.57.3 carries three advisories fixed only in 5.x, and the reachable one is CVE-2026-4372: a crafted config.json sets _attn_implementation_internal to an attacker's Hub repo and from_pretrained() executes code from it, bypassing trust_remote_code. tts_engine._load_model() is exactly that call. The vendored code is written against 4.57.3 internals, so the affected call sites now route through vendor/qwen_tts/_compat.py, which bridges both versions rather than rewriting against 5.x — upstream qwen-tts only runs on 4.57.3, so keeping 4.x working is what keeps tests/vendor_parity.py able to compare against upstream at all. Three changes are signature bridging (the check_model_inputs decorator form, the removed "default" RoPE entry, the mask builders' renamed/dropped arguments) and two are narrower than they look: config.pad_token_id is None on 4.57.3 for these checkpoints, so reading it via getattr is the same value, and rope_config_validation only validates, so it is skipped where 5.x expects a rope_parameters schema these configs do not use. Two were silent failures: - inv_freq is a non-persistent buffer, absent from every checkpoint. transformers 5 builds modules on meta and restores such buffers only for rotary classes its own _init_weights recognizes, by class name plus original_inv_freq. None of the three here qualify (this model overrides _init_weights without delegating, and the codec decoder's class is spelled ...RotatoryEmbedding upstream), so inv_freq kept uninitialized memory and the first forward produced NaN logits. reset_rotary_buffers() recomputes it after each from_pretrained; it is deterministic and idempotent, so a no-op on 4.57.3. - 4.x passed only the current step's position_ids; 5.x accumulates them, so while decoding one token the model got the whole sequence and attention output stopped matching the query length. align_position_ids() trims to the query length. Output does not move: codec token sequences and waveforms are bit-identical between upstream@4.57.3, vendored@4.57.3 and vendored@5.14.1 on one torch build, and all 144 shared model buffers match after loading. Advisories in the lock go 5 -> 2, the rest being the deliberate torch<2.13 ceiling and a setuptools sdist issue a service that never builds an sdist cannot reach. Cost is 99 -> 110 packages, since transformers 5 grew a CLI and pulls typer/rich. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * test(qwen3-tts): cover the transformers 4.x/5.x compat layer Unit tests for the two mechanisms whose absence produced NaN logits and a shape mismatch: reset_rotary_buffers() recomputes a poisoned inv_freq to exactly the value transformers 4.57.3 computed (and is idempotent), and align_position_ids() keeps the newest positions rather than the oldest. Both run without model weights or network, in ~0.1 s. Plus guards against a future re-vendor silently reintroducing the 4.x call sites: no direct import of ROPE_INIT_FUNCTIONS, check_model_inputs, rope_config_validation or the mask builders; no bare config.pad_token_id; and both from_pretrained paths must still call reset_rotary_buffers. Each failure names the file and the symbol, because the runtime symptoms are expensive to trace back. test_vendor.py's file inventory now covers _compat.py, which is ours rather than upstream's and so is exempt from the Apache-notice check. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * docs: record the transformers 5 port and the 18 -> 2 advisory arc Replaces the claim that the port fails with what it actually took: which of the six breakages were signature bridging, which two were silent (uninitialized inv_freq, accumulated position_ids), how each was found, and the verification chain that shows the output did not move. Also notes what did *not* need work, and the package-count cost of transformers 5. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * Update PyThorch to 2.13.0 Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Expand TTS e2e to cover voice creation and add test for NPU Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * refactor(demo): drop the "Need Help" button now that help mode covers it Help mode answers "what is this control" for any annotated element, which is what the demo-mode "Need Help" button did for a fixed set of three targets (current mode, plus whichever settings sidebar was open). Two help affordances sitting next to each other in the title bar is worse than one. Removing the button makes its whole path unreachable, so that goes too: `triggerHelpForCurrentMode`, `triggerContextHelp`, `resolveDriverStep`, `contextHelpConfig` and the `ContextHelpTarget` type. The auto-started demo tour and the per-button first-time popovers (`triggerFirstTimeHelp`, driven by the notification dots) are untouched — they are separate entry points. The tour's welcome step was anchored to the button's wrapper and its copy told people to use it, so it now anchors to the `?` toggle and points there instead. `TOUR_ONLY_ANCHORS` becomes an exported constant that both the `TourStep` id type and the parity test derive from, with a new test asserting a tour-only anchor is never also a click-to-learn topic — the help toggle must not be a topic, or help mode would try to explain itself. `DEMO_NEED_HELP` is left in the locale files: demoMode.ts documents the other DEMO_* keys as legacy leftovers that are kept, so removing just this one across 13 files would be inconsistent churn. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * Hide Phison models in other presets Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> --------- Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> Co-authored-by: Qiacheng Li <qiacheng.li@intel.com> Co-authored-by: Markus Schuettler <markus.schuettler@tngtech.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * Centralize audio features Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add standalone Whisper for Nvidia version Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve and fix bugs in TTS & STT Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix more bugs Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Implement review suggestions Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix further bugs Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Harden install-reinstall of BEs Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Further hardening of BE installation Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix mic availability with STT Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Implement review suggestions Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Allow running LAN client under Linux without encryption if not available. Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Make OpenVINO whisper equivalent in features to stand alone Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Extract leftover-venv helpers and wipe husks before uv venv Move interpreter-exists checks into venvState so every uv venv/install path deletes a leftover .venv without python.exe (uv --allow-existing would otherwise keep it). Short-circuit checkBackendWithDetails when the interpreter is missing, and treat ComfyUI needsInstallation as not installed so a lockfile 'create' cannot auto-start. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com> * Fall back from standalone Whisper when it is not ready resolveTranscription no longer returns the sidecar URL just because the backend is installed. If the model is missing or the service is not running, a configured external endpoint is used instead. Also mention standalone Whisper in the non-NVIDIA STT setup hint. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com> * Finish remaining CodeRabbit findings from PR 278 Keep disabled backends on the main-process registry so auto-start honors the list even when startAllSetUpServices is called without args, and sync it when settings change. Prefer standalone Whisper in effectiveSttEngine only when its model is present; otherwise keep a configured external fallback. Align the non-NVIDIA voice-input hint with the review wording. Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com> * Start TTS and STT backends only when a feature requests them (#281) Installed Qwen3-TTS and standalone Whisper sidecars no longer start at boot or after install. Device detection still runs so the picker is accurate; first synthesize/transcribe starts the process. OVMS Kokoro and Whisper servers similarly stay down until first use. Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> * Improve TTS/STT Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve TTS voice reproducibility Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Improve TTS voice reproducibility 2 Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix up TTS/STT after rebase onto dev The rebase pulled in the Agent Mode / game-agent work from dev (#276), which overlaps this branch in three places: - ai SDK v6 -> v7 renamed the tool execution context, so transcribeAudio moves from `experimental_context` to `context` like the other tools. - dev's speech-fallback fix ("a configured endpoint is the source, never touch OVMS") predates this branch's engine selection. Ported to the new model: the guard now keys off the External engine rather than a configured fallback, in the code paths that still exist (ensure*ServerRunning / initialize) — the `enabled`/`toggle` pair it originally guarded no longer has callers. Its test is rewritten against the same rule. - The mode helpers (presetModes, PromptStatusBar, PromptArea, setupWizard) now carry `audio` alongside dev's `agent` instead of duplicating the mappings. Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix on-demand backends resolving against a stale status Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> --------- Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> Signed-off-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Leslie Lazzarino <62446606+leslielazzarino@users.noreply.github.com> Co-authored-by: Qiacheng Li <qiacheng.li@intel.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com> Co-authored-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
feat: consolidate session / chat / media history
* Add e2e tests to new agentic presets Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix bug Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix vulnerabilities and upgrade dependencies Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Major upgrades ComfyUI v0.25.1→v0.34.0, llama.cpp b10472→b10666, @types/node 26, just-bash 3.4.2, pinia 4.0.3, ai 7.0.83 Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix e2e tests for harness items Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Update vite config Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Add reset preset to e2e tests to increase reliability Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> * Fix unit tests for Linux Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com> --------- Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
Signed-off-by: Leslie Lazzarino <leslie.lazzarino@tngtech.com>
Co-authored-by: Markus Schüttler <mschuettlerTNG@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agent Mode / Game Agent
Models
Speech
History
Phison
Backends
v0.25.1→v0.34.0b9763→b106662026.3.0.8022ddae3→2026.4.0.e5e9afa2UX / reliability
Tests
Testing Done:
Tested locally on B580 and PTL.
Checklist: