Add project-aware Coproducer and MCP server by bmatherly · Pull Request #1941 · mltframework/shotcut

bmatherly · 2026-10-20T01:32:17Z

Summary

This PR adds Coproducer, a project-aware AI assistant for Shotcut, and exposes its tools through a Streamable HTTP MCP endpoint. Coproducer and external MCP clients use the same endpoint.

Coproducer

Adds a QML-based Coproducer dock, opened with Ctrl+Shift+7.
Adds Chat and Search User Guide entries to Help.
Adds two interchangeable AI provider modes:
- OpenAI-compatible API: configure the API key, model, and base URL. HTTPS is required except for localhost endpoints.
- Local model: select a GGUF model; Shotcut starts and manages llama-server.
AI Chat sends model-requested tools through MCP for project inspection, timeline summaries, and forum search.
Search User Guide calls the forum-search MCP tool directly, without invoking an AI model.
Streams Markdown responses, supports citations, and includes cancellation and new-chat controls.

MCP

Implements a stateless Streamable HTTP endpoint at /mcp, with dynamic localhost ports and MCP initialization, tool discovery, and tool calls.
Adds Help → Copy MCP Endpoint so users can provide the current Shotcut instance’s URL to external MCP clients.
Registers three tools: shotcut_project_inspect, shotcut_timeline_summary, and shotcut_forum_search.
Project inspection returns read-only clip, playlist, timeline, and UI context. Timeline summaries can omit per-track details to reduce payload size.
Restricts connections to loopback and validates Host and Origin headers.
Moves MCP-only forum search and project inspection into the mcp/ module.

Testing

MCP tests cover initialization, tool listing and calls, per-instance isolation, HTTP Origin and GET handling, and timeline summary track inclusion.
MCP Inspector was used to verify connection, tool discovery, and tool calls.

Adds a new "Coproducer" dock (Ctrl+Shift+7) that lets users chat with an AI assistant about Shotcut and video editing, with two interchangeable backends: - OpenAI-compatible API: any https endpoint (OpenAI, OpenRouter, Groq, etc.), or http for localhost only. Streams responses via SSE, supports tool calling, and validates the base URL/API key before sending. - Local model: bundles and manages a llama-server subprocess (built statically from llama.cpp b11240 with Vulkan/OpenMP/LTO where applicable), downloadable via a new GGUF model picker reusing the existing QmlExtension download infrastructure (llamamodel.qml). Architecture: - ChatProvider: vendor-neutral interface (streaming tokens, tool calls, usage) implemented by OpenAiCompatibleProvider and LocalLlamaServerProvider, so neither provider depends on the other; both just depend on the shared OpenAI-protocol client. - ForumSearch: a "shotcut_forum_search" tool the model can invoke to search forum.shotcut.org and cite numbered sources, plus a forum-only "Help Search" mode that skips the model entirely. - CoproducerDock: Markdown-rendered streaming transcript with rounded prompt bubbles, citation links, cancel/new-chat controls, and a settings dialog for provider/model/API key/local model management. Build/packaging: - scripts/build-shotcut.sh: adds the llama.cpp (llama-server only, static) subproject, enabled via ENABLE_LLAMACPP and the new SHOTCUT_ENABLE_LLAMA_SERVER CMake option, with Release+LTO and OpenMP/Vulkan optimizations regardless of the overall debug setting. - New go-up/view-conversation-balloon icons for the dock's send button and toggle action.

Add a read-only project inspection tool that exposes the current clip, playlist, and timeline state to Coproducer in structured JSON. This includes active context, open producer metadata, clip catalog summaries, track information, and UI state needed for context-aware assistance. The tool is designed to be non-destructive and lightweight: it only reads project state and returns compact metadata for AI tool calls. It also resolves user-facing clip names from the correct producer source so recorded media and generated sources are described naturally in the model context. This enables richer project-aware AI interactions without modifying timeline contents or project data.

Also, add help entries in the help menu

Route Coproducer tools through MCP, bind each instance to a dynamic localhost port, and move MCP-only search and project inspection into the MCP module. Add transport, security, instance-isolation, and tool behavior tests.

ddennedy · 2026-10-20T16:46:23Z

 OPENBLAS_REVISION="v0.3.34"
 WHISPERCPP_HEAD=0
 WHISPERCPP_REVISION="v1.8.3"
+ENABLE_LLAMACPP=1


build-shotcut-msys2.sh is not updated, but llama.cpp is included with msys2. We often prefer their dependencies in the Windows builds since they are always very current. One thing we need to be concerned about is runtime CPU feature selection but per a chat query:

Yes. Runtime CPU feature selection is in the ggml CPU backend, and it does not depend on OpenBLAS.

Each ggml-cpu-*.dll (haswell, skylakex, zen4, sapphirerapids, and the rest) is built for one instruction set. At startup ggml loads those DLLs, each one reports a score through ggml_backend_score based on CPUID checks such as ggml_cpu_has_avx2, ggml_cpu_has_avx512, and ggml_cpu_has_avx_vnni, and the highest-scoring variant is used. Your UCRT64 package already ships that set next to ggml-blas.dll.

I will have to spend some time on Windows to see about this.

You can add it to the build script and then to test it, manually trigger the GitHub Actions build-shotcut-windows workflow, choosing this branch. When it is done, return the build summary page and you can download a zip artifact. That zip contains the portable zip. Kind of weird, but GitHub Actions always zips what you tell it to provide as an artifact even if it is already a zip.

ddennedy · 2026-10-20T18:55:35Z

I tried it with local llama.cpp in my PATH as well as MCP from a Grok Build CLI session, and it worked (as well as forum search).

ddennedy · 2026-10-20T19:22:20Z

+    m_history.clear();
+    m_history.append({Coproducer::ChatMessage::System, QString::fromLatin1(kMasterSystemPrompt)});


The + button in Coproducer is mysterious to me. It leads me to believe I can add separate chat sessions and switch between them, but instead I think it just clears the chat.

I changes the + to the cancel icon and added a tooltip. I wonder if we should also add sessions. It wouldn't be that hard to do.

We mainly use X elsewhere for Close-like actions. There is an "edit-clear" icon that is used by the LineEditClear widget that we use for search fields (except, I just noticed Markers search I am fixing now).

Leave sessions for a later PR, and a session list needs an automatic subject generation and a search function.

Use "AI Chat" consistently

Set reasoning_effort to none for the official OpenAI endpoint to allow function tools through Chat Completions. Leave other providers unchanged. Preserve HTTP error bodies instead of consuming them as streaming data. Include the model, HTTP status, API error details, and request ID in reported errors. Track the finish reason to help diagnose empty responses.

Now the llm can choose documentation search or user discussion search

Cancel chat and MCP requests and unload the provider before _Exit() bypasses destructors, preventing leftover llama-server processes.

ddennedy · 2026-10-29T18:56:05Z

Rebase or click the "Update branch" to merge master into here to resolve the clang-format check if so desired.

ddennedy · 2026-10-30T16:50:21Z

+    QString m_statusBase;
+    int m_thinkingSeconds = 0;
+    bool m_busy = false;
+    Mode m_mode = Mode::AiChat;


I want the mode to default from last-used in settings (sticky) because those who do not want AI can leave it on search. Simply defaulting to search would make it inconvenient for those who prefer the chat. I guess the settings default should be chat because search is in the Help menu.

bmatherly added 4 commits October 18, 2026 21:40

Convert Coproducer dock to QML

1bd77bd

Also, add help entries in the help menu

Add per-instance Streamable HTTP MCP support

0525eca

Route Coproducer tools through MCP, bind each instance to a dynamic localhost port, and move MCP-only search and project inspection into the MCP module. Add transport, security, instance-isolation, and tool behavior tests.

ddennedy requested changes Oct 20, 2026

View reviewed changes

ddennedy reviewed Oct 20, 2026

View reviewed changes

Comment thread src/qml/views/coproducer/coproducer.qml Resolved

ddennedy reviewed Oct 20, 2026

View reviewed changes

Comment thread src/coproducer/localllamaserverprovider.cpp Resolved

ddennedy reviewed Oct 21, 2026

View reviewed changes

Comment thread src/qml/views/coproducer/coproducer.qml Resolved

bmatherly added 11 commits October 27, 2026 22:03

Make combobox strings translatable

644df2a

Use "AI Chat" consistently

Make response test selectable (and therefore copyable)

0584d73

Fix icon for coproducer panel

bd46d0a

Enable GPU for llama.cpp server

938bb8a

ignore test artifacts

771fed4

Add user discussion search

568a1dc

Now the llm can choose documentation search or user discussion search

Clarify "clear conversation" button

e9bec0f

Change coproducer menu icon to cog for settings

bba9af7

Stop Coproducer's local server before application exit

5f051db

Cancel chat and MCP requests and unload the provider before _Exit() bypasses destructors, preventing leftover llama-server processes.

Specify that the user guide is English only

7163a7f

ddennedy reviewed Oct 30, 2026

View reviewed changes

		m_history.clear();
		m_history.append({Coproducer::ChatMessage::System, QString::fromLatin1(kMasterSystemPrompt)});

Conversation

bmatherly commented Oct 20, 2026

Summary

Coproducer

MCP

Testing

Uh oh!

Uh oh!

Uh oh!

ddennedy Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

bmatherly Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

ddennedy Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ddennedy commented Oct 20, 2026

Uh oh!

ddennedy Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

bmatherly Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

ddennedy Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ddennedy commented Oct 29, 2026

Uh oh!

ddennedy Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants