Skip to content

Commit 5cc5bb2

Browse files
committed
Add Google Gemini and custom OpenAI-compatible providers
Google Gemini: a genuine first-class provider (app/src/providers/gemini.ts) - its request/response shape is meaningfully different from the OpenAI-style APIs (system prompt is a separate field, "assistant" is called "model", no "tool" role), so this is real client code, not a thin wrapper. Agent mode tool-calling isn't wired up for it yet (same honest scoping as the llama.cpp backend) - a clear error explains this if you try. Custom OpenAI-compatible providers: Groq, Mistral, DeepSeek, xAI (Grok), OpenRouter, and self-hosted servers all speak the same chat-completions API that OpenAI does, differing only in base URL - so rather than five near-duplicate per-vendor clients, extracted the existing OpenAI chat logic into a reusable createOpenAiCompatibleChat(baseUrl, label) factory (app/src/providers/openai-compatible.ts) and let users add any number of named endpoints in Settings -> Integrations, each with its own stored API key. Quick-add preset buttons fill in the base URL for the popular ones. Custom providers are identified by "<customProviderId>::<model>" packed into the existing "provider:model" session-storage format (formatCustomModelRef/parseCustomModelId in frontend/src/lib/providers.ts) rather than adding a second identifier field everywhere sessions/messages already thread a single model string through. Verified end-to-end in a live browser session: added a Groq preset, confirmed it appears in the custom providers list with no duplication, and confirmed the quick-add form correctly prefills name/base URL from the preset before submission.
1 parent b86bc8a commit 5cc5bb2

15 files changed

Lines changed: 642 additions & 149 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,8 @@ Beyond chat, Modelforge includes an **agentic mode** — the model can read/writ
2727
## Features
2828

2929
**Chat & providers**
30-
- Local Ollama models, OpenAI, Anthropic, **and llama.cpp** in one interface, with token-by-token streaming.
30+
- Local Ollama models, OpenAI, Anthropic, Google Gemini, **and llama.cpp** in one interface, with token-by-token streaming.
31+
- **Custom OpenAI-compatible providers** — add Groq, Mistral, DeepSeek, xAI (Grok), OpenRouter, a self-hosted server, or any other endpoint that speaks the OpenAI chat-completions format, via one shared implementation rather than a dedicated client per vendor. Quick-add presets fill in the base URL for the popular ones; you just paste an API key.
3132
- **llama.cpp backend** — run GGUF models directly via [node-llama-cpp](https://github.com/withcatai/node-llama-cpp) instead of Ollama, with **Vulkan**, CUDA, or Metal GPU acceleration (auto-detected, selectable in Settings). Useful for Vulkan acceleration or models Ollama doesn't package. This is an *additional* backend, not a replacement — Ollama is still fully supported side by side. Agent mode tool-calling isn't wired up for this backend yet (a clear error explains this if you try); everything else (streaming, GPU offload, all the generation parameters) works the same as any other provider. Model weights load once and stay warm across messages, but each turn currently re-evaluates the full conversation from scratch rather than reusing a cache across turns — correct, but slower on long conversations than it could be.
3233
- Vision support — attach images (or extract frames from a video) for models that can see them. When an image is attached, an **"Analyze as..."** menu fills the composer with a ready-made prompt for common diagram/wireframe tasks — describe the UI, convert it to a Mermaid diagram, generate React + Tailwind code from it, list its components, or review it for usability/accessibility issues.
3334
- Live token usage and estimated cost per message and per session (Ollama is free/local; cloud providers show a running estimate).

‎app/src/main.ts‎

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,15 +24,22 @@ import type { McpServerConfig } from "./mcp-client";
2424
import type { AttachedFile } from "./file-reader";
2525
import * as openaiProvider from "./providers/openai";
2626
import * as anthropicProvider from "./providers/anthropic";
27+
import * as geminiProvider from "./providers/gemini";
28+
import { createOpenAiCompatibleChat } from "./providers/openai-compatible";
2729
import { setupMenu } from "./menu";
2830
import { setupAutoUpdater, checkForUpdatesManually } from "./updater";
2931
import type { ChatMessage, ChatChunk, ChatOptions, ProviderId, ToolDefinition } from "./providers/types";
3032

31-
const PROVIDER_SECRET_KEYS: Record<Exclude<ProviderId, "ollama" | "llamacpp">, string> = {
33+
const PROVIDER_SECRET_KEYS: Record<Exclude<ProviderId, "ollama" | "llamacpp" | "custom">, string> = {
3234
openai: "openai_api_key",
3335
anthropic: "anthropic_api_key",
36+
gemini: "gemini_api_key",
3437
};
3538

39+
function customProviderSecretKey(customProviderId: string): string {
40+
return `custom_${customProviderId}_api_key`;
41+
}
42+
3643
const activeChatRequests = new Map<string, AbortController>();
3744
let isBusy = false;
3845
let forceClose = false;
@@ -177,11 +184,32 @@ async function dispatchChat(
177184
} else if (provider === "llamacpp") {
178185
const modelPath = path.join(getLlamaCppModelsDir(), model);
179186
await llamacpp.chat(modelPath, messages, options, onToken, signal, tools);
187+
} else if (provider === "custom") {
188+
// model is "<customProviderId>::<actual model id>" — see
189+
// frontend/src/lib/providers.ts's formatCustomModelRef.
190+
const sep = model.indexOf("::");
191+
if (sep === -1) throw new Error(`Malformed custom model reference: ${model}`);
192+
const customProviderId = model.slice(0, sep);
193+
const actualModel = model.slice(sep + 2);
194+
const config = settingsStore.getSettings().customProviders?.find((p) => p.id === customProviderId);
195+
if (!config) throw new Error(`Custom provider "${customProviderId}" is no longer configured.`);
196+
const apiKey = secretsStore.getSecret(customProviderSecretKey(customProviderId));
197+
if (!apiKey) throw new Error(`No API key set for ${config.name}. Add one in Settings.`);
198+
await createOpenAiCompatibleChat(config.baseUrl, config.name)(
199+
apiKey,
200+
actualModel,
201+
messages,
202+
options,
203+
onToken,
204+
signal,
205+
tools
206+
);
180207
} else {
181208
const secretKey = PROVIDER_SECRET_KEYS[provider];
182209
const apiKey = secretsStore.getSecret(secretKey);
183210
if (!apiKey) throw new Error(`No API key set for ${provider}. Add one in Settings.`);
184-
const providerFn = provider === "openai" ? openaiProvider.chat : anthropicProvider.chat;
211+
const providerFn =
212+
provider === "openai" ? openaiProvider.chat : provider === "anthropic" ? anthropicProvider.chat : geminiProvider.chat;
185213
await providerFn(apiKey, model, messages, options, onToken, signal, tools);
186214
}
187215
}

‎app/src/providers/gemini.ts‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
import { describeHttpError, describeNetworkError } from "./errors";
2+
import { streamSSE } from "./sse";
3+
import type { ChatFn, ChatMessage } from "./types";
4+
5+
interface GeminiPart {
6+
text?: string;
7+
inlineData?: { mimeType: string; data: string };
8+
}
9+
10+
interface GeminiContent {
11+
role: "user" | "model";
12+
parts: GeminiPart[];
13+
}
14+
15+
// Gemini's request shape is meaningfully different from the OpenAI-style APIs
16+
// (system prompt is a separate top-level field, "assistant" is called
17+
// "model", and there's no "tool" role) — a real implementation, not a thin
18+
// wrapper over openai-compatible.ts.
19+
function toGeminiContents(messages: ChatMessage[]): { system?: string; contents: GeminiContent[] } {
20+
let system: string | undefined;
21+
const contents: GeminiContent[] = [];
22+
for (const m of messages) {
23+
if (m.role === "system") {
24+
system = system ? `${system}\n\n${m.content}` : m.content;
25+
continue;
26+
}
27+
if (m.role === "tool") continue; // not supported yet — see chat() below
28+
const parts: GeminiPart[] = [];
29+
if (m.content) parts.push({ text: m.content });
30+
if (m.images) {
31+
for (const img of m.images) parts.push({ inlineData: { mimeType: img.mimeType, data: img.data } });
32+
}
33+
contents.push({ role: m.role === "assistant" ? "model" : "user", parts });
34+
}
35+
return { system, contents };
36+
}
37+
38+
export const chat: ChatFn = async (apiKey, model, messages, options, onToken, signal, tools) => {
39+
if (tools && tools.length > 0) {
40+
throw new Error(
41+
"Agent mode isn't supported yet for Gemini — switch to OpenAI, Claude, or a local model for tool-calling, or turn Agent mode off."
42+
);
43+
}
44+
45+
const { system, contents } = toGeminiContents(messages);
46+
const url = `https://generativelanguage.googleapis.com/v1beta/models/${encodeURIComponent(model)}:streamGenerateContent?alt=sse&key=${encodeURIComponent(apiKey)}`;
47+
48+
let res: Response;
49+
try {
50+
res = await fetch(url, {
51+
method: "POST",
52+
headers: { "Content-Type": "application/json" },
53+
body: JSON.stringify({
54+
contents,
55+
...(system ? { systemInstruction: { parts: [{ text: system }] } } : {}),
56+
generationConfig: {
57+
temperature: options?.temperature ?? 0.7,
58+
topP: options?.topP,
59+
topK: options?.topK,
60+
maxOutputTokens: options?.maxTokens,
61+
stopSequences: options?.stop && options.stop.length > 0 ? options.stop : undefined,
62+
},
63+
}),
64+
signal,
65+
});
66+
} catch (err) {
67+
throw describeNetworkError("Gemini", err);
68+
}
69+
70+
if (!res.ok || !res.body) {
71+
throw new Error(await describeHttpError(res, "Gemini"));
72+
}
73+
74+
await streamSSE(res, (payload) => {
75+
try {
76+
const parsed = JSON.parse(payload);
77+
const text = parsed.candidates?.[0]?.content?.parts
78+
?.map((p: GeminiPart) => p.text ?? "")
79+
.join("");
80+
if (text) onToken({ message: { role: "assistant", content: text }, done: false });
81+
82+
if (parsed.usageMetadata) {
83+
onToken({
84+
done: false,
85+
usage: {
86+
promptTokens: parsed.usageMetadata.promptTokenCount,
87+
completionTokens: parsed.usageMetadata.candidatesTokenCount,
88+
},
89+
});
90+
}
91+
} catch {
92+
// ignore malformed payload
93+
}
94+
});
95+
onToken({ done: true });
96+
};
Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
import { streamSSE } from "./sse";
2+
import { describeHttpError, describeNetworkError } from "./errors";
3+
import type { ChatFn, ToolDefinition } from "./types";
4+
5+
function toOpenAiTools(tools: ToolDefinition[]): unknown[] {
6+
return tools.map((t) => ({
7+
type: "function",
8+
function: { name: t.name, description: t.description, parameters: t.parameters },
9+
}));
10+
}
11+
12+
// The OpenAI chat-completions request/response shape (and its SSE streaming
13+
// format) has become a de-facto standard — Groq, Mistral, DeepSeek, xAI, and
14+
// OpenRouter (among others) all speak it, differing only in base URL and
15+
// which of the optional fields they actually honor. One implementation
16+
// parameterized by base URL covers all of them instead of near-duplicate
17+
// per-vendor clients.
18+
export function createOpenAiCompatibleChat(baseUrl: string, providerLabel: string): ChatFn {
19+
const chat: ChatFn = async (apiKey, model, messages, options, onToken, signal, tools) => {
20+
const chatMessages = messages.map((m) => {
21+
if (m.role === "tool") {
22+
return { role: "tool", tool_call_id: m.toolCallId, content: m.content };
23+
}
24+
25+
const toolCalls =
26+
m.toolCalls && m.toolCalls.length > 0
27+
? m.toolCalls.map((tc) => ({
28+
id: tc.id,
29+
type: "function",
30+
function: { name: tc.name, arguments: JSON.stringify(tc.arguments) },
31+
}))
32+
: undefined;
33+
34+
if (!m.images || m.images.length === 0) {
35+
return { role: m.role, content: m.content, ...(toolCalls ? { tool_calls: toolCalls } : {}) };
36+
}
37+
return {
38+
role: m.role,
39+
content: [
40+
...(m.content ? [{ type: "text", text: m.content }] : []),
41+
...m.images.map((img) => ({
42+
type: "image_url",
43+
image_url: { url: `data:${img.mimeType};base64,${img.data}` },
44+
})),
45+
],
46+
};
47+
});
48+
49+
let res: Response;
50+
try {
51+
res = await fetch(`${baseUrl.replace(/\/+$/, "")}/chat/completions`, {
52+
method: "POST",
53+
headers: {
54+
"Content-Type": "application/json",
55+
Authorization: `Bearer ${apiKey}`,
56+
},
57+
body: JSON.stringify({
58+
model,
59+
messages: chatMessages,
60+
stream: true,
61+
stream_options: { include_usage: true },
62+
temperature: options?.temperature ?? 0.7,
63+
top_p: options?.topP,
64+
max_tokens: options?.maxTokens,
65+
frequency_penalty: options?.frequencyPenalty,
66+
presence_penalty: options?.presencePenalty,
67+
seed: options?.seed,
68+
stop: options?.stop && options.stop.length > 0 ? options.stop : undefined,
69+
...(tools && tools.length > 0 ? { tools: toOpenAiTools(tools) } : {}),
70+
}),
71+
signal,
72+
});
73+
} catch (err) {
74+
throw describeNetworkError(providerLabel, err);
75+
}
76+
77+
if (!res.ok || !res.body) {
78+
throw new Error(await describeHttpError(res, providerLabel));
79+
}
80+
81+
// Tool-call arguments arrive as fragmented JSON string deltas, indexed
82+
// by position in the tool_calls array, across many SSE events — only
83+
// valid to parse once the stream signals finish_reason "tool_calls".
84+
const pendingToolCalls = new Map<number, { id: string; name: string; argsText: string }>();
85+
86+
function flushToolCalls(): void {
87+
if (pendingToolCalls.size === 0) return;
88+
const toolCalls = [...pendingToolCalls.values()].map((tc) => {
89+
let args: Record<string, unknown> = {};
90+
try {
91+
args = JSON.parse(tc.argsText || "{}");
92+
} catch {
93+
// malformed arguments — surface as empty rather than crash the stream
94+
}
95+
return { id: tc.id, name: tc.name, arguments: args };
96+
});
97+
pendingToolCalls.clear();
98+
onToken({ done: false, toolCalls });
99+
}
100+
101+
await streamSSE(res, (payload) => {
102+
if (payload === "[DONE]") {
103+
flushToolCalls();
104+
onToken({ done: true });
105+
return;
106+
}
107+
try {
108+
const parsed = JSON.parse(payload);
109+
const choice = parsed.choices?.[0];
110+
const delta = choice?.delta;
111+
112+
if (delta?.content) {
113+
onToken({ message: { role: "assistant", content: delta.content }, done: false });
114+
}
115+
116+
if (delta?.tool_calls) {
117+
for (const tc of delta.tool_calls as Array<{
118+
index: number;
119+
id?: string;
120+
function?: { name?: string; arguments?: string };
121+
}>) {
122+
const existing = pendingToolCalls.get(tc.index) ?? { id: tc.id ?? "", name: "", argsText: "" };
123+
if (tc.id) existing.id = tc.id;
124+
if (tc.function?.name) existing.name = tc.function.name;
125+
if (tc.function?.arguments) existing.argsText += tc.function.arguments;
126+
pendingToolCalls.set(tc.index, existing);
127+
}
128+
}
129+
130+
if (choice?.finish_reason === "tool_calls") {
131+
flushToolCalls();
132+
}
133+
134+
if (parsed.usage) {
135+
onToken({
136+
done: false,
137+
usage: {
138+
promptTokens: parsed.usage.prompt_tokens,
139+
completionTokens: parsed.usage.completion_tokens,
140+
},
141+
});
142+
}
143+
} catch {
144+
// ignore malformed payload
145+
}
146+
});
147+
};
148+
return chat;
149+
}

0 commit comments

Comments
 (0)