diff --git a/.gitignore b/.gitignore index fb4fced..b608034 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,11 @@ playwright-report/ # Local embedding model weights downloaded by scripts/embed-smoke.mjs .embed-smoke-cache/ +# Unpacked ONNX generated from the committed gzip packs (see vendor/embed-models/README.md). +vendor/embed-models/**/*.onnx +vendor/embed-models/**/*.onnx_data +vendor/embed-models/**/*.tmp + # External folders connected for in-place reading (separate repos) .external/ diff --git a/docs/user/settings.md b/docs/user/settings.md index 7b49e82..7479159 100644 --- a/docs/user/settings.md +++ b/docs/user/settings.md @@ -49,7 +49,7 @@ Commands run on the machine Stem itself runs on: your own computer normally, or server if you [moved Stem to one](../running-on-a-server.md) — so they see the programs installed there, not the ones on the computer you happen to be typing on. On macOS and Linux they run under `zsh`, or `bash`/`sh` on a machine without it; on Windows under -`cmd.exe`. +Git Bash when it is installed, otherwise `cmd.exe`. - **Manual**: known-safe and always-allowed commands run; everything else asks first. - **Assisted**: an AI safety check passes routine commands and asks about uncertain @@ -60,6 +60,13 @@ Linux they run under `zsh`, or `bash`/`sh` on a machine without it; on Windows u On an approval card, **Always allow** saves a command prefix for future turns. Keep prefixes narrow; `git status` grants less access than `git`. +On Windows, **Windows shell** defaults to Git Bash when `bash.exe` is on disk, and +falls back to Command Prompt if it is not. Stem looks for Git Bash without using +PowerShell. If it is not in a usual place, paste the path to `bash.exe`. Commands +then run in that one shell — quoting and the always-allowed list follow it (`dir` +vs `ls`). Pick Command Prompt yourself if you want cmd.exe even though Git is +installed. + ### Commands on your own computer diff --git a/docs/windows-dev.md b/docs/windows-dev.md index da26fb2..8740bb2 100644 --- a/docs/windows-dev.md +++ b/docs/windows-dev.md @@ -81,23 +81,38 @@ node node_modules\electron\install.js ## Shell Stem uses for `run_command` -On Windows, approved commands run as: +On Windows, approved commands run in **Git Bash** when Git for Windows is +installed, and fall back to **Command Prompt** otherwise: + +Git Bash (when `bash.exe` is on disk): + +`bash.exe --noprofile --norc -c ""` + +`--noprofile --norc` skips `.bashrc` / `/etc/profile` (the same idea as cmd `/d`). +Git’s `usr\bin` is prepended to PATH so `ls` / `cat` / `grep` work. The safety +parser then follows **bash** quoting, not cmd’s — `ls` auto-runs, `dir` does not. + +Stem looks for `bash.exe` on disk (usual Git for Windows paths, then PATH) +without running PowerShell. If Git is installed somewhere unusual, paste the +path to `bash.exe` under Settings → Chat → Command execution. + +Command Prompt fallback (no Git Bash, or you pick it in Settings): `cmd.exe /d /s /c ""` - `/d` disables AutoRun (registry hooks that behave like a login profile). -- Stem does **not** load PowerShell’s `profile.ps1` for the default path. +- Stem does **not** load PowerShell’s `profile.ps1` for this path. - The command is wrapped in quotes and spawned with `windowsVerbatimArguments` so inner `"` (e.g. PowerShell `-Command "..."`) are not turned into `\"`. -### What auto-runs, and what doesn’t +### What auto-runs, and what doesn’t (cmd.exe) -The safety tiers are the same as on macOS, but the parser follows **cmd.exe** -rules, not zsh’s. That changes which commands can skip the safety check: +The safety tiers are the same as on macOS, but the **cmd.exe** parser is not zsh’s. +That changes which commands can skip the safety check: - Read-only probes auto-run: `dir`, `type`, `where`, `echo`, `cd`, `git status` - and friends. The POSIX names (`ls`, `cat`, `grep`) are not on the Windows - allowlist — under cmd they are not commands. + and friends. The POSIX names (`ls`, `cat`, `grep`) are not on the cmd allowlist + — under cmd they are not commands. - `'` is **not** a quote character to cmd, so anything containing one goes to the safety check rather than auto-running. `cmd` would read `type 'a & whoami'` as two commands, and Stem will not auto-run something it cannot bound. @@ -127,14 +142,22 @@ Or avoid pipes with `(...)` / property access when that is enough 1. `node -v` ≥ 24 and `npm -v` with portable Node on PATH. 2. `npm install` → `npm run preflight` → `npm run dev` opens Stem. 3. Complete onboarding / chat with a provider. -4. Ask Stem to run `echo hello`, `dir`, or `git status` — expect a normal result - (or an approval card), not a spawn/`zsh` error. +4. Ask Stem to run `echo hello`, `ls`, or `git status` — expect a normal result + (or an approval card), not a spawn/`zsh` error. Without Git Bash, `dir` is the + Command Prompt equivalent. 5. Confirm a broken `profile.ps1` did not fire for those default commands. 6. Optional: have Stem run the `-NoProfile` PowerShell one-liner above. -7. Assisted mode: ask for `type 'a & whoami & rem '`. It must show an approval - card, never run — cmd would split that into three commands. -8. Connect a folder read-only, then ask Stem to `type` a file inside it. Expect +7. Assisted mode: ask for `cat 'a & whoami'` (Git Bash) or `type 'a & whoami & rem '` + (cmd). It must show an approval card, never run. +8. Connect a folder read-only, then ask Stem to `cat` / `type` a file inside it. Expect the read-only refusal, not the file. 9. Check that `%APPDATA%\Stem\` appears and survives a restart. -10. Memory / search: if hybrid embeddings fail, check the main log for - `embed-endpoint` / named-pipe errors (FTS-only fallback is safe but weaker). +10. Memory / search: the default embedder is shipped as gzip parts under + `vendor/embed-models/` and unpacked into `%APPDATA%\Stem\embed-models` on + first launch (no Hugging Face). The reranker still downloads from the Hub + when it can; if that host is blocked, ranking falls back to embeddings + alone. Named-pipe errors in the log (`embed-endpoint`) only affect MCP + hybrid search; FTS-only fallback is safe but weaker. +11. Settings → Chat → Command execution → Windows shell should already be Git Bash + when `bash.exe` was found. Ask Stem to run `ls`. Switch to Command Prompt and + `dir` if you want the cmd parser. diff --git a/electron-builder.yml b/electron-builder.yml index 887178a..c234a3f 100644 --- a/electron-builder.yml +++ b/electron-builder.yml @@ -39,6 +39,9 @@ files: - dist/** - build/icon.png # loaded at runtime via app.getAppPath() (server/index.ts appIcon) - RELEASE_NOTES.md # ditto — the "what's new" popup reads it (workspace/release-notes.ts) + # Optional local ONNX cache (vendor/embed-models/README.md). Empty in git; a + # private fork that copies weights here ships them with the packaged app. + - vendor/embed-models/** # pdf.js is only used for folder-index text extraction (server/folder-index/pdf.ts), # which loads legacy/build/pdf.mjs + its fake-worker import pdf.worker.mjs. # The browser builds, rendering assets (cmaps/fonts/wasm/icc), minified diff --git a/package.json b/package.json index 0baa415..e4e3985 100644 --- a/package.json +++ b/package.json @@ -34,7 +34,8 @@ "eval:skills": "node scripts/skill-author-eval.mjs", "eval:skill-retrieval": "node scripts/skill-retrieval-eval.mjs", "fixtures:skills": "node scripts/skill-fixtures.mjs", - "gen:shortcuts-doc": "node scripts/gen-shortcuts-doc.mjs" + "gen:shortcuts-doc": "node scripts/gen-shortcuts-doc.mjs", + "vendor:embed-models": "node scripts/vendor-embed-models.mjs" }, "dependencies": { "@earendil-works/pi-coding-agent": "0.82.0", diff --git a/scripts/vendor-embed-models.mjs b/scripts/vendor-embed-models.mjs new file mode 100644 index 0000000..afe0613 --- /dev/null +++ b/scripts/vendor-embed-models.mjs @@ -0,0 +1,111 @@ +#!/usr/bin/env node +// Pack Stem's downloaded embedding weights into vendor/embed-models/ so a +// clone can load them without Hugging Face. GitHub rejects files over 100 MB +// and warns above 50 MB, so each ONNX is gzipped and split into 45 MB parts. +// +// npm run vendor:embed-models [sourceDir] +// +// Default source is this machine's Stem cache (override with STEM_EMBED_MODELS_DIR). +// Stem unpacks the parts into the app cache on first launch. +import { + copyFileSync, + createReadStream, + createWriteStream, + existsSync, + mkdirSync, + openSync, + readSync, + closeSync, + readdirSync, + renameSync, + statSync, + unlinkSync, + writeFileSync +} from 'node:fs'; +import { homedir } from 'node:os'; +import { dirname, join, relative } from 'node:path'; +import { pipeline } from 'node:stream/promises'; +import { createGzip } from 'node:zlib'; +import { fileURLToPath } from 'node:url'; + +const PART_BYTES = 45 * 1024 * 1024; +const GZIP_FROM_BYTES = 10 * 1024 * 1024; +const root = fileURLToPath(new URL('..', import.meta.url)); +const destRoot = join(root, 'vendor', 'embed-models'); + +function defaultCache() { + if (process.env.STEM_EMBED_MODELS_DIR) return process.env.STEM_EMBED_MODELS_DIR; + if (process.platform === 'darwin') return join(homedir(), 'Library', 'Application Support', 'Stem', 'embed-models'); + if (process.platform === 'win32') { + return join(process.env.APPDATA || join(homedir(), 'AppData', 'Roaming'), 'Stem', 'embed-models'); + } + return join(process.env.XDG_CONFIG_HOME || join(homedir(), '.config'), 'Stem', 'embed-models'); +} + +function walkFiles(dir) { + const out = []; + for (const ent of readdirSync(dir, { withFileTypes: true })) { + const p = join(dir, ent.name); + if (ent.isDirectory()) out.push(...walkFiles(p)); + else if (ent.isFile()) out.push(p); + } + return out; +} + +function shouldGzip(file, size) { + const lower = file.toLowerCase(); + return lower.endsWith('.onnx') || lower.endsWith('.onnx_data') || size >= GZIP_FROM_BYTES; +} + +async function gzipFile(src, destGz) { + mkdirSync(dirname(destGz), { recursive: true }); + const tmp = `${destGz}.tmp`; + await pipeline(createReadStream(src), createGzip({ level: 9 }), createWriteStream(tmp)); + const size = statSync(tmp).size; + if (size <= PART_BYTES) { + renameSync(tmp, destGz); + console.log(` packed ${relative(destRoot, destGz)} (${(size / 1024 / 1024).toFixed(1)} MB)`); + return; + } + const fd = openSync(tmp, 'r'); + let offset = 0; + let i = 0; + const buf = Buffer.alloc(PART_BYTES); + while (offset < size) { + const n = readSync(fd, buf, 0, PART_BYTES, offset); + const part = `${destGz}.${String(i).padStart(2, '0')}`; + writeFileSync(part, buf.subarray(0, n)); + console.log(` packed ${relative(destRoot, part)} (${(n / 1024 / 1024).toFixed(1)} MB)`); + offset += n; + i += 1; + } + closeSync(fd); + unlinkSync(tmp); +} + +const source = process.argv[2] || defaultCache(); +if (!existsSync(source)) { + console.error(`No embedding cache at ${source}\nRun Stem once on a machine that can reach huggingface.co, then retry.`); + process.exit(1); +} + +const files = walkFiles(source); +if (!files.length) { + console.error(`${source} has no model files yet.`); + process.exit(1); +} + +mkdirSync(destRoot, { recursive: true }); +for (const abs of files) { + const rel = relative(source, abs); + const size = statSync(abs).size; + if (shouldGzip(abs, size)) { + await gzipFile(abs, join(destRoot, `${rel}.gz`)); + } else { + const dest = join(destRoot, rel); + mkdirSync(dirname(dest), { recursive: true }); + copyFileSync(abs, dest); + console.log(` copied ${rel}`); + } +} +console.log(`OK → ${destRoot}`); diff --git a/src/preload/index.ts b/src/preload/index.ts index 3bb3357..71c25b2 100644 --- a/src/preload/index.ts +++ b/src/preload/index.ts @@ -219,6 +219,7 @@ const api: StemApi = { }, respondExecApproval: (id: string, decision: ExecDecision) => ipcRenderer.invoke('exec:resolveApproval', id, decision), + detectGitBash: () => ipcRenderer.invoke('exec:detectGitBash'), getScratchUsage: () => ipcRenderer.invoke('exec:scratchUsage'), clearScratch: (key: string) => ipcRenderer.invoke('exec:clearScratch', key), onMcpChanged: (listener: () => void) => { diff --git a/src/renderer/manage/tabs/settings/ChatSettings.tsx b/src/renderer/manage/tabs/settings/ChatSettings.tsx index a86f77c..37a6e82 100644 --- a/src/renderer/manage/tabs/settings/ChatSettings.tsx +++ b/src/renderer/manage/tabs/settings/ChatSettings.tsx @@ -6,7 +6,8 @@ import type { DeviceInfo, ExecSettings, ScratchUsageRow, - WebSearchSettings + WebSearchSettings, + WindowsShell } from '../../../../shared/types'; import { InfoTip } from '../../../ui/InfoTip'; import { ModelPicker } from '../../../ui/ModelPicker'; @@ -60,6 +61,9 @@ export function ChatSettings({ models, modelId, onSelectModel }: ModelTabProps) const [chats, setChats] = useState(null); const [exec, setExec] = useState(null); const [allowInput, setAllowInput] = useState(''); + const [detectedBash, setDetectedBash] = useState(null); + const [bashPathDraft, setBashPathDraft] = useState(''); + const [bashPathError, setBashPathError] = useState(''); // null while the walk is still running — sizing every chat's folder is a disk // walk on the server, so the block says "Measuring…" rather than "0 folders". const [scratch, setScratch] = useState(null); @@ -74,6 +78,7 @@ export function ChatSettings({ models, modelId, onSelectModel }: ModelTabProps) const [devices, setDevices] = useState([]); // Per-field debounce so typing doesn't spam the atomic settings writer. const ciMainTimer = useRef | null>(null); + const bashPathTimer = useRef | null>(null); useEffect(() => { void window.stem.getSettings().then((s) => { @@ -81,10 +86,14 @@ export function ChatSettings({ models, modelId, onSelectModel }: ModelTabProps) setCi(s.customInstructions); setChats(s.chats); setExec(s.exec); + setBashPathDraft(s.exec.gitBashPath ?? ''); }); // Its own request: a disk walk should not hold up the settings the rest of // this tab is made of. void window.stem.getScratchUsage().then(setScratch).catch(() => setScratch([])); + if (window.stem.platform === 'win32') { + void window.stem.detectGitBash().then(setDetectedBash).catch(() => setDetectedBash(null)); + } void window.stem.execHostState().then((s) => setExecHostEnabled(s.enabled)).catch(() => undefined); void window.stem .listDevices() @@ -116,7 +125,54 @@ export function ChatSettings({ models, modelId, onSelectModel }: ModelTabProps) function updateExec(patch: Partial) { setExec((cur) => (cur ? { ...cur, ...patch } : cur)); // optimistic; reconcile below - window.stem.updateExecSettings(patch).then((s) => setExec(s.exec)); + window.stem.updateExecSettings(patch).then((s) => { + setExec(s.exec); + if (patch.gitBashPath !== undefined || patch.windowsShell !== undefined) { + setBashPathDraft(s.exec.gitBashPath ?? ''); + } + }); + } + + async function chooseWindowsShell(next: WindowsShell) { + if (!exec) return; + if (next === 'cmd') { + setBashPathError(''); + updateExec({ windowsShell: 'cmd' }); + return; + } + const path = (bashPathDraft.trim() || exec.gitBashPath || detectedBash || '').trim(); + if (!path) { + setBashPathError('Git Bash was not found. Paste the path to bash.exe, then choose Git Bash again.'); + return; + } + setBashPathError(''); + updateExec({ windowsShell: 'git-bash', gitBashPath: path }); + } + + function saveGitBashPath(value: string) { + setBashPathDraft(value); + if (bashPathTimer.current) clearTimeout(bashPathTimer.current); + bashPathTimer.current = setTimeout(() => { + const trimmed = value.trim(); + if (!trimmed) { + // Empty path keeps Git Bash selected; spawn auto-detects or falls back to cmd. + updateExec({ gitBashPath: null }); + return; + } + updateExec({ + gitBashPath: trimmed, + windowsShell: exec?.windowsShell === 'git-bash' ? 'git-bash' : exec?.windowsShell + }); + }, 400); + } + + async function browseGitBash() { + const files = await window.stem.openFiles(); + const picked = files[0]; + if (!picked) return; + setBashPathError(''); + setBashPathDraft(picked); + updateExec({ windowsShell: 'git-bash', gitBashPath: picked }); } function clearScratch(key: string) { @@ -274,6 +330,52 @@ export function ChatSettings({ models, modelId, onSelectModel }: ModelTabProps) {exec?.enabled && ( <> + {window.stem.platform === 'win32' && ( +
+ + Windows shell{' '} + + Commands run in Git Bash when bash.exe is on disk, and fall back to Command + Prompt (cmd.exe) if it is not. Stem looks for Git for Windows in the usual + places (no PowerShell). If it is installed somewhere unusual, paste the path. + Switching shells changes which commands auto-run (dir vs ls) and how quotes + work. + + +
+ + +
+ {(exec.windowsShell === 'git-bash' || bashPathError) && ( + <> +
+ saveGitBashPath(e.target.value)} + /> + +
+ {bashPathError && {bashPathError}} + + )} +
+ )}
Approval mode{' '} diff --git a/src/renderer/styles.css b/src/renderer/styles.css index 2b9dde4..e2a8471 100644 --- a/src/renderer/styles.css +++ b/src/renderer/styles.css @@ -2872,6 +2872,13 @@ body.platform-win32 .hud-pill { background: color-mix(in srgb, var(--surface) 94 line-height: 1; } +.exec-bash-path { + display: flex; + gap: 8px; + align-items: center; +} +.exec-bash-path .ifield { flex: 1; min-width: 0; } + /* Settings → Chat → Command execution: what each chat's commands left on disk. */ .scratch-usage { margin-top: 8px; diff --git a/src/server/exec/executor.ts b/src/server/exec/executor.ts index 2888a66..e172da0 100644 --- a/src/server/exec/executor.ts +++ b/src/server/exec/executor.ts @@ -1,13 +1,17 @@ import { spawn } from 'node:child_process'; import { existsSync } from 'node:fs'; +import type { HostShell } from '../../shared/types'; +import { hostShellFromPlatform } from './host-shell'; // Spawns approved run_command commands. Runs in main (the privileged process). // On macOS/Linux: the host shell (see unixShell) `-c`, with the user's // LOGIN-shell PATH — a GUI app's environment lacks Homebrew/npm bin dirs, so // without this `agent-browser` & co. would be "command not found" even when // installed. -// On Windows: `cmd.exe /d /s /c` (no AutoRun; avoids a broken PowerShell -// profile.ps1). PATH comes from the process environment (Path/PATH). +// On Windows: `cmd.exe /d /s /c` by default (no AutoRun; avoids a broken +// PowerShell profile.ps1). Opt-in Git Bash is `bash.exe --noprofile --norc -c` +// (same idea: skip .bashrc). PATH comes from the process environment, plus +// Git's usr\bin when Git Bash is the host shell. export const DEFAULT_TIMEOUT_MS = 60_000; export const MAX_TIMEOUT_MS = 300_000; @@ -29,6 +33,11 @@ export interface ShellInvocation { args: string[]; /** POSIX process-group kill via negative PID; false on Windows (taskkill). */ detached: boolean; + /** + * Node must not rewrite quotes. True only for cmd's `/c "..."` form; + * Git Bash and zsh use normal argv quoting. + */ + verbatimArguments: boolean; } /** @@ -73,21 +82,33 @@ export function resetShellCacheForTests(): void { /** * Build the argv used to run one user command. Pure so Mac CI can assert the - * Windows shape without needing cmd.exe. + * Windows shape without needing cmd.exe or bash.exe. */ export function shellInvocation( command: string, - platform: NodeJS.Platform = process.platform + shell: HostShell = hostShellFromPlatform(), + gitBashPath?: string | null ): ShellInvocation { - if (platform === 'win32') { + if (shell === 'git-bash' && gitBashPath) { + // --noprofile --norc skips .bashrc / /etc/profile (mirrors cmd /d). PATH for + // unix tools is prepended by gitBashPathEnv, not by a login shell. + return { + command: gitBashPath, + args: ['--noprofile', '--norc', '-c', command], + detached: false, + verbatimArguments: false + }; + } + if (shell === 'cmd' || shell === 'git-bash') { + // git-bash without a path falls back to cmd — never spawn a missing bash. // /d = no AutoRun (registry hooks that mirror a broken profile). /s /c + a // quoted payload is the CreateProcess-safe form: cmd strips one outer quote // pair and runs the rest as-is (inner " and | stay intact for PowerShell). // Pair with windowsVerbatimArguments so Node does not turn " into \". const comspec = process.env.ComSpec || 'cmd.exe'; - return { command: comspec, args: ['/d', '/s', '/c', `"${command}"`], detached: false }; + return { command: comspec, args: ['/d', '/s', '/c', `"${command}"`], detached: false, verbatimArguments: true }; } - return { command: unixShell().path, args: ['-c', command], detached: true }; + return { command: unixShell().path, args: ['-c', command], detached: true, verbatimArguments: false }; } const loginPathCache = new Map>(); @@ -183,6 +204,8 @@ export interface RunCommandOptions { timeoutMs: number; env: NodeJS.ProcessEnv; signal?: AbortSignal; + shell?: HostShell; + gitBashPath?: string | null; } /** @@ -230,7 +253,7 @@ function killChildTree(child: ReturnType, detached: boolean): void */ export function runCommand(opts: RunCommandOptions): Promise { return new Promise((resolve, reject) => { - const shell = shellInvocation(opts.command); + const shell = shellInvocation(opts.command, opts.shell, opts.gitBashPath); let child: ReturnType; try { child = spawn(shell.command, shell.args, { @@ -239,9 +262,10 @@ export function runCommand(opts: RunCommandOptions): Promise { stdio: ['ignore', 'pipe', 'pipe'], detached: shell.detached, windowsHide: true, - // Keep the /c "..." quotes we built; Node's default Windows quoting - // would escape inner " as \" and break PowerShell -Command "...". - windowsVerbatimArguments: process.platform === 'win32' + // Keep the /c "..." quotes we built for cmd; Node's default Windows + // quoting would escape inner " as \" and break PowerShell -Command "...". + // Git Bash uses normal argv quoting (verbatimArguments is false). + windowsVerbatimArguments: shell.verbatimArguments }); } catch (e) { reject(e instanceof Error ? e : new Error(String(e))); diff --git a/src/server/exec/git-bash.ts b/src/server/exec/git-bash.ts new file mode 100644 index 0000000..59fcd76 --- /dev/null +++ b/src/server/exec/git-bash.ts @@ -0,0 +1,220 @@ +import { execFile } from 'node:child_process'; +import { existsSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { win32 as pathWin32 } from 'node:path'; +import type { ExecSettings, HostShell } from '../../shared/types'; + +// Git Bash detection and PATH helpers. Filesystem-first: never spawn PowerShell. +// where.exe / reg.exe are optional last resorts and fail closed if AppLocker +// blocks them. Existence of bash.exe is enough to pre-fill Settings; we do not +// run `bash --version` here (that spawn can fail under WDAC even when the file +// is present). + +const BASH_EXE = 'bash.exe'; +const LOOKUP_TIMEOUT_MS = 2000; + +export interface DetectGitBashDeps { + env?: NodeJS.ProcessEnv; + exists?: (path: string) => boolean; + platform?: NodeJS.Platform; +} + +/** True when `path` looks like bash.exe and the file is on disk. */ +export function isUsableGitBashPath( + path: string | null | undefined, + exists: (p: string) => boolean = existsSync +): boolean { + if (!path || typeof path !== 'string') return false; + const trimmed = path.trim(); + if (!trimmed.toLowerCase().endsWith(BASH_EXE)) return false; + try { + return exists(trimmed); + } catch { + return false; + } +} + +/** + * Usual Git for Windows install locations. Built with win32 joins so Mac CI can + * assert the same strings Windows would see. + */ +export function wellKnownGitBashCandidates(env: NodeJS.ProcessEnv = process.env): string[] { + const pf = env.ProgramFiles || 'C:\\Program Files'; + const pf86 = env['ProgramFiles(x86)'] || 'C:\\Program Files (x86)'; + const local = env.LOCALAPPDATA || pathWin32.join(homedir(), 'AppData', 'Local'); + const home = env.USERPROFILE || homedir(); + return [ + pathWin32.join(pf, 'Git', 'bin', BASH_EXE), + pathWin32.join(pf86, 'Git', 'bin', BASH_EXE), + pathWin32.join(local, 'Programs', 'Git', 'bin', BASH_EXE), + pathWin32.join(home, 'scoop', 'apps', 'git', 'current', 'bin', BASH_EXE) + ]; +} + +/** `Git\\cmd\\git.exe` → `Git\\bin\\bash.exe` (the default Git for Windows layout). */ +export function bashBesideGit(gitExe: string, exists: (p: string) => boolean = existsSync): string | null { + const cmdDir = pathWin32.dirname(gitExe); + const gitRoot = pathWin32.dirname(cmdDir); + const bash = pathWin32.join(gitRoot, 'bin', BASH_EXE); + return isUsableGitBashPath(bash, exists) ? bash : null; +} + +function bashFromPathEnv(pathEnv: string, exists: (p: string) => boolean): string | null { + for (const dir of pathEnv.split(';')) { + const trimmed = dir.trim(); + if (!trimmed) continue; + const bash = pathWin32.join(trimmed, BASH_EXE); + if (isUsableGitBashPath(bash, exists)) return bash; + const git = pathWin32.join(trimmed, 'git.exe'); + if (exists(git)) { + const beside = bashBesideGit(git, exists); + if (beside) return beside; + } + } + return null; +} + +/** + * Detect Git Bash without spawning anything: well-known paths, then PATH. + * Safe on administered machines where PowerShell (and even where.exe) is blocked. + */ +export function detectGitBashFromDisk(deps: DetectGitBashDeps = {}): string | null { + const env = deps.env ?? process.env; + const exists = deps.exists ?? existsSync; + for (const candidate of wellKnownGitBashCandidates(env)) { + if (isUsableGitBashPath(candidate, exists)) return candidate; + } + return bashFromPathEnv(env.Path || env.PATH || '', exists); +} + +function execFileQuiet(command: string, args: string[]): Promise { + return new Promise((resolve) => { + try { + execFile(command, args, { windowsHide: true, timeout: LOOKUP_TIMEOUT_MS }, (error, stdout) => { + if (error) resolve(null); + else resolve(typeof stdout === 'string' ? stdout : null); + }); + } catch { + resolve(null); + } + }); +} + +function firstWhereLine(stdout: string): string | null { + const first = stdout + .split(/\r?\n/) + .map((l) => l.trim()) + .find(Boolean); + return first ?? null; +} + +async function bashFromWhere(exists: (p: string) => boolean): Promise { + const gitOut = await execFileQuiet('where.exe', ['git']); + if (gitOut) { + const git = firstWhereLine(gitOut); + if (git) { + const beside = bashBesideGit(git, exists); + if (beside) return beside; + } + } + const bashOut = await execFileQuiet('where.exe', ['bash']); + if (bashOut) { + const bash = firstWhereLine(bashOut); + if (bash && isUsableGitBashPath(bash, exists)) return bash; + } + return null; +} + +/** `reg query` output: `InstallPath REG_SZ C:\Program Files\Git` */ +function installPathFromReg(stdout: string): string | null { + const match = stdout.match(/InstallPath\s+REG_\w+\s+(.+)/i); + const value = match?.[1]?.trim(); + return value || null; +} + +async function bashFromRegistry(exists: (p: string) => boolean): Promise { + const keys = [ + 'HKLM\\SOFTWARE\\GitForWindows', + 'HKLM\\SOFTWARE\\WOW6432Node\\GitForWindows', + 'HKCU\\SOFTWARE\\GitForWindows' + ]; + for (const key of keys) { + const out = await execFileQuiet('reg.exe', ['query', key, '/v', 'InstallPath']); + if (!out) continue; + const install = installPathFromReg(out); + if (!install) continue; + const bash = pathWin32.join(install, 'bin', BASH_EXE); + if (isUsableGitBashPath(bash, exists)) return bash; + } + return null; +} + +/** + * Full detection: disk, then PATH, then optional where.exe / reg.exe. + * where/reg never run on non-Windows, and a blocked binary is treated as + * "keep looking", never as "Git Bash is absent". + */ +export async function detectGitBash(deps: DetectGitBashDeps = {}): Promise { + const fromDisk = detectGitBashFromDisk(deps); + if (fromDisk) return fromDisk; + const platform = deps.platform ?? process.platform; + if (platform !== 'win32') return null; + const exists = deps.exists ?? existsSync; + try { + const fromWhere = await bashFromWhere(exists); + if (fromWhere) return fromWhere; + } catch { + // AppLocker / missing where.exe — keep going. + } + try { + return await bashFromRegistry(exists); + } catch { + return null; + } +} + +/** + * Prepend Git's unix-tool dirs so `ls`/`cat`/`grep` work without a login shell + * (`--noprofile --norc` skips /etc/profile, which is what would otherwise set PATH). + */ +export function gitBashPathEnv(bashPath: string, windowsPath: string): string { + const gitRoot = pathWin32.dirname(pathWin32.dirname(bashPath)); + const extras = [ + pathWin32.join(gitRoot, 'usr', 'bin'), + pathWin32.join(gitRoot, 'bin'), + pathWin32.join(gitRoot, 'cmd'), + pathWin32.join(gitRoot, 'mingw64', 'bin'), + pathWin32.join(gitRoot, 'mingw32', 'bin') + ]; + return [...extras, windowsPath].join(';'); +} + +/** + * bash.exe Stem will spawn: the saved path if it is still on disk, else a + * well-known Git for Windows install. Null means fall back to cmd.exe. + */ +export function resolveGitBashExecutable( + settings: Pick, + exists: (p: string) => boolean = existsSync, + env: NodeJS.ProcessEnv = process.env +): string | null { + if (isUsableGitBashPath(settings.gitBashPath, exists)) return settings.gitBashPath!.trim(); + return detectGitBashFromDisk({ exists, env }); +} + +/** + * The shell run_command will actually spawn. Git Bash when the user wants it + * (the Windows default) AND bash.exe is on disk — saved path or auto-detected. + * Otherwise cmd.exe. + */ +export function resolveHostShell( + settings: Pick, + platform: NodeJS.Platform = process.platform, + exists: (p: string) => boolean = existsSync +): HostShell { + if (platform !== 'win32') return 'zsh'; + if (settings.windowsShell === 'git-bash' && resolveGitBashExecutable(settings, exists)) { + return 'git-bash'; + } + return 'cmd'; +} diff --git a/src/server/exec/host-shell.ts b/src/server/exec/host-shell.ts new file mode 100644 index 0000000..15fc2ed --- /dev/null +++ b/src/server/exec/host-shell.ts @@ -0,0 +1,37 @@ +import type { HostShell } from '../../shared/types'; + +/** + * Default host shell from the OS when there are no ExecSettings (tests, a + * remote device's platform). Windows is cmd.exe here; the real Windows default + * in Settings is Git Bash, applied through resolveHostShell. + */ +export function hostShellFromPlatform(platform: NodeJS.Platform = process.platform): HostShell { + return platform === 'win32' ? 'cmd' : 'zsh'; +} + +/** True for cmd.exe's quoting rules (`'` is not a quote, `%` expands, `^` escapes). */ +export function isCmdShell(shell: HostShell): boolean { + return shell === 'cmd'; +} + +/** + * Per-turn hint so the model writes commands for the one shell that will run. + * Empty on zsh: the run_command tool description already covers that. + */ +export function hostShellAgentHint(shell: HostShell): string { + if (shell === 'cmd') { + return ( + 'run_command on this machine uses cmd.exe (/d /s /c, no AutoRun). Quote with double quotes: ' + + "cmd does not treat a single quote as a quote character. POSIX names like ls, cat, and grep " + + 'are not commands here — use dir, type, findstr. A bare | is a cmd pipe. If you need PowerShell, ' + + 'invoke powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "..." and put pipelines inside -Command.' + ); + } + if (shell === 'git-bash') { + return ( + 'run_command on this machine uses Git Bash (bash --noprofile --norc). POSIX quoting and commands ' + + '(ls, cat, grep) work. Paths may be Windows (C:\\Users\\...) or Git Bash (/c/Users/...).' + ); + } + return ''; +} diff --git a/src/server/exec/policy.ts b/src/server/exec/policy.ts index 0f74b88..097596c 100644 --- a/src/server/exec/policy.ts +++ b/src/server/exec/policy.ts @@ -1,5 +1,16 @@ -import type { ExecSettings } from '../../shared/types'; +import type { ExecSettings, HostShell } from '../../shared/types'; import { unixShell } from './executor'; +import { hostShellFromPlatform, isCmdShell } from './host-shell'; + +export { hostShellFromPlatform }; + +/** Local HostShell, or a device's Node platform (win32 → cmd, anything else → POSIX). */ +type ShellArg = HostShell | NodeJS.Platform; + +function toHostShell(shell: ShellArg = hostShellFromPlatform()): HostShell { + if (shell === 'cmd' || shell === 'git-bash' || shell === 'zsh') return shell; + return hostShellFromPlatform(shell); +} // The run_command auto-approve policy, kept pure so it is unit-testable: // @@ -15,9 +26,9 @@ import { unixShell } from './executor'; // entirely, and a command word containing a path separator never matches (an // allowlisted `git` must not admit `./git`). // -// The parse is PLATFORM-SPECIFIC because the two host shells disagree about +// The parse is SHELL-SPECIFIC because cmd.exe and POSIX shells disagree about // quoting, and a parser that models the wrong one hands out tier 1 for commands -// the shell will happily split. See WINDOWS_HARD_META below. +// the shell will happily split. See WINDOWS_HARD_META below. Git Bash is POSIX. /** Read-only probes that mean the same thing on both host shells. */ const SHARED_ALLOWLIST = ['rg', 'git status', 'git log', 'git diff', 'git show', 'git branch', 'agent-browser']; @@ -46,8 +57,10 @@ const POSIX_ALLOWLIST = [ */ const WINDOWS_ALLOWLIST = ['dir', 'type', 'where', 'echo', 'cd']; -function staticAllowlist(platform: NodeJS.Platform): Set { - return new Set([...SHARED_ALLOWLIST, ...(platform === 'win32' ? WINDOWS_ALLOWLIST : POSIX_ALLOWLIST)]); +function staticAllowlist(shell: HostShell): Set { + // Git Bash is a POSIX shell: ls/cat/grep exist. cmd.exe is the only one that + // gets dir/type — those names must not widen zsh or Git Bash tier 1. + return new Set([...SHARED_ALLOWLIST, ...(isCmdShell(shell) ? WINDOWS_ALLOWLIST : POSIX_ALLOWLIST)]); } /** One chained command within a compound (or the whole thing when not chained). */ @@ -110,14 +123,14 @@ function makeSegment(tokens: string[]): ParsedSegment { * Not a full shell parser — anything with shell semantics beyond "commands * chained with plain arguments" comes back `hasShellMeta` and is left to the judge. * - * `platform` selects which shell's quoting rules to model; it must match the - * shell `shellInvocation()` will actually spawn, or tier 1 is decided against a + * `shell` selects which quoting rules to model; it must match the shell + * `shellInvocation()` will actually spawn, or tier 1 is decided against a * grammar the host does not use. */ -export function parseCommand(command: string, platform: NodeJS.Platform = process.platform): ParsedCommand { - const win = platform === 'win32'; - const hardMeta = win ? WINDOWS_HARD_META : POSIX_HARD_META; - const dquoteMeta = win ? WINDOWS_DQUOTE_META : POSIX_DQUOTE_META; +export function parseCommand(command: string, shell: HostShell = hostShellFromPlatform()): ParsedCommand { + const cmd = isCmdShell(shell); + const hardMeta = cmd ? WINDOWS_HARD_META : POSIX_HARD_META; + const dquoteMeta = cmd ? WINDOWS_DQUOTE_META : POSIX_DQUOTE_META; const segments: ParsedSegment[] = []; let tokens: string[] = []; let current = ''; @@ -156,8 +169,9 @@ export function parseCommand(command: string, platform: NodeJS.Platform = proces continue; } // Under cmd.exe `'` opens nothing — it falls through to the hard-meta check - // below, so a single-quoted region can never hide a separator. - if (ch === '"' || (ch === "'" && !win)) { + // below, so a single-quoted region can never hide a separator. Git Bash and + // zsh honour single quotes. + if (ch === '"' || (ch === "'" && !cmd)) { quote = ch as "'" | '"'; inToken = true; continue; @@ -216,15 +230,18 @@ export interface Classification { export function classify( command: string, settings: Pick, - platform: NodeJS.Platform = process.platform, + shell: ShellArg = hostShellFromPlatform(), opts: { includeBuiltins?: boolean } = {} ): Classification { - const parsed = parseCommand(command, platform); + const host = toHostShell(shell); + const parsed = parseCommand(command, host); if (parsed.hasShellMeta || !parsed.segments.length) { return { tier: 'judge', prefixes: [], hasShellMeta: parsed.hasShellMeta }; } const user = new Set(settings.allowlist); - const allowed = opts.includeBuiltins === false ? new Set() : staticAllowlist(platform); + // includeBuiltins: false is the remote-target posture — that machine's tier 1 + // is only its learned allowlist, never ls/dir/git status from this host. + const allowed = opts.includeBuiltins === false ? new Set() : staticAllowlist(host); const uncovered = parsed.segments.filter( (seg) => !seg.candidates.some((c) => allowed.has(c) || user.has(c)) ); @@ -239,10 +256,12 @@ export function classify( * on a VPS those are different computers, and the judge is being asked about the * first one. */ -export function hostShellLabel(platform: NodeJS.Platform = process.platform): string { - if (platform === 'win32') return 'a Windows machine, under cmd.exe'; - const shell = unixShell().path.split('/').pop() || 'sh'; - return `the machine Stem runs on, under ${shell}`; +export function hostShellLabel(shell: ShellArg = hostShellFromPlatform()): string { + const host = toHostShell(shell); + if (host === 'cmd') return 'a Windows machine, under cmd.exe'; + if (host === 'git-bash') return 'a Windows machine, under Git Bash'; + const name = unixShell().path.split('/').pop() || 'sh'; + return `the machine Stem runs on, under ${name}`; } /** @@ -269,13 +288,13 @@ export function buildJudgePrompt( command: string, cwd: string, userIntent?: string, - platform: NodeJS.Platform = process.platform, + shell: ShellArg = hostShellFromPlatform(), shellLabel?: string ): string { const intent = (userIntent ?? '').trim().slice(0, 800); return [ `An AI assistant working on a request from its user wants to run a shell command on`, - `${shellLabel ?? hostShellLabel(platform)}. Classify whether the`, + `${shellLabel ?? hostShellLabel(shell)}. Classify whether the`, 'command is safe to run without asking the user first. Reply with exactly one word', '— safe, unsafe, or unsure — optionally followed on the same line by a very short reason.', '', diff --git a/src/server/exec/protected.ts b/src/server/exec/protected.ts index 8127acc..1a134c4 100644 --- a/src/server/exec/protected.ts +++ b/src/server/exec/protected.ts @@ -1,6 +1,8 @@ import { readFileSync, realpathSync } from 'node:fs'; import { win32 as pathWin32, posix as pathPosix } from 'node:path'; +import type { HostShell } from '../../shared/types'; import { protectedRootsPath } from '../workspace/paths'; +import { hostShellFromPlatform } from './host-shell'; // Main-side twin of the bridge extension's protected-roots gate (which cannot be // imported — it lives in the pi child's .mjs). Enforces read-only connected @@ -28,21 +30,28 @@ interface Host { resolve: (p: string) => string; sep: string; homeVar: string; - pathish: RegExp; } -function host(platform: NodeJS.Platform): Host { - const win = platform === 'win32'; +function host(shell: HostShell): Host { + // Git Bash still runs on NTFS with Windows cwd/roots; only the *tokens* we + // pull out of the command line are POSIX-shaped. + const win = shell !== 'zsh'; const p = win ? pathWin32 : pathPosix; return { win, resolve: (raw) => p.resolve(raw), sep: p.sep, - homeVar: win ? 'USERPROFILE' : 'HOME', - pathish: win ? WINDOWS_PATHISH_RE : POSIX_PATHISH_RE + homeVar: win ? 'USERPROFILE' : 'HOME' }; } +/** `/c/Users/foo` → `C:\Users\foo`. A bare `/b` is a flag, not drive B:. */ +export function msysToWindows(p: string): string | null { + const m = /^\/([a-zA-Z])\/(.+)$/.exec(p); + if (!m) return null; + return `${m[1]!.toUpperCase()}:\\${m[2]!.replace(/\//g, '\\')}`; +} + function canonicalish(p: string, h: Host): string { const resolved = h.resolve(p); try { @@ -75,9 +84,9 @@ export interface ProtectedScanResult { */ export function readProtectedRoots( path: string = protectedRootsPath(), - platform: NodeJS.Platform = process.platform + shell: HostShell = hostShellFromPlatform() ): string[] { - const h = host(platform); + const h = host(shell); let raw: string; try { raw = readFileSync(path, 'utf8'); @@ -90,14 +99,30 @@ export function readProtectedRoots( } /** Every path-looking token in a command, with ~ and (on Windows) %VAR% expanded. */ -function pathTokens(command: string, h: Host): string[] { +function pathTokens(command: string, shell: HostShell, h: Host): string[] { const home = process.env[h.homeVar] ?? ''; const text = h.win ? command.replace(WINDOWS_ENV_RE, (whole, name: string) => process.env[name] ?? whole) : command; const out: string[] = []; - for (const match of text.match(h.pathish) ?? []) { - out.push(match.startsWith('~') ? home + match.slice(1) : match); + const push = (raw: string): void => { + out.push(raw.startsWith('~') ? home + raw.slice(1).replace(/\//g, h.sep) : raw); + }; + if (shell === 'zsh') { + for (const match of text.match(POSIX_PATHISH_RE) ?? []) push(match); + return out; + } + // cmd.exe: Windows shapes only. Git Bash: those plus MSYS `/c/Users/...`. + for (const match of text.match(WINDOWS_PATHISH_RE) ?? []) push(match); + if (shell === 'git-bash') { + for (const match of text.match(POSIX_PATHISH_RE) ?? []) { + if (match.startsWith('~')) { + push(match); + continue; + } + const converted = msysToWindows(match); + if (converted) out.push(converted); + } } return out; } @@ -110,12 +135,12 @@ export function scanProtected( command: string, cwd: string, rootsPath: string = protectedRootsPath(), - platform: NodeJS.Platform = process.platform + shell: HostShell = hostShellFromPlatform() ): ProtectedScanResult { - const h = host(platform); + const h = host(shell); let roots: string[]; try { - roots = readProtectedRoots(rootsPath, platform); + roots = readProtectedRoots(rootsPath, shell); } catch { return { blocked: true, @@ -124,7 +149,7 @@ export function scanProtected( } if (!roots.length) return { blocked: false }; - const targets = [cwd, ...pathTokens(command, h)]; + const targets = [cwd, ...pathTokens(command, shell, h)]; for (const target of targets) { const canonical = canonicalish(target, h); const hit = roots.find((root) => isInside(canonical, root, h)); diff --git a/src/server/exec/service.ts b/src/server/exec/service.ts index 418a126..be3189d 100644 --- a/src/server/exec/service.ts +++ b/src/server/exec/service.ts @@ -6,6 +6,7 @@ import type { ExecApprovalRequest, ExecDecision, ExecSettings, + HostShell, ModelSummary, ServerSettings } from '../../shared/types'; @@ -14,6 +15,8 @@ import { resolveRoleEffort } from '../../shared/modelRoles'; import { log } from '../log'; import { ensureThreadScratch } from './scratch'; import { clampTimeout, execEnv, resolveLoginPath, runCommand } from './executor'; +import { gitBashPathEnv, resolveGitBashExecutable, resolveHostShell } from './git-bash'; +import { hostShellFromPlatform } from './host-shell'; import { buildJudgePrompt, classify, deviceShellLabel, parseJudgeVerdict, resolveJudgeModel } from './policy'; import { scanProtected } from './protected'; import { execDeviceRouter, resolveExecTarget } from '../exec-device/router'; @@ -96,6 +99,7 @@ export class ExecService implements ExecBridge { if (!settings.enabled) { return { ok: false, error: 'Command execution is disabled in Settings → Chat → Command execution.' }; } + const shell = resolveHostShell(settings); // A command aimed at a paired computer takes its own path: same tiers, but // classified against that machine's platform and its own allowlist, and @@ -119,14 +123,14 @@ export class ExecService implements ExecBridge { } // Fail-closed read-only guard: any reference to a protected root blocks. - const guard = scanProtected(command, cwd); + const guard = scanProtected(command, cwd, undefined, shell); if (guard.blocked) return { ok: false, error: guard.reason ?? 'Blocked by the read-only folder guard.' }; // Yolo mode: everything runs — the protected-roots guard above is the only gate. - if (settings.approvalMode === 'yolo') return this.run(command, cwd, req); + if (settings.approvalMode === 'yolo') return this.run(command, cwd, req, settings); // Tier 1: static + user allowlist (every chained segment must clear it). - const cls = classify(command, settings); + const cls = classify(command, settings, shell); if (cls.tier !== 'run') { // Tier 2 (assisted mode only): one-word LLM judge classification (intent-aware // when the turn's user message is known); errors/timeouts escalate. Manual mode @@ -134,8 +138,8 @@ export class ExecService implements ExecBridge { let judgeVerdict: 'unsafe' | 'unsure' | 'failed' | null = null; let judgeReason: string | undefined; if (settings.approvalMode === 'assisted') { - const verdict = await this.judge(command, cwd, settings, all.defaults, req.userText, req.currentModel); - if (verdict.verdict === 'safe') return this.run(command, cwd, req); + const verdict = await this.judge(command, cwd, settings, all.defaults, req.userText, req.currentModel, shell); + if (verdict.verdict === 'safe') return this.run(command, cwd, req, settings); judgeVerdict = verdict.verdict; judgeReason = verdict.reason; } @@ -166,7 +170,7 @@ export class ExecService implements ExecBridge { } } - return this.run(command, cwd, req); + return this.run(command, cwd, req, settings); } abortThread(threadId: string): void { @@ -251,10 +255,16 @@ export class ExecService implements ExecBridge { let judgeVerdict: 'unsafe' | 'unsure' | 'failed' | null = null; let judgeReason: string | undefined; if (settings.approvalMode === 'assisted') { - const verdict = await this.judge(command, cwdLabel, settings, all.defaults, req.userText, req.currentModel, { - platform: host.platform, - shellLabel: deviceShellLabel(host.platform, label) - }); + const verdict = await this.judge( + command, + cwdLabel, + settings, + all.defaults, + req.userText, + req.currentModel, + host.platform, + deviceShellLabel(host.platform, label) + ); if (verdict.verdict === 'safe') return dispatch(); judgeVerdict = verdict.verdict; judgeReason = verdict.reason; @@ -328,9 +338,10 @@ export class ExecService implements ExecBridge { defaults: DefaultsSettings, userText?: string, currentModel?: string | null, + shell: HostShell | NodeJS.Platform = hostShellFromPlatform(), // Set for a device-targeted command: the judge must reason about the shell // that will actually run it, on the machine it will actually run on. - target?: { platform: NodeJS.Platform; shellLabel: string } + shellLabel?: string ): Promise<{ verdict: 'safe' | 'unsafe' | 'unsure' | 'failed'; reason?: string }> { try { const runtime = this.deps.runtime(); @@ -340,7 +351,7 @@ export class ExecService implements ExecBridge { // and complete() then uses its own default, which is the best available // answer anyway. const model = resolveJudgeModel(settings, defaults, models, currentModel ?? null); - const reply = await runtime.complete(buildJudgePrompt(command, cwd, userText, target?.platform, target?.shellLabel), { + const reply = await runtime.complete(buildJudgePrompt(command, cwd, userText, shell, shellLabel), { model, // The judge sits between you and every command you run, so it feels the // effort setting more than any other role does — its own if it has been @@ -377,19 +388,30 @@ export class ExecService implements ExecBridge { return true; } - private async run(command: string, cwd: string, req: ExecRequest): Promise { + private async run( + command: string, + cwd: string, + req: ExecRequest, + settings: ExecSettings + ): Promise { await this.acquireSlot(); const controller = new AbortController(); const entry: RunningExec = { threadId: req.threadId ?? '', controller }; this.running.add(entry); try { + const shell = resolveHostShell(settings); + const gitBashPath = shell === 'git-bash' ? resolveGitBashExecutable(settings) : settings.gitBashPath; const loginPath = await resolveLoginPath(); + const pathForChild = + shell === 'git-bash' && gitBashPath ? gitBashPathEnv(gitBashPath, loginPath) : loginPath; const outcome = await runCommand({ command, cwd, timeoutMs: clampTimeout(req.timeoutMs), - env: execEnv(loginPath), - signal: controller.signal + env: execEnv(pathForChild), + signal: controller.signal, + shell, + gitBashPath }); if (controller.signal.aborted && !outcome.timedOut) { return { ok: false, error: 'The command was cancelled.' }; diff --git a/src/server/index.ts b/src/server/index.ts index 057cb73..97c9586 100644 --- a/src/server/index.ts +++ b/src/server/index.ts @@ -19,6 +19,7 @@ import { piHome } from './workspace/paths'; import type { TaskScheduler } from './scheduler'; import { initTaskScheduler } from './startup/scheduler'; import type { ExecService } from './exec/service'; +import { detectGitBash } from './exec/git-bash'; import { startScratchSweeper, stopScratchSweeper } from './exec/scratch'; import { initExecService } from './startup/exec'; import { initSkills } from './startup/skills'; @@ -450,6 +451,7 @@ function registerIpc(): void { registerServer('exec:resolveApproval', async (_e, id: string, decision: ExecDecision) => { execService?.resolveApproval(id, decision); }); + registerServer('exec:detectGitBash', async () => detectGitBash()); registerServer('settings:updateCustomInstructions', async (_e, patch: Partial) => { // Just persist — backend:startTurn reads the instructions fresh per turn (for // both surfaces), so the change applies to the next turn with no restart. diff --git a/src/server/pi/runtime.ts b/src/server/pi/runtime.ts index 4af3cc0..219a8af 100644 --- a/src/server/pi/runtime.ts +++ b/src/server/pi/runtime.ts @@ -43,6 +43,8 @@ import { log } from '../log'; import { isContextOverflowError } from '../backend/overflow'; import { PLAIN_MD_DIRECTIVE, stemAssistantInstructions } from '../workspace/bootstrap'; import { readSettings } from '../workspace/settings'; +import { resolveHostShell } from '../exec/git-bash'; +import { hostShellAgentHint } from '../exec/host-shell'; import { previewText } from '../chats/preview'; import { autoTitle, nameThread, nameThreadIfDue as nameIfDue, type SubjectDeps } from '../chats/subject'; import { setNaming } from '../workspace/chats'; @@ -2819,6 +2821,15 @@ export class PiRuntime extends EventEmitter implements ChatBackend { // from here. Mirrors the gate written above for this turn. const web = buildWebSearchContext(input.webSearch ?? true); if (web) blocks.push(web); + try { + const exec = (await readSettings()).exec; + if (exec.enabled) { + const hint = hostShellAgentHint(resolveHostShell(exec)); + if (hint) blocks.push(hint); + } + } catch { + // Shell hint is convenience for the model; a turn must still go out. + } if (input.format === 'md') blocks.push(PLAIN_MD_DIRECTIVE); // Images go to pi natively; text-like files and PDF text layers are inlined, diff --git a/src/server/pi/stem-mcp-extension.mjs b/src/server/pi/stem-mcp-extension.mjs index 0afa236..ae714cd 100644 --- a/src/server/pi/stem-mcp-extension.mjs +++ b/src/server/pi/stem-mcp-extension.mjs @@ -2033,7 +2033,10 @@ function registerExecTool(pi) { 'With `device`, the command runs on one of the user\'s own paired computers instead — see that parameter ' + 'for when and how. ' + 'On macOS/Linux Stem uses the host shell (zsh where there is one, otherwise bash or sh) with the login-shell ' + - 'PATH (Homebrew/npm CLIs like `agent-browser` work). On Windows Stem uses cmd.exe (/d /s /c — no AutoRun); ' + + 'PATH (Homebrew/npm CLIs like `agent-browser` work). On Windows Stem uses the shell chosen in Settings → ' + + 'Chat → Command execution (Command Prompt by default, or Git Bash). Each turn names the one local shell ' + + 'that will run — follow that, not both. ' + + 'When the shell is cmd.exe (the Windows default, and the shell on a paired Windows computer), ' + 'if you need PowerShell, invoke it explicitly as `powershell.exe -NoProfile -ExecutionPolicy Bypass ' + '-Command "..."` so a broken profile.ps1 cannot block the run. A bare `|` is a cmd pipe (it splits ' + 'before PowerShell) — put PowerShell pipelines inside `-Command "..."` (or use `(...)` / property ' + diff --git a/src/server/recall/embed-files.ts b/src/server/recall/embed-files.ts new file mode 100644 index 0000000..86766b2 --- /dev/null +++ b/src/server/recall/embed-files.ts @@ -0,0 +1,166 @@ +import { + copyFileSync, + createReadStream, + createWriteStream, + existsSync, + mkdirSync, + readdirSync, + renameSync, + unlinkSync +} from 'node:fs'; +import { dirname, join, relative } from 'node:path'; +import { Readable } from 'node:stream'; +import { pipeline } from 'node:stream/promises'; +import { createGunzip } from 'node:zlib'; +import type { LocalEmbedModelSpec } from './embed-catalog'; + +// Helpers for finding ONNX weights on disk and deciding whether the worker +// may call Hugging Face. Kept free of transformers.js so they are unit-testable +// and shareable with the manager (which only passes paths, not env flags). + +export type EmbedDtype = LocalEmbedModelSpec['dtype']; + +/** The weights filename transformers.js resolves for each catalog dtype. */ +export function weightsFile(dtype: EmbedDtype): string { + return dtype === 'q8' ? 'model_quantized.onnx' : dtype === 'q4' ? 'model_q4.onnx' : 'model.onnx'; +} + +/** + * True when `root` already holds the ONNX file transformers.js would load for + * this repo+dtype. Layout matches the Hugging Face cache: + * `{root}/{repo}/onnx/model_quantized.onnx` (or model_q4.onnx / model.onnx). + */ +export function weightsPresent(root: string, repo: string, dtype: EmbedDtype): boolean { + try { + return existsSync(join(root, repo, 'onnx', weightsFile(dtype))); + } catch { + return false; + } +} + +/** + * Whether the worker should talk to the Hub, and which extra local root to + * point transformers.js at. + * + * - Weights already in the userData cache → stay offline (a copied cache from + * another machine must not re-check huggingface.co). + * - Weights in the repo-shipped vendor dir → same, and set `localModelPath`. + * - Neither → download into cacheDir as before. + */ +export function hubAccessForLoad(opts: { + cacheDir: string; + bundledDir?: string | null; + repo: string; + dtype: EmbedDtype; +}): { allowRemoteModels: boolean; localModelPath: string | null } { + const bundled = + opts.bundledDir && weightsPresent(opts.bundledDir, opts.repo, opts.dtype) ? opts.bundledDir : null; + const cached = weightsPresent(opts.cacheDir, opts.repo, opts.dtype); + return { + allowRemoteModels: !cached && !bundled, + localModelPath: bundled + }; +} + +/** + * ONNX error text often uses the other slash than `path.join` on this OS. + * Match either so a truncated Windows download still counts as "our" file. + */ +export function pathAppearsInMessage(message: string, filePath: string): boolean { + if (message.includes(filePath)) return true; + const forward = filePath.replaceAll('\\', '/'); + const back = filePath.replaceAll('/', '\\'); + return message.includes(forward) || message.includes(back); +} + +/** Keep each git object under GitHub's 50 MB warning (hard cap is 100 MB). */ +export const PACK_PART_BYTES = 45 * 1024 * 1024; + +const GZ_PART = /\.gz\.(\d{2})$/; + +function walkFiles(dir: string): string[] { + if (!existsSync(dir)) return []; + const out: string[] = []; + for (const ent of readdirSync(dir, { withFileTypes: true })) { + const p = join(dir, ent.name); + if (ent.isDirectory()) out.push(...walkFiles(p)); + else if (ent.isFile()) out.push(p); + } + return out; +} + +function packedOnnxGroups(bundledDir: string): Map { + const groups = new Map(); + for (const abs of walkFiles(bundledDir)) { + const rel = relative(bundledDir, abs); + if (rel === 'README.md') continue; + const part = GZ_PART.exec(rel); + if (part) { + const base = rel.slice(0, -part[0].length); // strip .gz.00 + const list = groups.get(base) ?? []; + list.push(abs); + groups.set(base, list); + continue; + } + if (rel.endsWith('.gz')) { + const base = rel.slice(0, -3); + const list = groups.get(base) ?? []; + list.push(abs); + groups.set(base, list); + } + } + for (const [, parts] of groups) { + parts.sort((a, b) => a.localeCompare(b)); + } + return groups; +} + +/** + * Copy vendor sidecars (tokenizer, config) and gunzip packed ONNX weights into + * the userData cache. No-op when the cache already has the unpacked file. + * Packed layout: `model_quantized.onnx.gz` or split `…gz.00`, `…gz.01`, … + */ +export async function unpackBundledEmbedModels(bundledDir: string, cacheDir: string): Promise { + if (!existsSync(bundledDir)) return 0; + let unpacked = 0; + + for (const abs of walkFiles(bundledDir)) { + const rel = relative(bundledDir, abs); + if (rel === 'README.md' || rel.endsWith('.gz') || GZ_PART.test(rel)) continue; + const dest = join(cacheDir, rel); + if (existsSync(dest)) continue; + mkdirSync(dirname(dest), { recursive: true }); + copyFileSync(abs, dest); + unpacked += 1; + } + + for (const [relBase, parts] of packedOnnxGroups(bundledDir)) { + const dest = join(cacheDir, relBase); + if (existsSync(dest)) continue; + mkdirSync(dirname(dest), { recursive: true }); + const tmp = `${dest}.tmp`; + try { + await pipeline( + Readable.from( + (async function* () { + for (const part of parts) { + for await (const chunk of createReadStream(part)) yield chunk; + } + })() + ), + createGunzip(), + createWriteStream(tmp) + ); + renameSync(tmp, dest); + unpacked += 1; + } catch (err) { + try { + unlinkSync(tmp); + } catch { + // leftover tmp is harmless; next launch retries + } + throw err; + } + } + return unpacked; +} diff --git a/src/server/recall/embed-manager.ts b/src/server/recall/embed-manager.ts index 12f18ad..8d35f30 100644 --- a/src/server/recall/embed-manager.ts +++ b/src/server/recall/embed-manager.ts @@ -77,6 +77,8 @@ const STABLE_UPTIME_MS = 60_000; export function createEmbedWorkerManager(deps: { spawn: () => WorkerTransport; cacheDir: () => string; + /** Repo-shipped ONNX weights (vendor/embed-models). Optional so tests can omit it. */ + bundledDir?: () => string; embedTimeoutMs?: number; rerankTimeoutMs?: number; }): EmbedWorkerManager { @@ -277,8 +279,9 @@ export function createEmbedWorkerManager(deps: { if (rerankSpec) setRerankStatus({ model: rerankSpec.id, state: 'error', error }); } }); - if (spec) t.send({ type: 'load', spec, cacheDir: deps.cacheDir() }); - if (rerankSpec) t.send({ type: 'load-rerank', spec: rerankSpec, cacheDir: deps.cacheDir() }); + if (spec) t.send({ type: 'load', spec, cacheDir: deps.cacheDir(), bundledDir: deps.bundledDir?.() ?? null }); + if (rerankSpec) + t.send({ type: 'load-rerank', spec: rerankSpec, cacheDir: deps.cacheDir(), bundledDir: deps.bundledDir?.() ?? null }); } function stop(): void { @@ -316,7 +319,7 @@ export function createEmbedWorkerManager(deps: { if (transport && !spec) { spec = target; setStatus({ model: target.id, state: 'loading' }); - transport.send({ type: 'load', spec: target, cacheDir: deps.cacheDir() }); + transport.send({ type: 'load', spec: target, cacheDir: deps.cacheDir(), bundledDir: deps.bundledDir?.() ?? null }); return; } if (transport) stop(); @@ -363,7 +366,7 @@ export function createEmbedWorkerManager(deps: { if (transport) { rerankSpec = target; setRerankStatus({ model: target.id, state: 'loading' }); - transport.send({ type: 'load-rerank', spec: target, cacheDir: deps.cacheDir() }); + transport.send({ type: 'load-rerank', spec: target, cacheDir: deps.cacheDir(), bundledDir: deps.bundledDir?.() ?? null }); return; } rerankSpec = target; diff --git a/src/server/recall/embed-worker.ts b/src/server/recall/embed-worker.ts index ac9eecc..1b281e5 100644 --- a/src/server/recall/embed-worker.ts +++ b/src/server/recall/embed-worker.ts @@ -1,8 +1,9 @@ -import { existsSync, rmSync } from 'node:fs'; +import { rmSync } from 'node:fs'; import { join } from 'node:path'; import { applyPrefixes } from './embed-catalog'; import type { LocalEmbedModelSpec } from './embed-catalog'; import type { LocalRerankModelSpec } from './rerank-catalog'; +import { hubAccessForLoad, pathAppearsInMessage } from './embed-files'; import type { EmbedKind } from './embeddings'; import type { RerankResult } from './rerank'; import type { LocalEmbedStatus, LocalRerankStatus } from '../../shared/types'; @@ -15,9 +16,9 @@ import type { LocalEmbedStatus, LocalRerankStatus } from '../../shared/types'; // This file must stay free of Electron imports beyond the ambient parentPort. export type WorkerInMessage = - | { type: 'load'; spec: LocalEmbedModelSpec; cacheDir: string } + | { type: 'load'; spec: LocalEmbedModelSpec; cacheDir: string; bundledDir?: string | null } | { type: 'embed'; id: number; texts: string[]; kind: EmbedKind } - | { type: 'load-rerank'; spec: LocalRerankModelSpec; cacheDir: string } + | { type: 'load-rerank'; spec: LocalRerankModelSpec; cacheDir: string; bundledDir?: string | null } | { type: 'rerank'; id: number; query: string; docs: string[]; topN: number } | { type: 'dispose' }; @@ -117,11 +118,6 @@ function postRerankStatus(status: Omit): void { post({ type: 'rerank-status', status: { model: rerankSpec.id, ...status } }); } -/** The weights filename transformers.js resolves for each catalog dtype. */ -function weightsFile(dtype: LocalEmbedModelSpec['dtype']): string { - return dtype === 'q8' ? 'model_quantized.onnx' : dtype === 'q4' ? 'model_q4.onnx' : 'model.onnx'; -} - /** * transformers.js progress events → one throttled status callback. Files * download in parallel and each reports independently, so per-file @@ -169,18 +165,32 @@ function progressAggregator( * own healthy cache over it would throw away good bytes. */ function purgeIfCorrupt(message: string, repo: string, cacheDir: string): boolean { - if (!/protobuf parsing failed/i.test(message) || !message.includes(join(cacheDir, repo))) return false; + if (!/protobuf parsing failed/i.test(message) || !pathAppearsInMessage(message, join(cacheDir, repo))) return false; rmSync(join(cacheDir, repo), { recursive: true, force: true }); return true; } -async function load(nextSpec: LocalEmbedModelSpec, cacheDir: string): Promise { +function applyHubAccess( + env: { cacheDir: string; localModelPath?: string; allowRemoteModels?: boolean }, + cacheDir: string, + bundledDir: string | null | undefined, + repo: string, + dtype: LocalEmbedModelSpec['dtype'] +): boolean { + env.cacheDir = cacheDir; + const access = hubAccessForLoad({ cacheDir, bundledDir, repo, dtype }); + if (access.localModelPath) env.localModelPath = access.localModelPath; + env.allowRemoteModels = access.allowRemoteModels; + // True when bytes are already on disk (cache or vendor) — progress events are reads, not a download. + return !access.allowRemoteModels; +} + +async function load(nextSpec: LocalEmbedModelSpec, cacheDir: string, bundledDir?: string | null): Promise { spec = nextSpec; postStatus({ state: 'loading' }); try { const { pipeline, env } = await import('@huggingface/transformers'); - env.cacheDir = cacheDir; - const cached = existsSync(join(cacheDir, nextSpec.repo, 'onnx', weightsFile(nextSpec.dtype))); + const cached = applyHubAccess(env, cacheDir, bundledDir, nextSpec.repo, nextSpec.dtype); const pipe = (await pipeline('feature-extraction', nextSpec.repo, { dtype: nextSpec.dtype, progress_callback: progressAggregator(cached, postStatus) @@ -203,7 +213,7 @@ async function load(nextSpec: LocalEmbedModelSpec, cacheDir: string): Promise { +async function loadRerank(nextSpec: LocalRerankModelSpec, cacheDir: string, bundledDir?: string | null): Promise { // Replacing an already-loaded reranker (model switch): release the old ONNX // session first so two cross-encoders never sit in memory at once. const old = reranker; @@ -219,8 +229,7 @@ async function loadRerank(nextSpec: LocalRerankModelSpec, cacheDir: string): Pro const { AutoTokenizer, AutoModelForSequenceClassification, AutoModelForCausalLM, env } = await import( '@huggingface/transformers' ); - env.cacheDir = cacheDir; - const cached = existsSync(join(cacheDir, nextSpec.repo, 'onnx', weightsFile(nextSpec.dtype))); + const cached = applyHubAccess(env, cacheDir, bundledDir, nextSpec.repo, nextSpec.dtype); const onProgress = progressAggregator(cached, postRerankStatus); const tokenizer = (await AutoTokenizer.from_pretrained(nextSpec.repo, { progress_callback: onProgress @@ -364,9 +373,10 @@ let loadChain: Promise = Promise.resolve(); port.on('message', (e: { data: WorkerInMessage }) => { const msg = e.data; - if (msg.type === 'load') loadChain = loadChain.then(() => load(msg.spec, msg.cacheDir)); + if (msg.type === 'load') loadChain = loadChain.then(() => load(msg.spec, msg.cacheDir, msg.bundledDir)); else if (msg.type === 'embed') void embed(msg.id, msg.texts, msg.kind); - else if (msg.type === 'load-rerank') loadChain = loadChain.then(() => loadRerank(msg.spec, msg.cacheDir)); + else if (msg.type === 'load-rerank') + loadChain = loadChain.then(() => loadRerank(msg.spec, msg.cacheDir, msg.bundledDir)); else if (msg.type === 'rerank') void rerank(msg.id, msg.query, msg.docs, msg.topN); else if (msg.type === 'dispose') void dispose(); }); diff --git a/src/server/startup/exec.ts b/src/server/startup/exec.ts index 877f9c9..b12f33f 100644 --- a/src/server/startup/exec.ts +++ b/src/server/startup/exec.ts @@ -1,4 +1,5 @@ import { ExecService } from '../exec/service'; +import { detectGitBash, isUsableGitBashPath } from '../exec/git-bash'; import { readSettings, updateExecSettings } from '../workspace/settings'; import type { ChatBackend } from '../backend'; import type { ExecApprovalRequest } from '../../shared/types'; @@ -23,5 +24,24 @@ export function initExecService(deps: { emitApprovalResolved: deps.emitApprovalResolved }); deps.runtime.setExecBridge(service); + void seedWindowsGitBash(); return service; } + +/** + * Fresh Windows installs default to Git Bash with an empty path. Fill it from + * disk so Settings shows bash.exe and the first command does not have to wait + * on a later detect. Missing Git → leave the preference; spawn falls back to cmd. + */ +async function seedWindowsGitBash(): Promise { + if (process.platform !== 'win32') return; + try { + const cur = await readSettings(); + if (cur.exec.windowsShell !== 'git-bash') return; + if (isUsableGitBashPath(cur.exec.gitBashPath)) return; + const found = await detectGitBash(); + if (found) await updateExecSettings({ gitBashPath: found, windowsShell: 'git-bash' }); + } catch { + // Detection is best-effort; resolveHostShell still falls back to cmd. + } +} diff --git a/src/server/startup/retrieval.ts b/src/server/startup/retrieval.ts index 086595a..b95688f 100644 --- a/src/server/startup/retrieval.ts +++ b/src/server/startup/retrieval.ts @@ -1,6 +1,6 @@ import { host } from '../host'; import { readSettings } from '../workspace/settings'; -import { embedModelsDir, embedSocketPath, recallDbPath } from '../workspace/paths'; +import { embedModelsDir, bundledEmbedModelsDir, embedSocketPath, recallDbPath } from '../workspace/paths'; import { embedNewMessages } from '../recall/embed-episodic'; import { scanAllIndexedFolders } from '../folder-index'; @@ -8,6 +8,7 @@ import { getEmbeddingsClient, setRetrievalClients } from '../recall/retrieval'; import { startEmbedEndpoint } from '../recall/embed-endpoint'; import { createHttpEmbeddingsClient, type EmbeddingsClient } from '../recall/embeddings'; import { createHttpRerankClient } from '../recall/rerank'; +import { unpackBundledEmbedModels } from '../recall/embed-files'; import { EMBED_CATALOG, localModelCacheKey } from '../recall/embed-catalog'; import { createEmbedWorkerManager, type EmbedWorkerManager } from '../recall/embed-manager'; import { createEmbeddingsRouter, createLocalEmbeddingsClient } from '../recall/embed-local'; @@ -105,7 +106,11 @@ export function initRetrieval(deps: { /** Push on a client channel — the model download/load status streams. */ emit: (channel: string, payload: unknown) => void; }): RetrievalRuntime { - const embedManager = createEmbedWorkerManager({ spawn: spawnEmbedWorker, cacheDir: embedModelsDir }); + const embedManager = createEmbedWorkerManager({ + spawn: spawnEmbedWorker, + cacheDir: embedModelsDir, + bundledDir: bundledEmbedModelsDir + }); // Recall's O(N) cosine scans and episodic VACUUMs run in their own utility // process so they never block the main event loop; everything degrades to the // in-process implementations if the worker is unavailable (see recall/scan.ts). @@ -228,14 +233,20 @@ export function initRetrieval(deps: { // enough facts to matter. The delay only yields the window/backend startup // burst — keep it short, or the Manage panel shows a misleading idle state // ("not downloaded yet") on every restart until the kick lands. Skipped under - // E2E: hermetic runs must not hit the network. + // E2E: hermetic runs must not hit the network. Unpack repo-shipped gzip packs + // first so a machine that cannot reach Hugging Face still has local weights. if (!deps.e2e) { + const unpacked = unpackBundledEmbedModels(bundledEmbedModelsDir(), embedModelsDir()).catch((err) => { + console.warn(`[embed-models] unpack failed: ${err instanceof Error ? err.message : String(err)}`); + }); setTimeout(() => { - void getEmbedSettings().then((e) => { - if (e.mode === 'local') embedManager.ensure(EMBED_CATALOG[e.localModel]); - }); - void getRerankSettings().then((r) => { - if (r.mode === 'local') embedManager.ensureRerank(RERANK_CATALOG[r.localModel]); + void unpacked.then(() => { + void getEmbedSettings().then((e) => { + if (e.mode === 'local') embedManager.ensure(EMBED_CATALOG[e.localModel]); + }); + void getRerankSettings().then((r) => { + if (r.mode === 'local') embedManager.ensureRerank(RERANK_CATALOG[r.localModel]); + }); }); }, 1_500); } diff --git a/src/server/workspace/paths.ts b/src/server/workspace/paths.ts index 0af4f9e..e315292 100644 --- a/src/server/workspace/paths.ts +++ b/src/server/workspace/paths.ts @@ -273,6 +273,15 @@ export function embedModelsDir(): string { return process.env.STEM_EMBED_MODELS_DIR ?? join(userDataRoot(), 'embed-models'); } +/** + * ONNX weights shipped next to the clone (`vendor/embed-models`), so a machine + * that cannot reach Hugging Face still loads the local embedder. Empty until + * someone copies a cache in — see vendor/embed-models/README.md. + */ +export function bundledEmbedModelsDir(): string { + return process.env.STEM_BUNDLED_EMBED_MODELS_DIR ?? join(host().appRoot(), 'vendor', 'embed-models'); +} + /** * Stem-owned app settings (e.g. the global Quick Chat shortcut + its defaults). * Held in the main process because some of it — the global accelerator — can diff --git a/src/server/workspace/settings.ts b/src/server/workspace/settings.ts index d936d0a..e080b33 100644 --- a/src/server/workspace/settings.ts +++ b/src/server/workspace/settings.ts @@ -97,7 +97,10 @@ const DEFAULTS: ServerSettings = { judgeEffort: null, allowlist: [], deviceAllowlists: {}, - scratchTtlDays: DEFAULT_SCRATCH_TTL_DAYS + scratchTtlDays: DEFAULT_SCRATCH_TTL_DAYS, + // Prefer Git Bash on Windows (auto-detect bash.exe; cmd.exe if Git is missing). + windowsShell: 'git-bash', + gitBashPath: null }, // Embeddings + reranker for relevance-ranking facts at inject time. Embeddings // default to the bundled local model (multilingual, in-process, nothing leaves @@ -360,7 +363,19 @@ function coerce(parsed: Partial | null): ServerSettings { ? null : typeof rawExec.scratchTtlDays === 'number' && Number.isFinite(rawExec.scratchTtlDays) && rawExec.scratchTtlDays > 0 ? Math.floor(rawExec.scratchTtlDays) - : DEFAULTS.exec.scratchTtlDays + : DEFAULTS.exec.scratchTtlDays, + gitBashPath: + typeof rawExec.gitBashPath === 'string' && rawExec.gitBashPath.trim() + ? rawExec.gitBashPath.trim().slice(0, 500) + : null, + // git-bash without a saved path still means "prefer Git Bash": spawn-time + // resolveHostShell auto-detects bash.exe and falls back to cmd if missing. + windowsShell: + rawExec.windowsShell === 'cmd' + ? 'cmd' + : rawExec.windowsShell === 'git-bash' + ? 'git-bash' + : DEFAULTS.exec.windowsShell }; const rawRet = (parsed?.retrieval ?? {}) as Partial; const retrieval: RetrievalSettings = { diff --git a/src/shared/types.ts b/src/shared/types.ts index 3246b21..87d433c 100644 --- a/src/shared/types.ts +++ b/src/shared/types.ts @@ -1842,6 +1842,16 @@ export interface SkillsSettings { */ export type ExecApprovalMode = 'manual' | 'assisted' | 'yolo'; +/** + * The shell `run_command` actually spawns. Independent of `process.platform` so + * Windows can be cmd.exe or Git Bash. The parser, allowlist, and judge prompt + * must use this same value — a mismatch is a safety bug. + */ +export type HostShell = 'zsh' | 'cmd' | 'git-bash'; + +/** Windows-only setting: which host shell run_command uses. Ignored on macOS/Linux. */ +export type WindowsShell = 'cmd' | 'git-bash'; + /** * Command execution (the `run_command` tool): a tiered auto-approve policy. * A static safe allowlist and the user's learned prefixes run immediately; other @@ -1874,6 +1884,13 @@ export interface ExecSettings { * file and the chat's last message. See server/exec/scratch.ts. */ scratchTtlDays: number | null; + /** + * Windows host shell. Default `git-bash` when bash.exe is on disk; otherwise + * Stem falls back to cmd.exe at spawn time. `cmd` is an explicit choice. + */ + windowsShell: WindowsShell; + /** Absolute path to Git for Windows `bash.exe`. Ignored unless windowsShell is git-bash. */ + gitBashPath: string | null; } /** One chat's scratch folder in Settings → Chat → Command execution → Scratch files. */ @@ -2832,6 +2849,11 @@ export interface StemApi { onExecApprovalResolved(listener: (payload: ApprovalResolvedPayload) => void): () => void; /** Answer a pending exec approval ("Allow once" / "Always allow prefix" / "Deny"). */ respondExecApproval(id: string, decision: ExecDecision): Promise; + /** + * Filesystem-first Git Bash lookup (Windows). Returns the path to bash.exe or + * null. Never spawns PowerShell. + */ + detectGitBash(): Promise; /** What each chat's shell commands have left on disk, biggest first. */ getScratchUsage(): Promise; /** Empty one chat's scratch folder (or the unfiled pile); the chat itself stays. */ diff --git a/tests/unit/embed-files.test.ts b/tests/unit/embed-files.test.ts new file mode 100644 index 0000000..9a5e40e --- /dev/null +++ b/tests/unit/embed-files.test.ts @@ -0,0 +1,90 @@ +import { mkdirSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { gzipSync } from 'node:zlib'; +import { describe, expect, it } from 'vitest'; +import { + hubAccessForLoad, + pathAppearsInMessage, + unpackBundledEmbedModels, + weightsFile, + weightsPresent +} from '../../src/server/recall/embed-files'; + +const REPO = 'Xenova/multilingual-e5-small'; + +function cacheWithWeights(root: string, repo = REPO): void { + const dir = join(root, repo, 'onnx'); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, 'model_quantized.onnx'), 'fake'); +} + +describe('embed local files', () => { + it('names the transformers.js weights file per dtype', () => { + expect(weightsFile('q8')).toBe('model_quantized.onnx'); + expect(weightsFile('q4')).toBe('model_q4.onnx'); + expect(weightsFile('fp32')).toBe('model.onnx'); + }); + + it('finds a cached q8 model and stays offline', () => { + const cacheDir = mkdtempSync(join(tmpdir(), 'stem-embed-cache-')); + cacheWithWeights(cacheDir); + expect(weightsPresent(cacheDir, REPO, 'q8')).toBe(true); + expect(hubAccessForLoad({ cacheDir, repo: REPO, dtype: 'q8' })).toEqual({ + allowRemoteModels: false, + localModelPath: null + }); + }); + + it('uses the vendor dir and stays offline when the clone shipped weights', () => { + const cacheDir = mkdtempSync(join(tmpdir(), 'stem-embed-empty-')); + const bundledDir = mkdtempSync(join(tmpdir(), 'stem-embed-vendor-')); + cacheWithWeights(bundledDir); + expect(hubAccessForLoad({ cacheDir, bundledDir, repo: REPO, dtype: 'q8' })).toEqual({ + allowRemoteModels: false, + localModelPath: bundledDir + }); + }); + + it('allows Hugging Face when neither cache nor vendor has weights', () => { + const cacheDir = mkdtempSync(join(tmpdir(), 'stem-embed-miss-')); + expect(hubAccessForLoad({ cacheDir, bundledDir: cacheDir, repo: REPO, dtype: 'q8' })).toEqual({ + allowRemoteModels: true, + localModelPath: null + }); + }); + + it('matches ONNX error paths with either slash', () => { + const win = 'C:\\Users\\me\\AppData\\Roaming\\Stem\\embed-models\\Xenova\\multilingual-e5-small'; + const unixish = 'C:/Users/me/AppData/Roaming/Stem/embed-models/Xenova/multilingual-e5-small/onnx/model.onnx'; + expect(pathAppearsInMessage(unixish, win)).toBe(true); + expect(pathAppearsInMessage('Protobuf parsing failed: ' + win, win)).toBe(true); + expect(pathAppearsInMessage('other model', win)).toBe(false); + }); + + it('unpacks a gzipped ONNX and copies tokenizer sidecars into the cache', async () => { + const bundled = mkdtempSync(join(tmpdir(), 'stem-embed-pack-')); + const cache = mkdtempSync(join(tmpdir(), 'stem-embed-out-')); + const onnxDir = join(bundled, REPO, 'onnx'); + mkdirSync(onnxDir, { recursive: true }); + writeFileSync(join(bundled, REPO, 'config.json'), '{"ok":true}'); + const raw = Buffer.from('onnx-bytes'); + writeFileSync(join(onnxDir, 'model_quantized.onnx.gz'), gzipSync(raw)); + expect(await unpackBundledEmbedModels(bundled, cache)).toBe(2); + expect(readFileSync(join(cache, REPO, 'config.json'), 'utf8')).toBe('{"ok":true}'); + expect(readFileSync(join(cache, REPO, 'onnx', 'model_quantized.onnx'))).toEqual(raw); + expect(await unpackBundledEmbedModels(bundled, cache)).toBe(0); + }); + + it('reassembles split gzip parts under GitHub\'s 50 MB warning size', async () => { + const bundled = mkdtempSync(join(tmpdir(), 'stem-embed-parts-')); + const cache = mkdtempSync(join(tmpdir(), 'stem-embed-out-')); + const onnxDir = join(bundled, REPO, 'onnx'); + mkdirSync(onnxDir, { recursive: true }); + const gz = gzipSync(Buffer.from('split-onnx')); + writeFileSync(join(onnxDir, 'model_quantized.onnx.gz.00'), gz.subarray(0, 4)); + writeFileSync(join(onnxDir, 'model_quantized.onnx.gz.01'), gz.subarray(4)); + expect(await unpackBundledEmbedModels(bundled, cache)).toBe(1); + expect(readFileSync(join(cache, REPO, 'onnx', 'model_quantized.onnx')).toString()).toBe('split-onnx'); + }); +}); diff --git a/tests/unit/embed-manager.test.ts b/tests/unit/embed-manager.test.ts index dfbfa7c..638a0af 100644 --- a/tests/unit/embed-manager.test.ts +++ b/tests/unit/embed-manager.test.ts @@ -43,7 +43,7 @@ function fakeWorker(): FakeWorker { return w; } -function manager(opts: { embedTimeoutMs?: number; rerankTimeoutMs?: number } = {}) { +function manager(opts: { embedTimeoutMs?: number; rerankTimeoutMs?: number; bundledDir?: string } = {}) { const workers: FakeWorker[] = []; const mgr = createEmbedWorkerManager({ spawn: () => { @@ -52,7 +52,9 @@ function manager(opts: { embedTimeoutMs?: number; rerankTimeoutMs?: number } = { return w; }, cacheDir: () => '/tmp/models', - ...opts + bundledDir: opts.bundledDir ? () => opts.bundledDir! : undefined, + embedTimeoutMs: opts.embedTimeoutMs, + rerankTimeoutMs: opts.rerankTimeoutMs }); return { mgr, workers }; } @@ -73,11 +75,21 @@ describe('embed worker manager', () => { expect(workers).toHaveLength(0); mgr.ensure(SPEC); expect(workers).toHaveLength(1); - expect(workers[0].sent[0]).toMatchObject({ type: 'load', cacheDir: '/tmp/models' }); + expect(workers[0].sent[0]).toMatchObject({ type: 'load', cacheDir: '/tmp/models', bundledDir: null }); mgr.ensure(SPEC); // idempotent while the same model is up expect(workers).toHaveLength(1); }); + it('forwards the vendor models dir on load so the worker can stay offline', () => { + const { mgr, workers } = manager({ bundledDir: '/repo/vendor/embed-models' }); + mgr.ensure(SPEC); + expect(workers[0].sent[0]).toMatchObject({ + type: 'load', + cacheDir: '/tmp/models', + bundledDir: '/repo/vendor/embed-models' + }); + }); + it('queues embeds while loading and flushes them on ready', async () => { const { mgr, workers } = manager(); mgr.ensure(SPEC); diff --git a/tests/unit/exec-executor.test.ts b/tests/unit/exec-executor.test.ts index 3e37f5a..53314e4 100644 --- a/tests/unit/exec-executor.test.ts +++ b/tests/unit/exec-executor.test.ts @@ -52,20 +52,40 @@ describe('shellInvocation', () => { it('uses the host shell with -c on Unix platforms', () => { // Never a hardcoded /bin/zsh: a Linux server (the Docker image included) has // no zsh, and every command there died with `spawn /bin/zsh ENOENT`. - const expected = { command: unixShell().path, args: ['-c', 'echo hi'], detached: true }; - expect(shellInvocation('echo hi', 'darwin')).toEqual(expected); - expect(shellInvocation('echo hi', 'linux')).toEqual(expected); + const expected = { + command: unixShell().path, + args: ['-c', 'echo hi'], + detached: true, + verbatimArguments: false + }; + expect(shellInvocation('echo hi', 'zsh')).toEqual(expected); }); - it('uses cmd.exe /d /s /c on win32 (no AutoRun)', () => { - const inv = shellInvocation('echo hi', 'win32'); + it('uses cmd.exe /d /s /c on cmd (no AutoRun)', () => { + const inv = shellInvocation('echo hi', 'cmd'); // Quoted /c payload so cmd /s strips one outer pair; inner quotes stay intact. expect(inv.args).toEqual(['/d', '/s', '/c', '"echo hi"']); expect(inv.detached).toBe(false); + expect(inv.verbatimArguments).toBe(true); // ComSpec may be set; otherwise the default is cmd.exe. expect(inv.command.toLowerCase()).toMatch(/cmd\.exe$/); }); + it('uses bash --noprofile --norc -c for Git Bash (no login profile)', () => { + const bash = 'C:\\Program Files\\Git\\bin\\bash.exe'; + expect(shellInvocation('echo hi', 'git-bash', bash)).toEqual({ + command: bash, + args: ['--noprofile', '--norc', '-c', 'echo hi'], + detached: false, + verbatimArguments: false + }); + }); + + it('falls back to cmd when Git Bash is selected without a path', () => { + const inv = shellInvocation('echo hi', 'git-bash', null); + expect(inv.args).toEqual(['/d', '/s', '/c', '"echo hi"']); + expect(inv.verbatimArguments).toBe(true); + }); }); describe('resolveLoginPath', () => { diff --git a/tests/unit/exec-git-bash.test.ts b/tests/unit/exec-git-bash.test.ts new file mode 100644 index 0000000..ce57f5f --- /dev/null +++ b/tests/unit/exec-git-bash.test.ts @@ -0,0 +1,139 @@ +import { describe, expect, it } from 'vitest'; +import { win32 as pathWin32 } from 'node:path'; +import { + bashBesideGit, + detectGitBashFromDisk, + gitBashPathEnv, + isUsableGitBashPath, + resolveGitBashExecutable, + resolveHostShell, + wellKnownGitBashCandidates +} from '../../src/server/exec/git-bash'; +import { unixShell } from '../../src/server/exec/executor'; +import { hostShellAgentHint, hostShellFromPlatform } from '../../src/server/exec/host-shell'; +import { hostShellLabel } from '../../src/server/exec/policy'; + +const BASH = 'C:\\Program Files\\Git\\bin\\bash.exe'; +const GIT = 'C:\\Program Files\\Git\\cmd\\git.exe'; + +describe('isUsableGitBashPath', () => { + it('requires bash.exe and a file that exists', () => { + const exists = (p: string) => p === BASH; + expect(isUsableGitBashPath(BASH, exists)).toBe(true); + expect(isUsableGitBashPath('C:\\Program Files\\Git\\cmd\\git.exe', exists)).toBe(false); + expect(isUsableGitBashPath(null, exists)).toBe(false); + expect(isUsableGitBashPath(' ', exists)).toBe(false); + expect(isUsableGitBashPath(BASH, () => false)).toBe(false); + }); +}); + +describe('detectGitBashFromDisk', () => { + it('finds a well-known Program Files install without spawning', () => { + const env = { ProgramFiles: 'C:\\Program Files' }; + const exists = (p: string) => p === BASH; + expect(detectGitBashFromDisk({ env, exists })).toBe(BASH); + }); + + it('finds bash beside git.exe on PATH (Git\\cmd layout)', () => { + const env = { + ProgramFiles: 'D:\\none', + 'ProgramFiles(x86)': 'D:\\none86', + LOCALAPPDATA: 'D:\\local', + USERPROFILE: 'D:\\home', + Path: 'C:\\Windows;C:\\Program Files\\Git\\cmd' + }; + const exists = (p: string) => p === GIT || p === BASH; + expect(detectGitBashFromDisk({ env, exists })).toBe(BASH); + }); + + it('finds bash.exe itself on PATH', () => { + const env = { + ProgramFiles: 'D:\\none', + LOCALAPPDATA: 'D:\\local', + USERPROFILE: 'D:\\home', + Path: 'C:\\tools\\git\\bin' + }; + const bash = pathWin32.join('C:\\tools\\git\\bin', 'bash.exe'); + const exists = (p: string) => p === bash; + expect(detectGitBashFromDisk({ env, exists })).toBe(bash); + }); + + it('returns null when nothing is on disk (no spawn)', () => { + const env = { + ProgramFiles: 'D:\\none', + LOCALAPPDATA: 'D:\\local', + USERPROFILE: 'D:\\home', + Path: 'C:\\Windows' + }; + expect(detectGitBashFromDisk({ env, exists: () => false })).toBeNull(); + }); +}); + +describe('bashBesideGit', () => { + it('walks Git\\cmd\\git.exe up to Git\\bin\\bash.exe', () => { + expect(bashBesideGit(GIT, (p) => p === BASH)).toBe(BASH); + }); +}); + +describe('wellKnownGitBashCandidates', () => { + it('uses win32 joins so Mac CI sees Windows paths', () => { + const list = wellKnownGitBashCandidates({ + ProgramFiles: 'C:\\Program Files', + 'ProgramFiles(x86)': 'C:\\Program Files (x86)', + LOCALAPPDATA: 'C:\\Users\\me\\AppData\\Local', + USERPROFILE: 'C:\\Users\\me' + }); + expect(list[0]).toBe('C:\\Program Files\\Git\\bin\\bash.exe'); + expect(list.some((p) => p.includes('scoop'))).toBe(true); + }); +}); + +describe('gitBashPathEnv', () => { + it('prepends Git usr\\bin, bin, cmd, and mingw dirs', () => { + const path = gitBashPathEnv(BASH, 'C:\\Windows'); + expect(path.startsWith('C:\\Program Files\\Git\\usr\\bin;')).toBe(true); + expect(path).toContain('C:\\Program Files\\Git\\mingw64\\bin'); + expect(path.endsWith('C:\\Windows')).toBe(true); + }); +}); + +describe('resolveHostShell', () => { + it('is zsh off Windows, cmd on Windows when Settings asked for cmd', () => { + expect(resolveHostShell({ windowsShell: 'cmd', gitBashPath: null }, 'darwin')).toBe('zsh'); + expect(resolveHostShell({ windowsShell: 'cmd', gitBashPath: null }, 'win32')).toBe('cmd'); + }); + + it('uses Git Bash when opted in and bash.exe exists at the saved path', () => { + const exists = (p: string) => p === BASH; + expect(resolveHostShell({ windowsShell: 'git-bash', gitBashPath: BASH }, 'win32', exists)).toBe('git-bash'); + expect(resolveHostShell({ windowsShell: 'git-bash', gitBashPath: BASH }, 'darwin', exists)).toBe('zsh'); + }); + + it('falls back to cmd when Git Bash is selected but bash.exe is not on disk', () => { + expect(resolveHostShell({ windowsShell: 'git-bash', gitBashPath: BASH }, 'win32', () => false)).toBe('cmd'); + }); + + it('auto-detects bash.exe when git-bash is selected with no saved path', () => { + const exists = (p: string) => p === BASH; + expect(resolveHostShell({ windowsShell: 'git-bash', gitBashPath: null }, 'win32', exists)).toBe('git-bash'); + expect(resolveGitBashExecutable({ gitBashPath: null }, exists)).toBe(BASH); + }); +}); + +describe('hostShellLabel / hint', () => { + it('names one shell, never both', () => { + expect(hostShellLabel('cmd')).toContain('cmd.exe'); + expect(hostShellLabel('cmd')).not.toContain('Git Bash'); + expect(hostShellLabel('git-bash')).toContain('Git Bash'); + expect(hostShellLabel('git-bash')).not.toContain('cmd.exe'); + expect(hostShellLabel('zsh')).toContain(unixShell().path.split('/').pop()); + }); + + it('hints only on Windows shells (zsh is already in the tool description)', () => { + expect(hostShellAgentHint('zsh')).toBe(''); + expect(hostShellAgentHint('cmd')).toContain('cmd.exe'); + expect(hostShellAgentHint('git-bash')).toContain('Git Bash'); + expect(hostShellFromPlatform('linux')).toBe('zsh'); + expect(hostShellFromPlatform('win32')).toBe('cmd'); + }); +}); diff --git a/tests/unit/exec-policy.test.ts b/tests/unit/exec-policy.test.ts index da91713..a98992b 100644 --- a/tests/unit/exec-policy.test.ts +++ b/tests/unit/exec-policy.test.ts @@ -94,8 +94,8 @@ describe('classify', () => { const settings = { allowlist: ['git push', 'npm'] }; it('tier 1 for the static allowlist', () => { - expect(classify('ls -la', settings, 'darwin').tier).toBe('run'); - expect(classify('git status', settings, 'darwin').tier).toBe('run'); + expect(classify('ls -la', settings, 'zsh').tier).toBe('run'); + expect(classify('git status', settings, 'zsh').tier).toBe('run'); expect(classify('agent-browser open https://example.com', settings).tier).toBe('run'); // Double-quoted URLs/selectors must stay tier 1 (the agent-browser workflow). expect(classify('agent-browser open "https://youtube.com/watch?v=x&list=y"', settings).tier).toBe('run'); @@ -125,7 +125,7 @@ describe('classify', () => { // Windows smoke checklist uses this shape; it must hit the LLM judge in assisted mode. const cmd = 'powershell.exe -NoProfile -ExecutionPolicy Bypass -Command "1+1"'; - const cls = classify(cmd, settings, 'win32'); + const cls = classify(cmd, settings, 'cmd'); expect(cls.tier).toBe('judge'); expect(cls.prefixes).toEqual(['powershell.exe']); }); @@ -134,41 +134,50 @@ describe('classify', () => { // `dir`/`type`/`echo` exist to make cmd.exe usable; on zsh they would widen // tier 1 for no reason. `ls`/`cat` under cmd would auto-run into "not // recognized" — better to let the judge see an unknown command. - expect(classify('dir /b', settings, 'win32').tier).toBe('run'); - expect(classify('type notes.txt', settings, 'win32').tier).toBe('run'); - expect(classify('where git', settings, 'win32').tier).toBe('run'); - expect(classify('dir /b', settings, 'darwin').tier).toBe('judge'); - expect(classify('echo hello', settings, 'darwin').tier).toBe('judge'); - expect(classify('ls -la', settings, 'win32').tier).toBe('judge'); + expect(classify('dir /b', settings, 'cmd').tier).toBe('run'); + expect(classify('type notes.txt', settings, 'cmd').tier).toBe('run'); + expect(classify('where git', settings, 'cmd').tier).toBe('run'); + expect(classify('dir /b', settings, 'zsh').tier).toBe('judge'); + expect(classify('echo hello', settings, 'zsh').tier).toBe('judge'); + expect(classify('ls -la', settings, 'cmd').tier).toBe('judge'); // Shared entries hold on both. - expect(classify('git status', settings, 'win32').tier).toBe('run'); - expect(classify('rg needle', settings, 'darwin').tier).toBe('run'); + expect(classify('git status', settings, 'cmd').tier).toBe('run'); + expect(classify('rg needle', settings, 'zsh').tier).toBe('run'); }); it("does not let cmd.exe's non-quoting of ' smuggle a second command past tier 1", () => { // cmd.exe has no single-quote quoting: it sees the bare `&` and runs whoami. // A POSIX parse reads the whole thing as one protected argument to `cat`. const smuggle = "cat 'a & whoami & rem '"; - expect(classify(smuggle, settings, 'darwin').tier).toBe('run'); - expect(classify(smuggle, settings, 'win32').tier).toBe('judge'); + expect(classify(smuggle, settings, 'zsh').tier).toBe('run'); + expect(classify(smuggle, settings, 'cmd').tier).toBe('judge'); // Same shape through the entries this port added, and through a pipe. - expect(classify("type 'x & whoami & rem '", settings, 'win32').tier).toBe('judge'); - expect(classify("dir 'x | whoami | rem '", settings, 'win32').tier).toBe('judge'); + expect(classify("type 'x & whoami & rem '", settings, 'cmd').tier).toBe('judge'); + expect(classify("dir 'x | whoami | rem '", settings, 'cmd').tier).toBe('judge'); // %VAR% expands before cmd parses the line, so a variable can inject too. - expect(classify('echo %INJECT%', settings, 'win32').tier).toBe('judge'); - expect(classify('echo "%INJECT%"', settings, 'win32').tier).toBe('judge'); + expect(classify('echo %INJECT%', settings, 'cmd').tier).toBe('judge'); + expect(classify('echo "%INJECT%"', settings, 'cmd').tier).toBe('judge'); // ^ is cmd's escape character. - expect(classify('dir ^& whoami', settings, 'win32').tier).toBe('judge'); + expect(classify('dir ^& whoami', settings, 'cmd').tier).toBe('judge'); + }); + + it('Git Bash uses POSIX quoting and the POSIX allowlist', () => { + // Git Bash honours single quotes, so the cmd smuggle is a protected argument. + const smuggle = "cat 'a & whoami & rem '"; + expect(classify(smuggle, settings, 'git-bash').tier).toBe('run'); + expect(classify('ls -la', settings, 'git-bash').tier).toBe('run'); + expect(classify('dir /b', settings, 'git-bash').tier).toBe('judge'); + expect(classify('echo $HOME', settings, 'git-bash').tier).toBe('judge'); }); it('keeps Windows paths on tier 1 (\\ is a separator to cmd, not an escape)', () => { // The protected-roots scan is what gates paths on Windows; making `\` meta // here would push every `type C:\…` onto the judge and stop the allowlist // from doing anything useful. - expect(classify('type C:\\Users\\me\\notes.txt', settings, 'win32').tier).toBe('run'); - expect(classify('dir "C:\\Program Files"', settings, 'win32').tier).toBe('run'); + expect(classify('type C:\\Users\\me\\notes.txt', settings, 'cmd').tier).toBe('run'); + expect(classify('dir "C:\\Program Files"', settings, 'cmd').tier).toBe('run'); // Still meta on zsh, where it really is an escape. - expect(classify('cat a\\ b', settings, 'darwin').tier).toBe('judge'); + expect(classify('cat a\\ b', settings, 'zsh').tier).toBe('judge'); }); it('judges chains with any non-allowlisted segment', () => { @@ -268,13 +277,18 @@ describe('buildJudgePrompt', () => { it('names the one shell that will run the command, not both', () => { // What is destructive under cmd is not what is destructive under zsh; // describing both invites the model to hedge into `unsure`. - const win = buildJudgePrompt('del /q x', 'C:\\work', undefined, 'win32'); + const win = buildJudgePrompt('del /q x', 'C:\\work', undefined, 'cmd'); expect(win).toContain('cmd.exe'); expect(win).not.toContain('zsh'); - const posix = buildJudgePrompt('rm -rf build', '/tmp/work', undefined, 'darwin'); + expect(win).not.toContain('Git Bash'); + const posix = buildJudgePrompt('rm -rf build', '/tmp/work', undefined, 'zsh'); // The shell that will actually run it — zsh on a Mac, whatever a server has. expect(posix).toContain(unixShell().path.split('/').pop()); expect(posix).not.toContain('cmd.exe'); + const bash = buildJudgePrompt('ls -la', 'C:\\work', undefined, 'git-bash'); + expect(bash).toContain('Git Bash'); + expect(bash).not.toContain('cmd.exe'); + expect(bash).not.toContain('zsh'); }); it("embeds the user's request when available, and says so when not", () => { diff --git a/tests/unit/exec-protected.test.ts b/tests/unit/exec-protected.test.ts index 988b887..7407ac3 100644 --- a/tests/unit/exec-protected.test.ts +++ b/tests/unit/exec-protected.test.ts @@ -2,7 +2,7 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest'; import { mkdtempSync, realpathSync, rmSync, writeFileSync } from 'node:fs'; import { tmpdir } from 'node:os'; import { join } from 'node:path'; -import { scanProtected } from '../../src/server/exec/protected'; +import { scanProtected, msysToWindows } from '../../src/server/exec/protected'; // The main-side fail-closed guard for read-only connected folders: any command // or cwd referencing a protected root is blocked; unreadable gate state blocks @@ -64,7 +64,7 @@ describe('scanProtected on Windows paths', () => { }); const scan = (command: string, cwd = 'C:\\work') => - scanProtected(command, cwd, rootsPath, 'win32'); + scanProtected(command, cwd, rootsPath, 'cmd'); it('blocks drive-absolute paths inside a protected root', () => { // `type` and `dir` are tier 1 under cmd, so this is the gate's whole job. @@ -109,8 +109,39 @@ describe('scanProtected on Windows paths', () => { it('is what the POSIX scan misses — the regression this covers', () => { // Same command, POSIX rules: no `~` and no leading `/`, so nothing matched // and the read-only folder was wide open to `type`. - expect(scanProtected('type C:\\Users\\me\\vault\\secrets.txt', 'C:\\work', rootsPath, 'darwin').blocked).toBe( + expect(scanProtected('type C:\\Users\\me\\vault\\secrets.txt', 'C:\\work', rootsPath, 'zsh').blocked).toBe( false ); }); }); + +describe('scanProtected on Git Bash paths', () => { + const winVault = 'C:\\Users\\me\\vault'; + + beforeEach(() => { + writeFileSync(rootsPath, JSON.stringify({ roots: [winVault] })); + }); + + const scan = (command: string, cwd = 'C:\\work') => + scanProtected(command, cwd, rootsPath, 'git-bash'); + + it('blocks MSYS /c/Users/... paths mapped onto a Windows root', () => { + expect(scan('cat /c/Users/me/vault/secrets.txt').blocked).toBe(true); + }); + + it('still blocks Windows drive paths (the agent may emit either shape)', () => { + expect(scan('cat C:\\Users\\me\\vault\\secrets.txt').blocked).toBe(true); + }); + + it('does not treat a cmd /b flag as drive B:', () => { + expect(scan('ls /b').blocked).toBe(false); + }); +}); + +describe('msysToWindows', () => { + it('maps /c/Users/foo to C:\\Users\\foo', () => { + expect(msysToWindows('/c/Users/me/vault')).toBe('C:\\Users\\me\\vault'); + expect(msysToWindows('/b')).toBeNull(); + expect(msysToWindows('/usr/bin')).toBeNull(); + }); +}); diff --git a/tests/unit/exec-service.test.ts b/tests/unit/exec-service.test.ts index 5868ca0..f5e249b 100644 --- a/tests/unit/exec-service.test.ts +++ b/tests/unit/exec-service.test.ts @@ -31,7 +31,15 @@ function model(id: string, provider: string, isDefault = false): ModelSummary { function baseSettings(): AppSettings { return { - exec: { enabled: true, approvalMode: 'assisted', judgeModel: null, judgeEffort: null, allowlist: [] }, + exec: { + enabled: true, + approvalMode: 'assisted', + judgeModel: null, + judgeEffort: null, + allowlist: [], + windowsShell: 'cmd', + gitBashPath: null + }, // The judge reads these too: unpinned, it runs on the shared background model // if there is one, else the model of the chat that asked. defaults: { model: null, backgroundModel: null, backgroundEffort: 'low' } diff --git a/tests/unit/settings.test.ts b/tests/unit/settings.test.ts index 20bfbfd..b223fc0 100644 --- a/tests/unit/settings.test.ts +++ b/tests/unit/settings.test.ts @@ -548,7 +548,9 @@ describe('exec settings', () => { judgeEffort: null, allowlist: [], deviceAllowlists: {}, - scratchTtlDays: 30 + scratchTtlDays: 30, + windowsShell: 'git-bash', + gitBashPath: null }); }); @@ -600,6 +602,22 @@ describe('exec settings', () => { writeFileSync(path, JSON.stringify({ exec: { approvalMode: 'manual' } })); expect((await readSettings()).exec.approvalMode).toBe('manual'); }); + + it('defaults windowsShell to git-bash and keeps it without a saved path', async () => { + writeFileSync(path, JSON.stringify({ exec: { windowsShell: 'powershell', gitBashPath: 7 } })); + expect((await readSettings()).exec.windowsShell).toBe('git-bash'); + expect((await readSettings()).exec.gitBashPath).toBeNull(); + writeFileSync(path, JSON.stringify({ exec: { windowsShell: 'git-bash' } })); + expect((await readSettings()).exec.windowsShell).toBe('git-bash'); + writeFileSync(path, JSON.stringify({ exec: { windowsShell: 'cmd' } })); + expect((await readSettings()).exec.windowsShell).toBe('cmd'); + const next = await updateExecSettings({ + windowsShell: 'git-bash', + gitBashPath: 'C:\\Program Files\\Git\\bin\\bash.exe' + }); + expect(next.exec.windowsShell).toBe('git-bash'); + expect(next.exec.gitBashPath).toBe('C:\\Program Files\\Git\\bin\\bash.exe'); + }); }); describe('chats settings', () => { diff --git a/tests/unit/transport-http.test.ts b/tests/unit/transport-http.test.ts index 32d47f7..0dc510b 100644 --- a/tests/unit/transport-http.test.ts +++ b/tests/unit/transport-http.test.ts @@ -595,6 +595,10 @@ async function collectBlocks(res: Response, count: number): Promise { event: lines.find((line) => line.startsWith('event: '))?.slice(7) ?? null, data: JSON.parse(data) as Block['data'] }); + // One TCP read can contain several SSE blocks (pushTo then a broadcast + // in the same tick). Stop at `count` so callers asking for the first + // event do not also get whatever arrived in the same chunk. + if (blocks.length >= count) break; } split = buffer.indexOf('\n\n'); } diff --git a/vendor/embed-models/README.md b/vendor/embed-models/README.md new file mode 100644 index 0000000..20bbd5e --- /dev/null +++ b/vendor/embed-models/README.md @@ -0,0 +1,28 @@ +# Bundled embedding models + +Stem’s local memory search loads ONNX weights through transformers.js. On a +machine that can reach [Hugging Face](https://huggingface.co) they download +once into the app data folder (`embed-models/`). GitHub rejects files over +100 MB (and warns above 50 MB), so the default embedder is committed here +**gzipped and split into 45 MB parts**. Stem unpacks them into the app cache +on first launch and does not call the Hub when those files are present. + +The default reranker is ~570 MB even compressed, so it is not in git — it +still downloads from Hugging Face when the network allows it. + +## What is committed + +``` +vendor/embed-models/ + Xenova/multilingual-e5-small/config.json + Xenova/multilingual-e5-small/tokenizer.json.gz + Xenova/multilingual-e5-small/tokenizer_config.json + Xenova/multilingual-e5-small/onnx/model_quantized.onnx.gz.00 + Xenova/multilingual-e5-small/onnx/model_quantized.onnx.gz.01 +``` + +Rebuild the packs from a machine that already has a Stem cache: + +```bash +npm run vendor:embed-models +``` diff --git a/vendor/embed-models/Xenova/multilingual-e5-small/config.json b/vendor/embed-models/Xenova/multilingual-e5-small/config.json new file mode 100644 index 0000000..4104f38 --- /dev/null +++ b/vendor/embed-models/Xenova/multilingual-e5-small/config.json @@ -0,0 +1,25 @@ +{ + "_name_or_path": "intfloat/multilingual-e5-small", + "architectures": [ + "BertModel" + ], + "attention_probs_dropout_prob": 0.1, + "classifier_dropout": null, + "hidden_act": "gelu", + "hidden_dropout_prob": 0.1, + "hidden_size": 384, + "initializer_range": 0.02, + "intermediate_size": 1536, + "layer_norm_eps": 1e-12, + "max_position_embeddings": 512, + "model_type": "bert", + "num_attention_heads": 12, + "num_hidden_layers": 12, + "pad_token_id": 0, + "position_embedding_type": "absolute", + "tokenizer_class": "XLMRobertaTokenizer", + "transformers_version": "4.31.0.dev0", + "type_vocab_size": 2, + "use_cache": true, + "vocab_size": 250037 +} diff --git a/vendor/embed-models/Xenova/multilingual-e5-small/onnx/model_quantized.onnx.gz.00 b/vendor/embed-models/Xenova/multilingual-e5-small/onnx/model_quantized.onnx.gz.00 new file mode 100644 index 0000000..14628db Binary files /dev/null and b/vendor/embed-models/Xenova/multilingual-e5-small/onnx/model_quantized.onnx.gz.00 differ diff --git a/vendor/embed-models/Xenova/multilingual-e5-small/onnx/model_quantized.onnx.gz.01 b/vendor/embed-models/Xenova/multilingual-e5-small/onnx/model_quantized.onnx.gz.01 new file mode 100644 index 0000000..ebd138a Binary files /dev/null and b/vendor/embed-models/Xenova/multilingual-e5-small/onnx/model_quantized.onnx.gz.01 differ diff --git a/vendor/embed-models/Xenova/multilingual-e5-small/tokenizer.json.gz b/vendor/embed-models/Xenova/multilingual-e5-small/tokenizer.json.gz new file mode 100644 index 0000000..94d396f Binary files /dev/null and b/vendor/embed-models/Xenova/multilingual-e5-small/tokenizer.json.gz differ diff --git a/vendor/embed-models/Xenova/multilingual-e5-small/tokenizer_config.json b/vendor/embed-models/Xenova/multilingual-e5-small/tokenizer_config.json new file mode 100644 index 0000000..0592146 --- /dev/null +++ b/vendor/embed-models/Xenova/multilingual-e5-small/tokenizer_config.json @@ -0,0 +1,20 @@ +{ + "bos_token": "", + "clean_up_tokenization_spaces": true, + "cls_token": "", + "eos_token": "", + "mask_token": { + "__type": "AddedToken", + "content": "", + "lstrip": true, + "normalized": true, + "rstrip": false, + "single_word": false + }, + "model_max_length": 512, + "pad_token": "", + "sep_token": "", + "sp_model_kwargs": {}, + "tokenizer_class": "XLMRobertaTokenizer", + "unk_token": "" +}