From 28cc539134f9d90065fd672628bcb41e8eee7780 Mon Sep 17 00:00:00 2001 From: Daniel Bushman Date: Sat, 5 Sep 2026 22:24:13 +0000 Subject: [PATCH 1/2] docs(maproom): document Postgres multi-repo, Bedrock, and --help as source of truth MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The plugin docs described a single-repo SQLite deployment. Every diagnostic path assumed a local maproom.db, so a shared-Postgres fleet hit guidance that was not merely incomplete but actively wrong — connection-pool timeouts routed to chmod fixes for a file that does not exist on that backend. Storage model - Document both backends and how MAPROOM_DATABASE_URL selects between them - Cover the shared-Postgres fleet layout and the devcontainer trap where localhost:5433 is a throwaway instance, so indexing silently goes nowhere - Label the existing SQLite-only entries rather than deleting them Corrections to statements that were the opposite of reality - "Last scan: old date = stale index" — an unchanged git tree SHA short-circuits the scan and leaves the timestamp untouched, so an old date usually means nothing changed. Judging staleness by it causes pointless full re-scans. - "watch keeps the index fresh without manual re-scanning" — true for chunks, false for embeddings. The incremental processor deletes embeddings for changed chunks and never regenerates them, so coverage decays silently while search keeps working via full-text fallback. A periodic generate-embeddings job is required, not optional. - Provider-switch procedure had no deletion step. code_embeddings is UNIQUE on blob_sha, so blobs keep their old-dimension row, incremental runs skip them, and vector search at the new dimension cannot see them. - Ollama dimensions described as "auto-detected" and arbitrary models invited. Only 768/1024/1536 have columns and HNSW indexes; anything else cannot be stored at all. Embedding operations at scale - generate-embeddings degrades as it succeeds: its pending-chunk query is a NOT IN subquery whose cost grows with the table, against a statement_timeout fixed at 5s with no env override. It stalls when closest to finishing. Bound each pass and loop. - Note that work_mem changes that query's plan by three orders of magnitude, so reproducing it by hand in psql is misleading - One failing sub-batch fails the whole pipeline batch; keep batch size above the sub-batch threshold or provider parallelism never engages Providers - Full AWS Bedrock coverage: model dimensions and batch limits, the 256/512 rejection, Cohere v3's silent 512-token truncation, region/endpoint/FIPS config, the credential chain, and retry behavior — with an explicit caveat that Bedrock is absent from some builds - Google/Vertex uses ADC, not an API key; GEMINI_API_KEY and GOOGLE_API_KEY do nothing for it, and expired ADC fails non-interactively - Ollama's devcontainer endpoint trap (host.docker.internal, not localhost) Progressive disclosure - The entry-point skill now routes by task and by verbatim error string - Establishes the binary's own --help as the source of truth for the reader's build; the docs explain concepts, decisions and failure modes rather than mirroring flag lists that rot silently Also documents recovering a lost pm2 watcher fleet, partial scans left behind when a timeout kills one repo mid-run, and silent zero-hit results from a repo-name mismatch. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NCnCnSwkg7BFzyq7Bs8PHp --- plugins/maproom/README.md | 132 ++++- .../references/error-diagnosis.md | 211 ++++++- .../maproom/skills/maproom-search/SKILL.md | 333 +++++++++-- .../maproom-search/references/adc-setup.md | 92 ++- .../references/embedding-providers.md | 297 ++++++++-- .../references/multi-repo-guide.md | 170 +++++- .../references/troubleshooting.md | 527 +++++++++++++++++- 7 files changed, 1618 insertions(+), 144 deletions(-) diff --git a/plugins/maproom/README.md b/plugins/maproom/README.md index 45d60a63..697f59ed 100644 --- a/plugins/maproom/README.md +++ b/plugins/maproom/README.md @@ -19,13 +19,18 @@ The Maproom plugin provides semantic code search capabilities powered by the map Search across multiple repositories by configuring repo-specific search strategies and agent guidance. +Cross-repo search also needs every repo indexed into **one shared database** — in practice +one shared Postgres, because repos indexed into separate SQLite files can only be searched +one at a time. Export the same `MAPROOM_DATABASE_URL` for every repo before you scan (see +[Which backend?](#which-backend)). + 1. Copy the configuration template to your workspace root: ```bash cp plugins/maproom/skills/maproom-search/templates/maproom-repos.yaml ./maproom-repos.yaml ``` 2. Customize repo entries for your workspace (paths, descriptions, search guidance) 3. Set environment variables for path portability (`MAPROOM_REPOS_ROOT`, `MAPROOM_SPECS_ROOT`) -4. Index each repo with `maproom scan` +4. Index each repo with `maproom scan`, with that same shared `MAPROOM_DATABASE_URL` set See [multi-repo-guide.md](skills/maproom-search/references/multi-repo-guide.md) for detailed setup instructions, search strategies by repo type, and cross-repo search patterns. @@ -38,8 +43,44 @@ Before using the Maproom plugin, ensure you have: ```bash maproom --version ``` -3. **Indexed database**: Your codebase must be scanned using `maproom scan` before searching -4. **Database location**: The maproom database is typically located at `~/.maproom/maproom.db` (can be overridden with `MAPROOM_DATABASE_URL` environment variable) + Builds differ in which features and embedding providers they include, so treat + `maproom --help` — which lists the subcommands and environment variables your build + supports — as the source of truth rather than any doc, this one included. +3. **A database, on one of two backends**: maproom chooses its backend at runtime from + `MAPROOM_DATABASE_URL` (a `--database-url` flag overrides the env var): + - `sqlite://…` — or a plain filesystem path — selects **SQLite**. This is the default, + at `~/.maproom/maproom.db`. + - `postgres://…` / `postgresql://…` selects **PostgreSQL**. This requires a binary built + with `--features postgres`. +4. **Indexed content**: your codebase must be scanned with `maproom scan` before searching + +### Which backend? + +One repo on one machine: SQLite is fine — set nothing. + +A multi-repo fleet: point **every** repo at **one shared Postgres**. That single shared +database is what makes cross-repo search work; repos indexed into separate SQLite files +can only ever be searched one at a time. A typical devcontainer URL: + +```bash +export MAPROOM_DATABASE_URL=postgres://maproom:maproom@host.docker.internal:5433/maproom +``` + +### Devcontainer trap: `host.docker.internal`, never `localhost` + +The shared Postgres container runs on the **host** docker daemon, so from inside a +devcontainer it must be reached at `host.docker.internal`. Inside the container, +`localhost:5433` is frequently a *different*, throwaway Postgres — for example a +tmpfs-backed instance used by `cargo test` — whose data vanishes when the container stops. + +Pointing maproom at `localhost` raises no error. Scans succeed, `maproom status` looks +healthy, and the data is simply missing later. If indexed content keeps disappearing, +check the host in your URL first: + +```bash +echo $MAPROOM_DATABASE_URL +pg_isready -h host.docker.internal -p 5433 +``` To verify your setup: ```bash @@ -49,7 +90,7 @@ maproom --version # Index your repository maproom scan -# Verify indexing succeeded +# Verify indexing succeeded (also lists the indexed repo names) maproom status ``` @@ -111,7 +152,9 @@ Uses context expansion to show where validateCart is called throughout the codeb **Solution**: - Run `maproom scan` to index your codebase - Check indexing status: `maproom status` -- Verify database exists: `ls -la ~/.maproom/maproom.db` +- Confirm which database you are actually talking to: `echo $MAPROOM_DATABASE_URL`. On + SQLite, `ls -la ~/.maproom/maproom.db`; on Postgres, + `pg_isready -h host.docker.internal -p 5433` (see the devcontainer trap above) ### No Results Found **Problem**: Searches return no results or irrelevant matches @@ -122,6 +165,10 @@ Uses context expansion to show where validateCart is called throughout the codeb - Check if the repository is actually indexed: `maproom status` - Verify file types are indexed (use `--file-type` filter if needed) - Try different search modes: hybrid (default), fts, or vector +- `--repo` matches the **indexed repo name** (derived from the git origin, suffix + fuzzy-matched), not the directory name on disk. A directory called `django-olympics` + indexed as `django/django` is found by `--repo django`; `--repo django-olympics` + returns zero hits with no error. List the indexed names with `maproom status` - For very recent code changes, re-index: `maproom scan --force` ### Stale Results @@ -130,6 +177,8 @@ Uses context expansion to show where validateCart is called throughout the codeb **Solution**: - Re-index the repository: `maproom scan` - The daemon auto-refreshes but may need manual reindexing for major changes +- Scanning and watching refresh **chunks only**. Embeddings are a separate, scheduled + job — see [Index Maintenance](#index-maintenance) ### Performance Issues **Problem**: Searches are slow or timing out @@ -138,7 +187,15 @@ Uses context expansion to show where validateCart is called throughout the codeb - Reduce the number of results requested (use `k` parameter) - Use FTS mode for exact keyword matches (faster than semantic search) - Check database size: large databases may need optimization -- Ensure SQLite isn't locked by another process +- On the SQLite backend, ensure the database isn't locked by another process +- On the Postgres backend, `pool timed out while waiting for an open connection` from + maproom (with `Connection refused` from `psql`) almost always means the Postgres + container is not running, or you are pointed at the wrong host — it is *not* a file + permissions problem. Check `pg_isready -h host.docker.internal -p 5433`, then start + the host container. The same error appears transiently when something else is + saturating the database, so check `pg_stat_activity` for long-running statements + before concluding the container is down. See + [maproom-guide](skills/maproom-guide/SKILL.md) for the full diagnosis path ## Skills & Agents @@ -157,7 +214,10 @@ The maproom plugin includes specialized skills and agents for different tasks: ## Index Maintenance -The maproom semantic index requires periodic scanning to stay current with codebase changes. +Keeping the index healthy takes **two** jobs, and only one of them is scanning. Scanning +maintains chunks. It does not maintain embeddings. + +### 1. Chunks: periodic scanning **Recommended scan frequency:** - **Active repositories** (daily commits): Run `maproom scan` daily or before research sessions @@ -169,18 +229,60 @@ The maproom semantic index requires periodic scanning to stay current with codeb maproom scan [--repo-path /path/to/repo] ``` -**Index freshness check:** +### 2. Embeddings: a periodic `generate-embeddings` job (required) + +Neither `maproom scan` nor a file watcher regenerates embeddings. The incremental +processor **deletes** the embeddings for chunks that changed and never recreates them, so +embedding coverage decays continuously as you work. Documentation anywhere claiming that +"watch keeps the index fresh" is true for chunks and **false** for embeddings. + +The decay is silent, because losing embeddings does not break search: with zero +embeddings `maproom search` still returns results by falling back to full-text and +structural ranking. Only vector search and semantic ranking quietly go away. Never infer +coverage from "search still works" — check it: + ```bash -maproom status # Shows last scan timestamp +maproom status # per repo, e.g. "Embeddings: 0 (0.0%)" ``` +Hold coverage by scheduling `maproom generate-embeddings` on cron or your process +manager. Two things to know before you write that job: + +- **Bound each pass and loop until pending reaches zero.** An unbounded pass dies at + scale — and gets worse the more chunks you have already embedded — with + `Failed to fetch chunks ... canceling statement due to statement timeout`. The failure + therefore shows up *late*, just as the job is closest to finishing. +- **Batch size is a blast-radius decision.** A single failed provider sub-batch fails the + whole pipeline batch; these failures are transient and self-heal on the next pass, so a + bounded loop converges. + +Run `maproom generate-embeddings --help` for the current flags on your binary. For +provider configuration (Ollama, Google/Vertex, OpenAI, and — where your build includes it +— Bedrock), batch and concurrency tuning, and the model/dimension constraints on +switching embedding models, see +[maproom-guide](skills/maproom-guide/SKILL.md). + +### Index freshness check + +```bash +maproom status +``` + +**"Last scan" is not a freshness signal.** `maproom status` prints a per-worktree +"Last scan" timestamp that is *not* updated when an incremental scan finds the git tree +SHA unchanged — it logs `No changes detected (tree SHA match), skipping scan` and leaves +the old timestamp alone. A months-old "Last scan" therefore usually means "nothing has +changed since then", not "the index is stale"; judging staleness by this field leads to +pointless full re-scans. To really test freshness, search for a symbol you know was added +recently, or compare the repo's current HEAD against what the index recorded. + **Note:** The maproom-researcher agent does NOT automatically trigger index scans. Users must ensure the index is current before invoking the agent for accurate semantic search results. ## Maintenance ### Monthly CLI Verification -**Purpose:** Detect maproom CLI flag deprecation or behavior changes before agents encounter failures. The CLI is at v0.1.0 (pre-release), where breaking changes are allowed per semver. 52 command examples across plugin documentation depend on 6 CLI flags; if any flag is renamed or removed, agents will learn deprecated syntax and encounter command failures. +**Purpose:** Detect maproom CLI flag deprecation or behavior changes before agents encounter failures. The CLI is pre-1.0, where breaking changes are allowed per semver. 52 command examples across plugin documentation depend on 6 CLI flags; if any flag is renamed or removed, agents will learn deprecated syntax and encounter command failures. **Automation:** This procedure is automated via GitHub Actions (see `.github/workflows/monthly-cli-verification.yml`). The workflow runs on the first Friday of each month and creates a GitHub issue if drift is detected. Manual execution is still supported for ad-hoc verification using the `workflow_dispatch` trigger or by running the script directly: ```bash @@ -199,20 +301,20 @@ bash plugins/maproom/scripts/monthly-cli-verification.sh ```bash cd plugins/maproom ``` -- [ ] Run `maproom --version` and verify the version matches the documented version (currently 0.1.0): +- [ ] Run `maproom --version` and record it; if it has moved since the last verification, treat every flag check below as required: ```bash maproom --version ``` -- [ ] Run `maproom search --help` and verify all expected flags are present: +- [ ] Run `maproom search --help` and check the flags it prints against the baseline + deliverable named below — the baseline is where the expected flag list lives, so it is + not duplicated here: ```bash maproom search --help ``` - Confirm these 5 flags exist in the `search` subcommand: `--format`, `--kind`, `--lang`, `--preview`, `--preview-length` -- [ ] Run `maproom vector-search --help` and verify all expected flags are present: +- [ ] Run `maproom vector-search --help` and check it against the same baseline: ```bash maproom vector-search --help ``` - Confirm all 6 flags exist in the `vector-search` subcommand: `--format`, `--kind`, `--lang`, `--preview`, `--preview-length`, `--threshold` - [ ] Compare output against the baseline verification deliverable and check for any discrepancies (new flags, removed flags, renamed flags, changed defaults, changed accepted values): - **Baseline deliverable:** `planning/deliverables/cli-flag-verification.md` (located at the ticket level in your specs directory) - **Full path:** `/path/to/specs/tickets//planning/deliverables/cli-flag-verification.md` diff --git a/plugins/maproom/skills/maproom-guide/references/error-diagnosis.md b/plugins/maproom/skills/maproom-guide/references/error-diagnosis.md index 4b3b6ee3..bb057a07 100644 --- a/plugins/maproom/skills/maproom-guide/references/error-diagnosis.md +++ b/plugins/maproom/skills/maproom-guide/references/error-diagnosis.md @@ -14,7 +14,7 @@ Every maproom error falls into one of five categories. Identifying the category | **Index** | Repo not scanned, stale data, missing embeddings | Yes — scan or regenerate | "No repositories indexed", stale results | | **CLI** | Binary missing, wrong version, invalid flags | Yes — install or fix syntax | "command not found", invalid flag value | | **Query** | Wrong search type, bad filters, special chars | Yes — reformulate query | Zero results, irrelevant results | -| **Infrastructure** | Disk full, permissions, SQLite lock | Yes — fix environment | Permission denied, database locked | +| **Infrastructure** | Database unreachable, disk full, permissions, SQLite lock | Yes — fix environment | `pool timed out while waiting for an open connection`, `Connection refused`, database locked | **If you can't categorize the error** and it mentions panics, segfaults, or stack traces, it may be a **CLI bug** rather than a user-actionable issue. @@ -32,7 +32,7 @@ Every maproom error falls into one of five categories. Identifying the category ### Why Credentials Expire -Maproom's vector search and embedding generation call external APIs (Google Vertex AI by default, or OpenAI). These APIs require authentication tokens that expire periodically as a security measure. +Maproom's vector search and embedding generation call an embedding provider chosen at runtime by `MAPROOM_EMBEDDING_PROVIDER` — Google/Vertex, OpenAI, Cohere, Ollama, Bedrock. Which providers your build actually accepts varies between builds, so confirm against your own binary (`maproom --help`) rather than assuming. Every remote provider needs credentials, and credentials expire or get revoked. - **Google ADC tokens** expire after ~1 hour and are automatically refreshed — but the refresh token itself can expire after extended periods of inactivity - **OpenAI API keys** don't expire by time, but can be revoked or rate-limited @@ -45,6 +45,7 @@ Credential errors contain one of these patterns in the error message: - "No Google credentials found" (appears in the `Caused by` chain) - "invalid_rapt" - "quota_project_id is required" +- `Reauthentication failed. cannot prompt during non-interactive execution` — ADC expired and no human is present to re-consent - References to `OPENAI_API_KEY` when you're not using OpenAI **Note:** The top-level error message is usually "Failed to create embedding service". The specific credential detail (ADC, token provider, etc.) appears in the `Caused by` chain below it. An agent scanning only the first line of error output should match on "Failed to create embedding service" as the primary pattern. @@ -62,9 +63,71 @@ Credential errors contain one of these patterns in the error message: ### What to Do 1. **Don't investigate maproom source code** — credential errors are expected operational events -2. **Fall back to FTS** — `maproom search` works entirely locally, no credentials needed +2. **Fall back to FTS** — `maproom search` needs no embedding-provider credentials at all; full-text and structural ranking answer straight out of whichever backend you are on (local SQLite, or the shared Postgres a fleet points at) 3. **Refresh credentials** when convenient — see [ADC setup guide](../../maproom-search/references/adc-setup.md) +### Google/Vertex Uses ADC — an API Key Does Nothing + +`MAPROOM_EMBEDDING_PROVIDER=google` authenticates with **Application Default Credentials** plus a project id in `GOOGLE_PROJECT_ID` (or `MAPROOM_GOOGLE_PROJECT_ID`). Setting `GEMINI_API_KEY` or `GOOGLE_API_KEY` has **no effect whatsoever** on this provider. This is the most common and most confusing misconfiguration in the whole tool: the variables look like they should work, nothing complains that they are set, and the failure looks like a credential outage rather than a wiring mistake. + +ADC also expires. When it does, every embedding pass dies at config time — before a single chunk is processed: + +```text +Reauthentication failed. cannot prompt during non-interactive execution +``` + +Which fix you want depends on who runs the job: + +- **Interactive:** `gcloud auth application-default login` +- **Unattended** (cron, watcher, CI): set `GOOGLE_APPLICATION_CREDENTIALS` or `MAPROOM_GOOGLE_APPLICATION_CREDENTIALS` to a service-account key. A scheduled embedding job riding on a human's ADC will keep dying with the string above, on a cadence set by however long that ADC lasts. + +Other providers follow the same env-var shape: `openai` reads `OPENAI_API_KEY` (or `MAPROOM_OPENAI_API_KEY`), Cohere direct reads `MAPROOM_COHERE_API_KEY`, `ollama` needs no credentials at all, and `bedrock` resolves the standard AWS chain — static env keys, shared config/credentials files with `AWS_PROFILE`, web identity/IRSA, container credentials. Bedrock is not compiled into every maproom build; check that your binary accepts `MAPROOM_EMBEDDING_PROVIDER=bedrock` before planning around it. + +--- + +## Infrastructure Errors Explained + +Which infrastructure can break depends on which backend you are running, and maproom supports two. The store is chosen at runtime from `MAPROOM_DATABASE_URL`: a `sqlite://` URL or a plain filesystem path selects SQLite (the default, `~/.maproom/maproom.db`); a `postgres://` or `postgresql://` URL selects Postgres, which requires a build compiled with the `postgres` feature. A `--database-url` argument overrides the environment variable. `maproom --help` lists the subcommands and environment variables your build supports. + +### "pool timed out" / "Connection refused" — the Database Is Not There + +```text +pool timed out while waiting for an open connection +``` + +and, from `psql` against the same URL: + +```text +Connection refused +``` + +On a Postgres-backed install this pair almost always means **the Postgres container is not running**, or maproom is pointed at the wrong host. It is *not* a file-permissions problem — permission errors and `database is locked` belong to the SQLite backend, and reaching for `chmod` here spends the incident on the wrong system entirely. + +```bash +pg_isready -h host.docker.internal -p 5433 +``` + +If that fails, start the database container and retry. If it *succeeds* and maproom still times out, the pool is starving rather than absent: something is saturating the database's CPU — a runaway `ANALYZE`, a heavy hand-run query, several embedding jobs at once. Look before restarting anything: + +```sql +select pid, state, now() - query_start as runtime, left(query, 120) +from pg_stat_activity +where state <> 'idle' +order by runtime desc; +``` + +### The Devcontainer `localhost` Trap + +In a devcontainer the shared Postgres normally runs on the **host** docker daemon, and must be reached at `host.docker.internal`: + +```text +postgres://maproom:maproom@host.docker.internal:5433/maproom +``` + +Inside the container, `localhost:5433` may be a *different*, throwaway Postgres — for example a tmpfs-backed instance used by cargo tests — whose data vanishes when the container stops. Pointing maproom there raises **no error at all**: scans report success and the data is simply gone later. The symptom surfaces downstream instead, as repos that "were definitely indexed" reporting zero chunks, or cross-repo search that only ever sees whatever was scanned most recently. Check the URL maproom is actually using before you re-scan anything. + +A local Ollama endpoint has the same shape of trap: `OLLAMA_URL` (also `MAPROOM_OLLAMA_URL`) must be `http://host.docker.internal:11434` from inside a container, because `localhost:11434` is not the host's Ollama. + --- ## Reading `maproom status` Output @@ -74,11 +137,14 @@ Many troubleshooting steps say "run `maproom status`" — here's what to look fo | Field | Healthy Value | Problem Indicator | |---|---|---| | Chunks | > 0 | 0 chunks = repo not scanned | -| Embeddings % | > 0% (for vector search) | 0% = embeddings not generated; FTS still works | -| Last scan | Recent date | Old date or `never` = stale index | +| Embeddings % | > 0% (for vector search) | 0% = embeddings not generated; FTS still works and says nothing | +| Last scan | Any value, including months old | **Not a freshness signal** — see the note below | | Languages | Expected languages listed | Missing languages = incomplete scan | +| Repo name | The name your searches must filter on | Differs from the directory name on disk — the usual cause of silent zero hits | + +`maproom status --json` gives you output you can parse; `maproom status --help` lists the filtering flags your binary supports. -Use `maproom status --json` for programmatic parsing. Use `maproom status --repo ` to check a specific repo. +**`Last scan` does not measure staleness.** An incremental scan that finds the git tree SHA unchanged logs `No changes detected (tree SHA match), skipping scan` and leaves the previous timestamp untouched. A months-old `Last scan` on a quiet worktree therefore means "nothing has changed since then", *not* "this index has rotted". Judging staleness by this field is how people end up running long, pointless full re-scans of repos that were already current. See [Index Freshness Explained](#index-freshness-explained) for checks that actually mean something. --- @@ -86,10 +152,23 @@ Use `maproom status --json` for programmatic parsing. Use `maproom status --repo ### What "Stale Index" Means -Maproom maintains a local SQLite database containing indexed "chunks" of your codebase. This index is a **snapshot** — it reflects the code as it was when you last ran `maproom scan`. Code changes after the scan are invisible to maproom until the next scan. +The index is **not** a single point-in-time snapshot of one local SQLite file, and modelling it that way is the source of most bad staleness calls. Two things vary independently. + +**The backend.** SQLite or Postgres, chosen from `MAPROOM_DATABASE_URL` (see [Infrastructure Errors](#infrastructure-errors-explained)). A multi-repo fleet normally points every repo at one shared Postgres, which is exactly what makes cross-repo search work — so a "missing" repo there is often a repo indexed into a *different* database, not a stale one. + +**The two halves of freshness.** Chunks and embeddings age by different mechanisms, at different rates, and only one of them is refreshed by scanning: + +| Half of the index | Refreshed by | How it actually goes stale | +|---|---|---| +| Chunks (FTS, structure, `context`) | `maproom scan`, or a running `maproom watch` watcher | Only when code changes and nothing scans afterward | +| Embeddings (vector search) | `maproom generate-embeddings`, and nothing else | **Continuously, while you work** — the incremental processor deletes embeddings for changed chunks and never regenerates them | + +"Is my index fresh?" is therefore two questions with two different answers, and the second one degrades on its own even in a worktree that is being watched perfectly. See [The Watch Alternative](#the-watch-alternative). ### When Does the Index Become Stale? +For the **chunk** half, assuming no watcher is running on the worktree: + | Event | Index Impact | Action Needed | |---|---|---| | Editing a few files | Slightly stale — minor risk | Optional re-scan | @@ -98,10 +177,40 @@ Maproom maintains a local SQLite database containing indexed "chunks" of your co | Major refactor (renames, moves) | Definitely stale | Re-scan required | | No code changes | Fresh | No action | +### How to Actually Check Freshness + +Do **not** use the `Last scan` timestamp. An incremental scan that finds the git tree SHA unchanged logs: + +```text +No changes detected (tree SHA match), skipping scan +``` + +and leaves the old timestamp in place, so a perfectly current worktree can report a `Last scan` from months ago. Three checks that carry real information: + +1. **Search for a symbol you know is recent.** A function you added this week either comes back or it doesn't. Fastest honest answer for the chunk half. +2. **Compare the repo's current `HEAD` against the commit the index recorded** for that worktree. Different commit, genuinely behind; same commit, the chunks are current no matter what the timestamp says. +3. **Check embedding coverage explicitly.** `maproom status` reports it per repo as `Embeddings: N (X%)`. Never infer it from "search works" — see below. + +### Zero Embeddings Never Errors + +With no embeddings at all, `maproom search` still returns results: it falls back to full-text and structural ranking. Nothing fails, nothing warns, exit code is 0 — only vector search and semantic ranking are quietly unavailable. `maproom status` shows `Embeddings: 0 (0.0%)` for the repo and the `encoding_runs` table stays empty. + +That is why "search is returning hits, so the index is fine" is an unsafe inference, and why coverage has to be measured rather than assumed. On Postgres you can measure it directly: + +```sql +select r.name, count(distinct c.blob_sha) blobs, count(distinct e.blob_sha) embedded +from repos r +join worktrees w on w.repo_id = r.id +join chunk_worktrees cw on cw.worktree_id = w.id +join chunks c on c.id = cw.chunk_id +left join code_embeddings e on e.blob_sha = c.blob_sha +group by 1 order by 2 desc; +``` + ### Scanning vs Embedding Generation These are two distinct operations: -- **Scan** (`maproom scan`) — parses code with tree-sitter, extracts chunks, stores in SQLite. Fast (~2 seconds for small repos). Required for FTS search. +- **Scan** (`maproom scan`) — parses code with tree-sitter, extracts chunks, stores them in whichever backend is configured (SQLite or Postgres). Fast (~2 seconds for small repos). Required for FTS search. - **Embedding generation** (`maproom generate-embeddings`) — creates vector representations of each chunk via API. Slower (minutes for large repos). Required for vector search. Check progress with `maproom encoding-progress`. After a scan, FTS search works immediately. Vector search requires embeddings to also be up to date. Use `maproom encoding-progress` to monitor embedding generation completion percentage. @@ -116,11 +225,73 @@ By default, `maproom scan` attempts embedding generation after indexing. If cred ⚠️ Warning: Embedding generation failed: Configuration error: Missing required configuration: No Google credentials found. ``` -**This is a partial success, not a failure.** The scan itself succeeded — FTS search works immediately. Only vector search is affected. The scan exits with code 0. To skip embedding generation intentionally, use `--generate-embeddings=false` or `--no-generate-embeddings`. +**This is a partial success, not a failure.** The scan itself succeeded — FTS search works immediately. Only vector search is affected. The scan exits with code 0. Embedding generation can also be skipped deliberately; run `maproom scan --help` for the flag that disables it on your binary. + +### Partial Scan From a Timeout on a Slow Mount + +A nastier partial success: scanning a large repo on a slow mount (FUSE, virtiofs, a network share) can blow through a *wrapping* timeout — a `timeout` command, a job runner's cap — part-way through. That repo is left partially refreshed while every other repo in the run reports success, so the aggregate output looks clean. Nothing in the summary says "repo 7 only got 30% of the way". + +Measured example: a 1,328-file docs repo died at roughly 30% under an 1800s cap, and needed about 2,326s to finish when run unbounded. If one repo's results feel half-there after a batch scan, re-scan that repo on its own with the cap raised or removed, and watch it run to completion. + +### `generate-embeddings` Dies at Scale — and Gets Worse as It Succeeds + +```text +Failed to fetch chunks ... canceling statement due to statement timeout +``` + +This is the most confusing embedding failure, because it shows up *late* — the job stalls exactly when it is closest to finishing. + +The pending-chunk query is a `NOT IN` subquery whose cost grows with the size of `code_embeddings`, and maproom pins `statement_timeout` to 5000ms on every connection it opens (set in `after_connect`, with no environment override). So the more chunks you have successfully embedded, the more expensive it becomes just to ask "what is still pending" — until that question alone exceeds five seconds and every pass dies before embedding anything at all. + +Measured on ~195k chunks with ~113k rows already embedded: asking for 40,000 pending chunks blew the 5s timeout on every pass; asking for 10,000 took about 0.73s. + +**The fix is to bound each pass and loop until the pending count reaches zero**, instead of trying to drain the backlog in one unbounded run. Run `maproom generate-embeddings --help` for the flags that bound a pass on your binary, and see the batch-size tradeoff below. + +**Do not "reproduce" this by hand in psql and trust what you see.** maproom sets `work_mem` (256MB) on its own connections, and that changes this query's plan by three orders of magnitude: the same `NOT IN` plans at cost 218,119,741 (Materialize plus a per-row Seq Scan) under a small `work_mem`, versus 101,635 (a hashed SubPlan) at 64MB and above. A hand-run query in a default psql session looks catastrophically worse than what maproom actually executes — that is a measurement artifact, not the bug. + +### One Bad Sub-Batch Fails the Whole Batch + +```text +Sub-batch 2 failed: API error: Bad request: Batch of 50 texts rejected: {"error":"Post \"http://127.0.0.1:51950/tokenize\": EOF"} +``` + +Embedding requests go out in sub-batches (Ollama: 50 texts per sub-batch, 8 concurrent by default). If one sub-batch errors — the example above is the Ollama model runner dropping the connection — maproom marks the **entire pipeline batch** failed, not just those 50 texts. + +Your batch size is therefore the blast radius: at `--batch-size 1000` one dropped connection costs 1,000 chunks, at 500 it costs 500. These failures are **transient and self-healing** — the lost chunks are simply still pending next time — so a bounded loop converges regardless. A smaller batch just wastes less work per incident. + +Do not shrink it too far, though: sub-batch parallelism engages only when the pipeline batch size **exceeds** the sub-batch size, so a batch of 50 or below runs fully serialized. The knobs are `MAPROOM_EMBEDDING_PARALLEL_ENABLED`, `MAPROOM_EMBEDDING_PARALLEL_SUB_BATCH_SIZE` (default 50) and `MAPROOM_EMBEDDING_PARALLEL_MAX_CONCURRENCY` (default 8). Against a local Ollama, throughput improved up to 8 concurrent requests and got *worse* at 16. + +### Switching Embedding Models Corrupts Coverage Silently + +On Postgres, `code_embeddings` holds one row per `blob_sha` with a UNIQUE constraint on it, and fixed per-dimension columns `embedding_768` / `embedding_1024` / `embedding_1536`, each with its own HNSW index. Two consequences are worth knowing before they become an incident: + +- A blob already embedded at the old dimension **keeps its row** when you switch models. Incremental runs skip it (a row exists) and vector search at the new dimension cannot see it. Switching provider or model without first deleting the old-dimension rows raises no error and silently degrades recall. Verify homogeneity: + ```sql + select embedding_dim, model_version, count(*) from code_embeddings group by 1, 2; + ``` +- Only 768, 1024 and 1536 have columns. A model emitting any other dimension (Ollama `qwen3-embedding:4b` at 2560, or Titan v2's optional 256/512) cannot be stored at all, and is rejected by a dimension validation step rather than failing after a full run. Model choice is constrained by the schema, not by taste. + +`code_embeddings` also has no foreign key to `chunks`, so deleting chunks leaves orphan embedding rows behind. They are harmless in themselves, but they accumulate — and they inflate the `NOT IN` query described above. ### The Watch Alternative -`maproom watch` auto-indexes on file changes, keeping the index fresh without manual re-scanning. Useful during active development. +`maproom watch` auto-indexes on file changes, keeping the **chunk** half of the index fresh without manual re-scanning. Useful during active development. + +**It does not keep embeddings fresh — it consumes them.** The incremental processor *deletes* the embeddings belonging to changed chunks and never regenerates them. Coverage therefore decays continuously and silently for as long as you keep working, and nothing surfaces it, because search keeps returning results the whole way down (see [Zero Embeddings Never Errors](#zero-embeddings-never-errors)). + +Any documentation — including earlier revisions of this page — that says watching "keeps the index fresh without manual re-scanning" is describing chunks only. For embeddings the opposite is true. A watched fleet still needs a periodic `maproom generate-embeddings` job (cron, or a process-manager schedule), bounded and looped as described above, or coverage trends toward zero. + +### When the Watcher Fleet Disappears + +Where watchers run under a process manager — typically one watcher per repo under pm2 — the entire fleet can vanish while the manager itself is still alive and healthy: `pm2 list` prints an empty table, no watcher is running, and scanning silently stops for every repo at once. + +Recovery is one command, not N: + +```bash +pm2 resurrect +``` + +It restores every watcher from `~/.pm2/dump.pm2`. Without knowing that, the fix looks like hand-rebuilding each watcher from scratch. --- @@ -133,6 +304,19 @@ Zero results can mean: 2. **Wrong search type** — using FTS for a concept that needs vector search, or vice versa 3. **Filters too restrictive** — `--kind` or `--lang` excluding valid matches 4. **Query too specific** — too many terms diluting the search signal +5. **`--repo` doesn't match the indexed repo name** — the most common cause of a silent, error-free zero-hit result (below) + +### Silent Zero Hits From a `--repo` Mismatch + +`--repo` matches the **indexed repo name**, which is derived from the git origin, with suffix fuzzy-matching. It does not match the directory name on disk. A checkout living in `django-olympics` but indexed as `django/django` is found by `--repo django` (suffix match), while: + +```text +maproom search --repo django-olympics --query "..." +``` + +returns **zero hits, exit code 0, and no error message at all**. Nothing in that output distinguishes it from "this code does not exist". + +So when a search returns nothing and you expected hits, check the name before you re-scan anything: `maproom status` lists every indexed repo name, and those are the strings `--repo` accepts. ### Why FTS Misses Things Vector Search Finds @@ -169,7 +353,7 @@ The `maproom context` command can fail with: When one search type returns poor results, try the other: 1. FTS returns nothing? → Try vector search with a conceptual rephrasing 2. Vector search returns noise? → Try FTS with specific identifiers from the results -3. Both return nothing? → Verify repo is indexed, check filters, consider Grep fallback +3. Both return nothing? → Verify the indexed repo *name* first (see above), then that the repo is indexed, then filters, then fall back to Grep --- @@ -180,7 +364,10 @@ When one search type returns poor results, try the other: These errors have clear remediation steps the user can take: - Credential expiry → refresh ADC or set API key - Missing index → run `maproom scan` -- Missing embeddings → run `maproom generate-embeddings` +- Missing embeddings → run `maproom generate-embeddings`, bounded per pass and looped +- Embedding pass dying on `canceling statement due to statement timeout` → bound the pass, don't re-run it unbounded +- Zero hits with no error → check the indexed repo name, not the directory name +- `pool timed out` / `Connection refused` → start the database container; check the URL is not `localhost` - Bad query → reformulate with better terms - Wrong search type → switch between FTS and vector - Case-sensitive filters → use lowercase values diff --git a/plugins/maproom/skills/maproom-search/SKILL.md b/plugins/maproom/skills/maproom-search/SKILL.md index 7c1d72e3..df7bfaad 100644 --- a/plugins/maproom/skills/maproom-search/SKILL.md +++ b/plugins/maproom/skills/maproom-search/SKILL.md @@ -12,40 +12,218 @@ description: Semantic code search for exploring unfamiliar codebases and finding | Grep | Exact text/regex | | Glob | File paths | +## The `--help` Contract + +**The binary is the source of truth for flags. This skill is the source of truth for decisions.** + +maproom builds differ. Subcommands, flag names, defaults, storage backends, and +supported embedding providers all vary between versions and between builds compiled +with different feature sets. This skill and its reference files deliberately do **not** +mirror flag lists, because a copied flag list rots silently — it keeps looking correct +long after the binary changed. They explain concepts, the decisions you face, and the +failure modes. + +So before you rely on any flag or environment variable named anywhere in this skill, +confirm it against the binary you actually have: + +```bash +maproom --help # subcommands + environment variables this build supports +maproom --version +maproom search --help # flags for one subcommand +maproom vector-search --help +maproom scan --help +maproom generate-embeddings --help +maproom context --help +``` + +Reach for `--help` by reflex when: + +- a flag is rejected as unknown, or a flag you remember does not appear +- a default matters to your decision (batch size, preview length, traversal depth) +- you need the exact environment variable name for a setting +- you are checking whether a **feature exists in your build at all** — a Postgres + backend needs a build compiled with `--features postgres`, and provider support + (notably `MAPROOM_EMBEDDING_PROVIDER=bedrock`) is not present in every build + +Never assume "maproom supports X" from documentation alone, including this document. +Verify, then proceed. + +## Where to Look Next + +This file is the entry point. It covers setup, choosing a search type, and the +first-line fixes. Everything deeper lives in one of these: + +| If you need to... | Read | +|---|---| +| Run a **multi-repo fleet**, set up **shared Postgres**, search across repos, understand indexed repo names | [multi-repo-guide.md](./references/multi-repo-guide.md) | +| Choose and configure an **embedding provider** (Ollama, Google/Vertex, OpenAI, Cohere, AWS Bedrock), pick a model, understand dimension constraints, switch models safely | [embedding-providers.md](./references/embedding-providers.md) | +| Fix **expired Google ADC** (`Reauthentication failed`) or set up service-account credentials | [adc-setup.md](./references/adc-setup.md) | +| **Diagnose a failure** — dead index, connection pool timeouts, empty results, embedding passes dying, lost watchers | [troubleshooting.md](./references/troubleshooting.md) | +| Write better **queries**, avoid query anti-patterns | [search-best-practices.md](./references/search-best-practices.md) | + +Routing by symptom: + +| Symptom | Go to | +|---|---| +| `pool timed out while waiting for an open connection` | [troubleshooting.md](./references/troubleshooting.md) — Postgres container/host, not file permissions | +| `canceling statement due to statement timeout` during embedding generation | [Generate embeddings](#4-generate-embeddings-separate-recurring-and-bounded) below, then [troubleshooting.md](./references/troubleshooting.md) | +| `Reauthentication failed. cannot prompt during non-interactive execution` | [adc-setup.md](./references/adc-setup.md) | +| `Sub-batch N failed: API error: ...` | [embedding-providers.md](./references/embedding-providers.md) | +| Search returns zero hits for a repo you know is indexed | [Multi-Repo Search](#multi-repo-search) below — usually a repo-name mismatch | +| `Embeddings: 0 (0.0%)` in `maproom status`, but search still works | [Embeddings fail silently](#embeddings-fail-silently) below | + ## First-Time Setup -### 1. Initialize Database +### 1. Choose a storage backend + +maproom has **two** backends and picks one at runtime from `MAPROOM_DATABASE_URL`: + +| `MAPROOM_DATABASE_URL` | Backend | +|---|---| +| unset, a plain filesystem path, or `sqlite://...` | SQLite (default, `~/.maproom/maproom.db`) | +| `postgres://...` or `postgresql://...` | PostgreSQL — **requires a build compiled with `--features postgres`** | + +A `--database-url` flag overrides the environment variable. Run `maproom --help` to +confirm both are present on your build. + +**Single repo on a laptop:** SQLite. Nothing to configure. + +**A fleet of repos where you want cross-repo search:** one shared Postgres serving all +repos. In a devcontainer that typically looks like: + +```bash +export MAPROOM_DATABASE_URL=postgres://maproom:maproom@host.docker.internal:5433/maproom +pg_isready -h host.docker.internal -p 5433 +``` + +> **DevContainer trap — read this before you scan anything.** The Postgres container +> runs on the **host** docker daemon and must be reached at `host.docker.internal`. +> Inside the container, `localhost:5433` is frequently a *different*, throwaway +> Postgres (for example a tmpfs-backed instance used by `cargo test`) whose data +> vanishes when the container stops. Pointing maproom at `localhost` produces **no +> error** — it happily indexes into a disposable database, and you discover the loss +> hours later as missing data. Check the host, not just the port. + +Setup details for a shared fleet are in +[multi-repo-guide.md](./references/multi-repo-guide.md). + +### 2. Initialize Database ```bash maproom db migrate ``` -Run once per machine to create the local database. +Run once per database to create the schema. -### 2. Scan Repository +### 3. Scan Repository ```bash maproom scan ``` -Takes ~2s for small repos. Embeddings are generated by default (`--generate-embeddings true`). +Takes ~2s for small repos. A scan populates **chunks** — the full-text and structural +index. Whether a scan also produces embeddings for new chunks depends on your build's +defaults; check `maproom scan --help`. Do not rely on it: see step 4. For large repos, run in the background: ```bash nohup maproom scan > /tmp/maproom-scan.log 2>&1 & ``` -To regenerate embeddings separately (e.g., after model changes): -```bash -maproom generate-embeddings +> **Do not wrap a scan in a short timeout.** Scanning a large repo on a slow mount +> (FUSE, virtiofs, a network share) can exceed a wrapping timeout *mid-scan*. The repo +> is left partially indexed while every other repo in the same run reports success, so +> the aggregate run looks clean and nothing tells you a repo is half-indexed. Measured +> example: a 1,328-file docs repo died at 30% under an 1800s cap and needed ~2,326s to +> finish when run unbounded. If you must bound it, bound it generously and check each +> repo's result individually. + +### 4. Generate embeddings (separate, recurring, and bounded) + +**Scanning does not keep embeddings current.** This is the single most misunderstood +part of maproom. The incremental processor that watchers use **deletes** the embeddings +for changed chunks and never regenerates them. Embedding coverage therefore **decays +continuously as you work**, silently, because search keeps returning results the whole +time (see [Embeddings fail silently](#embeddings-fail-silently)). + +"Watch keeps the index fresh" is true for **chunks** and false for **embeddings**. A +periodic `maproom generate-embeddings` job — cron, pm2 schedule, whatever you have — is +**required** to hold coverage. + +**Bound every pass.** An unbounded `generate-embeddings` dies at scale, and gets worse +the more successful it has been. Its pending-chunk query is a `NOT IN` subquery whose +cost grows with the number of rows already embedded, and maproom pins +`statement_timeout` to 5000ms on every connection (set in `after_connect`, with no +environment override). Measured on ~195k chunks with ~113k already embedded, asking for +40,000 pending chunks blew past 5s and every pass died with: + ``` +Failed to fetch chunks ... canceling statement due to statement timeout +``` + +Bounding the same fetch to 10,000 measured ~0.73s. The failure appears **late** — the +job stalls exactly when it is closest to finishing. + +So: cap the work per pass and loop until the pending count reaches zero, in the +background. Get the flag names for capping and batching from +`maproom generate-embeddings --help` on your binary, then run it detached: -**Always run embedding generation in the background** — it makes API calls per chunk and can take minutes for large repos. Use the Bash tool's `run_in_background` parameter or `nohup`: ```bash nohup maproom generate-embeddings > /tmp/maproom-embeddings.log 2>&1 & ``` -### 3. Verify +Batch sizing, provider sub-batch behaviour, concurrency tuning +(`MAPROOM_EMBEDDING_PARALLEL_*`), and what happens when you switch models are all in +[embedding-providers.md](./references/embedding-providers.md). Two facts worth carrying +in your head now: + +- A pipeline batch size **at or below the sub-batch size (default 50) runs fully + serialized** — provider parallelism never engages. +- **Switching embedding models requires deleting the old rows.** Embeddings are stored + one row per blob with a unique constraint, so a blob already embedded at the old + dimension keeps its row, incremental runs skip it, and vector search at the new + dimension cannot see it. Coverage is silently corrupted. + +### 5. Verify ```bash maproom status ``` -FTS search works immediately after scan. Vector search requires embeddings to complete. +Confirm three separate things: + +1. **Your repo is listed** — and note the **indexed repo name**, which is derived from + the git origin and is often not the directory name (see + [Multi-Repo Search](#multi-repo-search)). +2. **Embedding coverage** — the per-repo `Embeddings: N (P%)` line. `Embeddings: 0 + (0.0%)` means vector search has nothing to work with. +3. **Not "Last scan"** — that timestamp is not a freshness signal. See below. + +FTS search works immediately after a scan. Vector search requires embeddings. + +#### Embeddings fail silently + +Zero embeddings **does not error**. With no embeddings, `maproom search` still returns +results by falling back to full-text and structural ranking; only vector search and +semantic ranking are unavailable. `maproom status` shows `Embeddings: 0 (0.0%)` and the +`encoding_runs` table stays empty. + +Never infer embedding health from "search works". Check coverage explicitly. On +Postgres you can check it per repo directly: + +```sql +select r.name, count(distinct c.blob_sha) blobs, count(distinct e.blob_sha) embedded +from repos r join worktrees w on w.repo_id=r.id +join chunk_worktrees cw on cw.worktree_id=w.id +join chunks c on c.id=cw.chunk_id +left join code_embeddings e on e.blob_sha=c.blob_sha group by 1 order by 2 desc; +``` + +#### "Last scan" is not a freshness signal + +`maproom status` prints a per-worktree **Last scan** timestamp that is **not updated** +when an incremental scan finds the git tree SHA unchanged — it logs +`No changes detected (tree SHA match), skipping scan` and leaves the old timestamp in +place. A months-old "Last scan" therefore usually means *nothing changed since then*, +not *the index is stale*. Judging staleness by this field leads to pointless full +re-scans. + +To actually test freshness: search for a symbol you know was added recently, or compare +the repo's current HEAD against what the index recorded. ## Choosing Search Type @@ -63,7 +241,7 @@ FTS search works immediately after scan. Vector search requires embeddings to co ```bash maproom vector-search --repo --query "" --format agent ``` -Requires embeddings (see First-Time Setup step 2). +Requires embeddings (see First-Time Setup step 4). ### Evidence from Testing @@ -80,7 +258,7 @@ For query optimization, see [search-best-practices.md](./references/search-best- The `search` and `vector-search` commands support two output formats via `--format`: **JSON (default):** Verbose structured output with full metadata. Preview requires explicit `--preview` flag. -**Agent (`--format agent`):** Compact one-line-per-result optimized for agent context windows. Preview is implicit (120 chars). +**Agent (`--format agent`):** Compact one-line-per-result optimized for agent context windows. Preview is implicit. ### JSON Format Example ```bash @@ -100,7 +278,7 @@ plugins/.../README.md:150 | code_block Code: text | 3.60 | ```text tests/ ├─ Structure: `{file}:{line} | {kind} {symbol} | {score} | {preview}...` ### Preview Behavior -Agent format implicitly enables preview — adding `--preview` is redundant. Use `--preview-length` to adjust (default: 120 chars for agent, 200 for json). For JSON format, `--preview` must be explicitly passed to include a `"preview"` field. +Agent format implicitly enables preview — adding `--preview` is redundant. For JSON format, preview must be explicitly requested before a `"preview"` field appears. The preview length is adjustable and its default differs per format; `maproom search --help` carries the current values for your build. ### Recommendation For agent use, always pass `--format agent`. It conserves context window tokens while preserving essential location, kind, score, and preview information. @@ -109,6 +287,12 @@ For agent use, always pass `--format agent`. It conserves context window tokens All filter values are **case-sensitive**. Combine multiple values with commas for OR logic. Filters are AND-combined across flags: `--kind func --lang py` returns only Python functions. +> **The list below is orientation, not a specification.** Chunk kinds and language +> tags come from the parser in *your* build and change between versions. Run +> `maproom search --help` for the values your binary accepts, and check what is +> actually present in your index rather than assuming a kind exists: +> `maproom search --repo --query --format json | jq -r '.hits[].kind' | sort -u` + | Flag | Value | Matches | |------|-------|---------| | `--kind` | `func` | Function definitions | @@ -130,15 +314,26 @@ All filter values are **case-sensitive**. Combine multiple values with commas fo | `--lang` | `md` | Markdown (.md) | | `--lang` | `json` | JSON (.json) | +The vocabulary above is a *taxonomy*, not a flag reference — the authoritative list of +filter flags and their defaults for your build is `maproom search --help`. + ```bash $ maproom search --repo --query "auth" --kind func --lang py --format agent $ maproom vector-search --repo --query "error handling" --threshold 0.7 --format agent ``` -**`--preview-length `** — Adjust preview character limit (default: 120 for agent, 200 for json). See [Output Formats](#output-formats) for preview behavior details. -**`--threshold `** — Vector-search only. Cosine similarity filter (0.0-1.0); only results >= threshold are returned. Omit for no filtering. -**`--worktree `** — Filter results to a specific worktree. Useful in multi-worktree environments to avoid cross-worktree noise. -**`--deduplicate `** — Deduplicate results across worktrees (default: true). Use `--no-deduplicate` to see all results including duplicates from different worktrees. +Beyond kind and language there are three more decisions worth knowing about. +**Similarity cut-off** is vector-search only: `--threshold` takes a cosine similarity +between 0.0 and 1.0 and drops everything below it — raise it when a concept query keeps +returning plausible-but-wrong neighbours, omit it to see the full ranking. +**Worktree scope** matters in a multi-worktree checkout: results can be narrowed to a +single worktree, and by default a chunk present in several worktrees collapses to one +hit — turn deduplication off only when you specifically need to know which worktrees +contain a match. **Preview length** trades context-window tokens for readable snippets +(see [Output Formats](#output-formats)). + +Exact spellings and current defaults for all of these come from `maproom search --help` +and `maproom vector-search --help`. | Task | Recommended Flags | |------|-------------------| @@ -156,15 +351,15 @@ Explore a chunk's relationships after finding it via search: maproom context --chunk-id [flags] ``` -| Flag | Purpose | Example | -|------|---------|---------| -| `--callers` | Include functions that call this chunk | `--callers` | -| `--callees` | Include functions called by this chunk | `--callees` | -| `--tests` | Include related test files | `--tests` | -| `--docs` | Include related documentation | `--docs` | -| `--config` | Include related configuration files | `--config` | -| `--max-depth` | Traversal depth (default: 2) | `--max-depth 3` | -| `--budget` | Token limit for context bundle (default: 6000) | `--budget 4000` | +Context expansion involves two decisions. **What to pull in** — the relationship axes: +the call graph around the chunk (its callers, its callees), plus related tests, docs and +configuration. Ask only for the axes you will actually read; each one widens the bundle. +**How far and how much** — a traversal depth and a token budget for the assembled +bundle. Depth is what you raise when tracing a call chain up to its entry point; budget +is what you lower when the bundle starts crowding out your context window. + +Run `maproom context --help` for the flag names and defaults on your binary — depth and +budget defaults in particular differ between older and newer builds. **Note:** The `--chunk-id` requires a numeric ID from `--format json` output, not the `file:line` format from `--format agent`. @@ -240,6 +435,25 @@ _(FTS because configuration keywords are known terms.)_ > All workflows above use `--repo ` placeholders. This section explains how to choose which repo to use. +Cross-repo search requires that every repo lives in **one shared database** — in +practice, one shared Postgres for the fleet (see +[First-Time Setup step 1](#1-choose-a-storage-backend)). Repos indexed into separate +SQLite files cannot be searched together. + +### `--repo` matches the indexed name, not the directory + +`--repo` matches the **indexed repo name**, which is derived from the git origin, with +suffix fuzzy-matching. It is **not** the directory name on disk. A directory named +`django-olympics` whose origin makes it `django/django` is found by `--repo django`; +`--repo django-olympics` returns **zero hits with no error message**. + +Silent empty results are usually this. Before blaming the query or the embeddings, list +the actual indexed names: + +```bash +maproom status +``` + ### Choosing the Right Repo | Question Type | Search In | Why | @@ -288,23 +502,78 @@ maproom context --chunk-id --callers --callees ``` _(Vector search across repos: specs for "why", code for "how".)_ -For detailed multi-repo strategies, cross-repo patterns, and chunk kind information, see [multi-repo-guide.md](./references/multi-repo-guide.md). +For detailed multi-repo strategies, shared-Postgres setup, cross-repo patterns, and chunk kind information, see [multi-repo-guide.md](./references/multi-repo-guide.md). ## Troubleshooting -For detailed error recovery steps, see [troubleshooting.md](./references/troubleshooting.md). +Start here; full error recovery steps are in [troubleshooting.md](./references/troubleshooting.md). + +**`pool timed out while waiting for an open connection`** (Postgres backend; `psql` +also reports `Connection refused`): +The Postgres **container is not running**, or you are pointed at the wrong host. This +is *not* a file-permissions problem — that diagnosis only applies to SQLite. Check +`pg_isready -h host.docker.internal -p 5433`, then start the host container. The same +error appears transiently when something is saturating the database's CPU (a runaway +`ANALYZE`, a heavy manual query), so check `pg_stat_activity` for long-running +statements before concluding the container is down. + +**`Failed to fetch chunks ... canceling statement due to statement timeout`** (during +`generate-embeddings`): +The pending-chunk query exceeded maproom's hard-pinned 5000ms `statement_timeout`. +Bound each pass to a smaller number of chunks (~10,000 measured ~0.73s where 40,000 +timed out) and loop until pending reaches zero. See +[First-Time Setup step 4](#4-generate-embeddings-separate-recurring-and-bounded). + +**`Reauthentication failed. cannot prompt during non-interactive execution`** +(Google/Vertex provider): +Application Default Credentials expired. Re-run `gcloud auth application-default login` +interactively, or use a service-account key for unattended runs. Note that +`GEMINI_API_KEY` / `GOOGLE_API_KEY` do nothing for this provider. See +[adc-setup.md](./references/adc-setup.md). + +**`Sub-batch N failed: API error: ...`** (one sub-batch errors, whole batch marked +failed): +These are usually transient provider hiccups and self-heal on the next pass, so a +bounded loop converges. A smaller pipeline batch size limits the blast radius per +incident — but keep it **above** the sub-batch size (default 50) or provider +parallelism never engages. See +[embedding-providers.md](./references/embedding-providers.md). **Token limit exceeded** (`input token count is ... but the model supports up to 20000`): -Re-run with smaller batches: `maproom generate-embeddings --batch-size 25` +The payload sent to the provider is too large. Reduce the provider sub-batch size +(`MAPROOM_EMBEDDING_PARALLEL_SUB_BATCH_SIZE`, default 50) rather than dropping the +pipeline batch size to a tiny value — a pipeline batch at or below the sub-batch size +runs fully serialized. Check `maproom generate-embeddings --help` for the batch flags +on your build. **Vector search returns no results** (search completes but returns empty): -Verify embeddings exist: `maproom status` — if missing, run `maproom generate-embeddings` +Check embedding coverage explicitly — `maproom status`, per-repo +`Embeddings: N (P%)`. Zero embeddings never raises an error; `search` keeps working via +full-text fallback while `vector-search` has nothing to match. If coverage is low, run a +bounded `maproom generate-embeddings` loop. If coverage is fine, suspect the repo name +(see [`--repo` matches the indexed name](#--repo-matches-the-indexed-name-not-the-directory)). + +**Zero hits from a repo you know is indexed:** +Almost always a `--repo` name mismatch — the indexed name comes from the git origin, +not the directory. Run `maproom status` and use the listed name. **No repositories indexed** (status shows no repositories): -Run First-Time Setup above to scan and index your repository. +Run First-Time Setup above. On Postgres, first confirm you are pointed at the shared +database and not a throwaway `localhost` instance. **Stale results after code changes** (results reference old or deleted code): -Re-scan the repository: `maproom scan` +Re-scan the repository with `maproom scan`. Do **not** use the `Last scan` timestamp as +evidence of staleness — it is not updated when an incremental scan finds the tree SHA +unchanged. And remember that re-scanning restores **chunks only**; embeddings for +changed chunks were deleted and need a separate `generate-embeddings` pass. + +**Embedding coverage keeps dropping while watchers run:** +Expected, not a bug. Watchers delete embeddings for changed chunks and never regenerate +them. Schedule a recurring bounded `generate-embeddings` job. + +**Every watcher disappeared** (`pm2 list` shows an empty table, pm2 daemon still alive): +Run `pm2 resurrect` — it restores the whole fleet from `~/.pm2/dump.pm2`. Do not +hand-rebuild watchers one by one. **Irrelevant results** (results don't match intent): Check Choosing Search Type above — FTS for exact terms, vector for concepts. Use 2-3 core terms. diff --git a/plugins/maproom/skills/maproom-search/references/adc-setup.md b/plugins/maproom/skills/maproom-search/references/adc-setup.md index 44b33b25..e8b9ca0c 100644 --- a/plugins/maproom/skills/maproom-search/references/adc-setup.md +++ b/plugins/maproom/skills/maproom-search/references/adc-setup.md @@ -2,6 +2,19 @@ Step-by-step guide for configuring Google Application Default Credentials (ADC) in a DevContainer environment. ADC is required when using Google Vertex AI as the embedding provider for maproom vector search. +> **ADC is not an API key.** The Google provider authenticates *only* through Application +> Default Credentials — either a user credential written by `gcloud auth application-default +> login`, or a service-account key pointed at by `GOOGLE_APPLICATION_CREDENTIALS`. Setting +> `GEMINI_API_KEY` or `GOOGLE_API_KEY` does **nothing** for this provider: maproom never reads +> them, and exporting them will not fix a credential error. This is the single most common +> mistake with the Google provider — if you have an API key and no ADC, you are not +> authenticated. (Bare API keys belong to *other* providers — `openai` reads `OPENAI_API_KEY`, +> Cohere direct reads `MAPROOM_COHERE_API_KEY` — never to `google`.) + +Provider availability differs between maproom builds. Confirm your binary accepts +`MAPROOM_EMBEDDING_PROVIDER=google` — `maproom --help` lists the subcommands and environment +variables your build supports. + ## Prerequisites - **gcloud CLI** installed (already available in the DevContainer) @@ -54,9 +67,12 @@ Configure maproom to use Google Vertex AI as the embedding provider: ```bash export MAPROOM_EMBEDDING_PROVIDER=google -export MAPROOM_GOOGLE_PROJECT_ID=YOUR_PROJECT_ID +export MAPROOM_GOOGLE_PROJECT_ID=YOUR_PROJECT_ID # GOOGLE_PROJECT_ID also works ``` +Note what is *not* here: there is no API-key variable to set. The project ID says *which* +project is called; ADC (Step 1) is what proves *who* is calling. + ### Step 4: Verify Credentials ```bash @@ -71,7 +87,10 @@ If this prints a long token string (starting with `ya29.`), ADC is configured co maproom vector-search --repo YOUR_REPO --query "test query" --format agent ``` -If this returns search results without credential errors, setup is complete. +If this returns search results without credential errors, setup is complete. Zero hits with no +error is *not* a credential failure: `--repo` matches the indexed repo name derived from the git +origin (with suffix fuzzy-matching), not the directory name on disk, so a mismatch returns an +empty result silently. List the indexed names with `maproom status`. ## Refreshing Expired Credentials @@ -89,7 +108,19 @@ Caused by: or errors containing `invalid_rapt`. -**Resolution:** +In a non-interactive context — a cron job, a pm2-managed process, a background +`maproom generate-embeddings` run — expired ADC surfaces as this instead, and every pass dies +at config time before a single embedding is produced: + +``` +Reauthentication failed. cannot prompt during non-interactive execution +``` + +That string means exactly one thing: gcloud wants a human to re-authenticate and there is no +terminal to ask. There are two fixes, and which one you want depends on whether a human is +around. + +**Resolution (interactive):** Re-run the login and quota project commands: @@ -104,6 +135,22 @@ gcloud auth application-default set-quota-project YOUR_PROJECT_ID gcloud auth application-default print-access-token ``` +**Resolution (unattended):** + +User credentials expire on a schedule you do not control, so they are the wrong credential for +scheduled or background jobs. Use a service-account key instead — it does not expire and never +prompts: + +```bash +export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json +# maproom also accepts MAPROOM_GOOGLE_APPLICATION_CREDENTIALS for the same purpose +``` + +The service account needs the same Vertex AI access as your user (see +[Permission denied](#error-permission-denied-on-vertex-ai-api) below). Make sure the variable is +exported in the environment the *job* runs in — a cron or pm2 process does not inherit your +interactive shell. + ## Troubleshooting ### Error: "Failed to create token provider from ADC" @@ -133,6 +180,31 @@ Caused by: gcloud auth application-default set-quota-project YOUR_PROJECT_ID ``` +### Error: "Reauthentication failed. cannot prompt during non-interactive execution" + +**Full error:** + +``` +Reauthentication failed. cannot prompt during non-interactive execution +``` + +**Root cause:** ADC user credentials have expired and gcloud cannot open an interactive +re-auth prompt (cron, pm2, CI, a background embedding run). Note that this is *not* fixable by +setting `GEMINI_API_KEY` or `GOOGLE_API_KEY` — see [the note at the top](#google-adc-setup-for-devcontainer). + +**Resolution:** either re-authenticate interactively — + +```bash +gcloud auth application-default login --no-launch-browser +gcloud auth application-default set-quota-project YOUR_PROJECT_ID +``` + +— or, for anything unattended, switch that job to a service-account key: + +```bash +export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json +``` + ### Error: "invalid_rapt" **Root cause:** The RAPT (Re-Auth Policy Token) has expired. This is the same as token expiry but specific to organizations with re-authentication policies. @@ -199,7 +271,7 @@ chmod 600 ~/.config/gcloud/application_default_credentials.json .config/gcloud/ ``` -**Token rotation:** ADC tokens expire periodically. Re-run `gcloud auth application-default login` when tokens expire. There is no way to set a permanent, non-expiring ADC token (this is a security feature). +**Token rotation:** ADC *user* credentials expire periodically, and there is no way to make one permanent (this is a security feature) — re-run `gcloud auth application-default login` when they expire. That is exactly why unattended jobs should use a service-account key (`GOOGLE_APPLICATION_CREDENTIALS`) instead of a user credential. **Service account keys:** If using service account JSON keys (`GOOGLE_APPLICATION_CREDENTIALS`), store them securely, rotate them regularly, and never commit them to version control. @@ -219,9 +291,19 @@ Use this checklist to confirm ADC is fully configured: - [ ] Access token is valid: `gcloud auth application-default print-access-token` (prints a token, not an error) - [ ] Vector search works: `maproom vector-search --repo YOUR_REPO --query "test" --format agent` (returns results without credential errors) +## If Google Auth Is Impractical + +If you cannot get ADC working — no browser, no service account you are allowed to create, a +re-auth policy that keeps expiring under a scheduled job — the Google provider is not your only +option. **Ollama** runs locally and is free (no cloud credentials at all), and **AWS Bedrock** +is an option where AWS credentials are easier to obtain than Google ones, though Bedrock support +is not present in every maproom build. See +[Embedding Provider Comparison](./embedding-providers.md) for how each is configured and how to +choose between them, and confirm what your binary actually accepts with `maproom --help`. + ## Related Documentation -- [Embedding Provider Comparison](./embedding-providers.md) - Compare Google Vertex AI, OpenAI, and Ollama providers +- [Embedding Provider Comparison](./embedding-providers.md) - Compare Google Vertex AI, OpenAI, Ollama, and (build permitting) AWS Bedrock providers - [Troubleshooting](./troubleshooting.md) - Common maproom error messages and recovery steps - [Google Cloud ADC Documentation](https://cloud.google.com/docs/authentication/provide-credentials-adc) - Official Google documentation - [Vertex AI Pricing](https://cloud.google.com/vertex-ai/pricing) - Google Cloud pricing for embedding API calls diff --git a/plugins/maproom/skills/maproom-search/references/embedding-providers.md b/plugins/maproom/skills/maproom-search/references/embedding-providers.md index c83dd631..fe4e1731 100644 --- a/plugins/maproom/skills/maproom-search/references/embedding-providers.md +++ b/plugins/maproom/skills/maproom-search/references/embedding-providers.md @@ -1,20 +1,34 @@ # Embedding Provider Comparison -Maproom supports three embedding providers for vector search: Google Vertex AI, OpenAI, and Ollama. This guide helps you choose the right provider and configure it for your environment. +Maproom can embed code chunks with several providers: Google Vertex AI, OpenAI, Ollama, AWS Bedrock, and (in some builds) Cohere directly. This guide helps you choose one, configure it, and — critically — switch between them without silently corrupting your index. + +**Provider availability differs between builds.** Do not assume a provider exists because it is documented here. Confirm against the binary you actually have: `maproom --help` lists the subcommands and environment variables your build supports, and the fastest check for a specific provider is to set `MAPROOM_EMBEDDING_PROVIDER=` and see whether a small run is accepted at config time. + +## Read This First: Dimension Is a Hard Schema Constraint + +**Only 768, 1024, and 1536 dimensional vectors can be stored. There is no fourth option.** + +The Postgres schema has fixed per-dimension columns — `embedding_768`, `embedding_1024`, `embedding_1536` — each with its own HNSW index. A model that emits any other dimension cannot be stored **at all**. Ollama's `qwen3-embedding:4b` (2560 dimensions), for example, is unusable no matter how good it is. Amazon Titan v2 can emit 256- or 512-dimensional vectors; those are rejected by a dimension validation step rather than failing after a full scan. + +So model choice is constrained by the schema, not by taste. Before you pull a model or point at a new endpoint, answer one question: *does it emit 768, 1024, or 1536?* If not, stop. + +`MAPROOM_EMBEDDING_DIMENSION` is **not** a free override. It does not reshape a model's output — a 2560-dimensional model does not become a 1024-dimensional model because you asked. Treat it as a declaration that must agree with both the model's real output width and the stored column set. ## Overview -Embedding providers convert code chunks into numerical vectors (embeddings) that enable semantic search. The provider you choose affects search quality, cost, and setup complexity. +Embedding providers convert code chunks into numerical vectors (embeddings) that enable semantic search. The provider you choose affects search quality, cost, throughput, and setup complexity. -**Important:** All code indexed with one provider must be searched with the same provider. Embeddings from different providers occupy incompatible vector spaces due to differing dimensions and training data. Switching providers requires re-indexing your entire repository. +**Important:** All code indexed with one provider must be searched with the same provider. Embeddings from different providers occupy incompatible vector spaces due to differing dimensions and training data. Switching providers requires regenerating embeddings **and deleting the old rows first** — see [Switching Providers or Models](#switching-providers-or-models). Skipping the delete does not error; it silently leaves blobs stranded at the old dimension. ## Provider Comparison Table | Provider | Authentication | Setup Steps | Cost | Dimensions | Pros | Cons | When to Use | |----------|---------------|-------------|------|------------|------|------|-------------| -| Google Vertex AI | ADC or service account (`GOOGLE_APPLICATION_CREDENTIALS`) | [ADC setup guide](./adc-setup.md) | Free tier available, then per-character pricing ([pricing](https://cloud.google.com/vertex-ai/pricing)) | 768 (`text-embedding-004`) | High quality embeddings; free tier; Google ecosystem integration | Requires GCP project; ADC credentials expire and need refresh | Teams already using Google Cloud; production environments with GCP infrastructure | +| Google Vertex AI | Application Default Credentials or service account — **not an API key** | [ADC setup guide](./adc-setup.md) | Free tier available, then per-character pricing ([pricing](https://cloud.google.com/vertex-ai/pricing)) | 768 (`text-embedding-004`) | High quality embeddings; free tier; Google ecosystem integration | Requires GCP project; ADC expires and kills unattended runs | Teams already using Google Cloud; production environments with GCP infrastructure | | OpenAI | API key (`OPENAI_API_KEY`) | Set environment variable | ~$0.02 per 1M tokens ([pricing](https://openai.com/api/pricing/)) | 1536 (`text-embedding-3-small`) | Simple API key setup; widely used; high quality | Requires paid API key; higher dimensions increase storage; network-dependent | Quick setup; teams already using OpenAI; when simplicity is preferred | -| Ollama | None (local) | Install Ollama and pull model | Free (runs locally) | Varies by model (768 for `nomic-embed-text`, 1024 for `mxbai-embed-large`) | Free; fully offline; no credentials needed; data stays local | Requires local compute resources; quality varies by model; slower on CPU | Air-gapped environments; cost-sensitive workflows; local development without API keys | +| Ollama | None (local) | Install Ollama, pull model, set `OLLAMA_URL` | Free (runs locally) | 1024 (`mxbai-embed-large`, default), 768 (`nomic-embed-text`) — other models only if they emit 768/1024/1536 | Free; fully offline; no credentials needed; data stays local | Requires local compute; model runner drops connections under load (see [Batching](#batching-parallelism-and-blast-radius)); devcontainer endpoint trap | Air-gapped environments; cost-sensitive workflows; local development without API keys | +| AWS Bedrock | AWS credential chain (env, profile, IRSA, container creds) | Set region + credentials | Per-token AWS pricing | 1024 (Titan v2, Cohere v3), 1536 (Titan v1) | Fits existing AWS IAM/VPC posture; no new credential system; managed scaling | **Not in every build**; Titan max batch is 1; Cohere silently truncates at 512 tokens | Teams already on AWS; environments where egress must stay inside a VPC | +| Cohere (direct) | API key (`MAPROOM_COHERE_API_KEY`) | Set environment variable | Per-token Cohere pricing | 1024 (v3 models) | Direct access without AWS | Availability varies by build; same 512-token truncation behavior as on Bedrock | You want Cohere v3 without an AWS account | > **Pricing disclaimer:** Costs shown are approximate as of Feb 2026 and are subject to change. Always check the linked pricing pages for current rates. @@ -23,10 +37,12 @@ Embedding providers convert code chunks into numerical vectors (embeddings) that ### Google Vertex AI **Model:** `text-embedding-004` -**Dimensions:** 768 +**Dimensions:** 768 (stored in `embedding_768`) **Endpoint:** `REGION-aiplatform.googleapis.com` (e.g., `us-central1-aiplatform.googleapis.com`) -Google Vertex AI uses Application Default Credentials (ADC) for authentication. See the [ADC Setup Guide](./adc-setup.md) for complete step-by-step instructions. +Google Vertex AI uses **Application Default Credentials (ADC)**, not an API key. See the [ADC Setup Guide](./adc-setup.md) for complete step-by-step instructions. + +> **The most common Google mistake:** setting `GEMINI_API_KEY` or `GOOGLE_API_KEY` does **nothing** for this provider. Neither variable is consulted. If you set one and embedding still fails to authenticate, that is why. **Environment variables:** @@ -34,36 +50,44 @@ Google Vertex AI uses Application Default Credentials (ADC) for authentication. # Set the embedding provider export MAPROOM_EMBEDDING_PROVIDER=google -# Set your Google Cloud project ID +# Set your Google Cloud project ID (either name works) +export GOOGLE_PROJECT_ID=YOUR_PROJECT_ID export MAPROOM_GOOGLE_PROJECT_ID=YOUR_PROJECT_ID # Optional: override model (default: text-embedding-004) export MAPROOM_EMBEDDING_MODEL=text-embedding-004 - -# Optional: override dimensions (default: 768) -export MAPROOM_EMBEDDING_DIMENSION=768 ``` **Authentication options:** -1. **ADC (recommended for development):** Run `gcloud auth application-default login` and set a quota project. See [ADC Setup Guide](./adc-setup.md). -2. **Service account JSON:** Set `GOOGLE_APPLICATION_CREDENTIALS` to the path of a service account key file. +1. **ADC (for interactive development):** Run `gcloud auth application-default login` and set a quota project. See [ADC Setup Guide](./adc-setup.md). +2. **Service account JSON (for unattended runs):** Set `GOOGLE_APPLICATION_CREDENTIALS` (or `MAPROOM_GOOGLE_APPLICATION_CREDENTIALS`) to the path of a service account key file. ```bash -# Service account authentication +# Service account authentication — required for cron/pm2 embedding jobs export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json ``` +**ADC expires, and it fails loudly at config time.** When the refresh token has lapsed, every embedding pass dies immediately with: + +``` +Reauthentication failed. cannot prompt during non-interactive execution +``` + +This is not a network, quota, or model problem — it means nothing can prompt you for a browser login. Fix it by running `gcloud auth application-default login` in an interactive shell, or by switching the job to a service-account key. Any scheduled embedding job authenticating with user ADC **will** eventually stop with this error. + **Generate embeddings:** ```bash maproom generate-embeddings ``` +Run `maproom generate-embeddings --help` for the current flags on your binary. + ### OpenAI **Model:** `text-embedding-3-small` -**Dimensions:** 1536 +**Dimensions:** 1536 (stored in `embedding_1536`) **Endpoint:** `api.openai.com/v1/embeddings` OpenAI uses a simple API key for authentication. @@ -81,11 +105,10 @@ export OPENAI_API_KEY=YOUR_OPENAI_API_KEY # Optional: override model (default: text-embedding-3-small) export MAPROOM_EMBEDDING_MODEL=text-embedding-3-small - -# Optional: override dimensions (default: 1536) -export MAPROOM_EMBEDDING_DIMENSION=1536 ``` +**Cohere direct:** builds with a direct Cohere provider read `MAPROOM_COHERE_API_KEY`. The Cohere v3 models are the same ones offered through Bedrock, including the 512-token truncation behavior described below. Confirm your build accepts the provider value before planning around it. + **Generate embeddings:** ```bash @@ -94,16 +117,18 @@ maproom generate-embeddings ### Ollama -**Models:** `nomic-embed-text` (768 dimensions), `mxbai-embed-large` (1024 dimensions), and others -**Dimensions:** Auto-detected from model name -**Endpoint:** `localhost:11434/api/embed` (auto-detects Docker networking) +**Models:** `mxbai-embed-large` (1024 dimensions, the default) and `nomic-embed-text` (768 dimensions) +**Dimensions:** determined by the model — and it must land on 768, 1024, or 1536 to be storable +**Endpoint:** from `OLLAMA_URL` (also `MAPROOM_OLLAMA_URL`) Ollama runs embedding models locally and requires no authentication. +> **Do not pick an arbitrary embedding model.** Anything outside 768/1024/1536 cannot be stored — `qwen3-embedding:4b` at 2560 dimensions is the usual trap. Check the model's output width before you pull it. + **Step 1: Install Ollama** ```bash -# macOS / Linux +# macOS / Linux — install on the HOST if you are working in a devcontainer curl -fsSL https://ollama.com/install.sh | sh ``` @@ -112,11 +137,11 @@ See the [Ollama installation guide](https://ollama.com/download) for other platf **Step 2: Pull an embedding model** ```bash -# Recommended: nomic-embed-text (768 dimensions, good quality-to-size ratio) -ollama pull nomic-embed-text - -# Alternative: mxbai-embed-large (1024 dimensions, higher quality, more resources) +# Default: mxbai-embed-large (1024 dimensions -> embedding_1024) ollama pull mxbai-embed-large + +# Alternative: nomic-embed-text (768 dimensions -> embedding_768) +ollama pull nomic-embed-text ``` **Step 3: Set environment variables** @@ -126,87 +151,240 @@ ollama pull mxbai-embed-large export MAPROOM_EMBEDDING_PROVIDER=ollama # Set the model name -export MAPROOM_EMBEDDING_MODEL=nomic-embed-text +export MAPROOM_EMBEDDING_MODEL=mxbai-embed-large -# Optional: override dimensions (auto-detected from model name) -export MAPROOM_EMBEDDING_DIMENSION=768 +# Endpoint — IN A DEVCONTAINER THIS MUST BE host.docker.internal +export OLLAMA_URL=http://host.docker.internal:11434 +``` + +**Devcontainer endpoint trap.** Inside a container, `localhost:11434` is *not* the host's Ollama — it is the container's own empty loopback. There is no auto-detection to rely on: set `OLLAMA_URL` (or `MAPROOM_OLLAMA_URL`) to `http://host.docker.internal:11434` explicitly. If embedding fails to connect at all, check this before anything else: + +```bash +curl -s http://host.docker.internal:11434/api/tags | head ``` **Step 4: Generate embeddings** +Ollama has to be running **on the host** — start it there (`ollama serve`), not inside the container. A server started in the container answers on the container's own `localhost:11434`, which is exactly the endpoint trap above. Confirm the host's Ollama is reachable, then run the pass: + ```bash -# Ensure Ollama is running -ollama serve & +# Verify the host's Ollama answers at the configured endpoint +curl -s http://host.docker.internal:11434/api/tags | head # Generate embeddings maproom generate-embeddings ``` -**Docker networking note:** The CLI auto-detects Docker environments and adjusts the Ollama endpoint accordingly. If Ollama runs on the Docker host and the CLI runs inside a container, the CLI resolves the endpoint automatically. +**Oversized chunks:** Ollama's own context limits apply, and oversized inputs are **truncated rather than rejected**. A ~107 KB chunk embedded without error in testing — it was simply cut off. You will not get a warning about lost tail content. + +### AWS Bedrock + +**Provider value:** `MAPROOM_EMBEDDING_PROVIDER=bedrock` + +> **Availability caveat — read before planning around this.** Bedrock support is **not present in every maproom build**. Confirm against your own binary (`maproom --help`, and whether `MAPROOM_EMBEDDING_PROVIDER=bedrock` is accepted) rather than assuming. Treat it as a provider *option*, not a guaranteed feature. + +#### Models + +| Model ID | Dimensions | Max batch | Notes | +|----------|-----------|-----------|-------| +| `amazon.titan-embed-text-v2:0` | 1024 | 1 | **Default.** Stored in `embedding_1024`. | +| `amazon.titan-embed-text-v1` | 1536 | 1 | Stored in `embedding_1536`. | +| `cohere.embed-english-v3` | 1024 | 96 | Embeds at most 512 tokens; silently truncates. | +| `cohere.embed-multilingual-v3` | 1024 | 96 | Embeds at most 512 tokens; silently truncates. | + +(Dimensions and batch limits are from the provider's own documentation.) + +**Titan v2's 256 and 512 dimensional modes are unusable.** The model can emit them, but maproom has column sets only for 768/1024/1536, so a dimension validation step rejects them up front — you find out at configuration time rather than after a full scan. Use 1024. + +**Cohere v3 silently truncates at 512 tokens.** Anything longer is embedded from its first 512 tokens with no error and no warning. For code chunks that routinely exceed that, the tail of the chunk contributes nothing to the vector, and search quality degrades in a way that looks like "semantic search is just bad here" rather than like a bug. If your chunks are large, prefer Titan or a provider without the limit. + +**Titan's max batch of 1 means there is no request batching.** One HTTP request per chunk. Throughput comes entirely from concurrency (see [Batching, Parallelism, and Blast Radius](#batching-parallelism-and-blast-radius)), not from raising the pipeline batch size. Cohere v3's max batch of 96 does batch, so it can amortize request overhead across chunks; that difference has not been measured here, so treat it as a reason to test rather than a promised speedup. + +#### Configuration + +```bash +export MAPROOM_EMBEDDING_PROVIDER=bedrock + +# Region — falls back to AWS_REGION, then AWS_DEFAULT_REGION +export MAPROOM_BEDROCK_REGION=us-east-1 + +# Model (default: amazon.titan-embed-text-v2:0) +export MAPROOM_EMBEDDING_MODEL=amazon.titan-embed-text-v2:0 + +# Optional: custom endpoint for VPC endpoints or local testing. +# Also honored: AWS_ENDPOINT_URL_BEDROCK_RUNTIME, AWS_ENDPOINT_URL +export MAPROOM_BEDROCK_ENDPOINT_URL=https://vpce-....bedrock-runtime.us-east-1.vpce.amazonaws.com + +# Optional: use FIPS endpoints +export MAPROOM_BEDROCK_USE_FIPS=true + +# Optional: select a named profile (also AWS_PROFILE) +export MAPROOM_AWS_PROFILE=my-profile +``` + +#### Credentials + +Credentials resolve through a standard AWS chain, so an environment that already runs AWS CLI or SDK workloads generally needs no new configuration: + +1. **Static environment keys** — `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`. +2. **Shared config and credentials files** — `AWS_CONFIG_FILE`, `AWS_SHARED_CREDENTIALS_FILE`, honoring `AWS_PROFILE`. +3. **Web identity / IRSA** — `AWS_WEB_IDENTITY_TOKEN_FILE` together with `AWS_ROLE_ARN` and `AWS_ROLE_SESSION_NAME`. This is the Kubernetes/EKS path. +4. **Container credentials** — `AWS_CONTAINER_CREDENTIALS_FULL_URI` or `AWS_CONTAINER_CREDENTIALS_RELATIVE_URI`, with `AWS_CONTAINER_AUTHORIZATION_TOKEN` or `AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE`. This is the ECS/Fargate path. + +If a run fails to authenticate, work down that list and confirm which step you *intended* to supply credentials — a stale `AWS_PROFILE` shadowing IRSA is the usual surprise. + +#### Request handling + +- **30 s** per-request timeout. +- **90 s** batch timeout. +- Up to **4 retries** with exponential backoff from a **500 ms** base. + +Throttling from Bedrock is therefore absorbed silently up to a point; if a pass is much slower than expected without failing, you are probably being throttled and retried rather than stalled. + +**Generate embeddings:** + +```bash +maproom generate-embeddings +``` + +## Batching, Parallelism, and Blast Radius + +Providers are called in **sub-batches** inside each pipeline batch. Three environment variables control this: + +| Variable | Default | Effect | +|----------|---------|--------| +| `MAPROOM_EMBEDDING_PARALLEL_ENABLED` | — | Turns sub-batch parallelism on or off. | +| `MAPROOM_EMBEDDING_PARALLEL_SUB_BATCH_SIZE` | 50 | Texts per provider request group. | +| `MAPROOM_EMBEDDING_PARALLEL_MAX_CONCURRENCY` | 8 | In-flight sub-batches. | + +**The rule that catches people: the pipeline batch size must EXCEED the sub-batch size, or parallelism never engages.** With the default sub-batch size of 50, a batch size of 50 or below runs fully serialized — you get no concurrency and no explanation. If throughput looks like one request at a time, check this ratio first. + +Measured against local Ollama, throughput improved up to **~8 concurrent requests** and got **worse at 16**. Eight is the sweet spot; raising `MAPROOM_EMBEDDING_PARALLEL_MAX_CONCURRENCY` past it is not free. + +**One bad sub-batch fails the whole batch.** If a single sub-batch errors, maproom marks the **entire pipeline batch** failed — every chunk in it, not just the 50 that failed. Observed with Ollama when the model runner drops the connection: + +``` +Sub-batch 2 failed: API error: Bad request: Batch of 50 texts rejected: {"error":"Post \"http://127.0.0.1:51950/tokenize\": EOF"} +``` + +These failures are **transient and self-heal on the next pass**, so a loop converges — but batch size sets the blast radius. With `--batch-size 1000` each incident cost 1,000 chunks; at 500 it cost 500. Pick a batch size that is comfortably above the sub-batch size (so parallelism engages) but not so large that one flaky sub-batch throws away a huge pass. + +Separately, keep each pass **bounded**: the pending-chunk query runs under a 5 s statement timeout that unbounded passes eventually exceed at scale. Bound the pass and loop until the pending count reaches zero. Run `maproom generate-embeddings --help` for the current flags on your binary. + +## Switching Providers or Models -## Re-Indexing When Switching Providers +Switching embedding providers — **or just switching models within one provider** — requires regenerating embeddings, because: -Switching embedding providers requires regenerating all embeddings because: +1. **Different dimensions:** Google produces 768-dimensional vectors, OpenAI 1536, Ollama `mxbai-embed-large` 1024, Bedrock Titan v1 1536 and Titan v2 / Cohere v3 1024. Vectors of different widths live in different columns and cannot be compared. +2. **Different vector spaces:** even at matching dimensions, each model maps concepts to different coordinates. Cosine similarity between vectors from different models is meaningless. -1. **Different dimensions:** Google produces 768-dimensional vectors, OpenAI produces 1536-dimensional vectors, and Ollama dimensions vary by model. These vectors cannot be compared. -2. **Different vector spaces:** Even if dimensions matched, each provider's model maps concepts to different coordinates. Cosine similarity scores between vectors from different providers are meaningless. +### You MUST delete the old rows first -**To switch providers:** +`code_embeddings` holds **one row per `blob_sha`, with a UNIQUE constraint on `blob_sha`**. A blob that already has a row at the old dimension **keeps it**: incremental runs see a row exists and skip the blob, while vector search at the new dimension cannot see it. The blob is stranded — embedded, but invisible. + +Nothing errors. Coverage numbers still look plausible. Search still returns results (non-vector ranking is unaffected). You simply get worse semantic results forever. + +**To switch:** ```bash -# 1. Set the new provider environment variables (see provider sections above) +# 1. Check what you currently have. Healthy = exactly one row of output. +psql "$MAPROOM_DATABASE_URL" -c \ + "select embedding_dim, model_version, count(*) from code_embeddings group by 1,2;" + +# 2. Set the new provider environment variables (see provider sections above) export MAPROOM_EMBEDDING_PROVIDER=openai export OPENAI_API_KEY=YOUR_OPENAI_API_KEY -# 2. Regenerate all embeddings with the new provider +# 3. DELETE the rows at the old dimension / old model. This step is mandatory. +psql "$MAPROOM_DATABASE_URL" -c \ + "delete from code_embeddings where embedding_dim = 1024;" +# (or, when staying at the same width but changing model:) +psql "$MAPROOM_DATABASE_URL" -c \ + "delete from code_embeddings where model_version = 'OLD_MODEL_VERSION';" + +# 4. Regenerate, in bounded passes, until the pending count reaches zero maproom generate-embeddings -# 3. Verify vector search works with the new provider +# 5. Verify homogeneity again — one row, the new dimension and model +psql "$MAPROOM_DATABASE_URL" -c \ + "select embedding_dim, model_version, count(*) from code_embeddings group by 1,2;" + +# 6. Verify vector search works with the new provider maproom vector-search --repo YOUR_REPO --query "test query" --format agent ``` -**Warning:** Until embeddings are regenerated, vector search results will be unreliable or empty. Full-text search (`maproom search`) is unaffected by provider changes since it does not use embeddings. +The verification query in steps 1 and 5 is the whole game: + +```sql +select embedding_dim, model_version, count(*) from code_embeddings group by 1,2; +``` + +**More than one row means your index is mixed** — some blobs are still at the old dimension or old model, and vector search silently ignores them. Delete and regenerate until this query returns a single row. + +Two notes on step 6: `--repo` matches the *indexed* repo name (derived from the git origin), not the directory on disk, so a zero-hit result may be a name mismatch rather than a bad switch — list indexed names with `maproom status`. And embedding coverage decays on its own as watchers process changes, so a fresh gap after a switch is not necessarily a switch problem; re-run `generate-embeddings` on a schedule regardless. + +**Warning:** until embeddings are regenerated, vector search results will be unreliable or empty. Non-vector search (`maproom search`) is unaffected by provider changes since it does not use embeddings. ## Environment Variable Summary +`maproom --help` lists the subcommands and environment variables your build supports; treat that as authoritative over this table. + | Variable | Description | Example | |----------|-------------|---------| -| `MAPROOM_EMBEDDING_PROVIDER` | Provider selection | `ollama`, `openai`, `google` | -| `MAPROOM_EMBEDDING_MODEL` | Model override | `text-embedding-004`, `text-embedding-3-small`, `nomic-embed-text` | -| `MAPROOM_EMBEDDING_DIMENSION` | Dimension override | `768`, `1536`, `1024` | -| `OPENAI_API_KEY` | OpenAI API key | `sk-...` (placeholder) | -| `MAPROOM_OPENAI_API_KEY` | OpenAI API key (prefixed) | `sk-...` (placeholder) | -| `GOOGLE_APPLICATION_CREDENTIALS` | Path to service account JSON | `/path/to/key.json` | -| `GOOGLE_PROJECT_ID` | Google Cloud project ID | `YOUR_PROJECT_ID` | -| `MAPROOM_GOOGLE_PROJECT_ID` | Google Cloud project ID (prefixed) | `YOUR_PROJECT_ID` | +| `MAPROOM_EMBEDDING_PROVIDER` | Provider selection | `ollama`, `openai`, `google`, `bedrock` | +| `MAPROOM_EMBEDDING_MODEL` | Model override | `mxbai-embed-large`, `text-embedding-004`, `amazon.titan-embed-text-v2:0` | +| `MAPROOM_EMBEDDING_DIMENSION` | Declared dimension — must be 768, 1024, or 1536 **and** match what the model emits | `768`, `1024`, `1536` | +| `MAPROOM_EMBEDDING_PARALLEL_ENABLED` | Enable sub-batch parallelism | `true` | +| `MAPROOM_EMBEDDING_PARALLEL_SUB_BATCH_SIZE` | Texts per sub-batch (default 50) | `50` | +| `MAPROOM_EMBEDDING_PARALLEL_MAX_CONCURRENCY` | In-flight sub-batches (default 8) | `8` | +| `OLLAMA_URL` / `MAPROOM_OLLAMA_URL` | Ollama endpoint — `host.docker.internal` in a devcontainer | `http://host.docker.internal:11434` | +| `OPENAI_API_KEY` / `MAPROOM_OPENAI_API_KEY` | OpenAI API key | `sk-...` (placeholder) | +| `MAPROOM_COHERE_API_KEY` | Cohere API key (direct, not via Bedrock) | placeholder | +| `GOOGLE_PROJECT_ID` / `MAPROOM_GOOGLE_PROJECT_ID` | Google Cloud project ID | `YOUR_PROJECT_ID` | +| `GOOGLE_APPLICATION_CREDENTIALS` / `MAPROOM_GOOGLE_APPLICATION_CREDENTIALS` | Path to service account JSON | `/path/to/key.json` | +| `MAPROOM_BEDROCK_REGION` | Bedrock region (falls back to `AWS_REGION`, `AWS_DEFAULT_REGION`) | `us-east-1` | +| `MAPROOM_BEDROCK_ENDPOINT_URL` | Custom Bedrock endpoint (also `AWS_ENDPOINT_URL_BEDROCK_RUNTIME`, `AWS_ENDPOINT_URL`) | VPC endpoint URL | +| `MAPROOM_BEDROCK_USE_FIPS` | Use FIPS endpoints | `true` | +| `MAPROOM_AWS_PROFILE` / `AWS_PROFILE` | AWS profile selection | `my-profile` | + +Note that `GEMINI_API_KEY` and `GOOGLE_API_KEY` are **not** in this table on purpose — they do nothing for the Google provider. ## FAQ ### Can I switch providers after indexing? -No, not without re-indexing. Embeddings from different providers are incompatible because they use different dimensions and map to different vector spaces. You must regenerate all embeddings after switching providers. See [Re-Indexing When Switching Providers](#re-indexing-when-switching-providers). +Not without regenerating — and not by regenerating alone. You must **delete the existing `code_embeddings` rows at the old dimension/model first**, because the UNIQUE constraint on `blob_sha` means an existing row causes the blob to be skipped rather than re-embedded. See [Switching Providers or Models](#switching-providers-or-models). + +### Can I use any Ollama model I like? + +No. The model must emit 768, 1024, or 1536 dimensions — those are the only stored columns. `mxbai-embed-large` (1024) and `nomic-embed-text` (768) work. Models like `qwen3-embedding:4b` (2560) cannot be stored at all. ### Can I use multiple providers simultaneously? -Not for the same repository. Each repository's embeddings must be generated with a single provider. However, you can use different providers for different repositories by setting the appropriate environment variables before running `generate-embeddings` for each repository. +Not within one searchable set of embeddings. Every blob must be embedded by the same model at the same dimension; the verification query in [Switching Providers or Models](#switching-providers-or-models) should return exactly one row. Mixing providers strands whichever half does not match the current search dimension. -### Does the provider affect full-text search (FTS)? +### Does the provider affect non-vector search? -No. Full-text search uses SQLite FTS5 indexes and does not involve embeddings at all. Provider configuration only affects `vector-search` and `generate-embeddings` commands. +No. Full-text and structural ranking do not involve embeddings at all — that is why an index with zero embeddings still returns search results. Provider configuration only affects `vector-search` and `generate-embeddings`. Do not infer embedding health from search working; check coverage explicitly. ### Which provider should I choose if I am unsure? -- If you have no API keys and want to get started quickly, use **Ollama** (free, local, no setup beyond installation). -- If you already have an OpenAI API key, use **OpenAI** (simplest cloud setup). -- If your team uses Google Cloud, use **Google Vertex AI** (best integration with GCP tooling). +- No API keys, want to start now: **Ollama** (free, local — just remember the `OLLAMA_URL` devcontainer trap). +- Already have an OpenAI key: **OpenAI** (simplest cloud setup). +- Team on Google Cloud: **Google Vertex AI** (but use a service account for any scheduled job). +- Team on AWS, egress must stay in a VPC: **Bedrock**, if your build has it. ### Do higher dimensions mean better search quality? -Not necessarily. Dimension count reflects the model's internal representation, not search quality directly. OpenAI's 1536-dimensional embeddings and Google's 768-dimensional embeddings both produce high-quality semantic search results. Higher dimensions do increase storage requirements and may slightly increase search latency. +Not necessarily. Dimension count reflects the model's internal representation, not search quality directly. OpenAI's 1536-dimensional embeddings and Google's 768-dimensional embeddings both produce high-quality semantic search results. Higher dimensions do increase storage requirements and may slightly increase search latency. Dimension is mostly a *storage compatibility* question here, not a quality dial. ### What happens if my API key or credentials expire during embedding generation? -The `generate-embeddings` command will fail mid-process. Refresh your credentials (see [ADC Setup Guide](./adc-setup.md) for Google, or update your API key for OpenAI), then re-run `generate-embeddings`. The CLI will resume or regenerate as needed. +The run fails. For Google, an expired ADC session fails at config time with `Reauthentication failed. cannot prompt during non-interactive execution` — refresh with `gcloud auth application-default login`, or move the job to a service-account key (see [ADC Setup Guide](./adc-setup.md)). For OpenAI/Cohere, update the key. For Bedrock, check which step of the credential chain was supposed to supply credentials. Then re-run `generate-embeddings`; already-embedded blobs are skipped, so a re-run picks up where it left off. + +### Why did one bad chunk kill 1,000 chunks? + +Because a failed sub-batch fails the entire pipeline batch. See [Batching, Parallelism, and Blast Radius](#batching-parallelism-and-blast-radius) — the failures are transient, so looping converges, but a smaller batch size limits the damage per incident. ## Related Documentation @@ -216,7 +394,8 @@ The `generate-embeddings` command will fail mid-process. Refresh your credential - [Google Vertex AI Pricing](https://cloud.google.com/vertex-ai/pricing) - Official Google Cloud pricing - [OpenAI API Pricing](https://openai.com/api/pricing/) - Official OpenAI pricing - [Ollama](https://ollama.com/) - Official Ollama website and documentation +- [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/) - Official Bedrock documentation --- -*Last Updated: Feb 2026* +*Last Updated: Sep 2026 — provider behavior verified against maproom CLI 0.3.0 with a PostgreSQL 16 + pgvector backend. Confirm provider availability against your own binary.* diff --git a/plugins/maproom/skills/maproom-search/references/multi-repo-guide.md b/plugins/maproom/skills/maproom-search/references/multi-repo-guide.md index bba86289..44a5da7d 100644 --- a/plugins/maproom/skills/maproom-search/references/multi-repo-guide.md +++ b/plugins/maproom/skills/maproom-search/references/multi-repo-guide.md @@ -254,6 +254,20 @@ These patterns combine searches across code and docs repos to answer questions t ## Troubleshooting Searches +### Zero Results -- Check the Repo Name First + +Before relaxing any filter, confirm that `--repo` names something that is actually indexed. `--repo` matches the **indexed repo name**, which is derived from the git origin -- not the directory name on disk -- and it matches on suffix. A directory called `django-olympics` whose origin indexes it as `django/django` is found by `--repo django`. `--repo django-olympics` returns **zero hits with no error**: maproom does not report that the repo filter matched nothing. + +```bash +# Authoritative list of indexed repo names +maproom status + +# Suffix match works; the on-disk directory name usually does not +maproom search --repo django --query "QuerySet" --format agent +``` + +Rule of thumb: if *every* query against one repo comes back empty, suspect a name mismatch. If only *some* queries come back empty, the name is fine -- move on to filter relaxation below. + ### Zero Results -- Progressive Filter Relaxation When a filtered search returns zero results, progressively relax filters to find relevant chunks. Remove one filter at a time to identify which constraint is too narrow. @@ -307,6 +321,22 @@ export MAPROOM_REPOS_ROOT=~/git export MAPROOM_SPECS_ROOT=~/_SPECS ``` +**Choose the storage backend before you scan anything.** maproom reads `MAPROOM_DATABASE_URL` at startup and picks its backend from the scheme: `sqlite://` (or a bare filesystem path) selects SQLite, the default at `~/.maproom/maproom.db`; `postgres://` or `postgresql://` selects PostgreSQL, which requires a binary built with `--features postgres`. A `--database-url` flag overrides the environment variable for a single invocation. Provider and feature availability differ between builds -- run `maproom --help` to confirm which subcommands and environment variables your binary actually supports before committing to a backend. + +**A multi-repo fleet wants ONE shared Postgres.** Cross-repo search only works across repos that live in the same database, so every repo in the fleet points at a single Postgres instance. Per-repo SQLite files cannot be searched together. + +```bash +# Every repo in the fleet shares this one database +export MAPROOM_DATABASE_URL=postgres://maproom:maproom@host.docker.internal:5433/maproom +``` + +**Devcontainer trap: `host.docker.internal`, never `localhost`.** The Postgres container runs on the *host* docker daemon, so from inside a devcontainer it is reachable only at `host.docker.internal`. Inside the container, `localhost:5433` is frequently a *different* Postgres -- for example a tmpfs-backed instance used for cargo tests -- whose data vanishes when the container stops. Pointing maproom at `localhost` raises no error: scans succeed, `maproom status` looks healthy, and the index is simply gone later. Verify the endpoint before the first scan: + +```bash +pg_isready -h host.docker.internal -p 5433 +psql postgres://maproom:maproom@host.docker.internal:5433/maproom -c 'select count(*) from chunks;' +``` + ### Step 2: Create the Configuration File Copy the YAML template to your workspace root: @@ -341,31 +371,73 @@ maproom scan --path /workspace/repos/claude-code-plugins --repo manifoldlogic/cl maproom scan --path /workspace/_SPECS --repo all-specs ``` -Each `--repo` name should match the key used in your `maproom-repos.yaml` configuration file. +Keep each `--repo` label aligned with the key used in your `maproom-repos.yaml` configuration file -- but the name that lands *in the index* is derived from the git origin, so it may not be the label you typed. + +**The name in the index is the name you search under.** At search time `--repo` matches the *indexed* repo name -- derived from the git origin, not from the directory name on disk -- with suffix fuzzy-matching. Whatever `maproom status` prints is authoritative; see "Zero Results -- Check the Repo Name First" above. + +**Do not wrap a scan in a timeout you have not measured.** A large repo on a slow mount (FUSE, virtiofs, a network share) can exceed a wrapping timeout mid-scan. That repo is left partially indexed while every other repo in the batch reports success, so the aggregate run looks clean. See "A clean-looking scan run can still leave one repo stale" below. ### Step 4: Verify Indexing -Confirm all repos are indexed and embeddings are available: +Confirm all repos are indexed, then check embedding coverage explicitly -- it is a separate question from whether chunks exist: ```bash maproom status ``` -Expected output shows each repository with its worktree and chunk count: +Real output carries more than chunk counts. Each worktree reports coverage and a last-scan timestamp: ``` Repository: crewchief Worktree: main Chunks: 24,333 + Embeddings: 24,333 (100.0%) + Last scan: 2026-04-11 09:22:41 UTC Repository: crewchief-specs Worktree: main Chunks: 1,200 + Embeddings: 0 (0.0%) + Last scan: 2026-08-30 17:04:08 UTC +``` + +**"Last scan" is not a freshness signal.** An incremental scan that finds the git tree SHA unchanged short-circuits: it logs + +``` +No changes detected (tree SHA match), skipping scan +``` + +and leaves the previous timestamp untouched. A months-old "Last scan" therefore usually means *nothing has changed since then*, not *the index is stale*. Treating that field as staleness leads to pointless full re-scans of repos that were already current. + +To actually check freshness, ask the index about something you know is recent, or compare HEAD against what the index recorded: + +```bash +# Does the index know about a symbol you added this week? +maproom search --repo crewchief --query "name_of_a_recently_added_symbol" --format agent + +# Or compare the working tree against what the index recorded for that worktree +git -C /workspace/repos/crewchief/crewchief rev-parse HEAD +psql "$MAPROOM_DATABASE_URL" -c 'select * from worktrees;' +``` + +**Zero embeddings never errors.** With no embeddings at all, `maproom search` still returns results by falling back to full-text and structural ranking; only `maproom vector-search` and semantic ranking are unavailable. So "search works" proves nothing about coverage. Check it directly: `maproom status` shows `Embeddings: 0 (0.0%)` per repo, and on Postgres the `encoding_runs` table stays empty when nothing has ever been embedded. + +Per-repo coverage, straight from the database: + +```sql +select r.name, count(distinct c.blob_sha) blobs, count(distinct e.blob_sha) embedded +from repos r join worktrees w on w.repo_id=r.id +join chunk_worktrees cw on cw.worktree_id=w.id +join chunks c on c.id=cw.chunk_id +left join code_embeddings e on e.blob_sha=c.blob_sha group by 1 order by 2 desc; ``` -If embeddings are missing (needed for vector-search), generate them: +If coverage is short, generate embeddings -- but treat this as the *first* run of a recurring job, not a one-time fix. Bound every pass and loop; see "Keeping the Index Fresh" below for why an unbounded run dies exactly when it is closest to finishing, and for the scheduled job that keeps coverage from decaying. ```bash +# Bound every pass. Run `maproom generate-embeddings --help` for the flag your build +# uses to cap chunks per pass (and for provider selection); ~10,000 is a measured-safe +# value against the 5s statement_timeout. An unbounded run dies late -- see below. maproom generate-embeddings --repo crewchief maproom generate-embeddings --repo crewchief-specs ``` @@ -388,3 +460,93 @@ maproom vector-search --repo crewchief --query "error handling" --k 3 --format a ### No Config File Fallback If `maproom-repos.yaml` is not present in the workspace, use `maproom status` to discover which repos are already indexed. The status output lists all repositories, worktrees, and chunk counts, providing enough information to construct search commands manually. + +## Keeping the Index Fresh + +### Watchers keep chunks fresh -- they do NOT keep embeddings fresh + +This is the fleet's quietest failure. The incremental processor **deletes** the embeddings belonging to changed chunks and never regenerates them. Nothing in the watch path calls the embedding provider. So as people work, embedding coverage decays continuously and silently -- silently because `maproom search` keeps returning results the whole time (it falls back to full-text and structural ranking), while `maproom vector-search` quietly loses reach over exactly the code that is being worked on most. + +Any documentation claiming "watch keeps the index fresh" is true for chunks and **false for embeddings**. + +A periodic `maproom generate-embeddings` job is therefore **required**, not optional. Schedule it however the fleet is already managed. With cron: + +```cron +# Watchers handle chunks. Nothing handles embeddings unless you schedule it. +17 * * * * /usr/local/bin/maproom-embed-topup >> /home/vscode/.maproom/embed-topup.log 2>&1 +``` + +Or, alongside the per-repo pm2 watchers, as a scheduled one-shot: + +```js +// pm2 ecosystem entry, next to the maproom:: watchers +{ name: "maproom:embed-topup", script: "/usr/local/bin/maproom-embed-topup", + autorestart: false, cron_restart: "17 * * * *" } +``` + +### Bound every embedding pass, then loop until pending hits zero + +An unbounded `generate-embeddings` run dies at scale, and it gets *worse the more it succeeds*. Its pending-chunk query is a `NOT IN` subquery whose cost grows with the size of `code_embeddings`, and maproom pins `statement_timeout` to 5000ms on every connection it opens (set in `after_connect`; there is no environment override). Measured on a ~195k-chunk database: with ~113k blobs already embedded, asking for 40,000 pending chunks blew past 5s and every pass died with + +``` +Failed to fetch chunks ... canceling statement due to statement timeout +``` + +Bounding the same pass to 10,000 chunks measured ~0.73s. The failure appears **late** -- the job stalls exactly when it is closest to finishing, which is why it reads as "embeddings are broken" rather than "that pass was too big". + +So: cap every pass, and loop until nothing is pending. Looping also absorbs transient provider failures -- one bad sub-batch marks the whole pipeline batch failed, but those chunks are simply picked up again on the next pass. + +```sh +#!/bin/sh +# maproom-embed-topup -- hold embedding coverage across the fleet. +set -u +DB="postgres://maproom:maproom@host.docker.internal:5433/maproom" + +# Cap on chunks per pass. Run `maproom generate-embeddings --help` to see what +# your build calls this flag; the number matters more than the spelling, and +# ~10,000 is a measured-safe working value against the 5s statement_timeout. +BOUND="-- 10000" + +pending() { + psql "$DB" -tAc "select count(distinct c.blob_sha) from chunks c + left join code_embeddings e on e.blob_sha = c.blob_sha + where e.blob_sha is null;" +} + +# pending() is fleet-wide, so sweep the whole fleet once per pass and re-check. +pass=0 +while [ "$(pending)" -gt 0 ] && [ "$pass" -lt 50 ]; do + for repo in crewchief crewchief-specs manifoldlogic/claude-code-plugins; do + maproom generate-embeddings --repo "$repo" $BOUND || true + done + pass=$((pass + 1)) +done +``` + +One caveat if you go query-hunting yourself: maproom sets `work_mem` (256MB) on its own connections, and that same `NOT IN` query plans at cost ~101,635 (hashed SubPlan) at 64MB+ versus ~218,119,741 (Materialize plus a per-row Seq Scan) under a small `work_mem`. Running it by hand in `psql` with default settings reproduces a problem maproom does not have. + +### The pm2 watcher fleet vanished + +**Symptom:** `pm2 list` prints an empty table. The pm2 God daemon is still alive, nothing errored, and no repo is being scanned any more -- so the index silently stops tracking new commits across the whole fleet. + +**Fix:** do not hand-rebuild N watchers. + +```bash +pm2 resurrect # restores every watcher from ~/.pm2/dump.pm2 +pm2 list # expect one maproom:: watcher per repo +``` + +Keep `~/.pm2/dump.pm2` current (`pm2 save`) after intentionally adding or removing watchers, since that dump is what `resurrect` replays. Note that resurrecting watchers restores *chunk* freshness only -- embeddings still depend on the scheduled top-up job above. + +### A clean-looking scan run can still leave one repo stale + +**Symptom:** a batch scan reports success for every repo, but one repo's results are missing code you know landed. Nothing in the run output points at it. + +**Cause:** a wrapping timeout killed that repo's scan partway through. The repo is left partially refreshed; the other repos succeeded, so the aggregate run looks clean. Slow mounts (FUSE, virtiofs, network shares) make this routine on large repos. Measured example: a 1,328-file docs repo died at 30% under an 1800s cap and needed ~2,326s to finish unbounded. + +**Fix:** re-scan the suspect repo on its own with no cap (or a cap you have measured against that repo on that mount), then confirm with a recent-symbol search -- not with "Last scan", which the tree-SHA short-circuit leaves stale on purpose. + +```bash +maproom scan --path /workspace/_SPECS/crewchief --repo crewchief-specs +maproom search --repo crewchief-specs --query "a_heading_added_recently" --format agent +``` diff --git a/plugins/maproom/skills/maproom-search/references/troubleshooting.md b/plugins/maproom/skills/maproom-search/references/troubleshooting.md index 957e76ff..a8ed5897 100644 --- a/plugins/maproom/skills/maproom-search/references/troubleshooting.md +++ b/plugins/maproom/skills/maproom-search/references/troubleshooting.md @@ -2,10 +2,89 @@ Detailed error recovery for common maproom issues, including edge case handling for boundary conditions and concurrency scenarios. This is a companion to the quick-reference troubleshooting section in [SKILL.md](../SKILL.md) — start there for quick fixes, and use this file when you need root cause analysis or step-by-step recovery. +--- + +## First: Which Storage Backend Are You On? + +Answer this before diagnosing anything else. maproom ships with **two storage backends**, and the +failure modes, the error strings and the fixes are different for each. Most of the older material +in this document was written against a single-repo SQLite database and is labelled accordingly. + +**The backend is chosen at runtime from `MAPROOM_DATABASE_URL`:** + +| Value of `MAPROOM_DATABASE_URL` | Backend | +|---|---| +| unset, a plain filesystem path, or `sqlite://...` | SQLite — the default, `~/.maproom/maproom.db` | +| `postgres://...` or `postgresql://...` | PostgreSQL — requires a build compiled with `--features postgres` | + +A `--database-url` flag overrides the environment variable. Postgres support is a **build feature, +not a guarantee**: run `maproom --help` to see the subcommands and environment variables your build +actually supports before assuming it is available. + +```bash +# What am I actually pointed at? +echo "$MAPROOM_DATABASE_URL" +maproom status +``` + +**Why it matters:** a SQLite database is a file you own, so permissions and local disk space are +real failure modes. A Postgres database is a server you reach over TCP, so *its* failure modes are +"container not running", "wrong host", "statement timeout" and "connection pool exhausted". +Applying a SQLite fix (`chmod`, deleting `maproom.db-wal`) to a Postgres problem accomplishes +nothing at all. + +### Multi-repo fleets share one Postgres database + +Cross-repo search only works when every repo is indexed into the **same** database, so a fleet +points every scanner, watcher and search at one shared Postgres instance: + +```bash +export MAPROOM_DATABASE_URL="postgres://maproom:maproom@host.docker.internal:5433/maproom" +``` + +### Devcontainer trap: `localhost:5433` is not the fleet database + +**This one destroys data silently and produces no error at all.** In a devcontainer the shared +Postgres container normally runs on the **host** docker daemon and must be reached at +`host.docker.internal`. Inside the container, `localhost:5433` may be a *different*, throwaway +Postgres — for example a tmpfs-backed instance used only by `cargo test` — whose data vanishes when +the container stops. + +Pointing maproom at `localhost` indexes happily into that disposable database. Nothing errors. You +find out later, when searches come back empty and `maproom status` no longer lists repos you know +you scanned. + +```bash +# Confirm the fleet database is reachable at the host address +pg_isready -h host.docker.internal -p 5433 +``` + +### Postgres schema facts worth knowing before you debug + +- `code_embeddings` holds **one row per `blob_sha`**, with a `UNIQUE` constraint on `blob_sha`. A + blob that already has a row is skipped by incremental runs — this is the root of the model + switching corruption described in + [Switching Embedding Models Requires Deleting the Old Rows](#switching-embedding-models-requires-deleting-the-old-rows). +- Vectors live in **fixed per-dimension columns** — `embedding_768`, `embedding_1024` and + `embedding_1536`, each with its own HNSW index. A model emitting any other dimension has nowhere + to be stored. +- `code_embeddings` has **no foreign key to `chunks`**, so deleting chunks leaves orphan embedding + rows behind. They are harmless, but they accumulate. + +--- + ## Debugging Workflow When a search command fails or produces unexpected results, follow this systematic workflow: +**0. Identify the storage backend** — every branch below depends on it +```bash +echo "$MAPROOM_DATABASE_URL" +# empty, or a path -> SQLite (~/.maproom/maproom.db); permissions and disk are in play +# postgres://... -> shared Postgres; connectivity and timeouts are in play, not permissions +``` +See [First: Which Storage Backend Are You On?](#first-which-storage-backend-are-you-on). + **1. Verify CLI installed** ```bash command -v maproom @@ -17,6 +96,11 @@ command -v maproom maproom --version # Expected: >= 0.1.0 (minimum version for this documentation) ``` +Behavior differs between builds. The edge-case tables at the end of this document were measured on +CLI v0.1.0 against SQLite; the Postgres and embedding-pipeline material was measured on CLI 0.3.0 +against Postgres 16 with pgvector. Provider and feature availability is **not** uniform across +builds — confirm anything load-bearing against your own binary with `maproom --help` rather than +trusting a list in any document. **3. Verify database status** ```bash @@ -39,10 +123,195 @@ maproom search --repo --query "test" --format agent --debug If all steps pass but your specific search still fails, check: - Query syntax (special characters may need quoting) - Filter values (case-sensitive: `func` not `Func`, `py` not `PY`) -- Repository name (must match indexed name exactly) +- Repository name — `--repo` matches the **indexed** name (derived from the git origin) with suffix + fuzzy-matching, not the directory name on disk; see + [Silent Zero Hits from a Repo-Name Mismatch](#silent-zero-hits-from-a-repo-name-mismatch) --- +### Connection Pool Timeout or "Connection refused" (Postgres backend) + +**Symptom:** maproom fails with a pool timeout. From maproom: + +``` +pool timed out while waiting for an open connection +``` + +and `psql` or `pg_isready` against the same URL reports: + +``` +Connection refused +``` + +**Root Cause:** On a Postgres setup this almost always means the **Postgres container is not +running**, or you are pointed at the wrong host. It is **not** a file-permissions problem. The +`chmod` recovery under [Permission Denied on Database (GAP-005)](#permission-denied-on-database-gap-005) +applies to the **SQLite backend only** — on Postgres there are no database files to `chmod` and +running those commands changes nothing. + +The same pool timeout also appears **transiently** when something else is saturating the database's +CPU: a runaway `ANALYZE`, a heavy hand-written query, a large scan. Rule that out before concluding +the server is down. + +**Fix:** +1. Is the server reachable at all? + ```bash + pg_isready -h host.docker.internal -p 5433 + ``` +2. If the connection is refused, start the Postgres container **on the host docker daemon** (not + inside the devcontainer), then retry. +3. If `pg_isready` succeeds but maproom still times out, the pool is being starved. Look for + long-running statements before restarting anything: + ```sql + select pid, now() - query_start as runtime, state, left(query, 120) + from pg_stat_activity + where state <> 'idle' + order by runtime desc; + ``` + Wait for or cancel the offending statement, then retry. +4. Confirm you are pointed at the fleet database and not a throwaway one — see + [Devcontainer trap: `localhost:5433` is not the fleet database](#devcontainer-trap-localhost5433-is-not-the-fleet-database). + +**Prevention:** Keep the shared Postgres container on a restart policy such as `unless-stopped`, and +serialize heavy maintenance (full re-scans, `ANALYZE`, bulk embedding passes) so it does not overlap +with interactive searching. + +### The Entire Watcher Fleet Disappeared + +**Symptom:** `pm2 list` prints an empty table. Every per-repo maproom watcher is gone at once, yet +the pm2 God daemon is still alive and answering. + +**Root Cause:** pm2 lost its in-memory process list. This is a pm2-level event, not a maproom one — +the watchers did not fail individually, the whole fleet vanished together. + +**Fix:** +```bash +pm2 resurrect # restores every watcher from ~/.pm2/dump.pm2 +pm2 list # confirm the fleet is back +``` + +**Prevention:** Run `pm2 save` after adding or changing watchers so `~/.pm2/dump.pm2` is current — +`pm2 resurrect` can only restore what was saved. Without knowing about `resurrect`, the obvious but +wrong recovery is to hand-rebuild N watchers one at a time, losing their original configuration. + +### Scan Run Reports Success but One Repo Is Only Half Indexed + +**Symptom:** A fleet-wide scan finishes and the aggregate run looks clean — every repo reports +success. Later, searches against one particular repo miss files you know exist. + +**Root Cause:** Scanning a large repo on a slow mount (FUSE, virtiofs, a network filesystem) can +exceed a wrapping timeout **mid-scan**. That repo is left partially refreshed while every other repo +in the run succeeds, so the run-level summary hides the failure. Measured example: a 1,328-file docs +repo died at roughly 30% under a 1800s cap, and needed about 2,326s to finish when run unbounded. + +**Fix:** +1. Re-scan the affected repo on its own, with no timeout wrapper (or a far larger one), and let it + run to completion. +2. Confirm coverage afterwards by searching for a file or symbol you know sits late in the traversal. + +**Prevention:** Budget scan timeouts against the *slowest* repo on the *slowest* mount, not the +average one. Treat any per-repo scan that ends at almost exactly your timeout value as a failure, +even when the surrounding job reports success. + +### Silent Zero Hits from a Repo-Name Mismatch + +**Symptom:** A search that should obviously match returns **zero results, with no error and exit +code 0**. + +**Root Cause:** `--repo` matches the **indexed repo name** — derived from the git origin, not the +directory name on disk — with suffix fuzzy-matching. A directory named `django-olympics` whose +origin is `django/django` is indexed as `django/django`: `--repo django` finds it, while +`--repo django-olympics` returns zero hits and says nothing at all. + +**Fix:** +```bash +# List the indexed names, then search using one of those +maproom status +maproom search --repo django --query "middleware" --format agent +``` + +**Prevention:** When a search comes back silently empty, suspect the repo name before you suspect +the index. A name that fuzzy-matches nothing can return an empty result set rather than +`Error: Repository not found`, so empty results are never proof that a repo is indexed and healthy. +See also [Zero Results with Valid Query](#zero-results-with-valid-query) for case-sensitive filter +values, which fail the same silent way. + +### generate-embeddings Stalls Near the Finish Line + +**Symptom:** Embedding generation ran fine for hours, then every pass began failing — and the closer +coverage gets to complete, the more reliably it dies: + +``` +Failed to fetch chunks ... canceling statement due to statement timeout +``` + +**Root Cause:** The pending-chunk query is a `NOT IN` subquery whose cost grows with the size of +`code_embeddings`, and maproom pins `statement_timeout` to **5000ms** on every connection (set in +`after_connect`; there is no environment override). The job therefore gets **slower as it succeeds**, +until the pending-chunk fetch alone exceeds 5 seconds. + +Measured on ~195k chunks with ~113k rows already embedded: fetching **40,000** pending chunks blew +the 5s timeout and every pass died. Bounding the pass to **10,000** completed the same fetch in +about 0.73s. + +**Fix:** +1. **Bound every pass** — on the order of 10,000 chunks — and loop until the pending count reaches + zero. Run `maproom generate-embeddings --help` to see the flag that bounds a pass on your binary. +2. Run bounded passes in a loop rather than trying to finish in one shot. Each pass is independent; + chunks that were not embedded simply stay pending for the next one. +3. Recognise the shape of the failure: it appears **late**, exactly when the job is closest to + finishing. A pass that worked yesterday failing today is expected behavior at scale, not a + regression, and not a reason to rebuild the index. + +**Do not reproduce this by hand in `psql` and trust what you see.** maproom sets `work_mem` itself +(256MB) on its own connections, and this query is exquisitely sensitive to it: the same `NOT IN` +query planned at **cost 218,119,741** (Materialize plus a per-row Seq Scan) under a small `work_mem` +versus **101,635** (a hashed SubPlan) at 64MB and above — three orders of magnitude apart. A stock +`psql` session will show you a far worse problem than maproom actually has. If you must plan it by +hand, match the setting first: + +```sql +set work_mem = '256MB'; +-- then EXPLAIN (ANALYZE, BUFFERS) the pending-chunk query +``` + +**Prevention:** Treat bounded-pass-plus-loop as the normal way to run `generate-embeddings` on a +large index, not as a workaround reached for after the first failure. + +### One Failing Sub-Batch Fails the Whole Batch + +**Symptom:** An embedding pass reports a batch failure that names a *sub-batch*: + +``` +Sub-batch 2 failed: API error: Bad request: Batch of 50 texts rejected: {"error":"Post \"http://127.0.0.1:51950/tokenize\": EOF"} +``` + +**Root Cause:** Providers are called in sub-batches (Ollama default 50, concurrency 8). If **one** +sub-batch errors, maproom marks the **entire pipeline batch** failed — every chunk in that batch is +lost for the pass, not just the 50 in the failing sub-batch. The error above is the Ollama model +runner dropping the connection mid-request. + +Blast radius scales with batch size: with `--batch-size 1000` each incident cost 1,000 chunks; at +500 it cost 500. + +**Fix:** +1. Do nothing dramatic. These failures are **transient and self-heal on the next pass** — the lost + chunks simply stay pending — so a loop of bounded passes converges on full coverage. +2. Lower `--batch-size` to limit the blast radius per incident, but keep it **above** the sub-batch + size (default 50), or sub-batch parallelism never engages and the pass runs fully serialized. +3. Tune parallelism with the environment variables below, confirming they are honored by your build + (`maproom --help` lists the environment variables your build supports): + - `MAPROOM_EMBEDDING_PARALLEL_ENABLED` + - `MAPROOM_EMBEDDING_PARALLEL_SUB_BATCH_SIZE` (default 50) + - `MAPROOM_EMBEDDING_PARALLEL_MAX_CONCURRENCY` (default 8) + + Measured against a local Ollama, throughput improved as concurrency rose to ~8 and got **worse** + at 16. Eight was the sweet spot. + +**Prevention:** Pick a batch size comfortably above the sub-batch size but well below "the whole +job" — 500 against Ollama's default sub-batch of 50 is a good starting point — and always run +embedding generation as a loop rather than a single heroic pass. + ### Token Limit Exceeded **Symptom:** Embedding generation fails with an error containing: @@ -62,26 +331,54 @@ Failed to generate code embeddings: Api(BadRequest("input token count is 20633 b maproom status ``` -**Prevention:** Use `--batch-size 25` when generating embeddings for repositories with large files or code chunks. +**Prevention:** Lower the batch size when generating embeddings against the Google API for +repositories with large files or code chunks. Mind the trade-off: a batch size at or below the +parallel sub-batch size (default 50) runs fully serialized — see +[One Failing Sub-Batch Fails the Whole Batch](#one-failing-sub-batch-fails-the-whole-batch). -### Vector Search Returns No Results +**Note:** this limit is provider-specific — it is the Google embedding API rejecting an oversized +request, not maproom. Other providers fail differently: Ollama truncates oversized inputs instead of +rejecting them (a ~107KB chunk embedded fine in testing), and Cohere v3 models on Bedrock silently +truncate beyond 512 tokens. A provider-side rejection of a whole batch is also distinct from +[One Failing Sub-Batch Fails the Whole Batch](#one-failing-sub-batch-fails-the-whole-batch), where a +transient connection error loses the batch and self-heals on the next pass. -**Symptom:** `maproom vector-search` completes without errors but returns an empty result set. +### Vector Search Returns No Results (Zero Embeddings) -**Root Cause:** Embeddings have not been generated for the repository. Vector search requires pre-computed embeddings; without them, there is nothing to match against. +**Symptom:** `maproom vector-search` completes without errors but returns an empty result set — +while `maproom search` keeps returning perfectly good results. + +**Root Cause:** Embeddings have not been generated. **Zero embeddings never raises an error.** With +no embeddings at all, `maproom search` still returns results by falling back to full-text and +structural ranking; only vector search and semantic ranking are actually unavailable. Never infer +embedding coverage from "search works" — that is precisely the signal this failure mode fakes. **Fix:** -1. Check embedding status: +1. Check coverage **explicitly**. `maproom status` reports it per repo, e.g. `Embeddings: 0 (0.0%)`: ```bash maproom status ``` -2. If embeddings are missing, generate them: + On Postgres, an empty `encoding_runs` table confirms no generation run has ever happened, and + coverage can be measured per repo directly: + ```sql + select r.name, count(distinct c.blob_sha) blobs, count(distinct e.blob_sha) embedded + from repos r join worktrees w on w.repo_id=r.id + join chunk_worktrees cw on cw.worktree_id=w.id + join chunks c on c.id=cw.chunk_id + left join code_embeddings e on e.blob_sha=c.blob_sha group by 1 order by 2 desc; + ``` +2. Generate embeddings in **bounded passes, looped until the pending count reaches zero** — an + unbounded pass dies at scale, see + [generate-embeddings Stalls Near the Finish Line](#generate-embeddings-stalls-near-the-finish-line): ```bash - maproom generate-embeddings + maproom generate-embeddings --batch-size 500 ``` -3. Re-run your vector search after embeddings complete. + Run `maproom generate-embeddings --help` for the bounding and batching flags your binary exposes. +3. Re-run your vector search once coverage is non-zero. -**Prevention:** Always run `maproom status` before your first vector search to confirm embeddings are available. +**Prevention:** Check embedding coverage explicitly before your first vector search, and on a +schedule afterwards — coverage **decays as you work**, silently, see +[Watchers Do Not Maintain Embeddings](#watchers-do-not-maintain-embeddings). ### No Repositories Indexed @@ -121,6 +418,20 @@ Failed to generate code embeddings: Api(BadRequest("input token count is 20633 b maproom generate-embeddings ``` +**Note — "Last scan" is not a freshness signal.** `maproom status` prints a per-worktree "Last +scan" timestamp that is **not** updated when an incremental scan finds the git tree SHA unchanged; +the scan logs `No changes detected (tree SHA match), skipping scan` and leaves the old timestamp in +place. A months-old "Last scan" therefore usually means "nothing has changed since then", **not** +"the index is stale". Judging staleness by this field leads to pointless full re-scans. To actually +test freshness, search for a symbol you know was added recently, or compare the repo's current HEAD +against what the index recorded. + +**Note — re-scanning does not restore embeddings, it removes them.** The incremental processor +**deletes** embeddings for changed chunks and never regenerates them, so a re-scan improves chunk +freshness while *reducing* embedding coverage. Follow any significant re-scan with bounded +`generate-embeddings` passes. See +[Watchers Do Not Maintain Embeddings](#watchers-do-not-maintain-embeddings). + **Prevention:** Re-scan after significant code changes (branch switches, large merges, refactors) to keep the index current. ### Irrelevant Results @@ -328,6 +639,58 @@ maproom search --repo --query "" maproom search --help ``` +### Pool Timed Out While Waiting for an Open Connection + +``` +pool timed out while waiting for an open connection +``` + +**Cause:** Depends entirely on the backend. On **Postgres** the server is unreachable (container not +running, wrong host) or its connections are saturated by a long-running statement. On **SQLite** it +is the tail end of a permissions problem, after ~25 seconds of `unable to open database file` +retries. + +**Recovery:** See +[Connection Pool Timeout or "Connection refused" (Postgres backend)](#connection-pool-timeout-or-connection-refused-postgres-backend) +for Postgres, or [Permission Denied on Database (GAP-005)](#permission-denied-on-database-gap-005) +for SQLite. Do not apply the SQLite `chmod` recovery to a Postgres deployment. + +### Canceling Statement Due to Statement Timeout + +``` +Failed to fetch chunks ... canceling statement due to statement timeout +``` + +**Cause:** Postgres backend only. maproom pins `statement_timeout` to 5000ms per connection, and the +pending-chunk query for `generate-embeddings` grows more expensive as `code_embeddings` fills up. + +**Recovery:** Bound each embedding pass and loop — see +[generate-embeddings Stalls Near the Finish Line](#generate-embeddings-stalls-near-the-finish-line). + +### Error: "Reauthentication failed" (ADC expired, non-interactive) + +``` +Reauthentication failed. cannot prompt during non-interactive execution +``` + +**Cause:** Google Application Default Credentials have expired, and the embedding pass is running +somewhere it cannot prompt you to log in — cron, pm2, CI, a background loop. Every pass dies at +config time, before any chunk is embedded. + +**Recovery:** +```bash +# Interactive sessions: refresh ADC +gcloud auth application-default login + +# Unattended jobs: use a service-account key instead of user ADC +export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json +# (MAPROOM_GOOGLE_APPLICATION_CREDENTIALS is also honored) +``` + +A user-ADC refresh token that expires every few hours is the wrong credential for a scheduled +embedding job; move unattended work to a service account rather than re-running `gcloud auth` by +hand each morning. + ### Error: "Failed to create token provider from ADC" ``` @@ -338,7 +701,7 @@ Caused by: Failed to create token provider from ADC ``` -**Cause:** Google Application Default Credentials (ADC) have expired. This is a **credential issue, not a code bug**. The `vector-search` subcommand requires valid credentials to call the embedding API (Vertex AI by default). When ADC tokens expire, the CLI cannot authenticate with the embedding provider. +**Cause:** Google Application Default Credentials (ADC) have expired. This is a **credential issue, not a code bug**. The `vector-search` subcommand requires valid credentials to call the embedding API (Google/Vertex when that is the configured provider). When ADC tokens expire, the CLI cannot authenticate with the embedding provider. **Recovery:** 1. Refresh ADC credentials: @@ -360,6 +723,9 @@ Caused by: ``` **Related Errors:** +- `Reauthentication failed. cannot prompt during non-interactive execution` is the same expiry seen + from a non-interactive context (cron, pm2, CI). See + [Error: "Reauthentication failed" (ADC expired, non-interactive)](#error-reauthentication-failed-adc-expired-non-interactive). - `invalid_rapt` in error output also indicates expired ADC credentials; use the same resolution steps above. - `quota_project_id is required` indicates the quota project is not configured; run `gcloud auth application-default set-quota-project YOUR_PROJECT_ID`. @@ -382,17 +748,53 @@ Error: Failed to create embedding service. Ensure OPENAI_API_KEY is set. ```bash echo "$MAPROOM_EMBEDDING_PROVIDER" ``` -2. If the variable is unset or set to `vertex-ai`, this is an ADC credential issue. Follow the ADC recovery steps above. +2. If the variable is unset, or set to the Google/Vertex provider (current builds accept + `MAPROOM_EMBEDDING_PROVIDER=google`; older builds may use a different value — check + `maproom --help`), this is an ADC credential issue. Follow the ADC recovery steps above. 3. If set to `openai`, set the API key: ```bash export OPENAI_API_KEY="" ``` 4. If you see `OPENAI_API_KEY` in the error but are not using OpenAI, the provider may be misconfigured. See [Embedding Providers](./embedding-providers.md) for correct configuration. +**Provider quick facts — confirm every one against your own binary, availability differs between +builds (`maproom --help`):** + +- **`ollama`** (local, free). Endpoint from `OLLAMA_URL` (also `MAPROOM_OLLAMA_URL`). **In a + devcontainer this must be `http://host.docker.internal:11434`** — `localhost:11434` is not the + host's Ollama. Default model `mxbai-embed-large` is 1024-dimensional (fits `embedding_1024`); + `nomic-embed-text` is 768. Oversized inputs are truncated rather than rejected. +- **`google`** (Vertex). Requires `GOOGLE_PROJECT_ID` (also `MAPROOM_GOOGLE_PROJECT_ID`) plus + **Application Default Credentials — not an API key**. Setting `GEMINI_API_KEY` or `GOOGLE_API_KEY` + does **nothing** for this provider; this is a common and confusing mistake, because the variables + look plausible and the failure arrives as an authentication error rather than a configuration one. +- **`openai`**: `OPENAI_API_KEY` (also `MAPROOM_OPENAI_API_KEY`). Cohere direct uses + `MAPROOM_COHERE_API_KEY`. +- **`bedrock`** (AWS): default model `amazon.titan-embed-text-v2:0` at 1024 dimensions with a + **maximum batch of 1**, so throughput comes from concurrency rather than batch size (see + [One Failing Sub-Batch Fails the Whole Batch](#one-failing-sub-batch-fails-the-whole-batch)); + `amazon.titan-embed-text-v1` is 1536-dimensional, also max batch 1; `cohere.embed-english-v3` and + `cohere.embed-multilingual-v3` are 1024-dimensional with max batch 96 and **silently truncate** + beyond 512 tokens. Credentials resolve through the standard AWS chain (static env keys, shared + config/credentials files honoring `AWS_PROFILE`, web identity / IRSA, container credentials). + **Bedrock support is not present in every maproom build** — verify with `maproom --help` and by + checking whether `MAPROOM_EMBEDDING_PROVIDER=bedrock` is accepted before planning around it. + Configuration details live in [Embedding Providers](./embedding-providers.md). + +Whichever provider you pick, its output dimension must match a stored column — see +[Embedding Dimension Must Match a Stored Column](#embedding-dimension-must-match-a-stored-column). + **Reference:** See [Embedding Providers](./embedding-providers.md) for the full list of supported providers and their required environment variables. ### Network Timeout During Vector Search +**Scope — this and the two sections that follow assume a *remote* embedding provider.** They were +measured against OpenAI; the same shapes apply to any hosted provider (Google/Vertex, Bedrock, +Cohere) with that provider's endpoint and credentials substituted. A local `ollama` provider has no +internet dependency and no rate limit at all — in a devcontainer it is reached at +`http://host.docker.internal:11434`, and it fails in the ways described under +[One Failing Sub-Batch Fails the Whole Batch](#one-failing-sub-batch-fails-the-whole-batch). + **Symptom:** `maproom vector-search` hangs or times out during embedding generation. The command does not return results or an error within the expected time frame. **Root Cause:** The OpenAI API is unreachable due to network connectivity issues. Vector search requires a live API call to generate query embeddings — unlike FTS search, it cannot operate offline. @@ -500,43 +902,123 @@ Caused by: Failed to create token provider from ADC ``` +From a non-interactive context — cron, pm2, CI, an overnight embedding loop — the same expiry +surfaces as: + +``` +Reauthentication failed. cannot prompt during non-interactive execution +``` + This is **not a code bug**. It is a credential expiry issue. Refresh credentials with: ```bash gcloud auth application-default login --no-launch-browser gcloud auth application-default set-quota-project YOUR_PROJECT_ID ``` +For unattended jobs, prefer a service-account key over user ADC: +`GOOGLE_APPLICATION_CREDENTIALS` (also `MAPROOM_GOOGLE_APPLICATION_CREDENTIALS`). + See [ADC Setup Guide](./adc-setup.md) for detailed instructions. ### text-embedding-004 Is Not Available via Gemini REST API The `text-embedding-004` model used by maproom for embeddings is only available through the **Vertex AI API**, not through the Gemini REST API. Attempting to use the Gemini REST API endpoint for embeddings will fail. This means: -- ADC credentials (Google Cloud authentication) are required for the default embedding provider. +- ADC credentials (Google Cloud authentication) are required whenever the Google/Vertex provider is + configured. - The `GOOGLE_API_KEY` environment variable (used for Gemini REST API) is **not sufficient** for embedding generation. -- You must use either the Vertex AI provider (with ADC) or the OpenAI provider (with `OPENAI_API_KEY`). +- For this model specifically, use the Vertex AI provider with ADC. It is not the only embedding + provider available — `ollama` (local), `openai`, `bedrock` and Cohere are separate options with + their own models and credentials; see the provider quick facts under + [Embedding Provider Misconfiguration](#embedding-provider-misconfiguration), and confirm what your + build accepts with `maproom --help`. See [Embedding Providers](./embedding-providers.md) for supported providers and configuration. ### Cross-Provider Re-indexing Required When Switching Providers -Embeddings generated by one provider (e.g., Vertex AI with `text-embedding-004`) are **not compatible** with embeddings from another provider (e.g., OpenAI with `text-embedding-ada-002`). If you switch embedding providers, you must regenerate all embeddings: +Embeddings generated by one provider (e.g., Vertex AI with `text-embedding-004`) are **not compatible** with embeddings from another provider (e.g., OpenAI with `text-embedding-ada-002`). If you switch embedding providers, you must regenerate all embeddings — on a large index as bounded +passes looped until the pending count reaches zero, not one unbounded run (see +[generate-embeddings Stalls Near the Finish Line](#generate-embeddings-stalls-near-the-finish-line)): ```bash -maproom generate-embeddings +maproom generate-embeddings --batch-size 500 ``` Failure to re-index after switching providers will cause vector-search to return poor or zero results, because the query embedding (from the new provider) will be compared against stored embeddings (from the old provider) that exist in a different vector space. +**On the Postgres backend, re-running `generate-embeddings` is not sufficient** — the old rows must +be deleted first. See the next entry. + See [Embedding Providers](./embedding-providers.md) for details on provider switching. +### Switching Embedding Models Requires Deleting the Old Rows + +`code_embeddings` is `UNIQUE` on `blob_sha`, so a blob that was already embedded at the old +dimension **keeps its row**. Incremental runs see that a row exists and skip the blob, while vector +search at the new dimension cannot see it. Switching model or provider without deleting the rows at +the old dimension **silently corrupts coverage** — no error, no warning, just a growing set of +chunks that are quietly unreachable by vector search. + +Verify homogeneity before and after any model change: + +```sql +select embedding_dim, model_version, count(*) from code_embeddings group by 1,2; +``` + +More than one `(embedding_dim, model_version)` pair means the index is mixed. Delete the rows at the +old dimension/model, then run bounded `generate-embeddings` passes until the pending count reaches +zero. + +Related housekeeping: `code_embeddings` has no foreign key to `chunks`, so deleting chunks leaves +orphan embedding rows behind. They are harmless, but they accumulate and they inflate the `NOT IN` +subquery described in +[generate-embeddings Stalls Near the Finish Line](#generate-embeddings-stalls-near-the-finish-line). + +### Embedding Dimension Must Match a Stored Column + +Only **768, 1024 and 1536** have columns (`embedding_768`, `embedding_1024`, `embedding_1536`) and +HNSW indexes. A model emitting any other dimension **cannot be stored at all** — for example Ollama's +`qwen3-embedding:4b` at 2560 dimensions. Model choice is constrained by the schema, not by taste. + +Where a model offers configurable output sizes, the same constraint applies: Titan v2 can emit 256- +and 512-dimensional vectors, but with no column set for those sizes they are rejected by a dimension +validation step rather than failing after a full scan. + +### Watchers Do Not Maintain Embeddings + +A per-repo watcher keeps **chunks** fresh. It does **not** keep embeddings fresh: the incremental +processor **deletes** embeddings for changed chunks and never regenerates them. Embedding coverage +therefore **decays continuously as you work** — silently, because search keeps returning results the +whole time by falling back to full-text ranking (see +[Vector Search Returns No Results (Zero Embeddings)](#vector-search-returns-no-results-zero-embeddings)). + +Holding coverage requires a **periodic `maproom generate-embeddings` job** — cron, a process-manager +schedule, or an equivalent — running bounded passes in a loop. Documentation elsewhere claiming +"watch keeps the index fresh" is true for chunks and **false for embeddings**. + --- ## Edge Case Handling This section documents boundary conditions, resource errors, and concurrency scenarios tested against `maproom` version 0.1.0. Each entry records empirical CLI behavior observed during testing. +**Backend scope:** these were measured against the **SQLite backend**, and the storage-related +entries (GAP-004, GAP-005, GAP-006) describe SQLite failure modes only. On the Postgres backend +there is no local database file, so file permissions, WAL files and `~/.maproom/` disk usage are not +in play at all — see +[First: Which Storage Backend Are You On?](#first-which-storage-backend-are-you-on) and +[Connection Pool Timeout or "Connection refused" (Postgres backend)](#connection-pool-timeout-or-connection-refused-postgres-backend). +The flag-behavior entries (GAP-001, GAP-002, GAP-003, GAP-007) are backend-independent, but were +measured on v0.1.0 — check your own binary with `maproom search --help` before relying on the exact +values below. + ### SQLite Lock Contention (GAP-006) -- HIGH PRIORITY +**Applies to: SQLite backend only.** Postgres handles concurrent readers and writers with MVCC and +does not produce `SQLITE_BUSY`; a pool timeout there means something else entirely (see +[Connection Pool Timeout or "Connection refused" (Postgres backend)](#connection-pool-timeout-or-connection-refused-postgres-backend)). +A shared Postgres fleet is in fact the standard way to run many concurrent watchers and searches. + **Symptom:** When multiple agents or processes run `maproom search` or `maproom scan` concurrently against the same repository, commands may fail with SQLite lock errors such as `SQLITE_BUSY` or connection pool timeouts. **Root Cause:** The maproom database uses SQLite, which has limited write concurrency. Concurrent read-only operations (searches) are handled well by SQLite's WAL mode. However, concurrent write operations (scan while searching) or access to a locked database can cause contention. @@ -604,7 +1086,7 @@ This section documents boundary conditions, resource errors, and concurrency sce | `--threshold=-0.5` (equals) | `vector-search` | Accepted by parser (fails at API key check before validation) | 1 | | `--threshold=1.5` (equals) | `vector-search` | Accepted by parser (fails at API key check before validation) | 1 | -**Note:** Full threshold range validation could not be tested empirically because `vector-search` requires `OPENAI_API_KEY` which was not available in the test environment. The parser accepts the values, but runtime behavior with actual embeddings is untested. +**Note:** Full threshold range validation could not be tested empirically because `vector-search` needs working credentials for the configured embedding provider (`OPENAI_API_KEY` in that test environment), which were not available. The parser accepts the values, but runtime behavior with actual embeddings is untested. **Fix:** 1. Only use `--threshold` with the `vector-search` subcommand, never with `search`: @@ -653,6 +1135,10 @@ This section documents boundary conditions, resource errors, and concurrency sce ### Disk Full During Scan (GAP-004) +**Applies to: SQLite backend only.** On Postgres, disk exhaustion is a property of the database +server's host and volume, not of `~/.maproom/`; the recovery below (deleting `maproom.db*` and +re-migrating) has no Postgres equivalent and must not be attempted there. + **Note:** This edge case has not been tested empirically due to the risk of destabilizing the shared development environment. Simulating disk-full conditions requires filling the filesystem, which could affect other processes and services. **Symptom (predicted):** `maproom scan` fails mid-operation when the filesystem runs out of space. The SQLite database may be left in an inconsistent state if the write-ahead log (WAL) cannot be flushed. @@ -688,6 +1174,13 @@ Error: database or disk is full ### Permission Denied on Database (GAP-005) +**Applies to: SQLite backend only.** This is the entry most often misapplied. On the Postgres +backend there is no database file to `chmod`, and an identical-looking connection pool timeout means +the server is unreachable or saturated — see +[Connection Pool Timeout or "Connection refused" (Postgres backend)](#connection-pool-timeout-or-connection-refused-postgres-backend). +Distinguish them by the accompanying log lines: repeated `unable to open database file` before the +timeout means SQLite permissions; `Connection refused` from `psql` or `pg_isready` means Postgres. + **Symptom:** Search or scan commands fail with repeated `ERROR unable to open database file` messages followed by a connection pool timeout. The CLI retries with exponential backoff for approximately 25 seconds before giving up. **Root Cause:** The maproom database file (`~/.maproom//maproom.db`) or its directory has insufficient filesystem permissions. SQLite requires read access for searches and write access for scans. SQLite also needs access to the WAL file (`maproom.db-wal`) and shared memory file (`maproom.db-shm`) in the same directory. From 3b69a5d32e4d24d27502f5b7f6346b4bc88abda7 Mon Sep 17 00:00:00 2001 From: Daniel Bushman Date: Sat, 5 Sep 2026 22:43:23 +0000 Subject: [PATCH 2/2] ci(maproom): scope the bare-search-command check to real invocations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Check 3 flagged every line in SKILL.md mentioning `maproom search` or `maproom vector-search` without `--format agent`. That over-matches in two ways, and both now trip on docs that are doing the right thing: - `maproom search --help` prints usage text, not search results, so `--format agent` is meaningless for it. The progressive-disclosure guidance deliberately points readers at `--help` as the source of truth for their own build, which the old pattern read as a violation. - Prose mentions ("`maproom search` still returns results...") were flagged the same as command invocations. Check 4 already had the right semantics for the second case: it only counts a line if it starts with the command or a `$` prompt. Check 3 now uses the same rule, and both checks skip `--help` invocations. The rule still fires on what it is meant to catch — a bare `maproom search --repo foo --query "bar"` in a code block outside the Output Formats section still fails the check (verified before committing). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01NCnCnSwkg7BFzyq7Bs8PHp --- plugins/maproom/scripts/verify-docs.sh | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/plugins/maproom/scripts/verify-docs.sh b/plugins/maproom/scripts/verify-docs.sh index 26e340f8..6a7471ca 100755 --- a/plugins/maproom/scripts/verify-docs.sh +++ b/plugins/maproom/scripts/verify-docs.sh @@ -188,7 +188,10 @@ fi # =========================================================================== human_echo "Check 3: Bare search commands in SKILL.md (only in Output Formats section)" # Find search/vector-search commands that do NOT include --format agent -bare_lines=$(grep -n "maproom search\|maproom vector-search" "$SKILL_MD" | grep -v "\-\-format agent" || true) +# --format agent exists to conserve agent context in command OUTPUT. `--help` +# prints usage text, not search results, so it is out of scope for this rule -- +# as are prose mentions of the command (filtered below, matching Check 4). +bare_lines=$(grep -n "maproom search\|maproom vector-search" "$SKILL_MD" | grep -v "\-\-format agent" | grep -v "\-\-help" || true) if [ -n "$bare_lines" ]; then # Find the line range for the Output Formats section # Start: line with "## Output Formats" @@ -208,7 +211,10 @@ if [ -n "$bare_lines" ]; then continue fi line_num=$(echo "$line" | cut -d: -f1) - if [ -n "$section_start" ] && [ "$line_num" -ge "$section_start" ] && [ "$line_num" -lt "$section_end" ]; then + content=$(echo "$line" | sed 's/^[0-9]*://') + if ! echo "$content" | grep -qE "^\s*(maproom|\\\$\s*maproom)"; then + : # Prose mention, not a command invocation (same rule as Check 4) + elif [ -n "$section_start" ] && [ "$line_num" -ge "$section_start" ] && [ "$line_num" -lt "$section_end" ]; then : # Inside Output Formats section, acceptable else bare_outside_section=$((bare_outside_section + 1)) @@ -235,7 +241,7 @@ fi # =========================================================================== human_echo "Check 4: Bare search commands in multi-repo-guide.md (zero in code blocks)" # Find lines with search commands missing --format agent -bare_lines=$(grep -n "maproom search\|maproom vector-search" "$MULTI_REPO_MD" | grep -v "\-\-format agent" || true) +bare_lines=$(grep -n "maproom search\|maproom vector-search" "$MULTI_REPO_MD" | grep -v "\-\-format agent" | grep -v "\-\-help" || true) if [ -z "$bare_lines" ]; then report_pass "No bare search commands found" json_add_result 4 "Bare search commands in multi-repo-guide.md" "pass" 0 0 "No bare search commands found"