feat(semantic): [alpha build] provider-aware typed embeddings, reranking, diagnostics, and eval harness

[Zireael]

Summary

Semantic search in AFT moves from a minimal embedding-and-cosine prototype to a provider-capability-aware retrieval subsystem with typed vectors, optional reranking, background lifecycle management, diagnostics, and evaluation tooling. This is a public preview — the feature is functional and tested (~93 new tests) but expects iteration based on real-world feedback.

What changed

The upgrade touches the full semantic pipeline — config, indexing, retrieval, diagnostics, and observability — without breaking the default fastembed experience.

Typed vector representations

Vectors are no longer opaque f32 blobs. Every stored vector carries explicit type metadata (DenseF32, Int8SourceDecoded, BinaryPacked) and is paired with its source kind so the correct distance metric is selected automatically. Binary packed vectors use Hamming search (native bitwise XOR + popcount) instead of cosine, which is both faster and semantically correct for quantized embeddings. This unlocks Perplexity's base64_binary and base64_int8 output modes alongside standard dense providers.

Provider capability profiles

Each embedding backend (fastembed, OpenAI-compatible, Ollama, Perplexity) declares what it supports: output encoding, distance metric, dimension range, max batch size. The config layer validates combinations at configure time — you cannot accidentally request binary vectors through a cosine-only provider. Profiles also carry fingerprint fields so switching providers triggers a clean index rebuild rather than silent corruption.

Fingerprint-driven index lifecycle

A SemanticIndexFingerprint captures every dimension that affects index correctness: backend, model, base_url, dimension, chunking_version, output_encoding, storage_strategy, vector kinds, normalization, and prompt hashes. diff() classifies changes as Rebuild (structural — re-embed everything), ClearQueryCache (query prompts changed — invalidate cached results only), or None. This replaces the previous "delete and hope" invalidation with precise, explainable rebuild decisions.

Non-blocking cold start

Index builds run in a background thread with cooperative cancellation (SemanticCancellationToken via AtomicU64 generation counter). The build checks the generation before each embedding batch and exits early when a reconfigure arrives. Priority ordering ensures high-value files (recently edited, high PageRank) get embedded first. Exponential backoff handles transient provider failures without blocking the session.

Stale-vector pruning

When files are edited, deleted, moved, excluded, or re-included, the index tracks which vectors are stale and prunes them during the next refresh cycle. Every vector record carries file/chunk ownership metadata (file path, version, chunk hash, index fingerprint) so pruning is traceable and deterministic.

File policy and docs chunking

A configurable file policy controls which files enter the index (include globs, exclude globs, max file size, max chunk count). The docs chunker splits Markdown and documentation files into semantic sections before embedding, improving recall for documentation-shaped queries.

Reranking pipeline

Optional reranking via any OpenAI-compatible /v1/rerank or chat-completion endpoint. The pipeline sends initial retrieval candidates to a reranker, parses the response (supporting multiple JSON shapes), and reorders results with safe fallback — if the reranker fails, the original cosine-similarity order is returned unchanged. Config fields: rerank.enabled, rerank.model, rerank.base_url, rerank.api_key_env, rerank.max_candidates.

Search pipeline metrics and diagnostics

Every aft_search call records timing, cache hits/misses, result counts, and reranker fallback events. Metrics are exposed through the status command and through JSONL diagnostic logs for offline analysis. The DiagnosticsOutputMode config controls verbosity in tool output (compact | verbose | off).

Semantic doctor

semantic_doctor is a health-check command that reports config summary, index summary, metrics summary, provider summary, and actionable suggestions. Use it to verify that the index is healthy, the provider is reachable, and the configuration is consistent.

Semantic eval harness

semantic_eval runs a JSONL-defined evaluation suite against the semantic index. Each case specifies a query, expected paths, expected symbols, and top-k. The harness computes recall@k and MRR (Mean Reciprocal Rank) for quantifying retrieval quality across config changes.

Status integration

The status command now includes semantic health metrics: lifecycle state, entry count, dimension, total queries, cache hit ratio, average query time, and provider info. The OpenCode TUI sidebar surfaces these alongside the existing index state.

Config trust boundary

backend, base_url, and api_key_env are user-only fields — project-level aft.jsonc cannot inject these. A hostile repository cannot redirect embeddings at an attacker-controlled endpoint or exfiltrate API keys. The plugin logs a warning when it strips a project-level setting.

Contextualized document-chunk embedding (partial)

Initial support for Perplexity-style document/chunk grouped embedding — chunks from the same source document are batched together rather than flattened. Oversized document handling and retry logic are still in progress (see roadmap).

How to test

Default fastembed (zero-config)

# Enable semantic search in your AFT config
# ~/.config/opencode/aft.jsonc or ~/.pi/agent/aft.jsonc:
{ "semantic_search": true }

# Start a session — index builds in background
# Run aft_search with a concept query:
aft_search({ "query": "authentication middleware" })

Verify: results appear with source: semantic or source: hybrid tags. Status shows [index: ready] after build completes.

Provider switching

// Switch to OpenAI-compatible
{
  "semantic_search": true,
  "semantic": {
    "backend": "openai_compatible",
    "model": "text-embedding-3-small",
    "base_url": "https://api.openai.com/v1",
    "api_key_env": "OPENAI_API_KEY"
  }
}

Verify: index rebuilds automatically on next session start. Status shows new provider/model.

Reranking

{
  "semantic_search": true,
  "semantic": {
    "backend": "openai_compatible",
    "model": "text-embedding-3-small",
    "base_url": "https://api.openai.com/v1",
    "api_key_env": "OPENAI_API_KEY"
  },
  "rerank": {
    "enabled": true,
    "model": "rerank-english-v3.0",
    "base_url": "https://api.cohere.com",
    "api_key_env": "COHERE_API_KEY"
  }
}

Verify: search results show reranker-sorted order. Disable reranker — results fall back to cosine order.

Semantic doctor

aft_search({ "query": "test" })  # trigger index build if cold
# Then check health via status command or semantic_doctor

Verify: health report shows ConfigSummary, IndexSummary, MetricsSummary, ProviderSummary.

Eval harness

// Create eval-cases.jsonl:
{"query": "authentication handler", "expected_paths": ["src/auth/middleware.ts"], "expected_symbols": ["authMiddleware"], "top_k": 10}
{"query": "database connection", "expected_paths": ["src/db/pool.ts"], "expected_symbols": ["createPool"], "top_k": 10}

Verify: returns recall@k and MRR scores.

Test coverage

~93 tests across 8 test sub-tasks covering:

• Config parsing and backward compatibility
• Fingerprint diff matrix (all field combinations → Rebuild/ClearQueryCache/None)
• File policy, docs chunking, and manifest handling
• VectorStore trait with DenseF32 and BinaryPacked implementations
• Binary packed-vector storage and Hamming search
• Lifecycle states, snapshots, and stale-vector pruning
• Search pipeline metrics, diagnostics, and DiagnosticsOutputMode
• Concurrency, race conditions, and cancellation token behavior
• Security trust boundary enforcement (project config stripping)
• Semantic doctor health report
• Semantic eval harness (JSONL parsing, scoring, recall/MRR)
• Reranking pipeline (parse multiple JSON shapes, fallback on failure)

Roadmap

Still in progress or planned for follow-up:

• aft-t6p.23: Complete contextualized document-chunk embedding (oversized docs, retry logic) — partially implemented
• aft-t6p.2.2: Configurable snippet truncation in reranking (currently hardcoded at 200 chars)
• aft-t6p.18: End-to-end verification across all backends
• aft-t6p.5: Configuration and operations documentation
• Performance benchmarking suite
• Migration tooling for index format upgrades

Architecture notes

Key new modules:

• crates/aft/src/semantic_rerank.rs — reranking pipeline with safe fallback
• crates/aft/src/semantic_diagnostics.rs — JSONL diagnostic logging
• crates/aft/src/semantic_doctor.rs — health-check report generation
• crates/aft/src/semantic_eval.rs — evaluation harness (JSONL parser, scoring)
• crates/aft/src/vector_store.rs — VectorStore trait with DenseF32 and BinaryPacked implementations
• crates/aft/src/commands/semantic_doctor.rs — doctor command handler
• crates/aft/src/commands/semantic_eval.rs — eval command handler

Modified significantly:

• crates/aft/src/semantic_index.rs — lifecycle management, fingerprint-driven invalidation, non-blocking build, stale pruning, typed vectors
• crates/aft/src/config.rs — provider profiles, rerank config, trust boundary fields
• crates/aft/src/commands/status.rs — semantic health metrics
• crates/aft/src/commands/semantic_search.rs — reranking integration, diagnostics output mode

---

^{Need help on this PR? Tag /codesmith with what you need. Autofix is disabled.}

---

Summary by cubic

Upgrades semantic search to a provider-aware pipeline with typed vectors, optional reranking, partial-index querying, and built-in diagnostics/eval. Adds Perplexity support and binary/int8 vectors with Hamming, and hardens reranking and config schema alignment for a smoother setup.

• New Features

  • Provider profiles and typed vectors (f32, int8, binary packed) with auto metric selection; enables Perplexity base64 binary/int8.
  • Fingerprint-driven lifecycle with background/partial builds and precise stale-vector pruning.
  • Optional reranking via OpenAI-compatible endpoints with safe fallback to original order.
  • Metrics and diagnostics with JSONL logs and configurable verbosity; status surfaces semantic health.
  • Tools: semantic_doctor for health checks and semantic_eval for recall@k/MRR.
  • File policy and docs chunking for better recall; trust boundary blocks project configs from setting backend/base_url/api_key.
  • VectorStore abstraction decouples storage/search; preview contextualized document-chunk embedding.
  • Dev/repo: Docker-based Rust validation scripts; restore upstream .alfonso/ and expand .gitignore to ignore local agent/tooling dirs.
  • Plugin schema now matches Rust enums and adds perplexity (output encoding, storage strategy, input mode, distance metric); project-level stripping extended with tests.
  • Reranker robustness: strip markdown fences in responses and prevent duplicate indices in reranked results.

• Migration

  • No changes needed for default fastembed.
  • Switching providers auto-triggers a clean index rebuild via fingerprints.
  • To use reranking, add a rerank block (model, base_url, api key env, limits).
  • Move backend/base_url/api_key to user-level config; project-level values are ignored.

^{Written for commit 95ea25c. Summary will update on new commits.}

Greptile Summary

This PR replaces the minimal embedding prototype with a provider-capability-aware retrieval subsystem: typed vectors (f32, int8, binary-packed), fingerprint-driven index lifecycle with background builds and cooperative cancellation, optional LLM-based reranking, JSONL diagnostics, semantic_doctor, and semantic_eval.

• Typed vector pipeline: TypedVector / StoredVector / VectorStore trait decouple encoding, storage, and search; Hamming distance is now used correctly for binary-packed vectors.
• Fingerprint-driven lifecycle: SemanticIndexFingerprint::diff() classifies changes as Rebuild, ClearQueryCache, or None, replacing the prior "delete and hope" invalidation.
• Reranking, diagnostics, and eval: optional reranking via any OpenAI-compatible endpoint with safe fallback; JSONL diagnostic logging with configurable retention; semantic_doctor and semantic_eval commands.
• Trust boundary hardened: 10 additional semantic config fields are now blocked from project-level aft.jsonc.

Confidence Score: 3/5

Safe to merge for alpha/preview use, but the reranking pipeline does not function as intended — the overfetch from the vector index is discarded before the reranker can see it.

The reranking feature fetches max(top_k, 50) candidates from the vector index specifically to give the reranker a wider pool, but fuse_hybrid_results immediately truncates back to top_k before calling rerank_candidates. For the default top_k=10, the reranker always operates on 10 items — the same result set a non-reranking search returns — so it can only reorder but cannot surface better matches from the wider pool.

crates/aft/src/commands/semantic_search.rs — the interaction between index.search(...clamp(50, MAX_TOP_K)) and fuse_hybrid_results(..., params.top_k) needs to be reworked so the wider pool reaches the reranker before truncation.

Important Files Changed

Filename	Overview
crates/aft/src/commands/semantic_search.rs	Reranking pipeline has a logic bug: vector search overfetches (clamp to 50) but fuse_hybrid_results truncates back to top_k before rerank_candidates is called, so the reranker never sees more candidates than top_k.
crates/aft/src/semantic_rerank.rs	New reranking pipeline with proper fallback, markdown-fence stripping, and multiple JSON shape support; prompt template has a minor double-expansion edge case.
crates/aft/src/config.rs	New typed enums and extensive SemanticBackendConfig fields; serde names align with TypeScript schema.
packages/opencode-plugin/src/config.ts	Trust boundary enforcement expanded to cover 10 new semantic fields; enum values now match Rust serde; reranking and diagnostics fields still absent from SemanticConfigSchema.
crates/aft/src/semantic_index.rs	Massive expansion: typed vector support, fingerprint-driven lifecycle, background build with cancellation token, stale pruning, partial index querying; new V7/V8 index versions.
crates/aft/src/vector_store.rs	New VectorStore trait with FlatF32 (cosine) and FlatBinaryHamming implementations; decouples storage/search from lifecycle management.

Sequence Diagram

sequenceDiagram
    participant Agent
    participant SemanticSearch
    participant VectorIndex
    participant LexicalIndex
    participant Reranker

    Agent->>SemanticSearch: "aft_search({ query, top_k })"
    SemanticSearch->>VectorIndex: search(query_vec, clamp(top_k, 50, 100))
    VectorIndex-->>SemanticSearch: semantic_results up to 50
    SemanticSearch->>LexicalIndex: lexical_rank(trigrams, 50)
    LexicalIndex-->>SemanticSearch: lexical_files up to 50
    SemanticSearch->>SemanticSearch: fuse_hybrid_results truncates to top_k
    Note over SemanticSearch: results.len() at most top_k default 10
    alt rerank_enabled
        SemanticSearch->>Reranker: POST chat/completions min(max_candidates,top_k) snippets
        Reranker-->>SemanticSearch: JSON indices
        SemanticSearch->>SemanticSearch: apply reorder + append missing
    end
    SemanticSearch-->>Agent: status results diagnostics

Loading

Comments Outside Diff (1)

1. packages/opencode-plugin/src/config.ts, line 37-54 (link)

   TypeScript enum values don't match the Rust serde strings — config will fail to deserialize

   Several new enum schemas use values that don't align with the Rust serde representation:

   • SemanticOutputEncodingEnum allows "binary", "ubinary", "int8", "uint8" but Rust OutputEncoding deserializes from "base64_binary" and "base64_int8".
   • SemanticStorageStrategyEnum allows "flat" and "binary_pack" but Rust StorageStrategy expects "native_f32" and "binary_packed".
   • SemanticInputModeEnum includes "chunk_extracts" and "contextualized" but Rust InputMode only has "flat_texts" and "document_chunks".
   • SemanticDistanceMetricEnum uses "dot" but Rust DistanceMetric expects "dot_product".
   • SemanticBackendEnum is missing the new "perplexity" variant added to Rust.

   A user who follows the TypeScript autocomplete and picks output_encoding: "int8" will pass TypeScript validation but receive a deserialization error (or silent fallback to default) from the Rust binary at runtime.

_{Reviews (3): Last reviewed commit: "fix: address greptile and qubic review c..." | Re-trigger Greptile}

[cubic-dev-ai]

4 issues found across 107 files

<sub>Note: This PR contains a large number of files. cubic only reviews up to 100 files per PR, so some files may not have been reviewed. cubic prioritizes the most important files to review.
On a pro plan you can use ultrareview for larger PRs.

Re-trigger cubic</sub>

[cubic-dev-ai]

1 issue found across 69 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name=".gitignore">

<violation number="1" location=".gitignore:95">
P2: Inconsistent .gitignore pattern: `omo/` should likely be `.omo/` to match the hidden tooling directory convention used by all other entries in this block.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Inconsistent .gitignore pattern: omo/ should likely be .omo/ to match the hidden tooling directory convention used by all other entries in this block.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At .gitignore, line 95:

<comment>Inconsistent .gitignore pattern: `omo/` should likely be `.omo/` to match the hidden tooling directory convention used by all other entries in this block.</comment>

<file context>
@@ -87,3 +87,11 @@ benchmarks/aft-search/.bench/
+.beads/
+.qartez/
+.claude/
+omo/
+.kiro/
+.lean-ctx/
</file context>

[greptile-apps]

Markdown-fenced JSON response never stripped in production path

The test rerank_parses_markdown_fenced_json demonstrates the need but strips the fences inline in the test body — the actual rerank_candidates function passes the LLM response straight to serde_json::from_str without stripping ```json … ``` wrappers. Many chat models consistently return JSON inside code fences regardless of response_format: json_object. When they do, both the Vec<usize> parse and every indices/rank/order field lookup fail, rerank_candidates returns RerankOutcome::Failed, and the reranker silently becomes a no-op for those models. The fix is to strip leading ```json / ``` and trailing ``` from content before the two parse attempts.

[greptile-apps]

Reranking config fields not declared in SemanticConfigSchema — silently stripped by Zod

SemanticBackendConfig on the Rust side gained 10+ new fields for reranking and diagnostics (rerank_enabled, rerank_model, rerank_base_url, rerank_api_key_env, rerank_timeout_ms, rerank_max_candidates, rerank_max_candidate_chars, diagnostics_enabled, output_mode, jsonl_logging, etc.), but none of them appear in the TypeScript SemanticConfigSchema. Zod's z.object({...}) strips unknown keys by default, so any user who adds rerank_enabled: true (or any other new field) to their semantic block in aft.jsonc will have it silently removed during parsing. The value never reaches Rust, reranking never activates, and there is no warning. This renders the reranking feature completely unconfigurable through the standard plugin config path.

[greptile-apps]

Successful search unconditionally overwrites Partial index status with Ready

Every successful search ends with *ctx.semantic_index_status().borrow_mut() = SemanticIndexStatus::Ready, even when the background build is still in progress. When a user queries while the index is Partial (e.g., 40% built), the search falls through (allowed by design), then this line overwrites the status — dropping the completeness percentage and the entry counts the build thread wrote. The background thread will overwrite it again on its next tick, but until then the status command and TUI sidebar show "ready" for an incomplete index.

[Zireael]

Source code for semantic search functionality for public preview.
Feature skeleton is there, needs finishing up, polishing static tests and functional testing.
One more thing that would need adding would be model2vec 'Potion Code 16M' support. If it performs well in tests, I think it could become fast, cheap and performant default semantic model.

Here's imlementation plans for sprints under this epic (in gastown beads format):
aft-semantic-search-upgrade.json

[cubic-dev-ai]

1 issue found across 4 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/opencode-plugin/src/config.ts">

<violation number="1" location="packages/opencode-plugin/src/config.ts:40">
P2: Semantic enum literals were renamed without backward-compatibility aliases or migration, breaking existing configs that use old values.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Semantic enum literals were renamed without backward-compatibility aliases or migration, breaking existing configs that use old values.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/opencode-plugin/src/config.ts, line 40:

<comment>Semantic enum literals were renamed without backward-compatibility aliases or migration, breaking existing configs that use old values.</comment>

<file context>
@@ -34,19 +34,19 @@ const CheckerEnum = z.enum([
 
 /** Output encoding mode for embeddings. */
-const SemanticOutputEncodingEnum = z.enum(["float", "binary", "ubinary", "int8", "uint8"]);
+const SemanticOutputEncodingEnum = z.enum(["float", "base64_int8", "base64_binary"]);
 
 /** Storage strategy for embedding vectors. */
</file context>