Vector search as an optional feature-gated vertical

[nicholasjpaterno]

Summary

TurboVault's built-in semantic search (TF-IDF) is a strong low-resource default and will stay the default. This issue tracks adding true embedding-based semantic search as an optional, feature-gated vertical for larger or more semantically diverse vaults, where dense embeddings meaningfully improve recall over lexical/TF-IDF approaches.

This is not a replacement for TF-IDF. It is an opt-in upgrade, off by default, so the common case (modest vault, zero-config, instant, offline, private) is unaffected.

Prior art

@ForrestThump has a working prototype on his feature/vector-db branch (CPU-friendly embeddings, paragraph-level incremental re-embedding, gated behind a vector-search feature) — excellent reference material. To be clear, this is open for anyone to pick up; no assumption about who authors it, and ForrestThump is very welcome to drive it if he has the bandwidth.

Architecture direction (from a Q2 2026 review)

Design it as a vertical that abstracts the embedder (and optional reranker) behind traits, defaults to a pure-Rust CPU stack, and gates heavier backends behind features.

Tier 1 — default backend (pure-Rust, CPU, edge-friendly)

• Static embeddings via model2vec-rs + hnsw_rs (pure Rust, zero FFI; cross-compiles cleanly to Pi/musl/WASM).
• Model: Potion potion-mxbai-256d-v2 (~71.45 MTEB v2 English, 15k+ sentences/sec on CPU, 256-dim so the index stays small); potion-mxbai-micro (~0.7 MB) for the edge floor.
• Static embeddings have closed most of the gap with small transformers while being orders of magnitude faster and smaller — a good fit for the local-first ethos.

Tier 2 — optional quality upgrade (heavier dependency)

• Quantized transformers via fastembed-rs (ONNX, int8): Qwen3-Embedding-0.6B (Matryoshka 256–2048 dims, Apache-2.0) or Nomic Embed v2 (CPU-purpose-built, single-model dense+sparse hybrid).

Tier 3 — optional reranker (highest quality per cost)

• FlashRank-class on CPU (sub-20 ms for ~50 candidates), or mxbai-rerank-xsmall / jina-reranker-v3 for higher quality.

Hybrid note: TurboVault already has Tantivy BM25. The strongest retrieval path is BM25 + dense + RRF, then optional rerank — mostly assembled from parts that already exist.

Suggested shape

• Lands as crates/plugins/turbovault-plugin-vector-search, behind a vector-search feature, default-off (depends on the Phase 1 provider scaffolding, [Phase 1] Decompose tools.rs into composable providers #28).
• Small / Matryoshka vectors by default (256d) to keep the index cheap.
• Incremental re-embedding driven by the vault-event/hook system (yolo #24) — re-embed only changed content.

Open questions

• Persistence: pure-Rust hnsw_rs (no deps) vs sqlite-vec / usearch (SQL + persistence, but C/C++ FFI).
• Whether to expose reranking from day one or as a follow-up.
• Eventually: is a static backend good and light enough to be promoted toward the default? Decide with real benchmarks, not up front.

Caveats

• MTEB v2 scores are not comparable to v1, and they are self-reported. The decisive test is performance on a real vault (MRR / NDCG on your own corpus).

[nicholasjpaterno]

Reopening this with concrete architecture direction in the updated description, based on a current (Q2 2026) review of static embeddings, quantized transformers, and rerankers. Short version: default to a pure-Rust static-embedding stack, with optional transformer and reranker tiers behind feature flags. Open to anyone who wants to take it on — @ForrestThump, you're very welcome to drive it given your prototype, but no pressure at all given everything else on your plate.

[ForrestThump]

Still interested in this. It ties in well with a local project I have. Once my local project matures a little more, I think I'll naturally gravitate toward a more hardened vector db feature. I'm thinking customizable embedding models so that a Raspberry Pi 4 user can select something lighter than what someone running on a more powerful machine would want. Also offsite inference for this kind of thing is dirt cheap, but has the obvious privacy caveats. I like the idea of providing lots of configurable options, and if it's a totally optional plugin on another repo, I think that adding that complexity and weight could be appropriate.

[nicholasjpaterno]

Phase 2 of the plugin architecture is now concrete in #39 and gives vector search its upstream integration target.

After #39, the compatible shape is: move the vertical under crates/plugins/, add a default-off vector-search feature that includes plugin-api, implement local provider tools and let the host advertise them under a vector_search_* namespace, consume note listing/reads through VaultApi, and use HookBus lag/resync semantics for incremental index maintenance. The vector crate may retain its broader public Rust API for direct non-MCP consumers; the host contract only governs mounting and access to TurboVault internals.

[ForrestThump]

There's now a full working end-to-end prototype on prototype/vector-e2e — building on the feature/vector-db engine but restructured as a compiled-in, default-off plugin module against the plugin API (#39).

What's there:

• The embedded engine (fastembed + usearch + SQLite) promoted to a first-class turbovault-vector crate, embedder + reranker behind traits (fastembed CPU default) — matches the tiered direction here. Chunk-level incremental re-embedding: update_note diffs chunk content hashes and only re-embeds what changed.
• A turbovault-plugin-vector module (default-off vector feature) exposing vector_search / reindex / status / config over the plugin boundary — the core 74 tools are untouched and the feature-off build excludes it entirely.
• Hybrid BM25 + vector with an optional cross-encoder reranker.

Live-validated over a real ~2k-note vault: real embeddings, sensible semantic results (a "deliberation" query surfaces the deliberation/ notes at ~0.72).

What's needed before a PR: the module leans on two small host capabilities I've proposed and prototyped alongside it — a per-vault plugin state dir (#42, where the index persists) and a reliable change-feed (#43, to keep the index current including external edits). Both are implemented on the same branch and validated by this consumer, so the clean path is: settle #42 + #43 (and #41 read_config for module config), then split out a focused engine-crate PR and the module PR. Happy to drive all of it.