Local Embeddings — Hybrid RAG (Phase 2) Design#

Date: 2026-07-28 Last revised: 2026-07-31 Status: Implemented in the development/desktop Node API; EmbeddingGemma migration and owner-accepted design review applied

Basis: docs/superpowers/specs/2026-07-28-ai-runtime-research.md (§ retrieval: local embeddings default; hybrid FTS5/BM25 + dense fused with RRF; structural chunking; optional rerank). Phase 1 (token budgeting, FTS5, citations, rolling memory, hardening) is already shipped in apps/api.

1. Intent#

Add optional, local, multilingual dense retrieval alongside the existing FTS5/BM25 retrieval so chat/RAG finds paraphrase and conceptual matches, especially across the library and within long articles. Sparse and dense retrieval are independent rankings; dense retrieval must still find and rank a chunk when FTS returns zero hits.

Dense indexing remains opt-in and OFF by default. When disabled, not ready, over budget, or unavailable, retrieval transparently uses today's FTS path. Local reading, token budgeting, and citation validation must never depend on the embedding runtime.

This is an embedding-first, not embedding-dependent design. Embeddings form the shared semantic memory layer for search, related passages, knowledge connections and grounded AI context. They do not replace original content, exact search, explicit rules or user-authored knowledge.

2. Scope (settled with owner)#

The current 5,000-chunk value is a deliberately conservative scan ceiling, not a product-scale claim. Before raising it, benchmark realistic mixed Chinese/English libraries at 10k, 50k, and 100k chunks on supported desktop hardware or introduce a verified native vector index. If the current count exceeds the configured value, or the deadline expires, library retrieval uses pure FTS; it does not substitute recent/starred items as the sole dense candidate pool. Cosine scans run in a worker thread and never on the request thread.

Phase 2 is not a packaged-client feature. apps/luma currently packages only the shared apps/web build and does not run the Node API. Therefore packaged macOS/Windows clients and iOS/Android clients do not receive local embeddings from this design. A desktop Node sidecar—including native onnxruntime-node binaries, signing/notarization, cache paths, process lifecycle, upgrades, and crash recovery—is a separate future effort. Nothing in this document implies mobile availability.

3. Data model, source identity, and index signature#

Use one canonical source selector everywhere:

type EmbeddingSource = {
  sourceKind: "extracted" | "feed";
  text: string;
  hash: string;
};

// Equivalent text choice:
// COALESCE(NULLIF(source_content, ''), content)
function selectEmbeddingSource(item): EmbeddingSource {
  return item.source_content.trim()
    ? {
        sourceKind: "extracted",
        text: item.source_content,
        hash: item.source_content_hash
      }
    : {
        sourceKind: "feed",
        text: item.content,
        hash: item.content_hash
      };
}

Chunks are always built from this selected text. Each row stores the matching source_kind and source_hash; stale checks compare against the current selector result, never blindly against items.content_hash. This also makes the feed-body → extracted-body transition an explicit reindex event.

CREATE TABLE IF NOT EXISTS item_chunks (
  id                   INTEGER PRIMARY KEY AUTOINCREMENT,
  chunk_key            TEXT NOT NULL UNIQUE, -- deterministic canonical chunk id
  item_id              INTEGER NOT NULL,
  ord                  INTEGER NOT NULL,
  text                 TEXT NOT NULL DEFAULT '',
  token_count          INTEGER NOT NULL DEFAULT 0,
  source_kind          TEXT NOT NULL DEFAULT '',
  source_hash          TEXT NOT NULL DEFAULT '',
  index_signature      TEXT NOT NULL DEFAULT '',
  paragraph_start      INTEGER NOT NULL DEFAULT 0,
  paragraph_end        INTEGER NOT NULL DEFAULT 0,
  char_start           INTEGER NOT NULL DEFAULT 0,
  char_end             INTEGER NOT NULL DEFAULT 0,
  heading_path_json    TEXT NOT NULL DEFAULT '[]',
  embedding            BLOB,
  dim                  INTEGER NOT NULL DEFAULT 0,
  created_at           TEXT NOT NULL,
  FOREIGN KEY(item_id) REFERENCES items(id) ON DELETE CASCADE,
  UNIQUE(item_id, ord)
);
CREATE INDEX IF NOT EXISTS idx_item_chunks_item ON item_chunks(item_id);
CREATE INDEX IF NOT EXISTS idx_item_chunks_current
  ON item_chunks(item_id, source_kind, source_hash, index_signature);

chunk_key is deterministically derived from item id, source kind/hash, active index signature, and ordinal. It is the common identity used by sparse and dense rankings. Paragraph indexes, character offsets, and heading paths map a chunk back to Phase-1 paragraph evidence and stable citations.

The active index_signature is a canonical serialization or hash containing:

• independent embeddingsVersion / pipeline version;
• provider;
• model id and immutable model revision;
• raw model output dimension and persisted vector dimension;
• active tokenizer and hard input limit;
• chunk target, minimum, maximum, and overlap token counts;
• model output and pooling contract;
• L2-normalization rule;
• runtime/model dtype and persisted vector dtype;
• document and query prefixes.

For the pinned default, the signature records local provider, onnx-community/embeddinggemma-300m-ONNX at revision 5090578d9565bb06545b4552f76e6bc2c93e4a66, q4 model weights, model-provided 768-dimensional sentence_embedding output, 256-dimensional Matryoshka truncation, L2 normalization, Float32 storage, a 512-token application hard limit, title: none | text: documents, and task: search result | query: queries.

A chunk is current only when both:

1. source_kind and source_hash equal the current selector result; and
2. index_signature equals the complete active signature.

Retrieval filters by those current values. It must never compare or fuse vectors from different signatures. Add an independent embeddingsVersion to apps/api/src/services/cache-policy.ts; app/package releases alone do not invalidate embeddings.

Vector wire format#

• Persist normalized vectors as little-endian Float32 bytes.
• Require blob.length === dim * 4.
• Reject a missing/wrong dimension and any non-finite component.
• Decode with DataView or an alignment-safe copied ArrayBuffer; do not construct Float32Array directly over a potentially unaligned Node Buffer.
• L2-normalize document and query vectors before storage/scoring and reject unusable zero/non-finite vectors.

Chunks are derived data, not user data. Keep item_chunks excluded from Sync and backup. The user-owned embedding_settings row is included in backup.

4. Settings and provider boundary#

V1 is local-only. Add a singleton embedding_settings table with:

• provider TEXT NOT NULL DEFAULT 'local';
• model, defaulting to the pinned model id;
• model_revision, defaulting to the pinned immutable revision;
• enabled;
• updated_at.

Keep the provider column for forward compatibility, but reject anything other than local in v1. Do not add or persist an embedding API key in v1. Add embedding_settings to backup-service.ts USER_TABLES; it contains no plaintext secret. Keep item_chunks excluded.

OpenAI-compatible and Ollama embeddings are deferred because they require embedding-specific payload handling, a second credential, and explicit disclosure that indexing may upload most of the corpus. If remote embeddings are added later:

• use a separate session-only embedding key supplied from OS secure storage, following the existing ai-settings-service.ts session-key rule;
• never persist the key in SQLite or include it in backup;
• require versioned, embedding-specific consent before both document indexing and query embedding;
• cancel indexing and stop query embedding immediately on consent revocation or provider change.

Existing general AI consent is not sufficient authorization for bulk embedding uploads.

5. Chunking, runtime lifecycle, and atomic indexing#

Tokenizer-based chunking#

chunking.ts splits the canonical selected source on headings, paragraphs, and sentences, then packs with the active model tokenizer, not the Phase-1 chat token estimator. No formatted document may exceed the model's hard 512-token input limit. The configured target/minimum/overlap values are signature inputs; initial values are tuned by tests and benchmarks rather than treated as a model-independent 350–600-token range.

The formatter prepends title: none | text: before token measurement and document embedding. Query embedding prepends task: search result | query:. CJK text has higher token density, so character counts are not accepted as a substitute for tokenizer counts. Chunk output is deterministic and records paragraph range, character offsets, and heading path.

Local runtime lifecycle#

embedding-service.ts exposes async batch embedding and:

• dynamically imports @huggingface/transformers only after local embeddings are enabled; the native runtime is not loaded while disabled;
• uses an explicit application cache directory such as <app-cache>/embeddings/models;
• reports model-download progress and treats offline/download failure as a recoverable background state;
• uses a single-flight initialization promise;
• clears the failed initialization promise so a later retry can succeed;
• disposes the old pipeline before/when changing model or revision;
• does not load the model during API startup;
• does not initiate model loading on the query critical path—retrieval uses the already-ready pipeline or falls back immediately to FTS.

The pinned model exposes a dedicated 768-dimensional sentence_embedding. Rinwa truncates it to the first 256 Matryoshka dimensions and then performs L2 normalization. Provider/model batch limits are explicit; queue batches never exceed the active runtime's measured safe limit.

Queue and atomic replacement#

embedding-index-queue.ts is throttled, resumable, and restart-safe:

1. Select an item and read { sourceKind, sourceHash, indexSignature }.
2. Chunk and compute all embeddings outside a database transaction.
3. Begin a transaction and re-read the canonical source selector plus active signature.
4. If any fingerprint component changed, roll back/discard the computed result and re-enqueue the item.
5. Otherwise replace that item's rows atomically with delete + insert.

The previous successful index remains visible until the replacement transaction commits. A failed inference or insert leaves it intact. Retrieval only reads rows matching the current selector fingerprint and active signature, so obsolete chunks are never used. UNIQUE(item_id, ord) prevents duplicate ordinal rows.

Initial CPU inference concurrency is exactly 1. The queue has bounded batches, retry with exponential backoff and jitter, persisted/recoverable pending work, restart recovery, and cancellation when embeddings are disabled. Future remote providers must also cancel on consent revocation/provider change. New or updated items are re-enqueued, and enabling embeddings starts a throttled backfill without blocking reading.

6. Independent hybrid retrieval and async chat integration#

hybrid-retrieval.ts exposes:

async function retrieve(query, scope): Promise<RankedPassage[]>

It preserves the Phase-1 passage return shape consumed by token budgeting and citation validation, but retrieval itself becomes async.

Common chunk-id space#

The current FTS implementation ranks articles and paragraphs, while dense retrieval ranks chunks. Before RRF, map sparse hits to canonical chunk_key values:

• use the existing article_search, searchItems, and searchArticlePassages sparse ranking;
• map each ranked paragraph to every deterministic chunk whose stored/transient paragraph or character range overlaps it;
• give each mapped chunk its best sparse rank, with deterministic tie-breaking by item id and ordinal;
• retain sparse-only transient chunks for current items that are not yet embedded, using the same deterministic chunker and chunk_key formula.

Dense rows use the same chunk_key. RRF therefore operates only on one canonical id space, deduplicates by chunk_key, and retains rank origin (sparse, dense, or both) for diagnostics/tests. Ranked chunks are converted back to Phase-1 article/paragraph evidence through their offsets and heading metadata; public citation ids remain the existing [P#] / [S#] ids assigned after final selection.

Independent rankings#

For each request:

1. Produce the sparse FTS/BM25 ranking.
2. Independently embed the query if the already-ready local pipeline is available.
3. Produce the dense cosine ranking without restricting it to FTS candidates:
   • within-article scope scans all current chunks for that article;
   • library scope scans all current indexed chunks in a worker thread when the count is within denseScanMaxChunks.
4. RRF the union of sparse and dense chunk rankings: score = Σ 1 / (k + rank), initially k = 60.
5. Apply deterministic ties (rrfScore, best source rank, item id, ordinal), deduplicate by chunk_key, and return the top evidence passages.

Recent/starred items may be ranking features later, but never define the sole dense pool. A semantic chunk with zero FTS hits can enter through the dense ranking and win a fused position.

Deadline and Phase-1 contracts#

Use a short overall retrieval deadline, initially 250 ms for a warm model and supported library size. Worker timeout, model failure, disabled/not-ready runtime, scan-budget overflow, or malformed vectors returns the pure-FTS ranking. Retrieval failure must not fail the conversation turn.

Today prepareConversationTurn and buildContextMessages are synchronous and shared by non-streaming and streaming routes. Make the context build and prepareConversationTurn async, await them in both replyToConversation and the streaming route, and keep their resolved return shape unchanged. Complete retrieval before the existing evidence-token allocation; then run the current token budgeting and citation-id validation unchanged over the selected passages.

7. Privacy and failure behavior#

• The local provider sends no article, chunk, or query text externally and needs no data-sharing consent.
• The one-time model download is a network action and requires a clear prompt; offline mode simply leaves indexing waiting.
• Never log article bodies, chunk text, prompts, vectors, credentials, or private cache paths.
• Model download, initialization, inference, scan deadline, and malformed-vector failures are recoverable states. Existing content and pure FTS remain usable.
• Disabling embeddings cancels queued work, ignores retained rows, and does not delete the previous successful index.
• Remote/Ollama providers and all associated upload consent are deferred as specified in §4.

8. Web UI#

In AI settings, add a development/desktop-Node-API-only “Semantic search (local embeddings)” section with:

• enable toggle;
• pinned model id and revision (model remains a settings value);
• one-time download disclosure/confirmation;
• download and indexing progress;
• subtle paused/retry/failure state;
• re-index action;
• clear FTS-only fallback explanation.

Do not show a remote provider selector in v1. The shared web UI must gate this section on an API capability response. Packaged Luma clients, which do not run the Node API, omit it or state that the capability is unavailable; they must not appear to support local embeddings. Keep i18n in all four locales.

9. Testing (API, node:test)#

• chunking.test.ts: structural boundaries; active-tokenizer target/overlap; hard 512-token limit including the active document prefix; deterministic output; paragraph, character, and heading metadata; CJK token density.
• embedding-service.test.ts: dynamic provider dispatch; pinned model configuration; batching; dedicated sentence_embedding output; 768d to 256d Matryoshka truncation; L2 normalization; mocked local model with no real download in CI.
• Vector codec tests: little-endian Float32 round-trip; alignment-safe decode; exact dim * 4 length; wrong dimension; non-finite/zero vector rejection.
• hybrid-retrieval.test.ts: a true dense-only semantic match with zero FTS hits still ranks; FTS-only matches remain; independent candidate sets; RRF rank origin, union/dedup, deterministic ties, and citation-offset mapping.
• Retrieval failure tests: disabled/not-ready model, model failure, worker timeout, scan-budget overflow, and malformed BLOB all fall back to pure FTS without changing token budgeting or citation validation.
• embedding-settings.test.ts: local-only CRUD; provider defaults to local; non-local providers rejected; backup includes settings and no plaintext secret. When remote providers are implemented, test versioned consent for both indexing and query embedding, session-only credentials, revocation, and provider-change cancellation.
• embedding-index-queue.test.ts: feed-body → extracted-body transition; model/provider/full-signature change; item update during inference; fingerprint recheck/re-enqueue; atomic replacement; transaction rollback; uniqueness; retry/backoff; restart recovery; batch limits; cancellation; previous successful rows retained on failure.
• Migration tests: additive creation, existing databases remain readable, UNIQUE(item_id, ord), and ON DELETE CASCADE.
• Cache-policy tests: independent embeddingsVersion; stale detection uses the current canonical selector plus complete index signature.

10. Dependencies, packaging, and cache paths#

Add @huggingface/transformers to apps/api. Fetch the pinned model revision only after enable/confirmation, store it under the explicit app cache directory, and report progress. The package and native runtime remain unloaded while embeddings are disabled.

This dependency belongs to the development/desktop Node API only. It is not bundled into the current apps/luma package, which builds and serves apps/web/dist and does not launch the Node API. Packaged desktop support requires the separate sidecar effort described in §2. Mobile support requires a different native/runtime design and remains deferred.

11. Correct decisions to preserve and deferrals#

Correct decisions to preserve#

• Dense indexing is opt-in and OFF by default.
• Pure FTS is always the fallback.
• The local provider sends nothing externally.
• Derived vectors are excluded from Sync and backup; user settings are backed up without plaintext secrets.
• Embeddings have an independent cache generation/signature; app releases do not invalidate them.
• Indexing is throttled, resumable, bounded, and preserves the previous successful index until replacement succeeds.
• CI never performs a real model download.
• Cross-encoder reranking and native vector extensions remain deferred.

Resolved defaults#

• Default model: onnx-community/embeddinggemma-300m-ONNX@5090578d9565bb06545b4552f76e6bc2c93e4a66.
• Formatting: task: search result | query: for queries and title: none | text: for documents.
• Model input limit: 512 tokenizer tokens including the prefix.
• Output/normalization: dedicated 768d sentence_embedding, truncated to 256 Matryoshka dimensions, then L2-normalized.
• Model/runtime dtype: q4 ONNX weights; persisted embeddings are little-endian Float32.
• Dense scope: all article chunks or all library chunks within the measured worker-thread budget—never FTS top-50 narrowing.
• Initial fusion: RRF k = 60, then top 8–10 evidence passages, subject to benchmark tuning without changing the independent-ranking invariant.

Deferred#

• OpenAI-compatible and Ollama embedding providers, their separate credentials, endpoint payloads, bulk-upload disclosures, and versioned consent.
• Packaged desktop Node sidecar and all mobile execution.
• Notes/highlights embedding; v1 indexes canonical article source text only.
• Native vector extensions, approximate nearest-neighbor indexes, cross-encoder reranking, and RAPTOR/hierarchical summaries.
• Increasing the supported chunk ceiling beyond measured desktop latency targets.

Revisions applied (from design review)#

1. Independent dense retrieval and true dense-only matches: §§1, 2, 6, 9, 11.
2. Canonical source selector and current-hash stale checks: §§3, 5, 9.
3. Complete index signature and independent cache generation: §§3, 5, 9, 11.
4. Common chunk-id RRF and citation-offset mapping: §§3, 6, 9.
5. Development Node API packaging boundary: §§2, 8, 10, 11.
6. Local-only v1, deferred remote providers, secure future credentials, and backup scope: §§2, 4, 7, 9, 11.
7. Atomic fingerprint-checked reindex and previous-index preservation: §§3, 5, 9, 11.
8. Async retrieval, both route paths, deadline, and FTS fallback: §§6, 9.
9. Active-tokenizer chunking, hard model limit, prefixes, and CJK density: §§2, 3, 5, 9, 11.
10. Measured chunk/latency budget and worker-thread scans: §§2, 6, 9, 11.
11. Little-endian Float32 BLOB validation and safe decode: §§3, 9, 11.
12. Dynamic runtime lifecycle and explicit model cache: §§5, 8, 10.
13. Queue resilience, cancellation, rollback, migration, and expanded tests: §§5, 7, 9.
14. Pinned default model and revision: §§2, 3, 5, 10, 11.