category: reference
tags: [meta, design, prd, semantic-search]
last_updated: 2026-03-12
confidence: high

Original PRD Semantic Search

This page is part of the original single-tenant PRD, split across five wiki pages: Design/Research Wiki | Design/Rest Api | Design/Semantic Search | Design/Mcp Server | Design/Note_Schema

Component 2: Chroma Semantic Search Plugin

Goal

Maintain a vector index of all wiki pages in ChromaDB, enabling semantic/similarity search. When a page is created, updated, or deleted, the index is updated automatically.

Implementation approach

This should hook into Otterwiki's page save/delete lifecycle. Again, investigate whether the plugin hook system supports after_save / after_delete style hooks.

• If hooks exist for page lifecycle events: Build as a plugin.
• If not: Add hook calls into the Page.save() and Page.delete() methods in wiki.py, and build the Chroma logic as a plugin that registers for those hooks. Or, alternatively, have the API plugin handle indexing as a side effect of PUT/DELETE operations, and add a /api/v1/reindex endpoint for bulk rebuild.

ChromaDB configuration

• Collection name: otterwiki_pages
• Embedding: Use Chroma's default all-MiniLM-L6-v2 sentence-transformer (runs locally, no API key needed, small footprint). Note: verify exact max sequence length at implementation time — if it's 256 tokens, the chunking approach below handles it correctly regardless.
• Metadata stored per chunk: {page_path, page_name, category, tags, last_updated, chunk_index} — extracted from YAML frontmatter. The page_path and chunk_index fields are used for deduplication and reassembly.

Chunking strategy

Pages are split into overlapping chunks for embedding. Each chunk is stored as a separate Chroma document. This ensures semantic search quality is independent of page length — a 300-word note and a 1500-word note are both fully indexed.

Chunking algorithm:

1. Strip YAML frontmatter from content (metadata is stored separately, not embedded).
2. Split on paragraph boundaries (double newline \n\n).
3. Accumulate paragraphs into chunks of ~200 tokens (~150 words). If a single paragraph exceeds 200 tokens, split it at sentence boundaries (. followed by a capital letter or newline).
4. Add ~50 tokens of overlap between adjacent chunks — repeat the last 1–2 sentences of the previous chunk at the start of the next. This prevents concepts spanning a boundary from being lost.
5. Assign each chunk an ID: {page_path}::chunk_{index} (e.g., Trends/Iran Attrition Strategy::chunk_0).

Short pages: If the entire page body (after frontmatter) is under 200 tokens, store it as a single chunk. No need to split.

Example: A 600-word page might produce 4 chunks of ~150 words each, with ~35 words of overlap between adjacent chunks.

def chunk_page(content: str, target_tokens: int = 200, overlap_tokens: int = 50) -> list[str]:
    """Split page content into overlapping chunks for embedding.

    Args:
        content: Page body text (frontmatter already stripped)
        target_tokens: Approximate tokens per chunk (~0.75 words per token)
        overlap_tokens: Approximate overlap between adjacent chunks

    Returns:
        List of chunk strings
    """
    # Implementation: split on paragraphs, accumulate to target size,
    # carry overlap from previous chunk. Fall back to sentence splitting
    # for oversized paragraphs.

Search result deduplication

Semantic search queries Chroma for the top n * 3 chunks (to account for multiple chunks from the same page), then deduplicates by page_path, keeping the best-matching (lowest distance) chunk per page, and returns the top n unique pages.

The snippet in the search response is the text of the best-matching chunk for that page, truncated to ~150 characters. This means the snippet is contextually relevant to the query, not just the page's opening paragraph.

API endpoints (added to the REST API)

Index maintenance

• On PUT /api/v1/pages/<path> (create/update): delete all existing chunks for that page path, then re-chunk and insert. This is simpler and safer than trying to diff chunks.
• On DELETE /api/v1/pages/<path>: delete all chunks for that page path.
• On page save via Otterwiki web UI: if hooks are available, also update Chroma. If not, run a periodic sync (see below).

Fallback: periodic sync

If lifecycle hooks are unavailable or unreliable, implement a background sync that runs every 60 seconds:

1. git log --since=<last_sync_time> --name-only to find changed files
2. Re-index only those files in Chroma
3. Update last_sync_time

State persistence: last_sync_time is stored in a small file at /app-data/chroma_sync_state.json containing {"last_sync": "2026-03-09T14:22:00Z"}. This persists across container restarts.

First boot / missing state: If the state file doesn't exist, or if the Chroma collection is empty, perform a full reindex of all pages. This is the same operation as POST /api/v1/reindex.

Race condition mitigation: If a page is saved via the web UI and queried via semantic search within the sync window (up to 60 seconds), the search may return stale results. This is acceptable — the full-text search endpoint (/api/v1/search) reads directly from Git and is always current. The MCP server can fall back to full-text search when recency matters.

Implementation: Use a background thread started on Flask app initialization (e.g., threading.Timer with a recurring callback), NOT a cron job. This keeps everything in one process and avoids external dependencies.

This ensures edits made via the web UI are reflected in semantic search even without hooks.