Blame
|
1 | --- |
||||||
| 2 | category: reference |
|||||||
| 3 | tags: [tasks, semantic-search, architecture] |
|||||||
|
4 | last_updated: 2026-03-20 |
||||||
|
5 | confidence: high |
||||||
| 6 | --- |
|||||||
| 7 | ||||||||
|
8 | # Semantic Search Architecture — IMPLEMENTED |
||||||
|
9 | |||||||
|
10 | ## Implementation (2026-03-20) |
||||||
|
11 | |||||||
|
12 | Semantic search is fully implemented and operational. All issues listed below have been resolved. |
||||||
|
13 | |||||||
|
14 | ### Architecture |
||||||
| 15 | - **FAISS backend** with `IndexFlatIP`, per-wiki indexes under `/srv/data/faiss/{slug}/` |
|||||||
| 16 | - **ONNX MiniLM-L6-v2 embeddings** (ChromaDB bundled model) |
|||||||
| 17 | - **Multi-tenant via `BackendRegistry`** — lazy per-wiki index creation |
|||||||
| 18 | - **Synchronous lifecycle hooks** — `HookListener` registers `page_saved`, `page_deleted`, `page_renamed` hooks that trigger immediate FAISS upsert/delete. No background worker or queue. |
|||||||
| 19 | - **Auto-reindex** on first wiki access back-fills existing pages |
|||||||
| 20 | - `reindex_all` is per-wiki scoped |
|||||||
|
21 | |||||||
|
22 | ### REST API |
||||||
| 23 | - `GET /api/v1/semantic-search` — query semantic search |
|||||||
| 24 | - `POST /api/v1/reindex` — trigger full reindex for a wiki |
|||||||
| 25 | - `GET /api/v1/reindex/status` — check reindex progress |
|||||||
|
26 | |||||||
|
27 | ### MCP integration |
||||||
| 28 | - `semantic_search` MCP tool calls the REST API |
|||||||
|
29 | |||||||
|
30 | ### Tests |
||||||
| 31 | - Tests exist and pass |
|||||||
|
32 | |||||||
|
33 | ## Resolved issues |
||||||
|
34 | |||||||
|
35 | ### 1. Multi-tenant indexing — RESOLVED |
||||||
| 36 | Per-wiki FAISS directories (`/srv/data/faiss/{slug}/`), per-wiki state managed by `BackendRegistry`. |
|||||||
|
37 | |||||||
|
38 | ### 2. In-process embedding risks — RESOLVED |
||||||
| 39 | Synchronous lifecycle hooks replace the daemon sync thread. No risk of mid-write corruption from SIGTERM killing a background thread. |
|||||||
|
40 | |||||||
|
41 | ### 3. Sync frequency — RESOLVED |
||||||
| 42 | Hook-based updates are immediate on page save/delete/rename. No polling. |
|||||||
| 43 | ||||||||
| 44 | ### 4. FAISS sidecar scalability — DEFERRED |
|||||||
| 45 | Not blocking. Current corpus sizes are well within the estimated ~10K chunk threshold. Can revisit if corpus grows significantly. |
|||||||
| 46 | ||||||||
| 47 | ## Related |
|||||||
| 48 | - [[Tasks/Semantic_Search_Multi_Tenant]] — multi-tenant implementation details |
|||||||
| 49 | - [[Design/Async_Embedding_Pipeline]] — original FAISS + MiniLM design (AWS, archived) |
|||||||