Blame
|
1 | --- |
||||||
|
2 | ## How to read this document |
||||||
| 3 | ||||||||
| 4 | - **Dependencies** list task IDs that must be complete before this task starts |
|||||||
| 5 | - **Parallel group** identifies tasks that can run simultaneously within a phase |
|||||||
| 6 | - **Target** identifies which repo and branch the work goes into |
|||||||
| 7 | - Tasks are numbered `P{phase}-{sequence}` (e.g., P0-3) |
|||||||
| 8 | - Acceptance criteria are binary — pass or fail, no judgment calls |
|||||||
| 9 | ||||||||
| 10 | --- |
|||||||
| 11 | ||||||||
| 12 | ## Phase 1: Single-User Serverless Wiki |
|||||||
| 13 | ||||||||
| 14 | **Goal:** Port the existing Otterwiki + API + semantic search + MCP stack to run on Lambda + EFS. Single user, single wiki, no multi-tenancy. |
|||||||
| 15 | ||||||||
|
16 | **Status: COMPLETE.** All tasks done, gate passed 2026-03-13. See [[Dev/Phase_1_Gate_Results]] for full validation results. |
||||||
| 17 | ||||||||
|
18 | ### P1-1: Mangum Adapter for Otterwiki |
||||||
| 19 | ||||||||
| 20 | **Parallel group:** Can start during Phase 0 (upstream work) |
|||||||
| 21 | **Dependencies:** None (upstream contribution, independent of wikibot-io infra) |
|||||||
| 22 | **Target:** `otterwiki` fork, PR to `main` |
|||||||
|
23 | **Status: COMPLETE** |
||||||
|
24 | |||||||
| 25 | **Description:** |
|||||||
| 26 | Add Lambda compatibility to Otterwiki via the Mangum adapter. Mangum wraps Flask/WSGI apps for Lambda invocation. The adapter should be opt-in — Otterwiki continues to work as a normal Flask app when not on Lambda. |
|||||||
| 27 | ||||||||
| 28 | Investigate Otterwiki's filesystem assumptions beyond the git repo: config files, static assets, session storage, temporary files. Document what needs to live on EFS vs. what can be ephemeral. |
|||||||
| 29 | ||||||||
|
30 | **Note:** Ended up using apig-wsgi (not Mangum) because Mangum is ASGI-only since v0.15+ and Otterwiki is a WSGI/Flask app. Custom `lambda_init.py` handler in `wikibot-io/app/otterwiki/`. |
||||||
| 31 | ||||||||
|
32 | **Deliverables:** |
||||||
|
33 | - `wikibot-io/app/otterwiki/lambda_init.py` — apig-wsgi handler with WSGI middleware for non-ASCII PATH_INFO |
||||||
|
34 | - Documentation of filesystem assumptions and EFS requirements |
||||||
|
35 | - No changes to existing Otterwiki behavior |
||||||
|
36 | |||||||
| 37 | **Acceptance criteria:** |
|||||||
|
38 | - [x] Otterwiki can be invoked via Lambda handler (apig-wsgi, not Mangum) |
||||||
| 39 | - [x] Existing test suite still passes |
|||||||
| 40 | - [x] Static assets served correctly |
|||||||
| 41 | - [x] Config file loading works from EFS path |
|||||||
| 42 | - [x] Git repo operations work with EFS-mounted repo path |
|||||||
| 43 | - [x] No import errors when running outside Lambda |
|||||||
|
44 | |||||||
| 45 | --- |
|||||||
| 46 | ||||||||
| 47 | ### P1-2: FAISS Backend for Semantic Search |
|||||||
| 48 | ||||||||
| 49 | **Parallel group:** Can start during Phase 0 (upstream work) |
|||||||
| 50 | **Dependencies:** None (upstream contribution, independent of wikibot-io infra) |
|||||||
| 51 | **Target:** `otterwiki-semantic-search` repo, PR to `main` |
|||||||
|
52 | **Status: COMPLETE** |
||||||
|
53 | |||||||
| 54 | **Description:** |
|||||||
| 55 | Add FAISS as an alternative vector store backend alongside the existing ChromaDB backend. FAISS stores indexes as files on disk (suitable for EFS). The backend is selected via configuration. |
|||||||
| 56 | ||||||||
| 57 | The embedding function must be pluggable: the existing `all-MiniLM-L6-v2` sentence-transformer for local/ChromaDB use, and a Bedrock `titan-embed-text-v2` adapter for Lambda use. The Bedrock adapter is a new addition but should be a clean interface implementation, not Lambda-specific. |
|||||||
| 58 | ||||||||
| 59 | Reference the PRD's FAISS details: `IndexFlatIP`, sidecar metadata in `embeddings.json`, search deduplication by page. |
|||||||
| 60 | ||||||||
| 61 | **Deliverables:** |
|||||||
| 62 | - `otterwiki_semantic_search/backends/faiss_backend.py` — FAISS index management (create, upsert, delete, search) |
|||||||
| 63 | - `otterwiki_semantic_search/backends/chroma_backend.py` — extracted from existing code |
|||||||
| 64 | - `otterwiki_semantic_search/backends/base.py` — abstract backend interface |
|||||||
| 65 | - `otterwiki_semantic_search/embeddings/bedrock.py` — Bedrock titan-embed-text-v2 adapter |
|||||||
| 66 | - `otterwiki_semantic_search/embeddings/base.py` — abstract embedding interface |
|||||||
| 67 | - Configuration switch: `VECTOR_BACKEND=faiss|chroma`, `EMBEDDING_MODEL=local|bedrock` |
|||||||
| 68 | - Unit tests for FAISS backend (using `/tmp` for index files) |
|||||||
| 69 | - Unit tests for Bedrock embedding adapter (mocked) |
|||||||
| 70 | - Existing ChromaDB tests still pass |
|||||||
| 71 | ||||||||
| 72 | **Acceptance criteria:** |
|||||||
|
73 | - [x] FAISS backend stores/retrieves/deletes vectors correctly |
||||||
| 74 | - [x] Sidecar metadata (`embeddings.json`) maps index positions to page paths and chunk text |
|||||||
| 75 | - [x] Search deduplicates by page, returns top N unique pages |
|||||||
| 76 | - [x] Bedrock adapter produces vectors of correct dimensionality |
|||||||
| 77 | - [x] Backend is selected by config, defaults to ChromaDB for backward compatibility |
|||||||
| 78 | - [x] Existing tests pass without modification |
|||||||
| 79 | - [x] New tests cover FAISS CRUD and search operations |
|||||||
|
80 | |||||||
| 81 | --- |
|||||||
| 82 | ||||||||
| 83 | ### P1-3: Otterwiki on Lambda |
|||||||
| 84 | ||||||||
| 85 | **Parallel group:** Phase 1 core |
|||||||
| 86 | **Dependencies:** P0-2 (EFS + Lambda infra), P1-1 (Mangum adapter) |
|||||||
| 87 | **Target:** `wikibot-io` repo, `feat/P1-3-otterwiki-lambda` |
|||||||
|
88 | **Status: COMPLETE** |
||||||
|
89 | |||||||
| 90 | **Description:** |
|||||||
|
91 | Deploy Otterwiki to Lambda using the apig-wsgi adapter from P1-1. Configure it to use a git repo on EFS. Verify the web UI works end-to-end: browse pages, create pages, edit pages, view history. |
||||||
|
92 | |||||||
| 93 | **Deliverables:** |
|||||||
| 94 | - Pulumi Lambda function for Otterwiki with EFS mount |
|||||||
| 95 | - Otterwiki configuration for Lambda environment (EFS paths, settings) |
|||||||
| 96 | - API Gateway route for web UI (`/{path+}`) |
|||||||
| 97 | - Integration test: create page via web UI API, read it back |
|||||||
| 98 | ||||||||
| 99 | **Acceptance criteria:** |
|||||||
|
100 | - [x] Otterwiki web UI loads in a browser |
||||||
| 101 | - [x] Can create, edit, and delete pages |
|||||||
| 102 | - [x] Page history works |
|||||||
| 103 | - [x] Static assets (CSS, JS) load correctly |
|||||||
| 104 | - [x] Git repo on EFS persists across invocations |
|||||||
|
105 | |||||||
| 106 | --- |
|||||||
| 107 | ||||||||
| 108 | ### P1-4: REST API on Lambda |
|||||||
| 109 | ||||||||
| 110 | **Parallel group:** Phase 1 (parallel with P1-5, P1-6 once P1-3 lands) |
|||||||
| 111 | **Dependencies:** P1-3 |
|||||||
| 112 | **Target:** `wikibot-io` repo, `feat/P1-4-api-lambda` |
|||||||
|
113 | **Status: COMPLETE** |
||||||
|
114 | |||||||
| 115 | **Description:** |
|||||||
| 116 | Deploy the existing `otterwiki-api` plugin on Lambda alongside Otterwiki. Verify all API endpoints work. The API plugin is already a Flask blueprint — it should load naturally in the Lambda environment. |
|||||||
| 117 | ||||||||
| 118 | **Deliverables:** |
|||||||
| 119 | - API plugin installed and loaded in Lambda Otterwiki instance |
|||||||
| 120 | - API Gateway routes for `/api/v1/*` |
|||||||
| 121 | - Integration tests: CRUD pages via API, search, link graph, changelog |
|||||||
| 122 | ||||||||
| 123 | **Acceptance criteria:** |
|||||||
|
124 | - [x] All existing API endpoints respond correctly |
||||||
| 125 | - [x] API key auth works via env var |
|||||||
| 126 | - [x] WikiLink index builds on startup |
|||||||
| 127 | - [x] Full-text search returns results |
|||||||
| 128 | - [x] Page create/read/update/delete cycle works |
|||||||
| 129 | - [x] Link graph queries return correct data |
|||||||
| 130 | ||||||||
| 131 | **Note:** Minor issue found during gate: `GET /pages/{path}/links` returns 404 due to Flask route priority. The MCP `get_links` tool works via a different URL pattern. Low priority fix. |
|||||||
|
132 | |||||||
| 133 | --- |
|||||||
| 134 | ||||||||
| 135 | ### P1-5: MCP Server on Lambda |
|||||||
| 136 | ||||||||
| 137 | **Parallel group:** Phase 1 (parallel with P1-4, P1-6) |
|||||||
| 138 | **Dependencies:** P1-3, P0-7 (MCP + WorkOS on Lambda) |
|||||||
| 139 | **Target:** `wikibot-io` repo, `feat/P1-5-mcp-lambda` |
|||||||
|
140 | **Status: COMPLETE** |
||||||
|
141 | |||||||
| 142 | **Description:** |
|||||||
| 143 | Deploy the existing `otterwiki-mcp` server on Lambda. Adapt from SSE to Streamable HTTP transport. The MCP server calls the REST API (P1-4) internally. Auth via WorkOS OAuth (from P0-7). |
|||||||
| 144 | ||||||||
|
145 | Decision: Separate Lambdas. Otterwiki needs VPC + EFS; MCP is stateless HTTP proxy. Coupling them would add VPC complexity to MCP for no benefit. |
||||||
|
146 | |||||||
| 147 | **Deliverables:** |
|||||||
|
148 | - MCP server deployed on separate Lambda (stateless, no VPC) |
||||||
| 149 | - Streamable HTTP transport via `json_response=True` (API Gateway doesn't support SSE) |
|||||||
|
150 | - All 12 MCP tools functional |
||||||
| 151 | - Integration test: read_note, write_note, search_notes via MCP protocol |
|||||||
| 152 | ||||||||
| 153 | **Acceptance criteria:** |
|||||||
|
154 | - [x] All MCP tools return correct results |
||||||
| 155 | - [x] OAuth authentication works (WorkOS AuthKit, fixed in P1-9) |
|||||||
| 156 | - [x] Bearer token authentication works |
|||||||
| 157 | - [x] Streamable HTTP transport functions correctly |
|||||||
| 158 | - [x] Architecture decision (same vs. separate Lambda) documented with rationale |
|||||||
|
159 | |||||||
| 160 | --- |
|||||||
| 161 | ||||||||
| 162 | ### P1-6: Semantic Search on Lambda |
|||||||
| 163 | ||||||||
| 164 | **Parallel group:** Phase 1 (parallel with P1-4, P1-5) |
|||||||
| 165 | **Dependencies:** P1-3, P1-2 (FAISS backend) |
|||||||
| 166 | **Target:** `wikibot-io` repo, `feat/P1-6-semantic-search-lambda` |
|||||||
|
167 | **Status: COMPLETE** |
||||||
|
168 | |||||||
| 169 | **Description:** |
|||||||
| 170 | Deploy the semantic search plugin on Lambda using the FAISS backend from P1-2. FAISS index stored on EFS alongside the git repo. Embedding via Bedrock `titan-embed-text-v2` (requires a VPC interface endpoint for Bedrock — add to Pulumi if not already present). |
|||||||
| 171 | ||||||||
| 172 | **Deliverables:** |
|||||||
| 173 | - Semantic search plugin deployed with FAISS backend |
|||||||
|
174 | - Bedrock VPC endpoint added to Pulumi |
||||||
|
175 | - FAISS index persists on EFS |
||||||
|
176 | - FAISS stripped to AVX2-only to fit 250MB Lambda package limit (163MB total) |
||||||
|
177 | |||||||
| 178 | **Acceptance criteria:** |
|||||||
|
179 | - [x] Semantic search endpoint returns relevant results |
||||||
| 180 | - [x] FAISS index persists across Lambda invocations |
|||||||
| 181 | - [x] Reindex endpoint works (full rebuild from repo) |
|||||||
| 182 | - [x] Bedrock embedding calls succeed from VPC Lambda |
|||||||
|
183 | |||||||
| 184 | --- |
|||||||
| 185 | ||||||||
| 186 | ### P1-7: Routing and TLS |
|||||||
| 187 | ||||||||
| 188 | **Parallel group:** Phase 1 |
|||||||
| 189 | **Dependencies:** P1-3 |
|||||||
| 190 | **Target:** `wikibot-io` repo, `feat/P1-7-routing-tls` |
|||||||
|
191 | **Status: COMPLETE** |
||||||
|
192 | |||||||
| 193 | **Description:** |
|||||||
| 194 | Configure API Gateway routing for all endpoints: web UI, REST API, MCP. Set up a custom domain with TLS via ACM. Route 53 DNS record pointing to API Gateway. |
|||||||
| 195 | ||||||||
| 196 | **Deliverables:** |
|||||||
| 197 | - `infra/components/api_gateway.py` — routes for web UI, API, MCP |
|||||||
| 198 | - `infra/components/dns.py` — Route 53 record + ACM certificate |
|||||||
| 199 | - Pulumi outputs for endpoint URLs |
|||||||
| 200 | ||||||||
| 201 | **Acceptance criteria:** |
|||||||
|
202 | - [x] `https://dev.wikibot.io/` serves Otterwiki web UI |
||||||
| 203 | - [x] `https://dev.wikibot.io/api/v1/health` returns 200 |
|||||||
| 204 | - [x] `https://dev.wikibot.io/mcp` accepts MCP connections |
|||||||
| 205 | - [x] TLS certificate valid and auto-renewing (ACM) — TLSv1.3, Amazon RSA 2048, expires 2026-09-26 |
|||||||
| 206 | - [x] All routes pass through to correct Lambda handlers |
|||||||
|
207 | |||||||
| 208 | --- |
|||||||
| 209 | ||||||||
| 210 | ### P1-8: Phase 1 E2E Test |
|||||||
| 211 | ||||||||
| 212 | **Parallel group:** Phase 1 (final — all other P1 tasks must complete) |
|||||||
| 213 | **Dependencies:** P1-4, P1-5, P1-6, P1-7 |
|||||||
| 214 | **Target:** `wikibot-io` repo, `feat/P1-8-e2e` |
|||||||
|
215 | **Status: COMPLETE** (automated smoke test + manual gate validation) |
||||||
|
216 | |||||||
| 217 | **Description:** |
|||||||
| 218 | End-to-end test covering the full single-user workflow: browser creates a page, API reads it, MCP reads it, semantic search finds it. |
|||||||
| 219 | ||||||||
| 220 | **Deliverables:** |
|||||||
|
221 | - `tests/test_e2e.py` — automated smoke test (API CRUD, search, semantic search endpoint checks, MCP initialize) |
||||||
| 222 | - Manual gate validation documented in [[Dev/Phase_1_Gate_Results]] |
|||||||
| 223 | ||||||||
| 224 | **Note on test coverage:** The automated test (`tests/test_e2e.py`) is a per-subsystem smoke test, not a cross-cutting E2E. The cross-write-path criteria below were validated manually during the Phase 1 gate by human + agent testing, not by the automated test. MCP tests are `xfail` (bearer token not configured in test environment). |
|||||||
|
225 | |||||||
| 226 | **Acceptance criteria:** |
|||||||
|
227 | - [x] Create page via Otterwiki web UI → readable via API and MCP — *validated manually during gate* |
||||||
| 228 | - [x] Create page via API → visible in web UI and searchable — *automated: API CRUD; manual: web UI visibility* |
|||||||
| 229 | - [x] Write via MCP → visible in web UI and API — *validated manually during gate* |
|||||||
| 230 | - [x] Semantic search finds pages by meaning, not just keywords — *automated: endpoint smoke; manual: relevance verified (distances 0.46–0.79)* |
|||||||
| 231 | - [x] Git history shows correct authorship for all write paths — *validated by gate agent: "Claude (MCP)" confirmed* |
|||||||
|
232 | |||||||
| 233 | ### P1-9: Self Hosting |
|||||||
| 234 | ||||||||
| 235 | **Dependencies**: P1-8 |
|||||||
|
236 | **Target:** `wikibot-io` repo (config), dev wiki repo (content) |
||||||
|
237 | **Status: COMPLETE** |
||||||
|
238 | |||||||
| 239 | **Description:** |
|||||||
|
240 | Migrate the local development wiki from docker-compose to dev.wikibot.io. Use Otterwiki's built-in Git HTTP server (`GIT_WEB_SERVER=True`) to `git push` the dev wiki repo directly, preserving full git history. |
||||||
| 241 | ||||||||
| 242 | Platform config already set: `RETAIN_PAGE_NAME_CASE=True`, `TREAT_UNDERSCORE_AS_SPACE_FOR_TITLES=True`, `GIT_WEB_SERVER=True`. |
|||||||
| 243 | ||||||||
|
244 | **Additional work done during P1-9:** |
||||||
| 245 | - Fixed WSGI PATH_INFO encoding for non-ASCII characters (em dashes, etc.) |
|||||||
| 246 | - Implemented space→underscore filename normalization (otterwiki fork `feat/space-underscore-normalization`) |
|||||||
| 247 | - Added `resolve_filename` migration fallback in otterwiki-api |
|||||||
| 248 | - Renamed all 30 wiki pages to use underscores |
|||||||
| 249 | - Fixed MCP OAuth discovery routing (P1-9 addendum, see [[Tasks/P1-9_MCP_OAuth_Routing]]) |
|||||||
|
250 | |||||||
| 251 | **Deliverables:** |
|||||||
|
252 | - dev.wikibot.io hosts the complete dev wiki with full git history |
||||||
| 253 | - Claude Code MCP config updated to use remote server |
|||||||
| 254 | - Local docker-compose dev wiki shut down |
|||||||
|
255 | |||||||
| 256 | **Acceptance Criteria:** |
|||||||
|
257 | - [x] `git push` succeeds with full history preserved |
||||||
| 258 | - [x] User spot checks `https://dev.wikibot.io/` to confirm content looks correct |
|||||||
| 259 | - [x] Agent spot checks via remote MCP to confirm content matches local |
|||||||
| 260 | - [x] Semantic search returns results on remote instance (233 chunks, 30 pages) |
|||||||
| 261 | - [x] Agent can read/write wiki pages via remote MCP to continue Phase 2 work |
|||||||