Blame
|
1 | --- |
||||||
| 2 | category: reference |
|||||||
| 3 | tags: [meta, design, prd, api] |
|||||||
| 4 | last_updated: 2026-03-12 |
|||||||
| 5 | confidence: high |
|||||||
| 6 | --- |
|||||||
| 7 | ||||||||
| 8 | # Original PRD API |
|||||||
| 9 | ||||||||
| 10 | > This page is part of the original single-tenant PRD, split across five wiki pages: |
|||||||
|
11 | > [[Design/Research_Wiki]] | [[Design/Rest Api]] | [[Design/Semantic_Search]] | [[Design/Mcp Server]] | [[Design/Note_Schema]] |
||||||
|
12 | |||||||
| 13 | --- |
|||||||
| 14 | ||||||||
| 15 | ## Component 1: REST API Plugin |
|||||||
| 16 | ||||||||
| 17 | ### Goal |
|||||||
| 18 | ||||||||
| 19 | Add a JSON REST API to Otterwiki so that pages can be created, read, updated, deleted, listed, and searched programmatically. |
|||||||
| 20 | ||||||||
| 21 | ### Implementation approach |
|||||||
| 22 | ||||||||
| 23 | **First, investigate the plugin system.** Examine `otterwiki/plugins.py` and `docs/plugin_examples/` in the fork to determine whether plugins can register Flask blueprints (i.e., add new routes). The plugin system has hooks and was extended in v2.17.3. |
|||||||
| 24 | ||||||||
| 25 | - **If plugins support blueprint registration:** Build the API as an Otterwiki plugin. This is the preferred path — no core modifications, clean separation, potentially upstreamable. |
|||||||
| 26 | - **If plugins do NOT support blueprint registration:** Add an `api.py` Flask blueprint directly to the Otterwiki codebase in the fork. Register it in `server.py`. This is a clean PR-able change. |
|||||||
| 27 | ||||||||
| 28 | ### Authentication |
|||||||
| 29 | ||||||||
| 30 | Use a simple API key passed via `Authorization: Bearer <key>` header. The key is configured via environment variable `OTTERWIKI_API_KEY`. This is a single-user research system, not a multi-tenant service, so this is sufficient. |
|||||||
| 31 | ||||||||
| 32 | ### Commit authorship and message conventions |
|||||||
| 33 | ||||||||
| 34 | All Git commits should clearly indicate their origin. This is important for reviewing history and understanding whether a change was made by the human, the AI, or a system process. |
|||||||
| 35 | ||||||||
| 36 | **Author identity for API commits:** |
|||||||
| 37 | ||||||||
| 38 | ``` |
|||||||
| 39 | Author: Claude (MCP) <claude-mcp@otterwiki.local> |
|||||||
| 40 | ``` |
|||||||
| 41 | ||||||||
| 42 | Configure this via environment variables `OTTERWIKI_API_AUTHOR_NAME` and `OTTERWIKI_API_AUTHOR_EMAIL`, defaulting to the above. |
|||||||
| 43 | ||||||||
| 44 | **Commit message format:** |
|||||||
| 45 | ||||||||
| 46 | ``` |
|||||||
| 47 | [source] action: page name — optional detail |
|||||||
| 48 | ``` |
|||||||
| 49 | ||||||||
| 50 | Where `source` is one of: |
|||||||
| 51 | - `mcp` — changes made via the MCP server / API |
|||||||
| 52 | - `web` — changes made via the Otterwiki web UI (Otterwiki handles this itself) |
|||||||
| 53 | - `system` — changes made by automated processes (e.g., reindex, bulk import) |
|||||||
| 54 | ||||||||
| 55 | Examples: |
|||||||
| 56 | ``` |
|||||||
| 57 | [mcp] Create: Events/2026-03-09 Day 10 — initial event log |
|||||||
| 58 | [mcp] Update: Trends/Iran Attrition Strategy — add Phase 2 radar blinding detail |
|||||||
| 59 | [mcp] Delete: Events/Draft Note |
|||||||
| 60 | [system] Bulk import: 15 pages from PDF migration |
|||||||
| 61 | ``` |
|||||||
| 62 | ||||||||
| 63 | If the caller provides a `commit_message` in the API request body, use it as-is but prepend the `[mcp]` prefix. If no message is provided, generate one from the action and page name. |
|||||||
| 64 | ||||||||
| 65 | ### Endpoints |
|||||||
| 66 | ||||||||
| 67 | All endpoints are prefixed with `/api/v1/`. |
|||||||
| 68 | ||||||||
| 69 | #### Pages |
|||||||
| 70 | ||||||||
| 71 | | Method | Endpoint | Description | |
|||||||
| 72 | |--------|----------|-------------| |
|||||||
| 73 | | `GET` | `/api/v1/pages` | List all pages. Optional query params: `?prefix=Actors/` (subdirectory), `?category=actor` (frontmatter category), `?tag=p2-interceptor-race` (frontmatter tag), `?updated_since=2026-03-08` (ISO date). Filters compose with AND logic. Returns array of `{name, path, category, tags, last_updated, content_length}`. | |
|||||||
| 74 | | `GET` | `/api/v1/pages/<path:pagepath>` | Get a single page. Returns `{name, path, content, metadata, frontmatter, links_to, linked_from}`. Optional `?revision=<sha>` for historical versions. | |
|||||||
| 75 | | `PUT` | `/api/v1/pages/<path:pagepath>` | Create or update a page. Body: `{content, commit_message}`. If `commit_message` is omitted, auto-generates per commit convention above. | |
|||||||
| 76 | | `DELETE` | `/api/v1/pages/<path:pagepath>` | Delete a page. Body: `{commit_message}` (optional). | |
|||||||
| 77 | | `GET` | `/api/v1/pages/<path:pagepath>/history` | Get revision history. Returns array of `{revision, author, date, message}`. Optional `?limit=N`. | |
|||||||
| 78 | ||||||||
| 79 | #### Search |
|||||||
| 80 | ||||||||
| 81 | | Method | Endpoint | Description | |
|||||||
| 82 | |--------|----------|-------------| |
|||||||
| 83 | | `GET` | `/api/v1/search?q=<query>` | Full-text search (uses Otterwiki's existing search). Returns array of `{name, path, snippet, score}`. | |
|||||||
| 84 | ||||||||
| 85 | #### Links (WikiLink graph) |
|||||||
| 86 | ||||||||
| 87 | | Method | Endpoint | Description | |
|||||||
| 88 | |--------|----------|-------------| |
|||||||
| 89 | | `GET` | `/api/v1/links/<path:pagepath>` | Get outgoing and incoming WikiLinks for a page. Returns `{links_to: [...], linked_from: [...]}`. | |
|||||||
| 90 | | `GET` | `/api/v1/links` | Get the full link graph. Returns `{nodes: [...], edges: [...]}`. | |
|||||||
| 91 | ||||||||
| 92 | #### Changelog |
|||||||
| 93 | ||||||||
| 94 | | Method | Endpoint | Description | |
|||||||
| 95 | |--------|----------|-------------| |
|||||||
| 96 | | `GET` | `/api/v1/changelog` | Recent changes across all pages. Returns array of `{revision, author, date, message, pages_affected}`. Optional `?limit=N`. | |
|||||||
| 97 | ||||||||
| 98 | ### WikiLink parsing and link graph |
|||||||
| 99 | ||||||||
| 100 | The API must parse `[[WikiLink]]` and `[[Display Text|WikiLink]]` syntax in page content to populate the `links_to` and `linked_from` fields. |
|||||||
| 101 | ||||||||
| 102 | **Parsing approach:** |
|||||||
| 103 | ||||||||
| 104 | The regex for extracting WikiLinks from markdown content: |
|||||||
| 105 | ||||||||
| 106 | ```python |
|||||||
| 107 | import re |
|||||||
| 108 | WIKILINK_RE = re.compile(r'\[\[([^\]|]+?)(?:\|([^\]]+?))?\]\]') |
|||||||
| 109 | ||||||||
| 110 | def extract_wikilinks(content: str) -> list[str]: |
|||||||
| 111 | """Returns list of target page paths from WikiLinks in content.""" |
|||||||
| 112 | return [match.group(2) or match.group(1) for match in WIKILINK_RE.finditer(content)] |
|||||||
| 113 | # For [[Display Text|Target]], returns "Target" |
|||||||
| 114 | # For [[Target]], returns "Target" |
|||||||
| 115 | ``` |
|||||||
| 116 | ||||||||
| 117 | Note: check whether Otterwiki uses `[[Target|Display Text]]` or `[[Display Text|Target]]` order — this varies between wiki engines. Otterwiki's syntax page shows `[[Text to display|WikiPage]]`, so the **target is the second element** when a pipe is present. |
|||||||
| 118 | ||||||||
| 119 | **Link index implementation:** |
|||||||
| 120 | ||||||||
| 121 | Maintain an in-memory reverse index (dict mapping page path → set of pages that link to it). This is built once on startup by scanning all pages, then updated incrementally: |
|||||||
| 122 | ||||||||
| 123 | - On page save: re-parse that page's WikiLinks, update the index for that page |
|||||||
| 124 | - On page delete: remove that page from the index |
|||||||
| 125 | ||||||||
| 126 | The startup scan is O(N) where N is total pages. For a wiki of 200–500 pages with ~500 words each, this takes under a second. The index lives in the Flask process memory — no external storage needed. |
|||||||
| 127 | ||||||||
| 128 | Otterwiki v2.17.3 added a broken WikiLinks checker in housekeeping (`#388`). Look at that implementation first — it likely already has WikiLink parsing that can be reused or imported. |
|||||||
| 129 | ||||||||
| 130 | **The `GET /api/v1/links/<path>` endpoint** reads directly from this index. It does NOT scan the repo on every request. |
|||||||
| 131 | ||||||||
| 132 | **The `GET /api/v1/links` endpoint** (full graph) serializes the entire index. This could be expensive on a very large wiki, but for our expected size (< 500 pages) it's fine. |
|||||||
| 133 | ||||||||
| 134 | ### Error responses |
|||||||
| 135 | ||||||||
| 136 | Standard HTTP status codes. JSON body: `{error: "description"}`. |
|||||||
| 137 | ||||||||
| 138 | - `401` — missing or invalid API key |
|||||||
| 139 | - `404` — page not found |
|||||||
| 140 | - `409` — conflict (e.g., concurrent edit) |
|||||||
| 141 | - `422` — invalid content or parameters |
|||||||
| 142 | ||||||||
| 143 | ### Example requests and responses |
|||||||
| 144 | ||||||||
| 145 | These examples are canonical — the implementation should match these JSON shapes exactly. |
|||||||
| 146 | ||||||||
| 147 | #### List pages: `GET /api/v1/pages?prefix=Trends/` |
|||||||
| 148 | ||||||||
| 149 | Response (note: NO content field — list operations return metadata only): |
|||||||
| 150 | ||||||||
| 151 | ```json |
|||||||
| 152 | { |
|||||||
| 153 | "pages": [ |
|||||||
| 154 | { |
|||||||
| 155 | "name": "Iran Attrition Strategy", |
|||||||
| 156 | "path": "Trends/Iran Attrition Strategy", |
|||||||
| 157 | "category": "trend", |
|||||||
| 158 | "tags": ["military", "p2-interceptor-race", "p3-infrastructure"], |
|||||||
| 159 | "last_updated": "2026-03-08", |
|||||||
| 160 | "content_length": 487 |
|||||||
| 161 | }, |
|||||||
| 162 | { |
|||||||
| 163 | "name": "Desalination Targeting Ratchet", |
|||||||
| 164 | "path": "Trends/Desalination Targeting Ratchet", |
|||||||
| 165 | "category": "trend", |
|||||||
| 166 | "tags": ["infrastructure", "p3-infrastructure"], |
|||||||
| 167 | "last_updated": "2026-03-08", |
|||||||
| 168 | "content_length": 312 |
|||||||
| 169 | } |
|||||||
| 170 | ], |
|||||||
| 171 | "total": 2 |
|||||||
| 172 | } |
|||||||
| 173 | ``` |
|||||||
| 174 | ||||||||
| 175 | The `content_length` field is word count. This lets the caller decide whether to fetch the full page or skip large ones. The `category` and `tags` fields are extracted from YAML frontmatter; they are `null` if frontmatter is missing or malformed. |
|||||||
| 176 | ||||||||
| 177 | #### Read page: `GET /api/v1/pages/Trends/Iran Attrition Strategy` |
|||||||
| 178 | ||||||||
| 179 | Response (full content, parsed frontmatter, resolved links): |
|||||||
| 180 | ||||||||
| 181 | ```json |
|||||||
| 182 | { |
|||||||
| 183 | "name": "Iran Attrition Strategy", |
|||||||
| 184 | "path": "Trends/Iran Attrition Strategy", |
|||||||
| 185 | "content": "---\ncategory: trend\ntags: [military, p2-interceptor-race, p3-infrastructure]\nlast_updated: 2026-03-08\nconfidence: high\n---\n\n# Iran Attrition Strategy\n\nIran is executing a multi-phase attrition campaign...", |
|||||||
| 186 | "frontmatter": { |
|||||||
| 187 | "category": "trend", |
|||||||
| 188 | "tags": ["military", "p2-interceptor-race", "p3-infrastructure"], |
|||||||
| 189 | "last_updated": "2026-03-08", |
|||||||
| 190 | "confidence": "high" |
|||||||
| 191 | }, |
|||||||
| 192 | "links_to": [ |
|||||||
| 193 | "Variables/Interceptor Stockpiles", |
|||||||
| 194 | "Propositions/Iran Rationing Ballistic Missiles", |
|||||||
| 195 | "Actors/Iran", |
|||||||
| 196 | "Trends/Desalination Targeting Ratchet" |
|||||||
| 197 | ], |
|||||||
| 198 | "linked_from": [ |
|||||||
| 199 | "Actors/Iran", |
|||||||
| 200 | "Propositions/Iran Rationing Ballistic Missiles" |
|||||||
| 201 | ], |
|||||||
| 202 | "revision": "a1b2c3d", |
|||||||
| 203 | "last_commit": { |
|||||||
| 204 | "revision": "a1b2c3d", |
|||||||
| 205 | "author": "Claude (MCP)", |
|||||||
| 206 | "date": "2026-03-08T14:22:00Z", |
|||||||
| 207 | "message": "Update Iran Attrition Strategy — add Phase 2 radar blinding detail" |
|||||||
| 208 | } |
|||||||
| 209 | } |
|||||||
| 210 | ``` |
|||||||
| 211 | ||||||||
| 212 | The `content` field is the **raw markdown file content including the frontmatter block**. The `frontmatter` field is the parsed YAML as a JSON object. If frontmatter is missing or invalid YAML, `frontmatter` is `null` and `content` still returns the raw file. |
|||||||
| 213 | ||||||||
| 214 | #### Write page: `PUT /api/v1/pages/Events/2026-03-09 Day 10` |
|||||||
| 215 | ||||||||
| 216 | Request body: |
|||||||
| 217 | ||||||||
| 218 | ```json |
|||||||
| 219 | { |
|||||||
| 220 | "content": "---\ncategory: event\ntags: [military, day-10]\nlast_updated: 2026-03-09\nconfidence: high\n---\n\n# Day 10 — March 9, 2026\n\n## Key developments\n\n...", |
|||||||
| 221 | "commit_message": "Create Day 10 event log" |
|||||||
| 222 | } |
|||||||
| 223 | ``` |
|||||||
| 224 | ||||||||
| 225 | Response: |
|||||||
| 226 | ||||||||
| 227 | ```json |
|||||||
| 228 | { |
|||||||
| 229 | "name": "2026-03-09 Day 10", |
|||||||
| 230 | "path": "Events/2026-03-09 Day 10", |
|||||||
| 231 | "revision": "d4e5f6a", |
|||||||
| 232 | "created": true |
|||||||
| 233 | } |
|||||||
| 234 | ``` |
|||||||
| 235 | ||||||||
| 236 | The `created` field is `true` if this is a new page, `false` if it's an update to an existing page. |
|||||||
| 237 | ||||||||
| 238 | #### Full-text search: `GET /api/v1/search?q=ballistic+missile+rationing` |
|||||||
| 239 | ||||||||
| 240 | Response (snippets are ~150 chars of context around the match, NOT full content): |
|||||||
| 241 | ||||||||
| 242 | ```json |
|||||||
| 243 | { |
|||||||
| 244 | "results": [ |
|||||||
| 245 | { |
|||||||
| 246 | "name": "Iran Rationing Ballistic Missiles", |
|||||||
| 247 | "path": "Propositions/Iran Rationing Ballistic Missiles", |
|||||||
| 248 | "snippet": "...the 86% drop in ballistic missile launch rates reflects deliberate rationing, not destroyed capability. Observable indicators: continued...", |
|||||||
| 249 | "score": 0.95 |
|||||||
| 250 | }, |
|||||||
| 251 | { |
|||||||
| 252 | "name": "Iran Attrition Strategy", |
|||||||
| 253 | "path": "Trends/Iran Attrition Strategy", |
|||||||
| 254 | "snippet": "...Phase 3 — Ballistic strikes on high-value targets. Once interceptor stockpiles are depleted and radar coverage is degraded, Iran commits ballistic missiles...", |
|||||||
| 255 | "score": 0.72 |
|||||||
| 256 | } |
|||||||
| 257 | ], |
|||||||
| 258 | "query": "ballistic missile rationing", |
|||||||
| 259 | "total": 2 |
|||||||
| 260 | } |
|||||||
| 261 | ``` |
|||||||
| 262 | ||||||||
| 263 | #### Semantic search: `GET /api/v1/semantic-search?q=strategy+for+depleting+Gulf+air+defenses&n=3` |
|||||||
| 264 | ||||||||
| 265 | Response (same shape as full-text search, but `distance` instead of `score` — lower is more similar): |
|||||||
| 266 | ||||||||
| 267 | ```json |
|||||||
| 268 | { |
|||||||
| 269 | "results": [ |
|||||||
| 270 | { |
|||||||
| 271 | "name": "Iran Attrition Strategy", |
|||||||
| 272 | "path": "Trends/Iran Attrition Strategy", |
|||||||
| 273 | "snippet": "Iran is executing a multi-phase attrition campaign designed to degrade Gulf state and US defensive capacity before committing high-value ballistic missile assets.", |
|||||||
| 274 | "distance": 0.34 |
|||||||
| 275 | }, |
|||||||
| 276 | { |
|||||||
| 277 | "name": "Interceptor Stockpiles", |
|||||||
| 278 | "path": "Variables/Interceptor Stockpiles", |
|||||||
| 279 | "snippet": "Tracking estimated remaining interceptor inventories across Gulf state Patriot and THAAD batteries...", |
|||||||
| 280 | "distance": 0.41 |
|||||||
| 281 | }, |
|||||||
| 282 | { |
|||||||
| 283 | "name": "Iran Rationing Ballistic Missiles", |
|||||||
| 284 | "path": "Propositions/Iran Rationing Ballistic Missiles", |
|||||||
| 285 | "snippet": "The 86% drop in ballistic missile launch rates reflects deliberate rationing, not destroyed capability...", |
|||||||
| 286 | "distance": 0.48 |
|||||||
| 287 | } |
|||||||
| 288 | ], |
|||||||
| 289 | "query": "strategy for depleting Gulf air defenses", |
|||||||
| 290 | "total": 3 |
|||||||
| 291 | } |
|||||||
| 292 | ``` |
|||||||
| 293 | ||||||||
| 294 | The `snippet` for semantic search is the **text of the best-matching chunk** for that page, truncated to ~150 characters. Unlike full-text search, this is contextually relevant to the query — it shows the passage that was closest to the query in embedding space, not just the page's opening. |
|||||||