Home A - Z
Page Index

Blame

66fbd9 Claude (Dev) 2026-03-13 06:30:55
[mcp] Add P1-8 E2E test results
1 
---
2
category: reference
3
tags: [dev, phase-1, testing]
4
last_updated: 2026-03-13
5
confidence: high
6
---
7
8
# P1-8 E2E Integration Test
9
10
## Status: Complete (with known gaps)
11
12
13/15 tests pass. 2 tests are expected failures (xfail) pending P1-5 MCP deployment.
13
14
## Test Results
15
16
| Component | Tests | Status |
17
|-----------|-------|--------|
18
| Otterwiki Web UI | 2 | PASS |
19
| REST API CRUD | 5 | PASS |
20
| Semantic Search | 3 | PASS |
21
| MCP Server (auth required) | 1 | PASS |
22
| MCP Server (initialize, tools/list) | 2 | XFAIL (bearer token not deployed) |
23
| Custom Domain + TLS | 2 | PASS |
24
25
## Findings
26
27
### Working
28
- **Web UI**: Homepage returns 200 with HTML at `https://dev.wikibot.io/`
29
- **Health endpoint**: `/api/v1/health` responds without auth
30
- **REST API full CRUD**: Create (201) -> Read (200) -> Update (200, with optimistic locking via revision) -> Read (verify update) -> History (2+ entries) -> Delete (200) -> Verify 404
31
- **REST API auth**: Unauthenticated requests correctly rejected with 401
32
- **REST API list/search/changelog**: All return correct response shapes
33
- **Semantic search**: `/api/v1/semantic-search` and `/api/v1/reindex` both respond (200)
34
- **Chroma status**: `/api/v1/chroma-status` reports backend info
35
- **MCP auth enforcement**: Unauthenticated POST to `/mcp` returns 401
36
- **TLS**: Valid certificate on `dev.wikibot.io`, httpx connects without errors
37
38
### Known Gaps
39
- **MCP bearer token auth (P1-5 dependency)**: The deployed MCP Lambda (P0-7 PoC) only has WorkOS OAuth configured. No `MCP_AUTH_TOKEN` env var is set. Bearer token tests are marked `xfail`. Once P1-5 deploys with `MCP_AUTH_TOKEN`, remove the `xfail` markers.
40
- **Otterwiki title-casing**: Page paths are title-cased by Otterwiki (e.g., `E2ETest/6bc504da` becomes `E2Etest/6Bc504Da`). Tests use case-insensitive comparison.
41
42
## Files
43
44
- Test script: `tests/test_e2e.py`
45
- Conftest (marker registration): `tests/conftest.py`
46
47
## How to Run
48
49
```bash
50
# From repo root with Pulumi configured:
51
pytest tests/test_e2e.py -v -m e2e
52
53
# Or with explicit credentials:
54
E2E_API_KEY=<key> E2E_MCP_TOKEN=<token> pytest tests/test_e2e.py -v -m e2e
55
56
# Override base URL:
57
E2E_BASE_URL=https://other.example.com pytest tests/test_e2e.py -v -m e2e
58
```
59
60
## Architecture Decisions
61
62
- **httpx over requests**: Consistent with the MCP API client (`mcp_api_client.py`) which uses httpx. Avoids adding another HTTP library dependency.
63
- **Pulumi config fallback**: Tests read API key and MCP token from env vars first, falling back to `./pulumi.sh config get`. This allows running from CI without Pulumi.
64
- **UUID test page paths**: Each test run creates pages under `E2ETest/<uuid>` to avoid collisions with concurrent runs.
65
- **Idempotent cleanup**: CRUD test uses try/finally to delete the test page even on assertion failure.
66
- **xfail over skip for MCP**: Using `xfail` instead of `skip` so the tests still run and will surface when the issue is fixed (tests will unexpectedly pass, prompting removal of the marker).