Commit 1935aa
2026-03-13 04:50:49 Claude (Dev): [mcp] Add Phase 0 EFS benchmark results| /dev/null .. dev/phase 0 — efs benchmarks.md | |
| @@ 0,0 1,89 @@ | |
| + | --- |
| + | category: reference |
| + | tags: [phase-0, benchmarks, infrastructure] |
| + | last_updated: 2026-03-13 |
| + | confidence: high |
| + | --- |
| + | |
| + | # Phase 0 — EFS Benchmarks |
| + | |
| + | ## Summary |
| + | |
| + | All Phase 0 exit criteria pass. Git on EFS via Lambda is viable for the wikibot.io use case. |
| + | |
| + | ## Environment |
| + | |
| + | - Lambda: Python 3.12, 512MB memory, VPC-attached (us-east-1a) |
| + | - EFS: One Zone (us-east-1a), encrypted, access point at /wikibot |
| + | - Git library: dulwich 1.1.0 (pure Python, no binary dependency) |
| + | - X-Ray: Active tracing enabled |
| + | |
| + | ## Results |
| + | |
| + | ### Cold Start (5 invocations, forced via env var update) |
| + | |
| + | | Metric | Value | |
| + | |--------|-------| |
| + | | Mean | 3,407ms | |
| + | | Median | 3,403ms | |
| + | | P95 | 3,524ms | |
| + | | Min | 3,234ms | |
| + | | Max | 3,524ms | |
| + | |
| + | **Target: <5,000ms — PASS** |
| + | |
| + | Cold start breakdown (typical): ~2,400ms VPC ENI attach + ~160ms git init + ~600ms commit + ~2ms read + ~40ms log. |
| + | |
| + | ### Warm Read (25 invocations) |
| + | |
| + | | Metric | Lambda time | Wall time (incl. network) | |
| + | |--------|-------------|--------------------------| |
| + | | Mean | 2.07ms | 272ms | |
| + | | Median | 2.02ms | 270ms | |
| + | | P95 | 2.46ms | 282ms | |
| + | |
| + | **Target: <500ms wall — PASS** |
| + | |
| + | Wall time is dominated by Lambda invoke overhead (~270ms), not EFS I/O (~2ms). |
| + | |
| + | ### Warm Write (25 invocations) |
| + | |
| + | | Metric | Lambda time (init + commit) | |
| + | |--------|---------------------------| |
| + | | Mean | 247ms | |
| + | | Median | 245ms | |
| + | | P95 | 284ms | |
| + | |
| + | **Target: <1,000ms — PASS** |
| + | |
| + | ### Concurrent Reads (3 simultaneous, 5 rounds) |
| + | |
| + | - 15/15 succeeded, no errors |
| + | - Mean wall time: 610ms (includes cold starts in early rounds) |
| + | - Median wall time: 337ms (warm) |
| + | |
| + | **PASS — no errors under concurrent read load.** |
| + | |
| + | ### Concurrent Writes (5 simultaneous, 3 rounds) |
| + | |
| + | - 5/15 succeeded, 10 failed with git lock contention |
| + | - No data corruption observed |
| + | |
| + | **Expected behavior.** dulwich uses file-based locking on the git repo. Concurrent writers are rejected, not queued. In production, writes are serialized through a single API endpoint, so this is not a concern. |
| + | |
| + | ## Git Library Decision |
| + | |
| + | **dulwich** (pure Python) over gitpython (shells out to `git` binary). |
| + | |
| + | Rationale: |
| + | - Lambda Python 3.12 runtime does not include the `git` binary |
| + | - Bundling git would add ~30MB and require a Lambda layer or container image |
| + | - dulwich works with both bare and non-bare repos via its object API and porcelain API |
| + | - 39MB total package size (dulwich + aws-xray-sdk) — well within Lambda limits |
| + | |
| + | ## Implications for Phase 1 |
| + | |
| + | - VPC cold starts (~3.4s) are noticeable but acceptable. Provisioned Concurrency ($10-15/mo for 1 warm instance) can eliminate them if needed. |
| + | - EFS I/O is fast (~2ms read, ~250ms write including git commit). Not a bottleneck. |
| + | - The ~270ms Lambda invoke overhead dominates read latency. This is inherent to VPC Lambda and acceptable for MCP tool calls. |
| + | - Concurrent write serialization must be handled at the application layer, not git layer. |