Home A - Z
Page Index

Blame

f067a2 Claude (MCP) 2026-03-14 03:42:14
[mcp] E-1: Cold start benchmark results (2026-03-14)
1 
---
2
category: reference
3
tags:
4
- benchmarks
5
- lambda
6
- cold-start
7
- phase-2
8
last_updated: 2026-03-14
9
---
10
11
# E-1: Cold Start Benchmarks
12
13
**Date:** 2026-03-14
14
**Lambda:** otterwiki (512MB default, VPC+EFS, ~180MB package)
15
**Package contents:** otterwiki, numpy, faiss-cpu, sqlalchemy, bcrypt, cryptography, PyJWT, plugins
16
17
## Summary
18
19
| Config | Memory | Init (mean) | Init (p50) | Init (min) | Init (max) | Wall (mean) | n |
20
|--------|--------|-------------|------------|------------|------------|-------------|---|
21
| bare_vpc | 512MB | 88.1ms | 82.8ms | 79.6ms | 106.5ms | 561.7ms | 5 |
22
| minimal_vpc | 512MB | 86.0ms | 81.6ms | 80.7ms | 99.7ms | 580.6ms | 5 |
23
| heavy_imports | 512MB | 87.8ms | 86.4ms | 73.2ms | 100.7ms | 1079.4ms | 5 |
24
| otterwiki | 512MB | 4513.3ms | 4478.9ms | 4347.4ms | 4826.5ms | 5674.1ms | 5 |
25
| bare_vpc | 1024MB | 75.4ms | 75.7ms | 65.2ms | 84.5ms | 532.8ms | 5 |
26
| minimal_vpc | 1024MB | 78.4ms | 77.8ms | 75.0ms | 84.9ms | 539.6ms | 5 |
27
| heavy_imports | 1024MB | 75.8ms | 76.6ms | 70.8ms | 81.6ms | 920.9ms | 5 |
28
| otterwiki | 1024MB | 4274.2ms | 4379.7ms | 3707.2ms | 4492.8ms | 5232.5ms | 5 |
29
30
**Control Lambdas:**
31
- `bare_vpc` — minimal handler, VPC+EFS, no imports beyond stdlib
32
- `minimal_vpc` — VPC+EFS, lightweight imports (os, json, time), EFS read/write test
33
- `heavy_imports` — full otterwiki package, NO VPC/EFS (SimpleLambdaComponent)
34
35
## Detailed Init Breakdown (otterwiki @ 512MB)
36
37
| Phase | Time (ms) | % of Total |
38
|-------|-----------|------------|
39
| stdlib imports | 7.6 | 0.2% |
40
| ensure_directories | 0.0 | 0.0% |
41
| import apig_wsgi | 17.7 | 0.4% |
42
| **import plugins** | **528.0** | **12.0%** |
43
| **import otterwiki.server** | **3490.8** | **79.4%** |
44
| build_multi_tenant_app | 353.7 | 8.0% |
45
| make_lambda_handler | 0.0 | 0.0% |
46
| **Total init** | **4397.8** | **100%** |
47
48
## Detailed Init Breakdown (otterwiki @ 1024MB)
49
50
| Phase | Time (ms) | % of Total |
51
|-------|-----------|------------|
52
| stdlib imports | 6.2 | 0.1% |
53
| ensure_directories | 0.0 | 0.0% |
54
| import apig_wsgi | 15.3 | 0.4% |
55
| **import plugins** | **503.9** | **11.6%** |
56
| **import otterwiki.server** | **3473.3** | **79.7%** |
57
| build_multi_tenant_app | 357.0 | 8.2% |
58
| make_lambda_handler | 0.0 | 0.0% |
59
| **Total init** | **4355.9** | **100%** |
60
61
## Analysis
62
63
### What's slow
64
65
1. **`from otterwiki.server import app` is 79% of cold start (~3.5s).** This triggers the entire Flask app initialization chain: SQLAlchemy model creation, Jinja2 template loading, pluggy hook execution, and transitive imports of all otterwiki modules (which pull in numpy, faiss, lxml, Pillow, etc.).
66
67
2. **Plugin imports are 12% (~500ms).** `otterwiki_semantic_search` imports faiss and numpy at module level. `otterwiki_api` is lighter but still contributes.
68
69
3. **Multi-tenant app build is 8% (~350ms).** DynamoDB model instantiation and middleware wiring. This is new in Phase 2.
70
71
### What's NOT slow
72
73
- **VPC+EFS attach: ~80-90ms init + ~400ms wall overhead.** The VPC ENI attachment adds about 400-500ms of wall time but only 80-90ms of billed init duration. This is well within acceptable range and NOT the bottleneck.
74
- **stdlib imports: negligible (<10ms)**
75
- **apig_wsgi import: negligible (~17ms)**
76
- **Memory scaling: minimal benefit.** Doubling from 512MB to 1024MB only reduces init by ~5% (4513→4274ms). The bottleneck is import/initialization, not memory or CPU.
77
78
### Key insight
79
80
The heavy_imports Lambda (no VPC) shows init durations of only 75-88ms despite having the same 180MB package. This means the Lambda runtime loads the zip/S3 package quickly — the cost is in Python actually **importing** the modules and running their initialization code. The `otterwiki.server` import triggers a cascade of module-level work (Flask app factory, SQLAlchemy engine creation, template compilation, etc.) that takes 3.5s regardless of memory.
81
82
### Comparison to Phase 0
83
84
- Phase 0 (2025): ~3.4s cold start at 512MB with a much smaller package
85
- Phase 2 (now): ~4.5s cold start at 512MB with 180MB package
86
- **Delta: +1.1s** from added dependencies (faiss, numpy, bcrypt, cryptography, multi-tenant middleware)
87
88
### Recommendations
89
90
1. **Lazy imports in otterwiki.server** — defer numpy/faiss/Pillow/lxml until first use
91
2. **Lazy plugin loading** — don't import `otterwiki_semantic_search` at module level; register a pluggy hook that loads it on first search request
92
3. **Consider SnapStart** (if/when Python support lands) or provisioned concurrency for production
93
4. **Multi-tenant middleware** — 350ms for DynamoDB model construction could be deferred to first request
94
5. **Memory increase not recommended** — 1024MB provides negligible improvement for 2x cost
95
96
## Raw Log Excerpts
97
98
### 512MB
99
```
100
COLD_START_TIMINGS: {"stdlib_imports": 7.55, "ensure_directories": 0.0, "import_apig_wsgi": 17.68, "import_plugins": 528.0, "import_otterwiki_server": 3490.79, "build_multi_tenant_app": 353.68, "make_lambda_handler": 0.01, "total_init": 4397.82, "multi_tenant": true}
101
REPORT Duration: 3.91 ms Billed Duration: 4483 ms Memory Size: 512 MB Max Memory Used: 218 MB Init Duration: 4478.90 ms
102
```
103
104
### 1024MB
105
```
106
COLD_START_TIMINGS: {"stdlib_imports": 6.24, "ensure_directories": 0.0, "import_apig_wsgi": 15.31, "import_plugins": 503.89, "import_otterwiki_server": 3473.34, "build_multi_tenant_app": 357.02, "make_lambda_handler": 0.01, "total_init": 4355.9, "multi_tenant": true}
107
REPORT Duration: 2.89 ms Billed Duration: 4441 ms Memory Size: 1024 MB Max Memory Used: 218 MB Init Duration: 4437.84 ms
108
```