category: reference
tags:
  - benchmarks
  - lambda
  - cold-start
  - phase-2
last_updated: 2026-03-14

E-1: Cold Start Benchmarks

Date: 2026-03-14 Lambda: otterwiki (512MB default, VPC+EFS, ~180MB package) Package contents: otterwiki, numpy, faiss-cpu, sqlalchemy, bcrypt, cryptography, PyJWT, plugins

Summary

Control Lambdas:

• bare_vpc — minimal handler, VPC+EFS, no imports beyond stdlib
• minimal_vpc — VPC+EFS, lightweight imports (os, json, time), EFS read/write test
• heavy_imports — full otterwiki package, NO VPC/EFS (SimpleLambdaComponent)

Detailed Init Breakdown (otterwiki @ 512MB)

Detailed Init Breakdown (otterwiki @ 1024MB)

Analysis

What's slow

1. from otterwiki.server import app is 79% of cold start (~3.5s). This triggers the entire Flask app initialization chain: SQLAlchemy model creation, Jinja2 template loading, pluggy hook execution, and transitive imports of all otterwiki modules (which pull in numpy, faiss, lxml, Pillow, etc.).

2. Plugin imports are 12% (~500ms). otterwiki_semantic_search imports faiss and numpy at module level. otterwiki_api is lighter but still contributes.

3. Multi-tenant app build is 8% (~350ms). DynamoDB model instantiation and middleware wiring. This is new in Phase 2.

What's NOT slow

• VPC+EFS attach: ~80-90ms init + ~400ms wall overhead. The VPC ENI attachment adds about 400-500ms of wall time but only 80-90ms of billed init duration. This is well within acceptable range and NOT the bottleneck.
• stdlib imports: negligible (<10ms)
• apig_wsgi import: negligible (~17ms)
• Memory scaling: minimal benefit. Doubling from 512MB to 1024MB only reduces init by ~5% (4513→4274ms). The bottleneck is import/initialization, not memory or CPU.

Key insight

The heavy_imports Lambda (no VPC) shows init durations of only 75-88ms despite having the same 180MB package. This means the Lambda runtime loads the zip/S3 package quickly — the cost is in Python actually importing the modules and running their initialization code. The otterwiki.server import triggers a cascade of module-level work (Flask app factory, SQLAlchemy engine creation, template compilation, etc.) that takes 3.5s regardless of memory.

Comparison to Phase 0

• Phase 0 (2025): ~3.4s cold start at 512MB with a much smaller package
• Phase 2 (now): ~4.5s cold start at 512MB with 180MB package
• Delta: +1.1s from added dependencies (faiss, numpy, bcrypt, cryptography, multi-tenant middleware)

Recommendations

1. Lazy imports in otterwiki.server — defer numpy/faiss/Pillow/lxml until first use
2. Lazy plugin loading — don't import otterwiki_semantic_search at module level; register a pluggy hook that loads it on first search request
3. Consider SnapStart (if/when Python support lands) or provisioned concurrency for production
4. Multi-tenant middleware — 350ms for DynamoDB model construction could be deferred to first request
5. Memory increase not recommended — 1024MB provides negligible improvement for 2x cost

Raw Log Excerpts

512MB

COLD_START_TIMINGS: {"stdlib_imports": 7.55, "ensure_directories": 0.0, "import_apig_wsgi": 17.68, "import_plugins": 528.0, "import_otterwiki_server": 3490.79, "build_multi_tenant_app": 353.68, "make_lambda_handler": 0.01, "total_init": 4397.82, "multi_tenant": true}
REPORT Duration: 3.91 ms  Billed Duration: 4483 ms  Memory Size: 512 MB  Max Memory Used: 218 MB  Init Duration: 4478.90 ms

1024MB

COLD_START_TIMINGS: {"stdlib_imports": 6.24, "ensure_directories": 0.0, "import_apig_wsgi": 15.31, "import_plugins": 503.89, "import_otterwiki_server": 3473.34, "build_multi_tenant_app": 357.02, "make_lambda_handler": 0.01, "total_init": 4355.9, "multi_tenant": true}
REPORT Duration: 2.89 ms  Billed Duration: 4441 ms  Memory Size: 1024 MB  Max Memory Used: 218 MB  Init Duration: 4437.84 ms