wasm-overhead-research/runtimes/README.md

WebAssembly Runtime SDK Evaluation

This directory contains implementations for evaluating the dependency size overhead of WebAssembly runtimes (Wasmer and Extism) across multiple programming languages.

Structure

runtimes/
├── test-modules/        # Minimal WASM modules for testing
├── wasmer/             # Wasmer runtime evaluations
│   ├── python/
│   ├── typescript/
│   ├── go/
│   ├── ruby/           # (planned)
│   └── java/           # (planned)
├── extism/             # Extism runtime evaluations
│   ├── python/
│   ├── typescript/
│   ├── go/
│   ├── ruby/           # (planned)
│   └── java/           # (planned)
└── measure_all.sh      # Aggregation script

Methodology

What We Measure

For each runtime/language combination, we measure:

1. Dependency Download Size: Total bytes fetched from package registries
2. Installed Footprint: Uncompressed size on disk after installation
3. Deployed Artifact Size: Size increase of deployable artifacts (binaries, JARs, bundles)
4. Native Libraries: Count and size of native libraries included
5. Lazy Downloads: Any runtime downloads on first execution

Measurement Approach

Python

• Baseline: Empty venv with minimal script
• With Runtime: venv with Wasmer/Extism SDK installed
• Metrics: venv size delta, pip download size, native .so/.dylib files

TypeScript/Node.js

• Baseline: Empty npm project
• With Runtime: Project with runtime SDK dependencies
• Metrics: node_modules size delta, npm pack size, native bindings

Go

• Baseline: Minimal Go binary
• With Runtime: Binary with runtime SDK statically linked
• Metrics: Binary size delta, module cache size

Test Modules

• add.wasm: Minimal 41-byte module exporting add(i32, i32) -> i32
• wasi_hello.wasm: Simple WASI module returning constant value
• extism_echo.wasm: Extism plugin with echo functionality (when built)

Running Measurements

Quick Start

# Measure all implementations
mise run measure:runtimes:all

# Measure specific runtime/language
mise run measure:wasmer:python
mise run measure:wasmer:typescript
mise run measure:wasmer:go
mise run measure:extism:python

Manual Execution

# Build test modules
cd runtimes/test-modules
python3 create_test_modules.py

# Run individual measurement
cd runtimes/wasmer/python
python3 measure.py

# Run all measurements
cd runtimes
./measure_all.sh

Results Format

Each measurement outputs JSON with:

{
  "runtime": "wasmer|extism",
  "language": "python|typescript|go|ruby|java",
  "os": "darwin|linux|windows",
  "arch": "arm64|x86_64",
  "versions": {
    "language": "3.11.0",
    "sdk": "1.0.0"
  },
  "baseline": {
    "size_bytes": 1000000
  },
  "with_runtime": {
    "size_bytes": 2000000,
    "download_size_bytes": 500000,
    "native_libs_count": 1
  },
  "delta": {
    "size_bytes": 1000000,
    "download_size_bytes": 500000
  },
  "native_libs": [...],
  "offline_viable": true,
  "notes": "Additional context"
}

Key Findings

Size Overhead Comparison

Runtime	Language	Install Overhead	Download Size	Notes
Wasmer	Python	3.9 KB	3.4 KB	⚠️ Core package only¹
Wasmer	TypeScript	12.8 MB	7.0 MB	@wasmer/sdk v0.9.0
Wasmer	Go	159.0 KB	N/A²	Dynamic linking to libwasmer.dylib
Extism	Python	21.9 MB	7.1 MB	Includes native libraries
Extism	TypeScript	Not tested	-	Partial implementation
Extism	Go	Not tested	-	Partial implementation

Detailed Results

Test Environment

• OS: macOS Darwin (arm64)
• Python: 3.12.6 / 3.13.3
• Node.js: v23.10.0
• Go: 1.24.5
• Date: August 2024

Wasmer Results

Python (wasmer 1.1.0)

• Download: 3.4 KB (wasmer-1.1.0-py3-none-any.whl)
• Install size delta: 3.9 KB
• Native libraries: None detected
• Issue: The wasmer package is pure Python without bundled native runtime

TypeScript (@wasmer/sdk 0.9.0)

• Download: 7.0 MB (npm packages)
• node_modules size: 12.8 MB
• Native libraries: None detected (WASM-based)
• Notes: Consolidated package, no longer requires @wasmer/wasi or @wasmer/wasmfs

Go (wasmer-go v1.0.4)

• Binary size increase: 159 KB
• Module cache: Included in binary
• Linking: Static
• Notes: Very efficient size overhead

Extism Results

Python (extism 1.0.0)

• Download: 7.1 MB
• Install size delta: 21.9 MB
• Native libraries: Included
• Notes: Significantly larger than Wasmer, includes full plugin runtime

Footnotes

¹ Wasmer Python Package Structure: The wasmer 1.1.0 package on PyPI is the core API package only. To actually compile and run WebAssembly, you need to install a compiler engine separately:

• pip install wasmer-compiler-cranelift==1.1.0 (recommended for development)
• pip install wasmer-compiler-llvm==1.1.0 (recommended for production)
• pip install wasmer-compiler-singlepass==1.1.0 (fastest compilation)

Our measurement only included the core package. With cranelift compiler, the total download would be significantly larger.

² Wasmer Go Dynamic Linking: The Go binary dynamically links to libwasmer.dylib which must be installed separately or bundled with the application. The binary size increase (159KB) doesn't include the ~40MB libwasmer library. For distribution, you need either:

• System-wide Wasmer installation
• Bundle libwasmer.dylib with your application
• Use CGO_ENABLED=0 for static linking (if supported)

Analysis & Recommendations

Key Takeaways

1. Go requires external dependencies: Despite small binary size increase (159KB), it needs libwasmer.dylib (~40MB) at runtime due to dynamic linking.

2. TypeScript/Node.js is self-contained: 12.8MB for @wasmer/sdk includes everything needed, making deployment simpler.

3. Python Wasmer is modular: Requires separate compiler package installation. Total size with compiler would be much larger than our 3.9KB measurement.

4. Extism is all-inclusive: At 21.9MB for Python, it bundles everything needed including native libraries.

Recommendations by Use Case

• Self-contained deployment: Use TypeScript with @wasmer/sdk (12.8MB all-inclusive)
• Minimal binary size: Use Go, but plan for libwasmer distribution
• Python applications: Install both wasmer and a compiler package, or use wasmtime
• Plugin systems: Extism provides the most complete out-of-box experience

Platform Considerations

• macOS ARM64: Some SDKs may not have pre-built wheels/binaries
• Python 3.13: Newer Python versions may lack wheel support
• Native Libraries: Dynamic vs static linking varies by language
• Lazy Downloads: Some SDKs download native components on first use

Adding New Languages

To add Ruby or Java support:

1. Create directory: runtimes/{runtime}/{language}/
2. Add implementation files:
   • main.{ext}: Runtime loader/executor
   • baseline.{ext}: Minimal app without runtime
   • measure.{sh|py|js}: Measurement script
3. Update measure_all.sh to include new language
4. Add mise task in .mise.toml

Known Issues

1. Wasmer Python on ARM64: Limited wheel availability
2. Extism Plugin Format: Requires Extism-specific WASM format
3. Version Compatibility: Some SDK versions may not support latest language versions

Future Improvements

• Add Ruby implementations
• Add Java implementations
• Create Docker containers for reproducible Linux measurements
• Add performance benchmarks (instantiation time, call overhead)
• Measure memory footprint at runtime
• Test with real-world WASM modules
• Add CI/CD automation for measurements