Skip to content

feat: add filesystem benchmark mode#105

Open
HeyGarrison wants to merge 5 commits intomasterfrom
opencode/curious-garden
Open

feat: add filesystem benchmark mode#105
HeyGarrison wants to merge 5 commits intomasterfrom
opencode/curious-garden

Conversation

@HeyGarrison
Copy link
Copy Markdown
Contributor

Summary

  • add a new fs benchmark mode that runs in-sandbox filesystem workloads (large-file read/write, small-file churn, and metadata operations) per provider
  • wire FS benchmarking into the CLI flow, result persistence (results/fs + latest.json), and composite scoring/success-rate reporting
  • document the new mode in README and methodology and add npm scripts for common FS benchmark invocations

Validation

  • attempted local smoke test with npm run bench:fs -- --iterations 1 --provider e2b
  • environment in this workspace blocked execution (tsx: Permission denied and missing local deps), so runtime validation is expected in CI where keys and deps are configured

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 1, 2026

Browser Benchmark Results

# Provider Score Create Connect Navigate Release Total Status
1 Browserbase 94.5 0.22s 0.12s 0.06s 0.13s 0.55s 10/10
2 Kernel 93.5 0.07s 0.40s 0.15s 0.07s 0.71s 10/10
3 Hyperbrowser 91.1 0.14s 0.19s 0.11s 0.16s 0.63s 10/10

View full run · SVG available as build artifact

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 1, 2026

Sandbox Benchmark Results

Sequential

# Provider Score Median TTI P95 P99 Status
1 declaw 99.3 0.04s 0.11s 0.11s 10/10
2 daytona 98.2 0.12s 0.26s 0.26s 10/10
3 upstash 95.2 0.40s 0.59s 0.59s 10/10
4 e2b 94.4 0.46s 0.70s 0.70s 10/10
5 vercel 93.5 0.56s 0.79s 0.79s 10/10
6 blaxel 92.9 0.51s 1.02s 1.02s 10/10
7 hopx 83.1 1.32s 2.25s 2.25s 10/10
8 modal 82.2 1.59s 2.06s 2.06s 10/10
9 namespace 81.2 1.83s 1.96s 1.96s 10/10
10 cloudflare 78.3 2.01s 2.40s 2.40s 10/10
11 runloop 75.5 1.80s 3.41s 3.41s 10/10
12 codesandbox 71.2 2.72s 3.13s 3.13s 10/10
13 archil 9.4 0.57s 0.57s 0.57s 1/10

Staggered

# Provider Score Median TTI P95 P99 Status
1 daytona 98.4 0.14s 0.20s 0.20s 10/10
2 declaw 98.0 0.04s 0.43s 0.43s 10/10
3 upstash 96.1 0.37s 0.41s 0.41s 10/10
4 blaxel 94.7 0.51s 0.56s 0.56s 10/10
5 e2b 93.8 0.48s 0.83s 0.83s 10/10
6 vercel 92.3 0.69s 0.88s 0.88s 10/10
7 hopx 84.9 1.37s 1.73s 1.73s 10/10
8 namespace 79.8 1.94s 2.14s 2.14s 10/10
9 modal 79.6 1.76s 2.45s 2.45s 10/10
10 cloudflare 75.9 2.11s 2.88s 2.88s 10/10
11 runloop 74.6 1.88s 3.52s 3.52s 10/10
12 codesandbox 71.7 2.73s 2.97s 2.97s 10/10
13 archil 9.7 0.28s 0.28s 0.28s 1/10

Burst

# Provider Score Median TTI P95 P99 Status
1 declaw 98.6 0.05s 0.28s 0.28s 10/10
2 daytona 98.6 0.13s 0.15s 0.15s 10/10
3 upstash 95.2 0.45s 0.53s 0.53s 10/10
4 e2b 94.7 0.50s 0.59s 0.59s 10/10
5 blaxel 94.3 0.56s 0.58s 0.58s 10/10
6 vercel 93.1 0.58s 0.85s 0.85s 10/10
7 modal 79.5 1.87s 2.32s 2.32s 10/10
8 namespace 79.3 1.91s 2.30s 2.30s 10/10
9 hopx 78.4 1.96s 2.46s 2.46s 10/10
10 cloudflare 75.8 2.06s 2.97s 2.97s 10/10
11 runloop 70.9 2.02s 4.25s 4.25s 10/10
12 codesandbox 70.8 2.82s 3.07s 3.07s 10/10
13 archil 9.7 0.29s 0.29s 0.29s 1/10

View full run · SVGs available as build artifacts

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 1, 2026

Storage Benchmark Results

1MB Files

# Provider Score Download Throughput Upload Status
1 Cloudflare R2 94.8 0.11s 78.5 Mbps 0.18s 1000/1000
2 AWS S3 94.2 0.34s 24.4 Mbps 0.06s 1000/1000
3 Tigris 93.5 0.52s 16.2 Mbps 0.19s 1000/1000

4MB Files

# Provider Score Download Throughput Upload Status
1 Cloudflare R2 94.9 0.18s 186.5 Mbps 0.32s 1000/1000

10MB Files

# Provider Score Download Throughput Upload Status
1 Cloudflare R2 94.1 0.34s 248.2 Mbps 0.81s 100/100
2 AWS S3 93.2 0.60s 139.9 Mbps 0.41s 100/100
3 Tigris 90.0 1.24s 67.7 Mbps 1.18s 100/100

16MB Files

# Provider Score Download Throughput Upload Status
1 Cloudflare R2 93.8 0.49s 276.4 Mbps 0.82s 100/100
2 AWS S3 90.5 1.53s 87.9 Mbps 0.43s 100/100
3 Tigris 88.4 1.72s 78.2 Mbps 1.22s 100/100

View full run · SVGs available as build artifacts

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 1, 2026

Filesystem Benchmark Results

# Provider Score Read Write Small Files Metadata Status
1 namespace 99.8 0.04s 0.04s 0.08s 0.00s 5/5
2 upstash 99.8 0.02s 0.01s 0.19s 0.01s 5/5
3 archil 99.7 0.04s 0.06s 0.21s 0.01s 5/5
4 vercel 99.7 0.07s 0.06s 0.17s 0.01s 5/5
5 blaxel 99.7 0.08s 0.06s 0.17s 0.01s 5/5
6 codesandbox 99.7 0.07s 0.06s 0.19s 0.00s 5/5
7 runloop 99.7 0.06s 0.05s 0.24s 0.01s 5/5
8 daytona 99.7 0.04s 0.01s 0.34s 0.01s 5/5
9 e2b 99.6 0.06s 0.07s 0.37s 0.01s 5/5
10 declaw 99.5 0.10s 0.11s 0.30s 0.01s 5/5
11 modal 99.3 0.06s 0.07s 0.66s 0.02s 5/5
12 cloudflare 97.9 0.45s 0.57s 1.08s 0.04s 5/5
13 hopx 93.6 2.19s 1.73s 1.72s 0.03s 5/5

View full run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant