Benchmark: GemStack AI orchestration vs Next.js (task speed + human interventions)

[suleimansh]

Idea from the Vike collab (Rom). Measure how "our AI" performs compared to Next.js: the same AI coding agent achieving the same real development tasks, with the GemStack orchestration layer (autopilot + skills + mcp) available versus a vanilla Next.js baseline that has none of it.

Metrics

1. Time for the AI to achieve a given task.
2. Number of human interventions required to complete it.

Scope to define

• A fixed task set (representative dev tasks) and a Next.js comparison harness.
• Reproducible runner + reporting (time + intervention count per task).
• A baseline run we can track over time.

Out of scope

This is not the self-healing loop (an app that watches its own tests/audits/errors and fixes itself). That is a separate direction. This benchmark only measures an AI agent building and changing apps.

Priority is medium; this is a strong story for what the orchestration layer is for, not blocking shipping. See the design spec in the comments.

Phases

• Benchmark Phase 0: one task, both apps, manual tally (prove the rubric) #78 Phase 0: one task, both apps, manual tally (prove the rubric)
• Benchmark Phase 1: semi-automated runner over a 3-5 task set #79 Phase 1: semi-automated runner over a 3-5 task set
• Benchmark Phase 2: full suite, aggregator, committed baseline #80 Phase 2: full suite, aggregator, committed baseline

[suleimansh]

Benchmark design: "our AI" vs Next.js (time-to-task + human interventions)

Scoping the "scope to define" section. This is the design spec only; the runnable harness is a follow-up.

Note: this benchmark is not the self-healing loop. It measures an AI coding agent building and changing apps. Self-healing is a separate direction and is out of scope here.

What we are actually measuring

Rom's framing: "how our AI performs compared to Next.js." Next.js has no AI orchestration layer, so the honest comparison is the same AI coding agent achieving the same tasks, with the GemStack orchestration layer available versus not:

• GemStack side ("our AI") - the agent operates with the orchestration layer in reach: @gemstack/ai-skills (invokable capabilities), @gemstack/ai-mcp / @gemstack/mcp (tools and resources the app exposes to the agent), and @gemstack/ai-autopilot (supervisor topology for multi-step plan -> dispatch -> synthesize).
• Next.js baseline - the same agent, same model, operating on a functionally equivalent Next.js app with none of that layer (vanilla framework + generic file editing).

Two headline metrics, both Rom's:

1. Time-to-task - wall-clock from task start to acceptance criteria passing.
2. Human interventions - count of times a human had to step in (correct, unblock, clarify, approve, retry).

Controlling for confounds (the hard part)

A "framework A vs framework B for AI" number is only credible if the only thing that varies is the framework. So both sides hold constant:

• Same agent + harness + model (pin the model version; fixed temperature/seed where the provider allows).
• Two functionally equivalent apps - a GemStack reference app and a Next.js reference app implementing the same product surface (small but real: auth + a CRUD resource + one AI feature). The Next.js app must be idiomatic and fair, not a strawman; that is the single biggest threat to validity.
• Same task prompts and the same machine-checkable acceptance criteria on both sides.
• Pinned everything - lockfiles, package versions, starting git commit per task.
• Variance - N runs per task (start at 5), report median + spread, not a single number. Record DNF separately.

The task set

Representative dev tasks an app author actually does, each with an objective acceptance check and a defined starting commit:

Category	Example	Acceptance check
Feature add	add a CRUD endpoint + minimal UI for a new resource	endpoint returns expected JSON + UI assertion passes
Schema / data change	add a field + migration, surface it	migration applies, field round-trips
Bug fix	induced failing test	the test goes green, no others break
AI integration	add an AI feature (tool/skill the agent wires in)	feature responds correctly to a fixed input
Multi-step feature	feature spanning data + endpoint + UI in one task	full acceptance script passes
Refactor	extract/rename across files	behavior unchanged (suite green) + structural assertion

Each task ships as: prompt, starting commit SHA, acceptance script (exit 0 = pass), intervention rubric, hard timeout, max-intervention cap.

Intervention rubric (so the count is objective, not vibes)

Counts as one human intervention:

• a manual code correction by a human
• unblocking a stuck agent with a hint
• a clarification the agent had to ask before it could proceed
• an approval gate that required a human
• a manual retry / rerun a human had to trigger

Does not count (this is the whole point of the orchestration layer):

• the agent's own internal retries, planning, and autopilot worker dispatch
• skill/MCP tool calls the agent makes autonomously

Termination: task ends at acceptance pass, or at the hard timeout / max-intervention cap (recorded as DNF). DNF is a result, not a discarded run.

Harness architecture

runner
  -> checkout starting commit
  -> launch agent with the task prompt (GemStack adapter | Next.js adapter)
  -> stream events; log every human-intervention event (structured)
  -> poll acceptance script; on exit 0 stop the timer
  -> emit report.json  { framework, task, runIndex, seconds, interventions, status }
aggregator
  -> median time, median interventions, success rate per (framework, task)
  -> committed baseline JSON, tracked over time (mirrors the Rudder perf baseline pattern)

Two thin adapters (GemStack / Next.js) keep the runner framework-agnostic. Interventions are captured as structured events so the count is reproducible rather than recalled.

Reporting

• Per-task table: framework | median time | median interventions | success rate | n.
• A single committed baseline file we re-run and diff over time (same discipline as the Rudder playground/bench baseline).
• Charts optional, later.

Phasing (keep it honest and incremental)

• Phase 0 - one task, both apps, manual stopwatch + manual intervention tally. Proves the method and the rubric before any automation.
• Phase 1 - 3 to 5 tasks, semi-automated runner + report.json.
• Phase 2 - full suite, automated runner + aggregator, first committed baseline run.

Threats to validity (call them out up front)

• Task selection bias - do not cherry-pick GemStack-favorable tasks; the set must include tasks where the layer gives no edge.
• Strawman baseline - the Next.js app must be idiomatic and competently built.
• Same agent both sides - any model/harness difference invalidates the number.
• Objective acceptance only - no human judgement in the pass/fail gate.

Priority stays medium; this is the measurable story for what the orchestration layer is for, not a shipping blocker. Phase 0 is the next concrete step whenever we pick this up.