feat: measured web-data provider contract — Firecrawl/Exa/Context.dev/Aside/local, selectable per task#40
feat: measured web-data provider contract — Firecrawl/Exa/Context.dev/Aside/local, selectable per task#40time-attack wants to merge 6 commits into
Conversation
gstack-web-data status/options/select/recommend over five providers (Firecrawl, Exa, Context.dev, Aside, local browser). Availability is detected (env keys, runtime consent, aside CLI, browse binary); per-task rankings come from the measured 2026-07-23 provider bakeoff. Recommend returns the best AVAILABLE provider and names better-but-unconfigured ones with their exact setup step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New webDataContract() reference emitted into all five skill trees, plus a dispatch-protocol trigger: read references/WEB-DATA.md before designing or performing bulk web-data acquisition (scraping, crawling, dataset building) — the gate the flightconnections plan never hit. The contract offers the provider menu once via gstack-web-data, recommends the best available provider per task, surfaces setup steps for better-but-unconfigured ones, keeps official APIs first and the hand-rolled path always valid, and generalizes the Context.dev public-URL egress rules to all off-machine providers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Registers bin/gstack-web-data in DEFAULT_RUNTIME_HELPERS and lib/web-data.ts in the helper dependency closure, so $GSTACK_HOME/bin/gstack-web-data resolves on installed runtimes as the WEB-DATA contract expects. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
There was a problem hiding this comment.
14 issues found across 18 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="scripts/gstack2/generate-skill-tree.ts">
<violation number="1" location="scripts/gstack2/generate-skill-tree.ts:526">
P2: Every bulk web-data workflow is instructed to violate the always-loaded authority policy because this command uses both redirection and `||`. Run only the helper command and handle a missing/nonzero result from the tool response.</violation>
<violation number="2" location="scripts/gstack2/generate-skill-tree.ts:528">
P2: The provider selector exceeds AskUserQuestion's four-option limit with five providers plus decline, so a single offer cannot expose the full decision space. Specify compliant ≤4-option batching/splitting so no provider or decline choice is silently omitted.</violation>
<violation number="3" location="scripts/gstack2/generate-skill-tree.ts:535">
P3: The generated `references/WEB-DATA.md` batch task order truncates at "local browser" but `TASK_RANKINGS.batch.order` includes `"aside"` as the 5th-ranked option after `"local-browser"`. While the serialization note covers both browsers conceptually, the doc lists only one browser by name, so the listed order doesn't match the full ranking. Either append `, Aside` or change the phrasing to group both browsers together (e.g., "any browser — serial, wrong tool past a handful of pages") for consistency.</violation>
<violation number="4" location="scripts/gstack2/generate-skill-tree.ts:536">
P2: The generated `references/WEB-DATA.md` hostile task order omits Exa, but `TASK_RANKINGS.hostile.order` in `lib/web-data.ts` includes `"exa"` as the 4th-ranked option. When a user has Exa configured but not Firecrawl, Aside, or Context.dev, the code-level `recommend("hostile", ...)` will rightfully return Exa as the best available provider. However, the generated reference doc says only "Firecrawl, then Aside…, Context.dev" and never mentions Exa, so an agent reading the doc as the authoritative contract would not know Exa is a valid fallback for hostile sites. Align the generated doc with the ranking: append `, Exa` to the hostile provider list (or add a note about why it's excluded if the omission is intentional).</violation>
</file>
<file name="lib/web-data.ts">
<violation number="1" location="lib/web-data.ts:21">
P0: Public scraping/search can now route to Firecrawl or Exa even though the repository privacy contract permits Context.dev as the only external web-context service. These providers and their egress paths should be removed rather than treating a selection in `web-data.json` as authorization to bypass that boundary.</violation>
<violation number="2" location="lib/web-data.ts:113">
P2: Provider state can be read from and written to a different directory than the rest of the runtime for supported GSTACK_HOME forms such as `~/.custom`. Reuse `resolveGstackHome({ env })` instead of duplicating path resolution.</violation>
<violation number="3" location="lib/web-data.ts:122">
P1: An offline setup or a config whose key was removed is reported as available, so `recommend()` can select Context.dev even though the runtime will reject every operation. Availability should come from the canonical runtime status check, including key presence, coherent selection, and verified validation.</violation>
<violation number="4" location="lib/web-data.ts:129">
P3: `onPath` calls `execFileSync` with no timeout option. If the `which`/`where` command hangs — for example on a system with slow PATH resolution over a network mount, or if a custom `which` wrapper blocks — the entire process hangs indefinitely. Add a reasonable timeout (e.g., 5 seconds) so that availability detection degrades gracefully instead of blocking the process.</violation>
<violation number="5" location="lib/web-data.ts:155">
P2: Authenticated-page work can be recommended to an unusable Aside installation because any executable named `aside` is treated as available. Availability should use the existing readiness contract rather than PATH presence alone.</violation>
<violation number="6" location="lib/web-data.ts:169">
P2: Users cannot persist different choices for search, scrape, and authenticated work because the store has one global `provider`. The selection schema and setter should key provider choices by `WebDataTask`.</violation>
</file>
<file name="CHANGELOG.md">
<violation number="1" location="CHANGELOG.md:10">
P1: This publishes a 1.65.0.0 release entry while the authoritative status remains `BLOCKED`, directly contradicting the changelog banner and release gate. The entry should be deferred until `/ship` after STATUS authorizes release.</violation>
<violation number="2" location="CHANGELOG.md:12">
P3: This entry does not follow the mandatory release-summary format: the headline exceeds 14 words and the numbers section lacks the required `BEFORE/AFTER/Δ` table. Rework the summary to the repository's release-doc structure before shipping.</violation>
<violation number="3" location="CHANGELOG.md:21">
P2: Users and contributors cannot reproduce or audit the ranking claims from developer-local `~` paths that are absent from the repository. Commit the benchmark/report and link repository-relative artifacts before describing these measurements as reproducible.</violation>
<violation number="4" location="CHANGELOG.md:44">
P1: The new `gstack-web-data` command lacks the accepted issue required by the repository's command/service guardrail. This feature needs issue-linked justification before its release documentation is accepted.</violation>
</file>
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
| @@ -0,0 +1,229 @@ | |||
| /** | |||
There was a problem hiding this comment.
P0: Public scraping/search can now route to Firecrawl or Exa even though the repository privacy contract permits Context.dev as the only external web-context service. These providers and their egress paths should be removed rather than treating a selection in web-data.json as authorization to bypass that boundary.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/web-data.ts, line 21:
<comment>Public scraping/search can now route to Firecrawl or Exa even though the repository privacy contract permits Context.dev as the only external web-context service. These providers and their egress paths should be removed rather than treating a selection in `web-data.json` as authorization to bypass that boundary.</comment>
<file context>
@@ -0,0 +1,229 @@
+import { homedir } from "os";
+import { dirname, join } from "path";
+
+export type WebDataProviderId = "firecrawl" | "exa" | "context" | "aside" | "local-browser";
+export type WebDataTask = "scrape" | "crawl" | "search" | "batch" | "hostile" | "authenticated";
+
</file context>
| ### Added | ||
|
|
||
| - `references/WEB-DATA.md` in all five skills: the optional scraping-provider contract. Official API first, provider menu once via AskUserQuestion, best-available recommendation per task, exact setup steps for better-but-unconfigured providers, decline persisted, hand-rolled path always valid. | ||
| - `gstack-web-data` CLI (`status`, `options`, `select`, `recommend`) with availability detection: Firecrawl/Exa env keys, Context.dev runtime consent, `aside` CLI on PATH, installed browse binary. Ships with the managed runtime. |
There was a problem hiding this comment.
P1: The new gstack-web-data command lacks the accepted issue required by the repository's command/service guardrail. This feature needs issue-linked justification before its release documentation is accepted.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At CHANGELOG.md, line 44:
<comment>The new `gstack-web-data` command lacks the accepted issue required by the repository's command/service guardrail. This feature needs issue-linked justification before its release documentation is accepted.</comment>
<file context>
@@ -7,6 +7,48 @@
+### Added
+
+- `references/WEB-DATA.md` in all five skills: the optional scraping-provider contract. Official API first, provider menu once via AskUserQuestion, best-available recommendation per task, exact setup steps for better-but-unconfigured providers, decline persisted, hand-rolled path always valid.
+- `gstack-web-data` CLI (`status`, `options`, `select`, `recommend`) with availability detection: Firecrawl/Exa env keys, Context.dev runtime consent, `aside` CLI on PATH, installed browse binary. Ships with the managed runtime.
+- Dispatch-protocol trigger in all five dispatchers: read WEB-DATA.md before designing or performing bulk web-data acquisition, whether the agent fetches directly or a plan designs the pipeline.
+
</file context>
| > completion state and remaining P0 gates. No version bump or release claim is | ||
| > made here while that status holds. | ||
|
|
||
| ## [1.65.0.0] - 2026-07-24 |
There was a problem hiding this comment.
P1: This publishes a 1.65.0.0 release entry while the authoritative status remains BLOCKED, directly contradicting the changelog banner and release gate. The entry should be deferred until /ship after STATUS authorizes release.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At CHANGELOG.md, line 10:
<comment>This publishes a 1.65.0.0 release entry while the authoritative status remains `BLOCKED`, directly contradicting the changelog banner and release gate. The entry should be deferred until `/ship` after STATUS authorizes release.</comment>
<file context>
@@ -7,6 +7,48 @@
> completion state and remaining P0 gates. No version bump or release claim is
> made here while that status holds.
+## [1.65.0.0] - 2026-07-24
+
+**Scraping now has a menu. Measured, ranked,**
</file context>
| try { | ||
| const cfg = JSON.parse(readFileSync(p, "utf-8")); | ||
| const net = cfg?.network; | ||
| if (net?.mode === "context" && net?.consent === true) return { ok: true, detail: "consented (network mode: context)" }; |
There was a problem hiding this comment.
P1: An offline setup or a config whose key was removed is reported as available, so recommend() can select Context.dev even though the runtime will reject every operation. Availability should come from the canonical runtime status check, including key presence, coherent selection, and verified validation.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/web-data.ts, line 122:
<comment>An offline setup or a config whose key was removed is reported as available, so `recommend()` can select Context.dev even though the runtime will reject every operation. Availability should come from the canonical runtime status check, including key presence, coherent selection, and verified validation.</comment>
<file context>
@@ -0,0 +1,229 @@
+ try {
+ const cfg = JSON.parse(readFileSync(p, "utf-8"));
+ const net = cfg?.network;
+ if (net?.mode === "context" && net?.consent === true) return { ok: true, detail: "consented (network mode: context)" };
+ return { ok: false, detail: "runtime present but Context.dev not selected/consented" };
+ } catch {
</file context>
| '- **crawl** (many pages from one site): Firecrawl, then Context.dev, Exa (cached subpages, freshness caveat), Aside, local browser.', | ||
| '- **search** (find the URLs first): Exa, then Firecrawl, Aside\'s agent. Context.dev search is deprecated upstream and the local browser has none.', | ||
| '- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser. Browsers serialize — wrong tool past a handful of pages.', | ||
| '- **hostile** (anti-bot sites): Firecrawl, then Aside (the user\'s real browser session), Context.dev. The plain local headless browser gets bot-walled.', |
There was a problem hiding this comment.
P2: The generated references/WEB-DATA.md hostile task order omits Exa, but TASK_RANKINGS.hostile.order in lib/web-data.ts includes "exa" as the 4th-ranked option. When a user has Exa configured but not Firecrawl, Aside, or Context.dev, the code-level recommend("hostile", ...) will rightfully return Exa as the best available provider. However, the generated reference doc says only "Firecrawl, then Aside…, Context.dev" and never mentions Exa, so an agent reading the doc as the authoritative contract would not know Exa is a valid fallback for hostile sites. Align the generated doc with the ranking: append , Exa to the hostile provider list (or add a note about why it's excluded if the omission is intentional).
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At scripts/gstack2/generate-skill-tree.ts, line 536:
<comment>The generated `references/WEB-DATA.md` hostile task order omits Exa, but `TASK_RANKINGS.hostile.order` in `lib/web-data.ts` includes `"exa"` as the 4th-ranked option. When a user has Exa configured but not Firecrawl, Aside, or Context.dev, the code-level `recommend("hostile", ...)` will rightfully return Exa as the best available provider. However, the generated reference doc says only "Firecrawl, then Aside…, Context.dev" and never mentions Exa, so an agent reading the doc as the authoritative contract would not know Exa is a valid fallback for hostile sites. Align the generated doc with the ranking: append `, Exa` to the hostile provider list (or add a note about why it's excluded if the omission is intentional).</comment>
<file context>
@@ -517,6 +517,32 @@ function webContextContract(): string {
+ '- **crawl** (many pages from one site): Firecrawl, then Context.dev, Exa (cached subpages, freshness caveat), Aside, local browser.',
+ '- **search** (find the URLs first): Exa, then Firecrawl, Aside\'s agent. Context.dev search is deprecated upstream and the local browser has none.',
+ '- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser. Browsers serialize — wrong tool past a handful of pages.',
+ '- **hostile** (anti-bot sites): Firecrawl, then Aside (the user\'s real browser session), Context.dev. The plain local headless browser gets bot-walled.',
+ '- **authenticated** (logged-in/account pages): Aside only. Never send authenticated pages, private addresses, or credentials to any API provider.',
+ '',
</file context>
| '- **hostile** (anti-bot sites): Firecrawl, then Aside (the user\'s real browser session), Context.dev. The plain local headless browser gets bot-walled.', | |
| '- **hostile** (anti-bot sites): Firecrawl, then Aside (the user\'s real browser session), Context.dev, then Exa. The plain local headless browser gets bot-walled.', |
| { id: "context", available: ctx.ok, detail: ctx.detail }, | ||
| { | ||
| id: "aside", | ||
| available: asideAvailable, |
There was a problem hiding this comment.
P2: Authenticated-page work can be recommended to an unusable Aside installation because any executable named aside is treated as available. Availability should use the existing readiness contract rather than PATH presence alone.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/web-data.ts, line 155:
<comment>Authenticated-page work can be recommended to an unusable Aside installation because any executable named `aside` is treated as available. Availability should use the existing readiness contract rather than PATH presence alone.</comment>
<file context>
@@ -0,0 +1,229 @@
+ { id: "context", available: ctx.ok, detail: ctx.detail },
+ {
+ id: "aside",
+ available: asideAvailable,
+ detail: asideAvailable ? "aside CLI on PATH" : "aside CLI not found",
+ },
</file context>
| // ---------- selection store ---------- | ||
|
|
||
| export interface WebDataSelection { | ||
| provider: WebDataProviderId | null; |
There was a problem hiding this comment.
P2: Users cannot persist different choices for search, scrape, and authenticated work because the store has one global provider. The selection schema and setter should key provider choices by WebDataTask.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/web-data.ts, line 169:
<comment>Users cannot persist different choices for search, scrape, and authenticated work because the store has one global `provider`. The selection schema and setter should key provider choices by `WebDataTask`.</comment>
<file context>
@@ -0,0 +1,229 @@
+// ---------- selection store ----------
+
+export interface WebDataSelection {
+ provider: WebDataProviderId | null;
+ /** User explicitly chose none — never offer again. */
+ declined: boolean;
</file context>
| '- **scrape** (single public page): Firecrawl, then Exa, Context.dev, Aside, local browser.', | ||
| '- **crawl** (many pages from one site): Firecrawl, then Context.dev, Exa (cached subpages, freshness caveat), Aside, local browser.', | ||
| '- **search** (find the URLs first): Exa, then Firecrawl, Aside\'s agent. Context.dev search is deprecated upstream and the local browser has none.', | ||
| '- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser. Browsers serialize — wrong tool past a handful of pages.', |
There was a problem hiding this comment.
P3: The generated references/WEB-DATA.md batch task order truncates at "local browser" but TASK_RANKINGS.batch.order includes "aside" as the 5th-ranked option after "local-browser". While the serialization note covers both browsers conceptually, the doc lists only one browser by name, so the listed order doesn't match the full ranking. Either append , Aside or change the phrasing to group both browsers together (e.g., "any browser — serial, wrong tool past a handful of pages") for consistency.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At scripts/gstack2/generate-skill-tree.ts, line 535:
<comment>The generated `references/WEB-DATA.md` batch task order truncates at "local browser" but `TASK_RANKINGS.batch.order` includes `"aside"` as the 5th-ranked option after `"local-browser"`. While the serialization note covers both browsers conceptually, the doc lists only one browser by name, so the listed order doesn't match the full ranking. Either append `, Aside` or change the phrasing to group both browsers together (e.g., "any browser — serial, wrong tool past a handful of pages") for consistency.</comment>
<file context>
@@ -517,6 +517,32 @@ function webContextContract(): string {
+ '- **scrape** (single public page): Firecrawl, then Exa, Context.dev, Aside, local browser.',
+ '- **crawl** (many pages from one site): Firecrawl, then Context.dev, Exa (cached subpages, freshness caveat), Aside, local browser.',
+ '- **search** (find the URLs first): Exa, then Firecrawl, Aside\'s agent. Context.dev search is deprecated upstream and the local browser has none.',
+ '- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser. Browsers serialize — wrong tool past a handful of pages.',
+ '- **hostile** (anti-bot sites): Firecrawl, then Aside (the user\'s real browser session), Context.dev. The plain local headless browser gets bot-walled.',
+ '- **authenticated** (logged-in/account pages): Aside only. Never send authenticated pages, private addresses, or credentials to any API provider.',
</file context>
| '- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser. Browsers serialize — wrong tool past a handful of pages.', | |
| '- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser, then Aside. Browsers serialize — wrong tool past a handful of pages.', |
| } | ||
| } | ||
|
|
||
| function onPath(bin: string, env: NodeJS.ProcessEnv): boolean { |
There was a problem hiding this comment.
P3: onPath calls execFileSync with no timeout option. If the which/where command hangs — for example on a system with slow PATH resolution over a network mount, or if a custom which wrapper blocks — the entire process hangs indefinitely. Add a reasonable timeout (e.g., 5 seconds) so that availability detection degrades gracefully instead of blocking the process.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/web-data.ts, line 129:
<comment>`onPath` calls `execFileSync` with no timeout option. If the `which`/`where` command hangs — for example on a system with slow PATH resolution over a network mount, or if a custom `which` wrapper blocks — the entire process hangs indefinitely. Add a reasonable timeout (e.g., 5 seconds) so that availability detection degrades gracefully instead of blocking the process.</comment>
<file context>
@@ -0,0 +1,229 @@
+ }
+}
+
+function onPath(bin: string, env: NodeJS.ProcessEnv): boolean {
+ try {
+ execFileSync(process.platform === "win32" ? "where" : "which", [bin], { stdio: "pipe", env: { ...env } });
</file context>
| @@ -7,6 +7,48 @@ | |||
| > completion state and remaining P0 gates. No version bump or release claim is | |||
There was a problem hiding this comment.
P3: This entry does not follow the mandatory release-summary format: the headline exceeds 14 words and the numbers section lacks the required BEFORE/AFTER/Δ table. Rework the summary to the repository's release-doc structure before shipping.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At CHANGELOG.md, line 12:
<comment>This entry does not follow the mandatory release-summary format: the headline exceeds 14 words and the numbers section lacks the required `BEFORE/AFTER/Δ` table. Rework the summary to the repository's release-doc structure before shipping.</comment>
<file context>
@@ -7,6 +7,48 @@
+## [1.65.0.0] - 2026-07-24
+
+**Scraping now has a menu. Measured, ranked,**
+**and it tells you what to install when the best tool is missing.**
+
</file context>
What
Optional web-data (scraping/crawling) provider contract, closing the gap where skills designed bespoke scrapers without ever offering options (observed: a flightconnections /plan hand-rolled a cheerio pipeline for ~1000 Wikipedia pages, no Context.dev check, no provider menu).
references/WEB-DATA.mdemitted into all five skill trees + a dispatch-protocol trigger: read it before designing or performing bulk web-data acquisition (agent-side fetch OR plan-designed pipeline).gstack-web-dataCLI (status|options|select|recommend): availability detection (Firecrawl/Exa env keys, Context.dev runtime consent,asideCLI, browse binary), persisted selection at$GSTACK_HOME/web-data.json, per-task recommendation. Registered in the managed-runtime helper manifest.Contract behavior
scrape|crawl|search|batch|hostile|authenticated).Testing
test/web-data.test.ts): rankings integrity, availability detection, selection store, recommend fallbacks, authenticated hard rail.bun run test:gstack2: 39/41 — the 2 failures (gstack2-runtime-install.test.ts, INSTALL_SOURCE_INCOMPLETE) reproduce identically on origin-main-era basetime-attack/datain a clean worktree (verified twice), pre-existing.bun run eval:select: 0/176 E2E, 0/28 LLM-judge tests affected by this diff.🤖 Generated with Claude Code
Summary by cubic
Adds an optional, measured web‑data provider contract with a one‑time menu and per‑task recommendations so scraping/crawling uses the best available tool instead of hand‑rolled fetches. Ships a
gstack-web-dataCLI, dispatch gates across skills, and strict egress rules.New Features
gstack-web-dataCLI:status|options|select|recommend; detects Firecrawl, Exa, Context.dev,aside, and the local browser; recommends per task (scrape|crawl|search|batch|hostile|authenticated).lib/web-data.tsfrom a live bakeoff; selection persists at$GSTACK_HOME/web-data.json; 14 unit tests cover rankings, availability, selection, and auth rails.references/WEB-DATA.mdbefore bulk web‑data work; official APIs win first; a one‑time AskUserQuestion offers providers with setup hints; decline persists and is respected.aside) or are skipped; helper is installed with the managed runtime.Migration
gstack-web-data optionsto see availability; thengstack-web-data select <firecrawl|exa|context|aside|local-browser|none>.asidefor logged‑in pages; never send authenticated pages to API providers.Written for commit d3987bb. Summary will update on new commits.