Skip to content

feat: measured web-data provider contract — Firecrawl/Exa/Context.dev/Aside/local, selectable per task#40

Open
time-attack wants to merge 6 commits into
mainfrom
time-attack/web-data-providers
Open

feat: measured web-data provider contract — Firecrawl/Exa/Context.dev/Aside/local, selectable per task#40
time-attack wants to merge 6 commits into
mainfrom
time-attack/web-data-providers

Conversation

@time-attack

@time-attack time-attack commented Jul 24, 2026

Copy link
Copy Markdown
Owner

What

Optional web-data (scraping/crawling) provider contract, closing the gap where skills designed bespoke scrapers without ever offering options (observed: a flightconnections /plan hand-rolled a cheerio pipeline for ~1000 Wikipedia pages, no Context.dev check, no provider menu).

  • references/WEB-DATA.md emitted into all five skill trees + a dispatch-protocol trigger: read it before designing or performing bulk web-data acquisition (agent-side fetch OR plan-designed pipeline).
  • gstack-web-data CLI (status|options|select|recommend): availability detection (Firecrawl/Exa env keys, Context.dev runtime consent, aside CLI, browse binary), persisted selection at $GSTACK_HOME/web-data.json, per-task recommendation. Registered in the managed-runtime helper manifest.
  • Measured rankings, not vibes: sourced from a 7-task live bakeoff (2026-07-23) across Firecrawl, Exa, Context.dev, Aside, local browse, and plain fetch. Firecrawl was the only API passing all 7 live (Amazon anti-bot in 1.1s clean markdown); the local headless browser got bot-walled; Aside is the only option for logged-in pages and is ranked never-for-bulk (22.6s / 8 pages).

Contract behavior

  • Official API for the source wins first (MediaWiki for Wikipedia — no scraper, no question).
  • One-time AskUserQuestion menu with availability + setup hints; best AVAILABLE provider recommended per task (scrape|crawl|search|batch|hostile|authenticated).
  • Best provider not installed → exact setup step (signup URL / env var / install) + best already-installed option for right now. Never blocks, never auto-installs.
  • Decline persists; never re-asked. Hand-rolled path stays fully valid.
  • Egress rails: off-machine providers get public URLs only — never authenticated pages, private addresses, project content, or credentials. Authenticated pages route to the user's own browser (Aside) or nowhere.

Testing

  • 14 new unit tests (test/web-data.test.ts): rankings integrity, availability detection, selection store, recommend fallbacks, authenticated hard rail.
  • bun run test:gstack2: 39/41 — the 2 failures (gstack2-runtime-install.test.ts, INSTALL_SOURCE_INCOMPLETE) reproduce identically on origin-main-era base time-attack/data in a clean worktree (verified twice), pre-existing.
  • Full free tier: 11 failures total, all reproduced on base with identical names/counts (golden ship skills, extension audit, office-hours carve, gbrain detection overrides, the 2 installer tests). None caused by this branch.
  • bun run eval:select: 0/176 E2E, 0/28 LLM-judge tests affected by this diff.
  • Live CLI verification: nothing-installed machine → setup-first messaging; partial install → best-available + setup hints for better options.

🤖 Generated with Claude Code


Summary by cubic

Adds an optional, measured web‑data provider contract with a one‑time menu and per‑task recommendations so scraping/crawling uses the best available tool instead of hand‑rolled fetches. Ships a gstack-web-data CLI, dispatch gates across skills, and strict egress rules.

  • New Features

    • gstack-web-data CLI: status|options|select|recommend; detects Firecrawl, Exa, Context.dev, aside, and the local browser; recommends per task (scrape|crawl|search|batch|hostile|authenticated).
    • Provider registry and rankings in lib/web-data.ts from a live bakeoff; selection persists at $GSTACK_HOME/web-data.json; 14 unit tests cover rankings, availability, selection, and auth rails.
    • All skills now read references/WEB-DATA.md before bulk web‑data work; official APIs win first; a one‑time AskUserQuestion offers providers with setup hints; decline persists and is respected.
    • Strict egress: off‑machine providers get public URLs only; authenticated pages route to the user’s browser (aside) or are skipped; helper is installed with the managed runtime.
  • Migration

    • Run gstack-web-data options to see availability; then gstack-web-data select <firecrawl|exa|context|aside|local-browser|none>.
    • Use aside for logged‑in pages; never send authenticated pages to API providers.
    • If you don’t select anything, gstack proceeds with an official API, a hand‑rolled fetcher, or the local browser.

Written for commit d3987bb. Summary will update on new commits.

Review in cubic

Sinabina and others added 6 commits July 23, 2026 18:05
gstack-web-data status/options/select/recommend over five providers
(Firecrawl, Exa, Context.dev, Aside, local browser). Availability is
detected (env keys, runtime consent, aside CLI, browse binary); per-task
rankings come from the measured 2026-07-23 provider bakeoff. Recommend
returns the best AVAILABLE provider and names better-but-unconfigured
ones with their exact setup step.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New webDataContract() reference emitted into all five skill trees, plus
a dispatch-protocol trigger: read references/WEB-DATA.md before
designing or performing bulk web-data acquisition (scraping, crawling,
dataset building) — the gate the flightconnections plan never hit. The
contract offers the provider menu once via gstack-web-data, recommends
the best available provider per task, surfaces setup steps for
better-but-unconfigured ones, keeps official APIs first and the
hand-rolled path always valid, and generalizes the Context.dev
public-URL egress rules to all off-machine providers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Registers bin/gstack-web-data in DEFAULT_RUNTIME_HELPERS and lib/web-data.ts
in the helper dependency closure, so $GSTACK_HOME/bin/gstack-web-data
resolves on installed runtimes as the WEB-DATA contract expects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cursor

cursor Bot commented Jul 24, 2026

Copy link
Copy Markdown

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

14 issues found across 18 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="scripts/gstack2/generate-skill-tree.ts">

<violation number="1" location="scripts/gstack2/generate-skill-tree.ts:526">
P2: Every bulk web-data workflow is instructed to violate the always-loaded authority policy because this command uses both redirection and `||`. Run only the helper command and handle a missing/nonzero result from the tool response.</violation>

<violation number="2" location="scripts/gstack2/generate-skill-tree.ts:528">
P2: The provider selector exceeds AskUserQuestion's four-option limit with five providers plus decline, so a single offer cannot expose the full decision space. Specify compliant ≤4-option batching/splitting so no provider or decline choice is silently omitted.</violation>

<violation number="3" location="scripts/gstack2/generate-skill-tree.ts:535">
P3: The generated `references/WEB-DATA.md` batch task order truncates at "local browser" but `TASK_RANKINGS.batch.order` includes `"aside"` as the 5th-ranked option after `"local-browser"`. While the serialization note covers both browsers conceptually, the doc lists only one browser by name, so the listed order doesn't match the full ranking. Either append `, Aside` or change the phrasing to group both browsers together (e.g., "any browser — serial, wrong tool past a handful of pages") for consistency.</violation>

<violation number="4" location="scripts/gstack2/generate-skill-tree.ts:536">
P2: The generated `references/WEB-DATA.md` hostile task order omits Exa, but `TASK_RANKINGS.hostile.order` in `lib/web-data.ts` includes `"exa"` as the 4th-ranked option. When a user has Exa configured but not Firecrawl, Aside, or Context.dev, the code-level `recommend("hostile", ...)` will rightfully return Exa as the best available provider. However, the generated reference doc says only "Firecrawl, then Aside…, Context.dev" and never mentions Exa, so an agent reading the doc as the authoritative contract would not know Exa is a valid fallback for hostile sites. Align the generated doc with the ranking: append `, Exa` to the hostile provider list (or add a note about why it's excluded if the omission is intentional).</violation>
</file>

<file name="lib/web-data.ts">

<violation number="1" location="lib/web-data.ts:21">
P0: Public scraping/search can now route to Firecrawl or Exa even though the repository privacy contract permits Context.dev as the only external web-context service. These providers and their egress paths should be removed rather than treating a selection in `web-data.json` as authorization to bypass that boundary.</violation>

<violation number="2" location="lib/web-data.ts:113">
P2: Provider state can be read from and written to a different directory than the rest of the runtime for supported GSTACK_HOME forms such as `~/.custom`. Reuse `resolveGstackHome({ env })` instead of duplicating path resolution.</violation>

<violation number="3" location="lib/web-data.ts:122">
P1: An offline setup or a config whose key was removed is reported as available, so `recommend()` can select Context.dev even though the runtime will reject every operation. Availability should come from the canonical runtime status check, including key presence, coherent selection, and verified validation.</violation>

<violation number="4" location="lib/web-data.ts:129">
P3: `onPath` calls `execFileSync` with no timeout option. If the `which`/`where` command hangs — for example on a system with slow PATH resolution over a network mount, or if a custom `which` wrapper blocks — the entire process hangs indefinitely. Add a reasonable timeout (e.g., 5 seconds) so that availability detection degrades gracefully instead of blocking the process.</violation>

<violation number="5" location="lib/web-data.ts:155">
P2: Authenticated-page work can be recommended to an unusable Aside installation because any executable named `aside` is treated as available. Availability should use the existing readiness contract rather than PATH presence alone.</violation>

<violation number="6" location="lib/web-data.ts:169">
P2: Users cannot persist different choices for search, scrape, and authenticated work because the store has one global `provider`. The selection schema and setter should key provider choices by `WebDataTask`.</violation>
</file>

<file name="CHANGELOG.md">

<violation number="1" location="CHANGELOG.md:10">
P1: This publishes a 1.65.0.0 release entry while the authoritative status remains `BLOCKED`, directly contradicting the changelog banner and release gate. The entry should be deferred until `/ship` after STATUS authorizes release.</violation>

<violation number="2" location="CHANGELOG.md:12">
P3: This entry does not follow the mandatory release-summary format: the headline exceeds 14 words and the numbers section lacks the required `BEFORE/AFTER/Δ` table. Rework the summary to the repository's release-doc structure before shipping.</violation>

<violation number="3" location="CHANGELOG.md:21">
P2: Users and contributors cannot reproduce or audit the ranking claims from developer-local `~` paths that are absent from the repository. Commit the benchmark/report and link repository-relative artifacts before describing these measurements as reproducible.</violation>

<violation number="4" location="CHANGELOG.md:44">
P1: The new `gstack-web-data` command lacks the accepted issue required by the repository's command/service guardrail. This feature needs issue-linked justification before its release documentation is accepted.</violation>
</file>

Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic

Comment thread lib/web-data.ts
@@ -0,0 +1,229 @@
/**

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P0: Public scraping/search can now route to Firecrawl or Exa even though the repository privacy contract permits Context.dev as the only external web-context service. These providers and their egress paths should be removed rather than treating a selection in web-data.json as authorization to bypass that boundary.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/web-data.ts, line 21:

<comment>Public scraping/search can now route to Firecrawl or Exa even though the repository privacy contract permits Context.dev as the only external web-context service. These providers and their egress paths should be removed rather than treating a selection in `web-data.json` as authorization to bypass that boundary.</comment>

<file context>
@@ -0,0 +1,229 @@
+import { homedir } from "os";
+import { dirname, join } from "path";
+
+export type WebDataProviderId = "firecrawl" | "exa" | "context" | "aside" | "local-browser";
+export type WebDataTask = "scrape" | "crawl" | "search" | "batch" | "hostile" | "authenticated";
+
</file context>
Fix with cubic

Comment thread CHANGELOG.md
### Added

- `references/WEB-DATA.md` in all five skills: the optional scraping-provider contract. Official API first, provider menu once via AskUserQuestion, best-available recommendation per task, exact setup steps for better-but-unconfigured providers, decline persisted, hand-rolled path always valid.
- `gstack-web-data` CLI (`status`, `options`, `select`, `recommend`) with availability detection: Firecrawl/Exa env keys, Context.dev runtime consent, `aside` CLI on PATH, installed browse binary. Ships with the managed runtime.

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: The new gstack-web-data command lacks the accepted issue required by the repository's command/service guardrail. This feature needs issue-linked justification before its release documentation is accepted.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At CHANGELOG.md, line 44:

<comment>The new `gstack-web-data` command lacks the accepted issue required by the repository's command/service guardrail. This feature needs issue-linked justification before its release documentation is accepted.</comment>

<file context>
@@ -7,6 +7,48 @@
+### Added
+
+- `references/WEB-DATA.md` in all five skills: the optional scraping-provider contract. Official API first, provider menu once via AskUserQuestion, best-available recommendation per task, exact setup steps for better-but-unconfigured providers, decline persisted, hand-rolled path always valid.
+- `gstack-web-data` CLI (`status`, `options`, `select`, `recommend`) with availability detection: Firecrawl/Exa env keys, Context.dev runtime consent, `aside` CLI on PATH, installed browse binary. Ships with the managed runtime.
+- Dispatch-protocol trigger in all five dispatchers: read WEB-DATA.md before designing or performing bulk web-data acquisition, whether the agent fetches directly or a plan designs the pipeline.
+
</file context>
Fix with cubic

Comment thread CHANGELOG.md
> completion state and remaining P0 gates. No version bump or release claim is
> made here while that status holds.

## [1.65.0.0] - 2026-07-24

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: This publishes a 1.65.0.0 release entry while the authoritative status remains BLOCKED, directly contradicting the changelog banner and release gate. The entry should be deferred until /ship after STATUS authorizes release.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At CHANGELOG.md, line 10:

<comment>This publishes a 1.65.0.0 release entry while the authoritative status remains `BLOCKED`, directly contradicting the changelog banner and release gate. The entry should be deferred until `/ship` after STATUS authorizes release.</comment>

<file context>
@@ -7,6 +7,48 @@
 > completion state and remaining P0 gates. No version bump or release claim is
 > made here while that status holds.
 
+## [1.65.0.0] - 2026-07-24
+
+**Scraping now has a menu. Measured, ranked,**
</file context>
Fix with cubic

Comment thread lib/web-data.ts
try {
const cfg = JSON.parse(readFileSync(p, "utf-8"));
const net = cfg?.network;
if (net?.mode === "context" && net?.consent === true) return { ok: true, detail: "consented (network mode: context)" };

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: An offline setup or a config whose key was removed is reported as available, so recommend() can select Context.dev even though the runtime will reject every operation. Availability should come from the canonical runtime status check, including key presence, coherent selection, and verified validation.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/web-data.ts, line 122:

<comment>An offline setup or a config whose key was removed is reported as available, so `recommend()` can select Context.dev even though the runtime will reject every operation. Availability should come from the canonical runtime status check, including key presence, coherent selection, and verified validation.</comment>

<file context>
@@ -0,0 +1,229 @@
+  try {
+    const cfg = JSON.parse(readFileSync(p, "utf-8"));
+    const net = cfg?.network;
+    if (net?.mode === "context" && net?.consent === true) return { ok: true, detail: "consented (network mode: context)" };
+    return { ok: false, detail: "runtime present but Context.dev not selected/consented" };
+  } catch {
</file context>
Fix with cubic

'- **crawl** (many pages from one site): Firecrawl, then Context.dev, Exa (cached subpages, freshness caveat), Aside, local browser.',
'- **search** (find the URLs first): Exa, then Firecrawl, Aside\'s agent. Context.dev search is deprecated upstream and the local browser has none.',
'- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser. Browsers serialize — wrong tool past a handful of pages.',
'- **hostile** (anti-bot sites): Firecrawl, then Aside (the user\'s real browser session), Context.dev. The plain local headless browser gets bot-walled.',

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The generated references/WEB-DATA.md hostile task order omits Exa, but TASK_RANKINGS.hostile.order in lib/web-data.ts includes "exa" as the 4th-ranked option. When a user has Exa configured but not Firecrawl, Aside, or Context.dev, the code-level recommend("hostile", ...) will rightfully return Exa as the best available provider. However, the generated reference doc says only "Firecrawl, then Aside…, Context.dev" and never mentions Exa, so an agent reading the doc as the authoritative contract would not know Exa is a valid fallback for hostile sites. Align the generated doc with the ranking: append , Exa to the hostile provider list (or add a note about why it's excluded if the omission is intentional).

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At scripts/gstack2/generate-skill-tree.ts, line 536:

<comment>The generated `references/WEB-DATA.md` hostile task order omits Exa, but `TASK_RANKINGS.hostile.order` in `lib/web-data.ts` includes `"exa"` as the 4th-ranked option. When a user has Exa configured but not Firecrawl, Aside, or Context.dev, the code-level `recommend("hostile", ...)` will rightfully return Exa as the best available provider. However, the generated reference doc says only "Firecrawl, then Aside…, Context.dev" and never mentions Exa, so an agent reading the doc as the authoritative contract would not know Exa is a valid fallback for hostile sites. Align the generated doc with the ranking: append `, Exa` to the hostile provider list (or add a note about why it's excluded if the omission is intentional).</comment>

<file context>
@@ -517,6 +517,32 @@ function webContextContract(): string {
+    '- **crawl** (many pages from one site): Firecrawl, then Context.dev, Exa (cached subpages, freshness caveat), Aside, local browser.',
+    '- **search** (find the URLs first): Exa, then Firecrawl, Aside\'s agent. Context.dev search is deprecated upstream and the local browser has none.',
+    '- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser. Browsers serialize — wrong tool past a handful of pages.',
+    '- **hostile** (anti-bot sites): Firecrawl, then Aside (the user\'s real browser session), Context.dev. The plain local headless browser gets bot-walled.',
+    '- **authenticated** (logged-in/account pages): Aside only. Never send authenticated pages, private addresses, or credentials to any API provider.',
+    '',
</file context>
Suggested change
'- **hostile** (anti-bot sites): Firecrawl, then Aside (the user\'s real browser session), Context.dev. The plain local headless browser gets bot-walled.',
'- **hostile** (anti-bot sites): Firecrawl, then Aside (the user\'s real browser session), Context.dev, then Exa. The plain local headless browser gets bot-walled.',
Fix with cubic

Comment thread lib/web-data.ts
{ id: "context", available: ctx.ok, detail: ctx.detail },
{
id: "aside",
available: asideAvailable,

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Authenticated-page work can be recommended to an unusable Aside installation because any executable named aside is treated as available. Availability should use the existing readiness contract rather than PATH presence alone.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/web-data.ts, line 155:

<comment>Authenticated-page work can be recommended to an unusable Aside installation because any executable named `aside` is treated as available. Availability should use the existing readiness contract rather than PATH presence alone.</comment>

<file context>
@@ -0,0 +1,229 @@
+    { id: "context", available: ctx.ok, detail: ctx.detail },
+    {
+      id: "aside",
+      available: asideAvailable,
+      detail: asideAvailable ? "aside CLI on PATH" : "aside CLI not found",
+    },
</file context>
Fix with cubic

Comment thread lib/web-data.ts
// ---------- selection store ----------

export interface WebDataSelection {
provider: WebDataProviderId | null;

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Users cannot persist different choices for search, scrape, and authenticated work because the store has one global provider. The selection schema and setter should key provider choices by WebDataTask.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/web-data.ts, line 169:

<comment>Users cannot persist different choices for search, scrape, and authenticated work because the store has one global `provider`. The selection schema and setter should key provider choices by `WebDataTask`.</comment>

<file context>
@@ -0,0 +1,229 @@
+// ---------- selection store ----------
+
+export interface WebDataSelection {
+  provider: WebDataProviderId | null;
+  /** User explicitly chose none — never offer again. */
+  declined: boolean;
</file context>
Fix with cubic

'- **scrape** (single public page): Firecrawl, then Exa, Context.dev, Aside, local browser.',
'- **crawl** (many pages from one site): Firecrawl, then Context.dev, Exa (cached subpages, freshness caveat), Aside, local browser.',
'- **search** (find the URLs first): Exa, then Firecrawl, Aside\'s agent. Context.dev search is deprecated upstream and the local browser has none.',
'- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser. Browsers serialize — wrong tool past a handful of pages.',

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The generated references/WEB-DATA.md batch task order truncates at "local browser" but TASK_RANKINGS.batch.order includes "aside" as the 5th-ranked option after "local-browser". While the serialization note covers both browsers conceptually, the doc lists only one browser by name, so the listed order doesn't match the full ranking. Either append , Aside or change the phrasing to group both browsers together (e.g., "any browser — serial, wrong tool past a handful of pages") for consistency.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At scripts/gstack2/generate-skill-tree.ts, line 535:

<comment>The generated `references/WEB-DATA.md` batch task order truncates at "local browser" but `TASK_RANKINGS.batch.order` includes `"aside"` as the 5th-ranked option after `"local-browser"`. While the serialization note covers both browsers conceptually, the doc lists only one browser by name, so the listed order doesn't match the full ranking. Either append `, Aside` or change the phrasing to group both browsers together (e.g., "any browser — serial, wrong tool past a handful of pages") for consistency.</comment>

<file context>
@@ -517,6 +517,32 @@ function webContextContract(): string {
+    '- **scrape** (single public page): Firecrawl, then Exa, Context.dev, Aside, local browser.',
+    '- **crawl** (many pages from one site): Firecrawl, then Context.dev, Exa (cached subpages, freshness caveat), Aside, local browser.',
+    '- **search** (find the URLs first): Exa, then Firecrawl, Aside\'s agent. Context.dev search is deprecated upstream and the local browser has none.',
+    '- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser. Browsers serialize — wrong tool past a handful of pages.',
+    '- **hostile** (anti-bot sites): Firecrawl, then Aside (the user\'s real browser session), Context.dev. The plain local headless browser gets bot-walled.',
+    '- **authenticated** (logged-in/account pages): Aside only. Never send authenticated pages, private addresses, or credentials to any API provider.',
</file context>
Suggested change
'- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser. Browsers serialize — wrong tool past a handful of pages.',
'- **batch** (many known URLs): Firecrawl, then Exa, Context.dev, local browser, then Aside. Browsers serialize — wrong tool past a handful of pages.',
Fix with cubic

Comment thread lib/web-data.ts
}
}

function onPath(bin: string, env: NodeJS.ProcessEnv): boolean {

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: onPath calls execFileSync with no timeout option. If the which/where command hangs — for example on a system with slow PATH resolution over a network mount, or if a custom which wrapper blocks — the entire process hangs indefinitely. Add a reasonable timeout (e.g., 5 seconds) so that availability detection degrades gracefully instead of blocking the process.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/web-data.ts, line 129:

<comment>`onPath` calls `execFileSync` with no timeout option. If the `which`/`where` command hangs — for example on a system with slow PATH resolution over a network mount, or if a custom `which` wrapper blocks — the entire process hangs indefinitely. Add a reasonable timeout (e.g., 5 seconds) so that availability detection degrades gracefully instead of blocking the process.</comment>

<file context>
@@ -0,0 +1,229 @@
+  }
+}
+
+function onPath(bin: string, env: NodeJS.ProcessEnv): boolean {
+  try {
+    execFileSync(process.platform === "win32" ? "where" : "which", [bin], { stdio: "pipe", env: { ...env } });
</file context>
Fix with cubic

Comment thread CHANGELOG.md
@@ -7,6 +7,48 @@
> completion state and remaining P0 gates. No version bump or release claim is

@cubic-dev-ai cubic-dev-ai Bot Jul 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This entry does not follow the mandatory release-summary format: the headline exceeds 14 words and the numbers section lacks the required BEFORE/AFTER/Δ table. Rework the summary to the repository's release-doc structure before shipping.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At CHANGELOG.md, line 12:

<comment>This entry does not follow the mandatory release-summary format: the headline exceeds 14 words and the numbers section lacks the required `BEFORE/AFTER/Δ` table. Rework the summary to the repository's release-doc structure before shipping.</comment>

<file context>
@@ -7,6 +7,48 @@
 
+## [1.65.0.0] - 2026-07-24
+
+**Scraping now has a menu. Measured, ranked,**
+**and it tells you what to install when the best tool is missing.**
+
</file context>
Fix with cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant