diff --git a/.claude/skills/kild-cli/SKILL.md b/.claude/skills/kild-cli/SKILL.md index bcd6ca11..056247e2 100644 --- a/.claude/skills/kild-cli/SKILL.md +++ b/.claude/skills/kild-cli/SKILL.md @@ -44,6 +44,8 @@ driven from the cockpit UI over the engine's WebSocket, not the CLI.) | `kild worktree ls --project

` | List the project's `kild/*` worktrees | | `kild worktree rm --project

` | Remove a worktree (frees disk; the `kild/` branch persists) | | `kild worktree prune --project

` | Remove **and `-d`-delete the branch of** each `kild/*` worktree merged into the default branch (clean trees only; dirty/in-use ones are kept) | +| `kild web fetch [--format markdown\|html]` | Fetch a URL as markdown (in-process; keyless) | +| `kild web search ""` | Query the web via the configured SearXNG (`KILD_SEARXNG_URL`) | Add `--json` to any command for machine-readable output on stdout. @@ -139,6 +141,10 @@ kild agent ls --project ~/projects/myapp --json | jq -r '.[].name' `.pi/agents/`, or globally in `~/.config/kild/agents/` or `~/.claude/agents/`. The built-in `default` agent uses pi's own prompt. To add an agent, drop a file — kild only reads them. +- **Web access.** A `kild run` agent automatically gets `webfetch` (read a URL as + markdown — keyless, in-process) and, when `KILD_SEARXNG_URL` points at a SearXNG + instance, `web_search`. Works for any model. `KILD_WEB=off` disables both. Use + `kild web fetch/search` to exercise them directly without spending agent tokens. - **One-shot only.** `kild run` blocks until the agent finishes. For a long task, expect it to take a while; tool progress streams to stderr so you can see it working. There is no way yet to attach to or steer a running agent from the CLI. diff --git a/CLAUDE.md b/CLAUDE.md index 6a742b93..38ee7e54 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -136,6 +136,7 @@ kild/ │ │ │ ├── agents.ts # agents read from .kild/.claude/.pi convention dirs │ │ │ ├── sessions.ts # SessionManager: coding-agent SDK sessions → UiEvent stream │ │ │ ├── worktree.ts # [kild-owned] git worktree CRUD + ensureWorktree + merge-prune (NO @flue) +│ │ │ ├── web/ # [kild-owned] web_search (SearXNG seam) + webfetch (in-process) tools │ │ │ ├── run.ts # [Flue layer] one-shot run via Flue │ │ │ ├── comms-bus.ts # [Flue layer] agent-to-agent peer comms bus (Flue-demo; NOT the Room primitive) │ │ │ ├── brain.ts # [Flue layer] operator-mirror agent (kild capabilities as tools) @@ -173,6 +174,13 @@ kild/ self-contained (no `kild/*` imports) so it can be lifted into Flue verbatim. Worktrees **persist**: a session closing never removes one; only an explicit `kild worktree rm` / UI action or the automatic merge-prune does. +- **The web boundary.** kild owns the `web_search` / `webfetch` *tools* + (`engine/src/kild/web/`); the search *backend* is external — a self-hosted SearXNG + kild only points at via `KILD_SEARXNG_URL`. The engine never spawns or manages the + container. The backend sits behind a `SearchProvider` seam, so DDG/fastCRW/Tavily + are a new impl + a switch arm later — no tool or worker changes. `webfetch` is + in-process (turndown), keyless, and needs no backend. Works for any model (e.g. + MiniMax) since it's a plain pi tool, not a provider-native capability. ### Naming conventions diff --git a/engine/README.md b/engine/README.md index 28e6717c..d6d79359 100644 --- a/engine/README.md +++ b/engine/README.md @@ -15,12 +15,30 @@ bun run cli -- project ls --json # the kild CLI (secondary interface) bun run cli -- run --model anthropic/claude-haiku-4-5 "what files are here?" bun run cli -- run --worktree fix-auth "…" # run isolated in a kild/fix-auth worktree bun run cli -- worktree ls --project

# list/rm/prune kild worktrees +bun run cli -- web fetch https://example.com # read a url as markdown (in-process) +bun run cli -- web search "claude opus" # query the web (needs KILD_SEARXNG_URL) bun run typecheck && bun run lint # tsc + biome ``` `pi` must be on PATH and authenticated (`~/.pi/agent/auth.json` — Claude Max / ChatGPT OAuth work natively). +## Web tools + +Agents get `webfetch` (URL → markdown, in-process, keyless) and — when a search +backend is configured — `web_search`. Search is backed by a **self-hosted SearXNG** +kild only points at; it never runs the container: + +```bash +cd ../infra/searxng && docker compose up -d +export KILD_SEARXNG_URL=http://localhost:8888 +``` + +`webfetch` works regardless; without `KILD_SEARXNG_URL`, `web_search` is simply not +offered (the worker logs a one-line notice). `KILD_WEB=off` disables both. The search +backend sits behind a `SearchProvider` seam (`kild/web/search.ts`) — DDG/fastCRW/Tavily +are a new impl + a switch arm later, no tool/worker changes. + ## Layout ``` @@ -34,6 +52,7 @@ src/ agents.ts agents from .kild/.claude/.pi convention dirs sessions.ts SessionManager: coding-agent SDK sessions → UiEvent stream worktree.ts [kild-owned] git worktree CRUD + ensureWorktree + merge-prune (no @flue) + web/ web_search (SearXNG seam) + webfetch (in-process turndown) tools run/rooms/brain/observability/auth.ts [Flue layer] flue/ worktree-sandbox.ts [Flue layer] worktree() SandboxFactory — the upstream contribution diff --git a/engine/bun.lock b/engine/bun.lock index d4618df9..6876490d 100644 --- a/engine/bun.lock +++ b/engine/bun.lock @@ -11,12 +11,14 @@ "@flue/runtime": "^0.9.2", "hono": "^4.8.3", "just-bash": "^3.0.1", + "turndown": "^7.2.4", "valibot": "^1.1.0", }, "devDependencies": { "@biomejs/biome": "^2.0.0", "@flue/cli": "^0.9.2", "@types/node": "^22.0.0", + "@types/turndown": "^5.0.6", "typescript": "^5.6.0", }, }, @@ -392,6 +394,8 @@ "@types/retry": ["@types/retry@0.12.0", "", {}, "sha512-wWKOClTTiizcZhXnPY4wikVAwmdYHp8q6DmC+EJUzAMsycb7HB32Kh9RN4+0gExjmPmZSAQjgURXIGATPegAvA=="], + "@types/turndown": ["@types/turndown@5.0.6", "", {}, "sha512-ru00MoyeeouE5BX4gRL+6m/BsDfbRayOskWqUvh7CLGW+UXxHQItqALa38kKnOiZPqJrtzJUgAC2+F0rL1S4Pg=="], + "@valibot/to-json-schema": ["@valibot/to-json-schema@1.7.0", "", { "peerDependencies": { "valibot": "^1.4.0" } }, "sha512-Y3pPVibbIOHzohrlxSINvO7w/bvXkoYS3BQHoImV9ynE+bXKf171bdMucPurV2zp7gdmt0L1HCcNAsbo7cFRQw=="], "@vercel/detect-agent": ["@vercel/detect-agent@1.2.3", "", {}, "sha512-VYNCgUc0nOmC4WJmWw9GkrKdfr8Zl4/rxhC5SvgacBgxiW9W/9NRttUoHHXV8xdII3MaRgkZZVX8Ikzc/Jmjag=="], diff --git a/engine/package.json b/engine/package.json index c06f7ee0..88a5a7a8 100644 --- a/engine/package.json +++ b/engine/package.json @@ -25,12 +25,14 @@ "@flue/runtime": "^0.9.2", "hono": "^4.8.3", "just-bash": "^3.0.1", + "turndown": "^7.2.4", "valibot": "^1.1.0" }, "devDependencies": { "@biomejs/biome": "^2.0.0", "@flue/cli": "^0.9.2", "@types/node": "^22.0.0", + "@types/turndown": "^5.0.6", "typescript": "^5.6.0" } } diff --git a/engine/src/cli.ts b/engine/src/cli.ts index 3dce256d..dc775054 100644 --- a/engine/src/cli.ts +++ b/engine/src/cli.ts @@ -12,6 +12,9 @@ import { AuthStorage, createAgentSession, ModelRegistry } from '@earendil-works/ import { listAgents, resolveAgentInstructions } from './kild/agents.ts'; import { resolveModel, withRole } from './kild/models.ts'; import { addProject, findProject, loadProjects, removeProject } from './kild/projects.ts'; +import { fetchUrl } from './kild/web/fetch.ts'; +import { webSearchProvider } from './kild/web/search.ts'; +import { webTools } from './kild/web/tools.ts'; import { ensureWorktree, listWorktrees, @@ -30,6 +33,7 @@ const { values, positionals } = parseArgs({ model: { type: 'string' }, worktree: { type: 'string' }, participants: { type: 'string' }, // `kild room` participants, e.g. orchestrator,worker,reviewer + format: { type: 'string' }, // `kild web fetch` output format: markdown (default) | html }, }); @@ -53,12 +57,14 @@ async function dispatch(): Promise { return agent(action, rest); case 'worktree': return worktree(action, rest); + case 'web': + return web(action, rest); case 'run': return run([action, ...rest].filter(Boolean).join(' ')); case 'room': return room([action, ...rest].filter(Boolean).join(' ')); default: - console.error('usage: kild …'); + console.error('usage: kild …'); process.exit(2); } } @@ -168,6 +174,32 @@ async function worktree(action: string | undefined, args: string[]): Promise { + if (action === 'search') { + const query = args.join(' '); + if (!query) throw new Error('usage: kild web search ""'); + const provider = webSearchProvider(); + if (!provider) throw new Error('web search needs KILD_SEARXNG_URL (see infra/searxng)'); + const hits = await provider.search(query, 8); + if (json) return void console.log(JSON.stringify(hits, null, 2)); + if (hits.length === 0) return void console.error('no results'); + for (const [i, h] of hits.entries()) { + console.log(`${i + 1}. ${h.title}\n ${h.url}${h.snippet ? `\n ${h.snippet}` : ''}`); + } + } else if (action === 'fetch') { + const [url] = args; + if (!url) throw new Error('usage: kild web fetch [--format markdown|html]'); + const format = values.format === 'html' ? 'html' : 'markdown'; + const text = await fetchUrl(url, format); + if (json) console.log(JSON.stringify({ url, format, text }, null, 2)); + else console.log(text); + } else { + throw new Error('usage: kild web '); + } +} + async function run(prompt: string): Promise { if (!prompt) throw new Error('usage: kild run '); // If the engine is up, run THROUGH it so the session shows up in the cockpit; @@ -383,11 +415,13 @@ async function runInProcess(prompt: string): Promise { const registry = ModelRegistry.create(authStorage); const model = resolveModel(registry, values.model); + const tools = webTools(); const { session } = await createAgentSession({ model, authStorage, modelRegistry: registry, cwd, + customTools: tools.length ? tools : undefined, }); let text = ''; diff --git a/engine/src/kild/config.ts b/engine/src/kild/config.ts index bd4eec79..5fc6768f 100644 --- a/engine/src/kild/config.ts +++ b/engine/src/kild/config.ts @@ -11,3 +11,14 @@ export function kildHome(): string { const base = process.env.XDG_CONFIG_HOME ?? path.join(process.env.HOME ?? '', '.config'); return path.join(base, 'kild'); } + +/** Whether agents get the web tools. On by default; `KILD_WEB=off` is the kill-switch. */ +export function webEnabled(): boolean { + return (process.env.KILD_WEB ?? '').toLowerCase() !== 'off'; +} + +/** The self-hosted SearXNG base URL backing `web_search`, or undefined if unset. + * kild only *points at* this instance — it never spawns or manages it. */ +export function searxngUrl(): string | undefined { + return process.env.KILD_SEARXNG_URL || undefined; +} diff --git a/engine/src/kild/web/fetch-tool.ts b/engine/src/kild/web/fetch-tool.ts new file mode 100644 index 00000000..e1182c21 --- /dev/null +++ b/engine/src/kild/web/fetch-tool.ts @@ -0,0 +1,32 @@ +import type { ToolDefinition } from '@earendil-works/pi-coding-agent'; +import { Type } from 'typebox'; + +import { fetchUrl } from './fetch.ts'; + +/** + * The agent-facing `webfetch` tool. Plain HTTP + HTML→markdown (no headless browser, + * no service, no key). Always available when web is enabled — it needs no backend. + */ +export function createWebFetchTool(): ToolDefinition { + return { + name: 'webfetch', + label: 'Web Fetch', + description: + 'Fetch a public http(s) URL and return its content as markdown (default) or raw ' + + 'html. Use this to read a page found via web_search.', + promptSnippet: 'webfetch — read a url as markdown', + parameters: Type.Object({ + url: Type.String({ description: 'The http(s) URL to fetch.' }), + format: Type.Optional( + Type.Union([Type.Literal('markdown'), Type.Literal('html')], { + description: 'Output format (default markdown).', + }), + ), + }), + async execute(_toolCallId, params) { + const { url, format } = params as { url: string; format?: 'markdown' | 'html' }; + const text = await fetchUrl(url, format ?? 'markdown'); + return { content: [{ type: 'text' as const, text }], details: { url } }; + }, + }; +} diff --git a/engine/src/kild/web/fetch.test.ts b/engine/src/kild/web/fetch.test.ts new file mode 100644 index 00000000..3f886eaf --- /dev/null +++ b/engine/src/kild/web/fetch.test.ts @@ -0,0 +1,44 @@ +import { afterEach, expect, test } from 'bun:test'; + +import { fetchUrl } from './fetch.ts'; + +const realFetch = globalThis.fetch; +afterEach(() => { + globalThis.fetch = realFetch; +}); + +const serve = (body: string, contentType = 'text/html'): typeof fetch => + (async () => + new Response(body, { status: 200, headers: { 'content-type': contentType } })) as typeof fetch; + +test('converts HTML to markdown by default', async () => { + globalThis.fetch = serve('

Hi

Para

'); + const md = await fetchUrl('https://example.com'); + expect(md).toContain('# Hi'); + expect(md).toContain('Para'); +}); + +test('format=html returns the raw html', async () => { + globalThis.fetch = serve('

Hi

'); + expect(await fetchUrl('https://example.com', 'html')).toContain('

Hi

'); +}); + +test('non-html content is returned as-is (turndown would mangle it)', async () => { + globalThis.fetch = serve('{"a":1}', 'application/json'); + expect(await fetchUrl('https://api.example.com/x')).toBe('{"a":1}'); +}); + +test('truncates very large output', async () => { + globalThis.fetch = serve(`${'x'.repeat(60 * 1024)}`); + const out = await fetchUrl('https://example.com'); + expect(out).toContain('[truncated at'); + expect(out.length).toBeLessThan(55 * 1024); +}); + +test('rejects non-http(s) and private/loopback hosts before fetching', async () => { + await expect(fetchUrl('ftp://x/y')).rejects.toThrow(/http/); + await expect(fetchUrl('http://localhost/x')).rejects.toThrow(/private|loopback/); + await expect(fetchUrl('http://127.0.0.1/x')).rejects.toThrow(/private|loopback/); + await expect(fetchUrl('http://192.168.1.5/x')).rejects.toThrow(/private|loopback/); + await expect(fetchUrl('http://10.0.0.1/x')).rejects.toThrow(/private|loopback/); +}); diff --git a/engine/src/kild/web/fetch.ts b/engine/src/kild/web/fetch.ts new file mode 100644 index 00000000..7921b27d --- /dev/null +++ b/engine/src/kild/web/fetch.ts @@ -0,0 +1,100 @@ +import TurndownService from 'turndown'; + +export type FetchFormat = 'markdown' | 'html'; + +const MAX_BYTES = 5 * 1024 * 1024; // 5 MB hard cap (mirrors opencode webfetch) +const MAX_OUTPUT = 50 * 1024; // truncate returned text to ~50 KB +const DEFAULT_TIMEOUT_MS = 30_000; +const MAX_TIMEOUT_MS = 120_000; +// A realistic desktop Chrome UA gets past most basic bot filters; some walls (e.g. +// Cloudflare) instead trust a plain identifying agent, so we retry once with that. +const BROWSER_UA = + 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36'; +const FALLBACK_UA = 'kild'; + +const turndown = new TurndownService({ + headingStyle: 'atx', + hr: '---', + bulletListMarker: '-', + codeBlockStyle: 'fenced', + emDelimiter: '*', +}); +// Drop non-content elements entirely — otherwise their text (CSS, JS, meta) +// leaks into the markdown. +turndown.remove(['script', 'style', 'head', 'noscript', 'iframe']); + +/** Block fetches to loopback / link-local / private ranges so the agent can't be + * steered into the local network (basic SSRF guard). Literal-host check — does not + * defend against DNS rebinding; acceptable for a single-user loopback tool. */ +function assertPublicHost(hostname: string): void { + const h = hostname.toLowerCase().replace(/^\[|\]$/g, ''); // strip IPv6 brackets + const blocked = + h === 'localhost' || + h.endsWith('.localhost') || + h === '0.0.0.0' || + h === '::1' || + h.startsWith('127.') || + h.startsWith('10.') || + h.startsWith('169.254.') || + h.startsWith('192.168.') || + /^172\.(1[6-9]|2\d|3[01])\./.test(h) || + h.startsWith('fc') || // IPv6 unique-local + h.startsWith('fd') || + h.startsWith('fe80:'); // IPv6 link-local + if (blocked) throw new Error(`refusing to fetch a private/loopback host: ${hostname}`); +} + +async function doFetch(url: string, ua: string, timeoutMs: number): Promise { + return fetch(url, { + headers: { 'user-agent': ua, accept: 'text/html,application/xhtml+xml,*/*' }, + redirect: 'follow', + signal: AbortSignal.timeout(timeoutMs), + }); +} + +/** Fetch a public URL and return its content as markdown (default) or raw html/text. + * Plain HTTP — no headless browser. Enforces http(s), a 5 MB cap, and a timeout. */ +export async function fetchUrl( + url: string, + format: FetchFormat = 'markdown', + timeoutSeconds?: number, +): Promise { + let parsed: URL; + try { + parsed = new URL(url); + } catch { + throw new Error(`invalid url: ${url}`); + } + if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') { + throw new Error(`only http(s) urls are supported: ${url}`); + } + assertPublicHost(parsed.hostname); + + const timeoutMs = Math.min(MAX_TIMEOUT_MS, (timeoutSeconds ?? DEFAULT_TIMEOUT_MS / 1000) * 1000); + + let res = await doFetch(url, BROWSER_UA, timeoutMs); + if ((res.status === 403 || res.status === 429 || res.status === 503) && res.body) + res.body.cancel?.(); + if (res.status === 403 || res.status === 429 || res.status === 503) { + res = await doFetch(url, FALLBACK_UA, timeoutMs); // some walls trust a plain UA + } + if (!res.ok) throw new Error(`fetch ${res.status} for ${url}`); + + const len = Number(res.headers.get('content-length') ?? '0'); + if (len > MAX_BYTES) throw new Error(`response too large (${len} bytes > ${MAX_BYTES})`); + + const contentType = res.headers.get('content-type') ?? ''; + const raw = await res.text(); + if (raw.length > MAX_BYTES) throw new Error(`response too large (> ${MAX_BYTES} bytes)`); + + // Non-HTML (json, plain text, etc.) is returned as-is — turndown would mangle it. + const isHtml = contentType.includes('html') || /]/i.test(raw); + let out: string; + if (format === 'html' || !isHtml) out = raw; + else out = turndown.turndown(raw); + + if (out.length > MAX_OUTPUT) { + out = `${out.slice(0, MAX_OUTPUT)}\n\n…[truncated at ${MAX_OUTPUT} bytes]`; + } + return out; +} diff --git a/engine/src/kild/web/search-tool.ts b/engine/src/kild/web/search-tool.ts new file mode 100644 index 00000000..ca44cd48 --- /dev/null +++ b/engine/src/kild/web/search-tool.ts @@ -0,0 +1,39 @@ +import type { ToolDefinition } from '@earendil-works/pi-coding-agent'; +import { Type } from 'typebox'; + +import type { SearchProvider } from './search.ts'; + +/** + * The agent-facing `web_search` tool. Backed by an injected {@link SearchProvider} + * (SearXNG in v1) so the model — any model, including ones with no provider-native + * search — can look things up. Registered only when a provider is configured. + */ +export function createWebSearchTool(provider: SearchProvider): ToolDefinition { + return { + name: 'web_search', + label: 'Web Search', + description: + 'Search the web for current/external information. Returns a ranked list of ' + + '{title, url, snippet}. Follow up with `webfetch` on a result URL to read the page.', + promptSnippet: 'web_search — query the web; then webfetch a result url to read it', + parameters: Type.Object({ + query: Type.String({ description: 'The search query.' }), + numResults: Type.Optional( + Type.Number({ description: 'How many results to return (default 5, max 10).' }), + ), + }), + async execute(_toolCallId, params) { + const { query, numResults } = params as { query: string; numResults?: number }; + const limit = Math.min(10, Math.max(1, Math.floor(numResults ?? 5))); + const hits = await provider.search(query, limit); + const text = hits.length + ? hits + .map( + (h, i) => `${i + 1}. [${h.title}](${h.url})${h.snippet ? `\n ${h.snippet}` : ''}`, + ) + .join('\n') + : `No results for "${query}".`; + return { content: [{ type: 'text' as const, text }], details: { count: hits.length } }; + }, + }; +} diff --git a/engine/src/kild/web/search.test.ts b/engine/src/kild/web/search.test.ts new file mode 100644 index 00000000..6904301a --- /dev/null +++ b/engine/src/kild/web/search.test.ts @@ -0,0 +1,49 @@ +import { afterEach, expect, test } from 'bun:test'; + +import { searxng, webSearchProvider } from './search.ts'; + +const realFetch = globalThis.fetch; +afterEach(() => { + globalThis.fetch = realFetch; +}); + +const respond = (body: unknown, status = 200): typeof fetch => + (async () => + new Response(JSON.stringify(body), { + status, + headers: { 'content-type': 'application/json' }, + })) as typeof fetch; + +test('searxng maps results to {title,url,snippet} and caps to the limit', async () => { + globalThis.fetch = respond({ + results: [ + { title: 'A', url: 'https://a.com', content: 'snippet a' }, + { title: 'B', url: 'https://b.com', content: 'snippet b' }, + { title: 'C', url: 'https://c.com', content: 'snippet c' }, + ], + }); + const hits = await searxng('http://searx.local').search('q', 2); + expect(hits).toHaveLength(2); + expect(hits[0]).toEqual({ title: 'A', url: 'https://a.com', snippet: 'snippet a' }); +}); + +test('searxng falls back title→url + empty snippet, and drops rows without a url', async () => { + globalThis.fetch = respond({ results: [{ url: 'https://c.com' }, { title: 'no url here' }] }); + const hits = await searxng('http://searx.local').search('q', 10); + expect(hits).toEqual([{ title: 'https://c.com', url: 'https://c.com', snippet: '' }]); +}); + +test('searxng throws a helpful error on a non-200', async () => { + globalThis.fetch = respond({}, 403); + await expect(searxng('http://searx.local').search('q', 5)).rejects.toThrow(/searxng 403/); +}); + +test('webSearchProvider is null without KILD_SEARXNG_URL, set with it', () => { + const prev = process.env.KILD_SEARXNG_URL; + delete process.env.KILD_SEARXNG_URL; + expect(webSearchProvider()).toBeNull(); + process.env.KILD_SEARXNG_URL = 'http://searx.local'; + expect(webSearchProvider()).not.toBeNull(); + if (prev === undefined) delete process.env.KILD_SEARXNG_URL; + else process.env.KILD_SEARXNG_URL = prev; +}); diff --git a/engine/src/kild/web/search.ts b/engine/src/kild/web/search.ts new file mode 100644 index 00000000..40c7f1ea --- /dev/null +++ b/engine/src/kild/web/search.ts @@ -0,0 +1,49 @@ +import { searxngUrl } from '../config.ts'; + +/** One web-search result. The model-facing tool renders these as a list. */ +export interface SearchHit { + title: string; + url: string; + snippet: string; +} + +/** The search backend seam. v1 ships one impl (SearXNG); DDG / fastCRW / Tavily are + * user-selectable later by adding an impl + a `webSearchProvider()` switch arm — + * additive, no tool/worker rework. */ +export interface SearchProvider { + search(query: string, limit: number): Promise; +} + +/** A SearXNG-backed provider. kild talks to a self-hosted instance over its keyless + * JSON API (`/search?q=…&format=json`) — it does NOT run or manage the container. */ +export function searxng(baseUrl: string): SearchProvider { + return { + async search(query, limit) { + const u = new URL('/search', baseUrl); + u.searchParams.set('q', query); + u.searchParams.set('format', 'json'); + const r = await fetch(u, { signal: AbortSignal.timeout(25_000) }); + if (!r.ok) { + throw new Error( + `searxng ${r.status} — is it up and is the JSON format enabled? (see infra/searxng)`, + ); + } + const body = (await r.json()) as { + results?: { title?: string; url?: string; content?: string }[]; + }; + return (body.results ?? []) + .filter( + (h): h is { url: string; title?: string; content?: string } => typeof h.url === 'string', + ) + .slice(0, limit) + .map((h) => ({ title: h.title ?? h.url, url: h.url, snippet: h.content ?? '' })); + }, + }; +} + +/** The configured search provider, or null when web search is unavailable (no + * `KILD_SEARXNG_URL`). Callers register `web_search` only when this is non-null. */ +export function webSearchProvider(): SearchProvider | null { + const url = searxngUrl(); + return url ? searxng(url) : null; +} diff --git a/engine/src/kild/web/tools.ts b/engine/src/kild/web/tools.ts new file mode 100644 index 00000000..f0421b4b --- /dev/null +++ b/engine/src/kild/web/tools.ts @@ -0,0 +1,15 @@ +import type { ToolDefinition } from '@earendil-works/pi-coding-agent'; + +import { webEnabled } from '../config.ts'; +import { createWebFetchTool } from './fetch-tool.ts'; +import { webSearchProvider } from './search.ts'; +import { createWebSearchTool } from './search-tool.ts'; + +/** The web tools an agent session should get, in any context (engine worker or the + * CLI's in-process fallback). `webfetch` is always present when web is enabled (it's + * in-process, no backend); `web_search` is added only when a provider is configured. */ +export function webTools(): ToolDefinition[] { + if (!webEnabled()) return []; + const provider = webSearchProvider(); + return [createWebFetchTool(), ...(provider ? [createWebSearchTool(provider)] : [])]; +} diff --git a/engine/src/worker.ts b/engine/src/worker.ts index 5be8321a..b3275ead 100644 --- a/engine/src/worker.ts +++ b/engine/src/worker.ts @@ -1,10 +1,18 @@ -import { AuthStorage, createAgentSession, ModelRegistry } from '@earendil-works/pi-coding-agent'; +import { + AuthStorage, + createAgentSession, + ModelRegistry, + type ToolDefinition, +} from '@earendil-works/pi-coding-agent'; import { resolveAgentInstructions } from './kild/agents.ts'; +import { webEnabled } from './kild/config.ts'; import { type RawAgentEvent, translate, type UiEvent } from './kild/events.ts'; import { resolveModel, withRole } from './kild/models.ts'; import { createInviteAgentTool } from './kild/room/invite-agent-tool.ts'; import { createPostMessageTool } from './kild/room/post-message-tool.ts'; +import { webSearchProvider } from './kild/web/search.ts'; +import { webTools } from './kild/web/tools.ts'; import { ensureWorktree } from './kild/worktree.ts'; /** @@ -49,14 +57,16 @@ export async function runWorker(): Promise { let postedThisTurn = false; let turnText = ''; - // A given-but-unknown model errors (resolveModel throws); no model = pi default. - let model: ReturnType; - let session: Awaited>['session']; - try { - model = resolveModel(registry, modelPattern); + // Web tools (any model): `webfetch` is in-process and always available when web is + // enabled; `web_search` needs a backend, so it registers only when one is configured. + if (webEnabled() && !webSearchProvider()) { + process.stderr.write('kild: web_search disabled — set KILD_SEARXNG_URL (see infra/searxng)\n'); + } + const customTools: ToolDefinition[] = [ + ...webTools(), // A room participant gets `post_message` + `invite_agent`; their calls become // control lines on stdout the engine routes back into the room. - const customTools = inRoom + ...(inRoom ? [ createPostMessageTool((text) => { postedThisTurn = true; @@ -64,13 +74,20 @@ export async function runWorker(): Promise { }), createInviteAgentTool(emitInvite), ] - : undefined; + : []), + ]; + + // A given-but-unknown model errors (resolveModel throws); no model = pi default. + let model: ReturnType; + let session: Awaited>['session']; + try { + model = resolveModel(registry, modelPattern); ({ session } = await createAgentSession({ model, authStorage, modelRegistry: registry, cwd, - customTools, + customTools: customTools.length ? customTools : undefined, })); } catch (err) { emit({ kind: 'error', message: errText(err) }); diff --git a/infra/searxng/README.md b/infra/searxng/README.md new file mode 100644 index 00000000..6e3a26d0 --- /dev/null +++ b/infra/searxng/README.md @@ -0,0 +1,37 @@ +# SearXNG — the search backend for kild's `web_search` + +kild's `web_search` tool calls a **self-hosted SearXNG** over its keyless JSON API. +kild only *points at* it (`KILD_SEARXNG_URL`); it never runs or supervises the +container — that's you, here. + +## Start it + +```bash +cd infra/searxng +docker compose up -d +export KILD_SEARXNG_URL=http://localhost:8888 # add to your shell profile +``` + +Verify (no agent tokens spent): + +```bash +curl -s "http://localhost:8888/search?q=hello&format=json" | head -c 300 +cd ../../engine && bun run cli -- web search "anthropic claude opus" --json +``` + +## Notes + +- **Without `KILD_SEARXNG_URL`**, `web_search` is simply not offered to agents (the + worker logs a one-line notice). `webfetch` still works — it's in-process, no backend. +- Edit `settings.yml` to tune engines (it merges with SearXNG's defaults). Drop + `google` if it gets CAPTCHA-throttled; brave / duckduckgo / startpage are reliable. +- Change `server.secret_key` for anything past local dev. +- The instance is loopback-only and the limiter is disabled because it serves a + single local user/agent. Do **not** expose it publicly without re-enabling the + limiter + adding auth/TLS. + +## Other backends (later) + +`web_search` sits behind a `SearchProvider` seam (`engine/src/kild/web/search.ts`). +Adding DuckDuckGo, fastCRW, Tavily, etc. is a new impl + a `webSearchProvider()` +switch arm — no changes to the tool or the worker. diff --git a/infra/searxng/docker-compose.yml b/infra/searxng/docker-compose.yml new file mode 100644 index 00000000..4ae73de1 --- /dev/null +++ b/infra/searxng/docker-compose.yml @@ -0,0 +1,19 @@ +# Optional, self-hosted SearXNG backing kild's `web_search` tool. +# kild only POINTS at this over HTTP (KILD_SEARXNG_URL) — the engine never spawns or +# manages it. Bring it up yourself: +# +# cd infra/searxng && docker compose up -d +# export KILD_SEARXNG_URL=http://localhost:8888 +# +# Then any agent gets `web_search` (and `webfetch` works regardless). +services: + searxng: + image: searxng/searxng:latest + container_name: kild-searxng + ports: + - "8888:8080" + volumes: + - ./settings.yml:/etc/searxng/settings.yml:ro + environment: + - SEARXNG_BASE_URL=http://localhost:8888/ + restart: unless-stopped diff --git a/infra/searxng/settings.yml b/infra/searxng/settings.yml new file mode 100644 index 00000000..d04333f7 --- /dev/null +++ b/infra/searxng/settings.yml @@ -0,0 +1,18 @@ +# Minimal SearXNG override for kild. Merges with SearXNG's shipped defaults +# (use_default_settings: true), changing only what kild needs: +# - json format ON (the keyless API kild's web_search calls) +# - limiter OFF (this instance is loopback-only and serves a single user/agent) +# Tune `engines` here if you want to drop CAPTCHA-prone sources (e.g. google) or +# favor brave/duckduckgo/startpage. +use_default_settings: true + +server: + # CHANGE THIS for anything beyond local dev. + secret_key: "kild-local-dev-change-me" + limiter: false + image_proxy: false + +search: + formats: + - html + - json