feat(algolia-crawler): add crawler skill for RAG-optimized indexing#27
Merged
Conversation
Adds a CLI-only skill that guides customers to crawl web page(s) into a RAG-optimized Algolia index: inspect the page (incl. JS-rendered), write a RAG-shaped recordExtractor, validate with `algolia crawler test` before indexing, apply index settings explicitly, then reindex. Includes references (workflow, record-extractor, rag-index-settings, javascript-rendering, cli) and evals (with-skill vs baseline: 100% vs ~71%). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Point at one URL, index all sub-pages, and shape the config from ALL page templates (not just the start page): new references/site-crawls.md (discover -> group templates -> sample & `crawler test` each -> branching extractor or multiple actions -> unified settings + cross-template verify), plus `sitemaps` in the config example, reframed scoping (cover the section, not one page), and a 4th eval for the whole-site case. Eval 4 (with-skill 6/6 vs baseline 5/6): the baseline configures sub-page discovery but skips validating the extractor per page template — the gap this adds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…not a name) Dogfooding the skill on the Agent Studio docs surfaced that `algolia crawler get` accepts a UUID only — passing a crawler name returns `malformed_id`, so the documented `get <name>` recovery path doesn't work. And `crawler list` (the other suggested path) breaks account-wide when any crawler uses a non-boolean renderJavaScript. Corrected SKILL.md / cli.md / workflow.md: recover the id from `crawler list`, with a REST `GET /1/crawlers?name=` fallback when `list` is broken; `get` takes a UUID only. Added a real-world validation note to EVAL_RESULTS.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gbelain
approved these changes
Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this skill do?
Guides Algolia customers to crawl one or more web pages into a RAG-optimized Algolia index using the Algolia CLI — inspect the page (including JS-rendered), write a RAG-shaped
recordExtractor, validate withalgolia crawler testbefore indexing, apply index settings explicitly, then reindex.Why
None of the existing skills cover the Crawler (ingestion).
algolia-crawlerfills the gap betweenalgolia-cli(index ops) andalgobot-cli(the RAG/agent layer): it gets web content into Algolia in a shape that retrieves well for RAG, then hands off toalgobot-clifor the agent.What's inside
SKILL.md— the RAG record mental model + the end-to-end workflow + the two gotchas that bite (JS-rendered data, values hidden in attributes).references/—workflow,record-extractor(RAG record patterns, reading values from attributes, chunking),rag-index-settings(incl. NeuralSearch),javascript-rendering,cli.evals/— 3 realistic scenarios with expectations +EVAL_RESULTS.md.CLI-only by design — every step is an
algolia crawler …/algolia settings/algolia searchcommand. It bakes in CLI-specific realities verified in practice: the CLI accepts only booleanrenderJavaScript(the object form breaksget/list/create -F),crawler createprints nothing (recover the id vialist), andinitialIndexSettingsfrequently doesn't apply so settings must be applied explicitly withsettings import.Evaluation
With-skill vs baseline across the 3 scenarios: 100% vs ~71% assertion pass rate. The skill's value concentrates on CLI correctness + non-obvious gotchas — a capable model already handles generic RAG record shaping. See
skills/algolia-crawler/evals/EVAL_RESULTS.md.Checklist
python3 scripts/validate_skills.py .passes (0 errors)evals/evals.jsonexists with at least 1 realistic prompt../other-skill/references.claude-plugin/marketplace.json(new skills only)skills/here (useful to customers)🤖 Generated with Claude Code