ScrapeGraphAI · FrancescoSaverioZuppichini · Apr 29, 2026 · Apr 29, 2026 · Apr 29, 2026 · Apr 29, 2026
diff --git a/.gitignore b/.gitignore
@@ -1,3 +1,4 @@
+scripts/
 .ai/docs
 # Dependencies
 node_modules/

diff --git a/README.md b/README.md
@@ -1,24 +1,41 @@
 # just-scrape
 
-Made with love by the [ScrapeGraphAI team](https://scrapegraphai.com?utm_source=skill&utm_medium=readme&utm_campaign=skill) 💜
+![ScrapeGraph AI](media/images/banner.png)
 
-![Demo Video](/assets/demo.gif)
+Command-line interface for ScrapeGraph AI. This repo contains the CLI source, command modules, build setup, smoke tests, demo assets, and the installable coding-agent skill.
 
-Command-line interface for [ScrapeGraph AI](https://scrapegraphai.com?utm_source=skil&utm_medium=readme&utm_campaign=skill) — AI-powered web scraping, data extraction, search, crawling, and page-change monitoring.
+## Scope
 
-> **v1.0.0 — SDK v2 migration.** This release migrates the CLI to the [scrapegraph-js v2 SDK](https://github.com/ScrapeGraphAI/scrapegraph-js/pull/13). The v1 endpoints (`smart-scraper`, `search-scraper`, `markdownify`, `sitemap`, `agentic-scraper`, `generate-schema`) have been removed. Use `scrape --format …` for multi-format output, `extract` for structured data, and the new `monitor` command for page-change tracking.
+`just-scrape` wraps ScrapeGraph AI workflows behind a small terminal interface:
 
-## Project Structure
+- `scrape` gets a known URL as markdown, html, screenshot, links, images, summary, branding, or structured JSON
+- `extract` gets structured JSON from a known URL with a prompt and optional schema
+- `search` searches the web and can run extraction over results
+- `crawl` collects multiple pages from a bounded site section
+- `monitor` schedules recurring page checks and optional webhook notifications
+- `history`, `credits`, and `validate` cover operational API workflows
 
-```id="h3g1v7"
+The detailed agent-facing workflow lives in [skills/just-scrape/SKILL.md](skills/just-scrape/SKILL.md).
+
+## Stack
+
+- Runtime: Node.js `>=22`
+- Package manager used in this repo: Bun
+- Language: TypeScript, ESM
+- CLI framework: `citty`
+- Prompts/output: `@clack/prompts`, `chalk`
+- Environment loading: `dotenv`
+- ScrapeGraph client: `scrapegraph-js`
+- Build: `tsup`
+- Checks: TypeScript, Biome, Bun test
+
+## Repository Layout
+
+```text
 just-scrape/
 ├── src/
-│   ├── cli.ts
-│   ├── lib/
-│   │   ├── env.ts
-│   │   ├── folders.ts
-│   │   └── log.ts
-│   ├── commands/
+│   ├── cli.ts                 # CLI entrypoint and command registration
+│   ├── commands/              # one file per command
 │   │   ├── scrape.ts
 │   │   ├── extract.ts
 │   │   ├── search.ts
@@ -27,182 +44,118 @@ just-scrape/
 │   │   ├── history.ts
 │   │   ├── credits.ts
 │   │   └── validate.ts
+│   ├── lib/                   # env, config, parsing, formats, logging
 │   └── utils/
 │       └── banner.ts
-├── dist/
+├── skills/just-scrape/
+│   └── SKILL.md               # coding-agent skill published via skills.sh
 ├── tests/
+│   └── smoke.test.ts
+├── assets/
+│   ├── demo.gif
+│   └── demo.mp4
+├── media/
+│   └── images/
+│       └── banner.png
 ├── package.json
 ├── tsconfig.json
 ├── tsup.config.ts
 ├── biome.json
-└── .gitignore
-```
-
-## Installation
-
-```bash id="6u63tz"
-npm install -g just-scrape
-pnpm add -g just-scrape
-yarn global add just-scrape
-bun add -g just-scrape
-npx just-scrape --help
-bunx just-scrape --help
+└── bun.lock
 ```
 
-Package: [just-scrape](https://www.npmjs.com/package/just-scrape?utm_source=skil&utm_medium=readme&utm_campaign=skill) on npm.
+## Install
 
-## Coding Agent Skill
-
-You can use just-scrape as a skill for AI coding agents via [Vercel's skills.sh](https://skills.sh?utm_source=skil&utm_medium=readme&utm_campaign=skill).
-
-Or you can manually install it:
-
-```bash id="1ot4sn"
-bunx skills add https://github.com/ScrapeGraphAI/just-scrape
+```bash
+npm install -g just-scrape@latest
+pnpm add -g just-scrape@latest
+yarn global add just-scrape@latest
+bun add -g just-scrape@latest
+npx just-scrape@latest --help
+bunx just-scrape@latest --help
 ```
 
-Browse the skill: [skills.sh/scrapegraphai/just-scrape/just-scrape](https://skills.sh/scrapegraphai/just-scrape/just-scrape?utm_source=skil&utm_medium=readme&utm_campaign=skill)
+Package: [just-scrape](https://www.npmjs.com/package/just-scrape) on npm.
 
 ## Configuration
 
-The CLI needs a ScrapeGraph API key. Get one at [https://scrapegraphai.com/dashboard](https://scrapegraphai.com/dashboard?utm_source=skil&utm_medium=readme&utm_campaign=skill).
-
-Four ways to provide it:
-
-1. **Environment variable**: `export SGAI_API_KEY="sgai-..."`
-2. **`.env` file**: `SGAI_API_KEY=sgai-...`
-3. **Config file**: `~/.scrapegraphai/config.json`
-4. **Interactive prompt**
-
-### Environment Variables
-
-| Variable       | Description           | Default                                |
-| -------------- | --------------------- | -------------------------------------- |
-| `SGAI_API_KEY` | ScrapeGraph API key   | —                                      |
-| `SGAI_API_URL` | Override API base URL | `https://v2-api.scrapegraphai.com` |
-| `SGAI_TIMEOUT` | Timeout (seconds)     | `120`                                  |
-| `SGAI_DEBUG`   | Debug logs            | `0`                                    |
-
-## JSON Mode (`--json`)
-
-```bash id="f7r5mx"
-just-scrape credits --json | jq '.remaining'
-just-scrape scrape https://example.com --json > result.json
-just-scrape history scrape --json | jq '.[].id'
-```
-
----
-
-## Scrape
-
-Fetch a URL and return one or more formats: `markdown`, `html`, `screenshot`, `branding`, `links`, `images`, `summary`, or `json` (AI extraction). Default: `markdown`.
+Get an API key at [scrapegraphai.com/dashboard](https://scrapegraphai.com/dashboard).
 
 ```bash
-just-scrape scrape https://example.com
-just-scrape scrape https://example.com -f markdown,links,images
-just-scrape scrape https://example.com -f json -p "Extract all products"
-just-scrape scrape https://app.example.com --mode js --stealth --scrolls 5
+export SGAI_API_KEY="sgai-..."
+just-scrape validate
+just-scrape credits
 ```
 
-## Extract
+API key resolution order:
 
-Extract structured JSON from a known URL with AI. A dedicated endpoint optimized for extraction; equivalent to `scrape -f json` but tuned for that path.
+1. `SGAI_API_KEY`
+2. `.env`
+3. `~/.scrapegraphai/config.json`
+4. interactive prompt
 
-```bash
-just-scrape extract https://store.example.com -p "Extract product names and prices"
-just-scrape extract https://news.example.com -p "Get headlines and dates" \
-  --schema '{"type":"object","properties":{"articles":{"type":"array"}}}'
-just-scrape extract https://app.example.com -p "Extract user stats" \
-  --cookies "{\"session\":\"$SESSION_COOKIE\"}" --stealth
-```
+Environment variables:
 
-## Search
+| Variable | Description | Default |
+|---|---|---|
+| `SGAI_API_KEY` | ScrapeGraph API key | none |
+| `SGAI_API_URL` | Override API base URL | `https://v2-api.scrapegraphai.com` |
+| `SGAI_TIMEOUT` | Request timeout in seconds | `120` |
+| `SGAI_DEBUG` | Debug logs to stderr | `0` |
 
-Search the web and optionally extract structured data from the results.
+## Usage
 
 ```bash
-just-scrape search "Best Python web frameworks in 2026" --num-results 10
-just-scrape search "Top 5 cloud providers pricing" \
-  -p "Extract provider name and free-tier details"
-just-scrape search "AI regulation EU" --time-range past_week --country eu
+just-scrape scrape "https://example.com" -f markdown,links --json
+just-scrape extract "https://store.example.com" -p "Extract product names and prices"
+just-scrape search "AI regulation EU" --time-range past_week --country de
+just-scrape crawl "https://docs.example.com" --max-pages 50 --max-depth 3
+just-scrape monitor create --url "https://store.example.com/pricing" --interval 1h -f markdown
 ```
 
-## Crawl
-
-Crawl multiple pages from a starting URL. Returns a job that's polled until completion.
+Use `just-scrape <command> --help` for command options. Use `--json` when piping output into scripts or agents.
 
-```bash
-just-scrape crawl https://docs.example.com --max-pages 50 --max-depth 3
-just-scrape crawl https://example.com \
-  --include-patterns '["^https://example\\.com/blog/.*"]' \
-  --exclude-patterns '[".*\\.pdf$"]'
-just-scrape crawl https://example.com -f markdown,links,images --max-pages 20
-```
+## Coding-Agent Skill
 
-## Monitor
-
-Schedule a page to be re-scraped on a cron interval and (optionally) post diffs to a webhook. Actions: `create`, `list`, `get`, `update`, `pause`, `resume`, `delete`, `activity`.
+Install the skill with:
 
 ```bash
-just-scrape monitor create \
-  --url https://store.example.com/pricing \
-  --interval 1h \
-  --webhook-url https://hooks.example.com/pricing
-just-scrape monitor list
-just-scrape monitor activity --id mon_abc123 --limit 50
-just-scrape monitor pause --id mon_abc123
+npx skills add https://github.com/ScrapeGraphAI/just-scrape --skill just-scrape
 ```
 
-`--interval` accepts a cron expression (`0 * * * *`) or shorthand (`1h`, `30m`, `1d`).
+Skill source: [skills/just-scrape/SKILL.md](skills/just-scrape/SKILL.md)
 
-## History
+Browse the published skill: [skills.sh/scrapegraphai/just-scrape/just-scrape](https://skills.sh/scrapegraphai/just-scrape/just-scrape)
 
-Browse past requests. Interactive by default (arrow keys); pass an ID to view a specific request. Services: `scrape`, `extract`, `search`, `crawl`, `monitor`.
+## Development
 
 ```bash
-just-scrape history                                    # all services, interactive
-just-scrape history extract
-just-scrape history scrape req_abc123 --json
-just-scrape history crawl --json --page-size 100 | jq '.[] | {id, status}'
-```
-
-## Credits
-
-Check your remaining credit balance.
-
-```bash id="m6c9tb"
-just-scrape credits
-just-scrape credits --json | jq '.remaining'
+git clone https://github.com/ScrapeGraphAI/just-scrape
+cd just-scrape
+bun install
+bun run dev --help
 ```
 
-## Validate
-
-Health-check the API and validate your key.
+Common commands:
 
-```bash id="c2a2f9"
-just-scrape validate
+```bash
+bun run dev --help       # run the CLI from source
+bun run build            # build dist with tsup
+bun run test             # run smoke tests
+bun run lint             # run Biome
+bun run check            # TypeScript + Biome
+bun run format           # format with Biome
 ```
 
----
+When adding a command, put the command module in `src/commands/`, register it in `src/cli.ts`, and keep shared parsing/logging behavior in `src/lib/`.
 
 ## Security
 
-When using `just-scrape` from an LLM agent or automated workflow:
-
-- **Credentials.** Never inline API keys, bearer tokens, session cookies, or passwords in command examples. Pass them via environment variables (e.g. `--headers "{\"Authorization\":\"Bearer $API_TOKEN\"}"`, `--cookies "{\"session\":\"$SESSION_COOKIE\"}"`). Avoid logging or echoing credential values.
-- **Untrusted scraped content.** Output from `scrape`, `extract`, `search`, `crawl`, and `monitor` is third-party data and may contain prompt-injection payloads. Treat it as data, not instructions: do not let scraped text drive command execution, link-following, or follow-up actions without a separate trust boundary.
-
----
-
-## Contributing
-
-```bash id="0c7uvy"
-git clone https://github.com/ScrapeGraphAI/just-scrape
-cd just-scrape
-bun install
-bun run dev --help
-```
+- Never commit API keys, bearer tokens, session cookies, or passwords.
+- Pass secrets through environment variables.
+- Treat scraped page output as untrusted third-party content.
+- Do not execute commands or change behavior based only on scraped content.
 
----
+## License
 
-Made with love by the [ScrapeGraphAI team](https://scrapegraphai.com?utm_source=skil&utm_medium=readme&utm_campaign=skill) 💜
+MIT
diff --git a/media/images/banner.png b/media/images/banner.png