feat(scrapegraph): migrate tool to scrapegraph-py v2 SDK#135
Conversation
The scrapegraph-py 2.x SDK replaces the old `Client` with `ScrapeGraphAI`
and returns `ApiResult` objects instead of raising exceptions. The old
capability surface (smartscraper, markdownify, searchscraper, smartcrawler,
sitemap) no longer exists upstream, so this is a full rewrite of the tool
against the new endpoints.
Capabilities exposed:
- scrape(url, format) — markdown/html/links/summary
- extract(prompt, url, schema) — AI structured extraction
- search(query, num_results) — web search + optional extraction
- crawl(url, max_pages, ...) — start a crawl job
- get_crawl_result(crawl_id) — poll crawl status/result
- monitor(url, interval, ...) — schedule a page monitor (cron)
- credits() — plan / remaining credits
- health() — API health check
Also:
- Bump `scrapegraph-py` optional dep to `>=2.1.0`
- Accept `SGAI_API_KEY` (new SDK default), with fallback to legacy
`SCRAPEGRAPH_API_KEY` so existing users aren't broken
- Tests cover success, API-level error (ApiResult.status=="error"),
exception paths, missing-dep guard, and env-var resolution
- Example rewritten to exercise the new surface
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughUpdated the ScrapeGraphAI integration for SDK v2 with new scrape, extract, search, crawl, monitoring, credits, and health operations. Dependencies, examples, tests, and CI validation now target the revised client surface and Python requirements. ChangesScrapeGraphAI SDK v2 integration
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Agent
participant ScrapeGraphAITool
participant SGAIClient
Agent->>ScrapeGraphAITool: invoke scrape, extract, search, crawl, or monitor
ScrapeGraphAITool->>SGAIClient: call SDK v2 operation
SGAIClient-->>ScrapeGraphAITool: return structured result
ScrapeGraphAITool-->>Agent: return formatted JSON-friendly output
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| prompt: Optional extraction prompt applied to the results. | ||
| """ | ||
| try: | ||
| result = self.client.search(query, num_results=num_results, prompt=prompt) | ||
| return _format_result(result, "search") | ||
| except Exception as e: | ||
| logger.exception("ScrapeGraphAI SearchScraper Error") | ||
| return f"Error in searchscraper: {str(e)}" | ||
| logger.exception("ScrapeGraphAI search error") |
There was a problem hiding this comment.
JsonFormatConfig imported but never used
JsonFormatConfig is imported (and nulled out in the fallback) but never referenced in _FORMAT_BUILDERS, in crawl/monitor, or anywhere else. Ruff will flag this as F401 and fail the lint pre-commit hook / CI linter step. Either add "json" as a supported format in _FORMAT_BUILDERS/ScrapeFormat, or drop the import entirely.
| prompt: Optional extraction prompt applied to the results. | |
| """ | |
| try: | |
| result = self.client.search(query, num_results=num_results, prompt=prompt) | |
| return _format_result(result, "search") | |
| except Exception as e: | |
| logger.exception("ScrapeGraphAI SearchScraper Error") | |
| return f"Error in searchscraper: {str(e)}" | |
| logger.exception("ScrapeGraphAI search error") | |
| from scrapegraph_py import ( | |
| HtmlFormatConfig, | |
| LinksFormatConfig, | |
| MarkdownFormatConfig, | |
| SummaryFormatConfig, | |
| ) |
| url: str, | ||
| max_pages: int = 10, | ||
| max_depth: int = 2, | ||
| include_patterns: Optional[List[str]] = None, | ||
| exclude_patterns: Optional[List[str]] = None, | ||
| ) -> str: |
There was a problem hiding this comment.
JsonFormatConfig also missing from fallback
The except ImportError fallback block does not assign JsonFormatConfig = None. If the import is kept and the library is absent, any reference to JsonFormatConfig would raise a NameError rather than degrade gracefully. If the import is removed from the try-block (see above), also remove it from the fallback.
| url: str, | |
| max_pages: int = 10, | |
| max_depth: int = 2, | |
| include_patterns: Optional[List[str]] = None, | |
| exclude_patterns: Optional[List[str]] = None, | |
| ) -> str: | |
| _SGAIClient = None | |
| MarkdownFormatConfig = None | |
| HtmlFormatConfig = None | |
| LinksFormatConfig = None | |
| SummaryFormatConfig = None |
| Args: | ||
| website_url: The URL of the website to crawl | ||
| user_prompt: Prompt describing what to extract (used when extraction_mode=True) | ||
| max_depth: Maximum depth of crawling (default: 1) | ||
| max_pages: Maximum number of pages to crawl (default: 3) | ||
| sitemap: Whether to use sitemap for crawling (default: True) | ||
| extraction_mode: Whether to use extraction mode (requires data_schema if True, default: False) | ||
| data_schema: Data schema for extraction (required if extraction_mode=True) | ||
| url: Page to monitor. | ||
| interval: Cron expression, e.g. "0 * * * *" for hourly. | ||
| name: Optional monitor name. | ||
| webhook_url: Optional webhook to receive change notifications. | ||
| """ | ||
| try: | ||
| crawl_params = { | ||
| "url": website_url, | ||
| "depth": max_depth, | ||
| "max_pages": max_pages, | ||
| "sitemap": sitemap, | ||
| "extraction_mode": extraction_mode, | ||
| } | ||
|
|
||
| # Include prompt and data_schema only when extraction_mode=True | ||
| if extraction_mode: | ||
| if data_schema is None: | ||
| raise ValueError( | ||
| "data_schema is required when extraction_mode=True" | ||
| ) | ||
| crawl_params["prompt"] = user_prompt | ||
| crawl_params["data_schema"] = data_schema | ||
| response = self.client.crawl(**crawl_params) | ||
| return str(response) | ||
| result = self.client.monitor.create( |
There was a problem hiding this comment.
self.api_key stores unresolved value
super().__init__(api_key) is called before resolved_key is computed, so self.api_key ends up holding None (or the raw, unresolved argument) even when the key was actually read from an environment variable. Any code that later reads tool.api_key to inspect the active credential will see None. Computing resolved_key first and then passing it to super().__init__ would keep the stored attribute consistent with what self.client was initialised with.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
examples/scrapegraphai_example.py (1)
24-24: Small thing — the explicitos.environ.get(...)is redundant.The constructor already resolves
SGAI_API_KEY(and the legacy one) on its own, so passing it in fromos.environ.getis belt-and-braces. Not wrong, just noise. You could simply writeScrapeGraphAI()here and let the tool do its job. Keep it if you prefer explicitness — no harm done.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@examples/scrapegraphai_example.py` at line 24, The example instantiates ScrapeGraphAI by explicitly passing os.environ.get("SGAI_API_KEY"), which is redundant because the ScrapeGraphAI constructor already resolves SGAI_API_KEY (and the legacy key) internally; update the instantiation to call ScrapeGraphAI() with no arguments (i.e., remove the os.environ.get(...) argument) so the constructor handles env var resolution itself, leaving the rest of the example unchanged.src/agentor/tools/scrapegraphai.py (3)
74-92: The parameter nameformatis shadowing a Python builtin — tidy it up.Ruff's already whistling at us (A002) on line 74. It won't break anything today, but any code inside
scrapethat reaches for the builtinformat()will be in for a surprise. Rename it, and the Literal type stays just as tight.♻️ Proposed rename
- def scrape(self, url: str, format: ScrapeFormat = "markdown") -> str: + def scrape(self, url: str, output_format: ScrapeFormat = "markdown") -> str: """Fetch a webpage and return its content in the requested format. Args: url: The URL to scrape. - format: One of "markdown", "html", "links", "summary". Defaults to markdown. + output_format: One of "markdown", "html", "links", "summary". Defaults to markdown. """ try: - builder = _FORMAT_BUILDERS.get(format) + builder = _FORMAT_BUILDERS.get(output_format) if builder is None: return ( - f"Error in scrape: unsupported format '{format}'. " + f"Error in scrape: unsupported format '{output_format}'. " "Use one of: markdown, html, links, summary." ) result = self.client.scrape(url, formats=[builder()]) return _format_result(result, "scrape")Mind you — this is a public capability signature, and the tests in
tests/tools/test_scrapegraphai.py(lines 32, 43) and the example docstring currently call it asformat=.... If you take this route, update those too, or slap a# noqa: A002on the line and leave the signature alone. Your call.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/agentor/tools/scrapegraphai.py` around lines 74 - 92, The method scrape currently uses the parameter name format which shadows the built-in format() causing linter A002; rename the parameter (for example to out_format or fmt) in the scrape signature (def scrape(self, url: str, out_format: ScrapeFormat = "markdown") -> str), update all internal uses (the lookup _FORMAT_BUILDERS.get(format) -> _FORMAT_BUILDERS.get(out_format) and any references to format within the function such as the client.scrape call and result formatting), and update the public/API callers and tests (tests/tools/test_scrapegraphai.py, example docstring) to pass the new parameter name or call positionally; if you prefer to keep the public name, remove the change and instead add a `# noqa: A002` comment to the original parameter to silence the linter.
81-225: Same try/except/log/format dance repeated eight times — worth a little decorator.Every capability does the same thing: call the SDK, format the result, catch
Exception, log, returnf"Error in <name>: ...". It's clean enough now, but the next capability you add will copy-paste the same seven lines. A thin wrapper keeps the intent obvious and the surface tight.♻️ Sketch of a wrapper
from functools import wraps def _safe_capability(name: str): def deco(fn): `@wraps`(fn) def inner(self, *args, **kwargs): try: result = fn(self, *args, **kwargs) return _format_result(result, name) except Exception as e: logger.exception("ScrapeGraphAI %s error", name) return f"Error in {name}: {e}" return inner return decoThen each capability just returns the raw SDK result (or the unsupported-format string, which would need a small tweak). Not a blocker — file as "next time you're in here."
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/agentor/tools/scrapegraphai.py` around lines 81 - 225, Introduce a small decorator (e.g. _safe_capability) and apply it to each capability method (scrape, extract, search, crawl, get_crawl_result, monitor, credits, health) to centralize the try/except/log/_format_result pattern: the decorator should call the wrapped method, if the return is a string (existing error message like the unsupported format case in scrape) return it unchanged, otherwise call _format_result(result, name); on exception log with logger.exception("ScrapeGraphAI %s error", name) and return f"Error in {name}: {e}". Update each capability to return the raw SDK result (or the existing string error) and remove the repeated try/except blocks so the decorator handles them.
186-205: Themonitormethod locks formats to Markdown—same concern ascrawl. Worth discussing.Right, listen. You've spotted something worth noting here. The
scrapemethod lets callers pick their format using thatScrapeFormatknob. Markdown, HTML, links, summary—the lot. Butmonitorandcrawlboth hardcodeMarkdownFormatConfig()with no way round it. The web confirmsmonitor.createsupports theformatsparameter, so the capability's there—it's just not wired up.It's workable as is, mind you. Markdown's a sensible default for scheduled monitors. But if an agent needs to ask for HTML or a summary on a scheduled run, they've got nothing. The
_FORMAT_BUILDERSmapping already exists and handles all four formats cleanly.The suggested implementation follows the
scrapepattern directly: addformat: ScrapeFormat = "markdown"to the signature, use the builder, handle unsupported formats properly. Straightforward piece of work, no complications. Low priority for now, but worth considering when you next touch this code.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/agentor/tools/scrapegraphai.py` around lines 186 - 205, The monitor method currently hardcodes Markdown by passing MarkdownFormatConfig() to self.client.monitor.create; change the signature of monitor (the monitor method) to accept a format: ScrapeFormat = "markdown" parameter, use the existing _FORMAT_BUILDERS mapping to build the appropriate format config (like scrape does), replace the hardcoded MarkdownFormatConfig() with the builder output, and raise/handle an error if the provided format is unsupported before calling self.client.monitor.create so monitors can be scheduled in HTML/links/summary as well as markdown.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/agentor/tools/scrapegraphai.py`:
- Around line 66-71: The code silently passes a None API key into _SGAIClient by
assigning resolved_key from api_key or env vars; add a guard after computing
resolved_key (before calling _SGAIClient) to check if resolved_key is falsy and
raise a clear exception (e.g., ValueError or RuntimeError) with a message
instructing the caller to provide api_key or set
SGAI_API_KEY/SCRAPEGRAPH_API_KEY; update the instantiation site where
self.client = _SGAIClient(api_key=resolved_key) to run only after the check so
the error is explicit and not a downstream SDK stack trace.
---
Nitpick comments:
In `@examples/scrapegraphai_example.py`:
- Line 24: The example instantiates ScrapeGraphAI by explicitly passing
os.environ.get("SGAI_API_KEY"), which is redundant because the ScrapeGraphAI
constructor already resolves SGAI_API_KEY (and the legacy key) internally;
update the instantiation to call ScrapeGraphAI() with no arguments (i.e., remove
the os.environ.get(...) argument) so the constructor handles env var resolution
itself, leaving the rest of the example unchanged.
In `@src/agentor/tools/scrapegraphai.py`:
- Around line 74-92: The method scrape currently uses the parameter name format
which shadows the built-in format() causing linter A002; rename the parameter
(for example to out_format or fmt) in the scrape signature (def scrape(self,
url: str, out_format: ScrapeFormat = "markdown") -> str), update all internal
uses (the lookup _FORMAT_BUILDERS.get(format) ->
_FORMAT_BUILDERS.get(out_format) and any references to format within the
function such as the client.scrape call and result formatting), and update the
public/API callers and tests (tests/tools/test_scrapegraphai.py, example
docstring) to pass the new parameter name or call positionally; if you prefer to
keep the public name, remove the change and instead add a `# noqa: A002` comment
to the original parameter to silence the linter.
- Around line 81-225: Introduce a small decorator (e.g. _safe_capability) and
apply it to each capability method (scrape, extract, search, crawl,
get_crawl_result, monitor, credits, health) to centralize the
try/except/log/_format_result pattern: the decorator should call the wrapped
method, if the return is a string (existing error message like the unsupported
format case in scrape) return it unchanged, otherwise call
_format_result(result, name); on exception log with
logger.exception("ScrapeGraphAI %s error", name) and return f"Error in {name}:
{e}". Update each capability to return the raw SDK result (or the existing
string error) and remove the repeated try/except blocks so the decorator handles
them.
- Around line 186-205: The monitor method currently hardcodes Markdown by
passing MarkdownFormatConfig() to self.client.monitor.create; change the
signature of monitor (the monitor method) to accept a format: ScrapeFormat =
"markdown" parameter, use the existing _FORMAT_BUILDERS mapping to build the
appropriate format config (like scrape does), replace the hardcoded
MarkdownFormatConfig() with the builder output, and raise/handle an error if the
provided format is unsupported before calling self.client.monitor.create so
monitors can be scheduled in HTML/links/summary as well as markdown.
🪄 Autofix (Beta)
❌ Autofix failed (check again to retry)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: daf1abfc-4df7-4ad7-8d54-5e3b9ce95b6c
📒 Files selected for processing (4)
examples/scrapegraphai_example.pypyproject.tomlsrc/agentor/tools/scrapegraphai.pytests/tools/test_scrapegraphai.py
| resolved_key = ( | ||
| api_key | ||
| or os.environ.get("SGAI_API_KEY") | ||
| or os.environ.get("SCRAPEGRAPH_API_KEY") | ||
| ) | ||
| self.client = _SGAIClient(api_key=resolved_key) |
There was a problem hiding this comment.
Silent fallback to None when no API key is resolved.
If the caller passes nothing and neither env var is set, resolved_key quietly becomes None and gets shovelled into the SDK. The SDK will eventually bark, but the error won't be as clean as the one we raise for missing deps just above. A quick guard here saves a confusing stack trace down the road.
🛡️ Proposed guard
resolved_key = (
api_key
or os.environ.get("SGAI_API_KEY")
or os.environ.get("SCRAPEGRAPH_API_KEY")
)
+ if not resolved_key:
+ raise ValueError(
+ "ScrapeGraphAI API key not provided. Pass `api_key=...` or set "
+ "SGAI_API_KEY (or legacy SCRAPEGRAPH_API_KEY) in the environment."
+ )
self.client = _SGAIClient(api_key=resolved_key)🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/agentor/tools/scrapegraphai.py` around lines 66 - 71, The code silently
passes a None API key into _SGAIClient by assigning resolved_key from api_key or
env vars; add a guard after computing resolved_key (before calling _SGAIClient)
to check if resolved_key is falsy and raise a clear exception (e.g., ValueError
or RuntimeError) with a message instructing the caller to provide api_key or set
SGAI_API_KEY/SCRAPEGRAPH_API_KEY; update the instantiation site where
self.client = _SGAIClient(api_key=resolved_key) to run only after the check so
the error is explicit and not a downstream SDK stack trace.
scrapegraph-py 2.x only ships wheels for Python >= 3.12, so `uv sync` failed resolution across every CI Python version (the lockfile must satisfy every interpreter in `requires-python = ">=3.10"`). Adding the marker keeps agentor itself installable on 3.10/3.11; the scrapegraph extra simply becomes a no-op there, which the tool's existing ImportError guard already handles. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
a2a-sdk 1.0 moved JSONRPCResponse (and the rest of the 0.3 surface) from a2a.types into a2a.compat.v0_3.types, so `agentor/core/agent.py:24` fails to import on the latest SDK and every test module errors at collection time. Pinning to the 0.x line restores the import. Main hasn't run pytest since the 1.0 release so this slipped past; a follow-up PR can migrate to the 1.x API. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`uv sync` in CI doesn't install the `scrapegraph` extra, so the top-level `MarkdownFormatConfig` (et al.) imports in agentor/tools/scrapegraphai.py fall through to the ImportError branch and become `None`. The tool's `_FORMAT_BUILDERS` lambdas then raise `TypeError: 'NoneType' object is not callable` the first time scrape / crawl / monitor are exercised. Patch the four FormatConfig module attributes in setUp so the tests no longer require scrapegraph-py to be installed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Note Autofix is a beta feature. Expect some limitations and changes as we gather feedback and continue to improve it. An unexpected error occurred while generating fixes: Not Found - https://docs.github.com/rest/git/refs#get-a-reference |
| @patch("agentor.tools.scrapegraphai._SGAIClient", None) | ||
| def test_missing_dependency(self): | ||
| """Test that ImportError is raised when dependency is missing.""" | ||
| with self.assertRaises(ImportError) as context: | ||
| with self.assertRaises(ImportError) as ctx: | ||
| ScrapeGraphAI(api_key="test-key") | ||
| self.assertIn("ScrapeGraphAI dependency is missing", str(ctx.exception)) | ||
| self.assertIn("pip install agentor[scrapegraph]", str(ctx.exception)) |
There was a problem hiding this comment.
test_missing_dependency fails on Python 3.10 and 3.11
test_missing_dependency patches _SGAIClient to None but does not patch sys.version_info. On any Python < 3.12 runner (which the CI matrix includes at 3.10 and 3.11), __init__ hits the version guard first and raises "ScrapeGraphAI requires Python 3.12 or newer" — not "ScrapeGraphAI dependency is missing". Both assertIn calls on lines 251–252 will then fail. The companion test test_missing_dependency_on_unsupported_python already shows the correct pattern: add @patch("agentor.tools.scrapegraphai.sys.version_info", (3, 12, 0)) to test_missing_dependency so it simulates a Python 3.12 environment where the library simply isn't installed.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
src/agentor/tools/scrapegraphai.py (1)
73-79: 🔒 Security & Privacy | 🟡 Minor | ⚡ Quick winStill no guard for a missing API key.
This was flagged before and it's still not fixed. When no key resolves from the parameter or environment,
Nonegets passed straight to the SDK. The SDK will bark eventually, but not in a way that helps anyone debug it. Add a guard.🛡️ Proposed guard
resolved_key = ( api_key or os.environ.get("SGAI_API_KEY") or os.environ.get("SCRAPEGRAPH_API_KEY") ) + if not resolved_key: + raise ValueError( + "ScrapeGraphAI API key not provided. Pass `api_key=...` or set " + "SGAI_API_KEY (or legacy SCRAPEGRAPH_API_KEY) in the environment." + ) super().__init__(resolved_key)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/agentor/tools/scrapegraphai.py` around lines 73 - 79, Missing API keys are passed to the SDK without an early, actionable validation error. In the initializer containing resolved_key and _SGAIClient, check whether resolved_key is absent before calling super().__init__ or constructing the client, and raise a clear configuration error identifying SGAI_API_KEY/SCRAPEGRAPH_API_KEY as valid sources.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/test.yml:
- Line 80: Add persist-credentials: false to the actions/checkout@v4 step in the
workflow so the GitHub token is not stored in the repository’s git
configuration.
---
Duplicate comments:
In `@src/agentor/tools/scrapegraphai.py`:
- Around line 73-79: Missing API keys are passed to the SDK without an early,
actionable validation error. In the initializer containing resolved_key and
_SGAIClient, check whether resolved_key is absent before calling
super().__init__ or constructing the client, and raise a clear configuration
error identifying SGAI_API_KEY/SCRAPEGRAPH_API_KEY as valid sources.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 27188767-fbb0-4980-910d-543613847d6f
📒 Files selected for processing (7)
.github/workflows/test.ymlexamples/scrapegraphai_example.pyexamples/tools/README.mdexamples/tools/scrapegraphai.pysrc/agentor/tools/scrapegraphai.pytests/tools/test_scrapegraphai.pytests/tools/test_scrapegraphai_contract.py
✅ Files skipped from review due to trivial changes (2)
- examples/tools/README.md
- examples/tools/scrapegraphai.py
🚧 Files skipped from review as they are similar to previous changes (1)
- examples/scrapegraphai_example.py
| env: | ||
| AGENTOR_REQUIRE_SCRAPEGRAPH: "1" | ||
| steps: | ||
| - uses: actions/checkout@v4 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
Set persist-credentials: false on the checkout step.
The checkout action persists the GitHub token in git config by default. That token sits there for anyone who can access runner artifacts. Shut the door.
🛡️ Proposed fix
- uses: actions/checkout@v4
+ with:
+ persist-credentials: false📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - uses: actions/checkout@v4 | |
| - uses: actions/checkout@v4 | |
| with: | |
| persist-credentials: false |
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 80-80: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/test.yml at line 80, Add persist-credentials: false to the
actions/checkout@v4 step in the workflow so the GitHub token is not stored in
the repository’s git configuration.
Source: Linters/SAST tools
2 participants
Summary
ScrapeGraphAItool to the new scrapegraph-py 2.x SDK. The oldClientclass and its methods (smartscraper,markdownify,searchscraper,smartcrawler,sitemap) no longer exist upstream, so this is a full rewrite against the new endpoint surface.scrapegraphextra toscrapegraph-py>=2.1.0.SGAI_API_KEY(the new SDK default) and falls back to the legacySCRAPEGRAPH_API_KEYso existing users aren't broken.New capabilities
scrape(url, format)client.scrape(...)withMarkdown/Html/Links/SummaryFormatConfigextract(prompt, url, schema)client.extract(...)— AI structured extractionsearch(query, num_results, prompt)client.search(...)crawl(url, max_pages, max_depth, include/exclude)client.crawl.start(...)get_crawl_result(crawl_id)client.crawl.get(...)monitor(url, interval, name, webhook_url)client.monitor.create(...)credits()client.credits()health()client.health()Responses from the SDK are
ApiResultobjects; the tool turns successful results into a JSON string and surfacesresult.erroras"Error in <capability>: ..."so the LLM gets a consistent string return.Breaking change note
The previous capability names (
smartscraper,markdownify, etc.) are removed. Any agent prompt that hard-coded those names needs to be updated — seeexamples/scrapegraphai_example.pyfor the new surface.Test plan
pytest tests/tools/test_scrapegraphai.py— 16/16 pass (success paths, API error paths, exception paths, missing-dep guard, env-var resolution including legacy fallback)health,credits,scrape,extract,search,crawl(start +get_crawl_resultpoll to completion), and error path for invalid URLSummary by CodeRabbit
scrape(markdown/html/links/summary),extract,search,crawl, andmonitor, pluscreditsandhealth.SGAI_API_KEYusage.scrapegraph-py2.1+ on Python 3.12+.Greptile Summary
This PR replaces the defunct scrapegraph-py v1
ClientAPI with the v2ScrapeGraphAISDK surface, expanding the tool from 5 capabilities to 16 and adding a legacySCRAPEGRAPH_API_KEYenv-var fallback for backwards compatibility.src/agentor/tools/scrapegraphai.py): full rewrite; env-var resolution, Python-version guard, and_format_resulthelper are all clean; the previously flaggedJsonFormatConfig,super().__init__ordering, and test version-mock issues are all resolved in this PR.test_scrapegraphai.py): 16 unit tests covering success, API-error, exception, env-var precedence, and Python-version guard paths.test_scrapegraphai_contract.py): new job in CI verifies the SDK's method surface and format-builder types against a realscrapegraph-pyinstall on Python 3.12.Confidence Score: 5/5
Safe to merge — the migration is complete, all previously flagged issues are resolved, and the test suite is thorough.
The rewrite is well-scoped: env-var fallback, version guard, and error serialization are all correct. All three issues from prior review threads are addressed in this PR. The remaining observations are purely about crawl()/monitor() not exposing a format parameter and minor contract test coverage gaps — neither affects correctness.
No files require special attention. The only things worth a second look are the hardcoded MarkdownFormatConfig in crawl() and monitor(), which limit user flexibility but do not cause incorrect behaviour.
Important Files Changed
Flowchart
%%{init: {'theme': 'neutral'}}%% flowchart TD A[ScrapeGraphAI Tool] --> B{__init__} B -->|_SGAIClient is None| C{sys.version_info < 3.12?} C -->|Yes| D[ImportError: requires Python 3.12+] C -->|No| E[ImportError: dependency missing] B -->|_SGAIClient available| F[resolve api_key SGAI_API_KEY then SCRAPEGRAPH_API_KEY] F --> G[_SGAIClient instance] G --> H[scrape / extract / search / crawl / monitor / credits / health] H --> I{ApiResult.status == success?} I -->|Yes| J[serialize result.data to JSON string] I -->|No| K[Error in capability: result.error]%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% flowchart TD A[ScrapeGraphAI Tool] --> B{__init__} B -->|_SGAIClient is None| C{sys.version_info < 3.12?} C -->|Yes| D[ImportError: requires Python 3.12+] C -->|No| E[ImportError: dependency missing] B -->|_SGAIClient available| F[resolve api_key SGAI_API_KEY then SCRAPEGRAPH_API_KEY] F --> G[_SGAIClient instance] G --> H[scrape / extract / search / crawl / monitor / credits / health] H --> I{ApiResult.status == success?} I -->|Yes| J[serialize result.data to JSON string] I -->|No| K[Error in capability: result.error]Reviews (5): Last reviewed commit: "test: fix cross-version CI assertions" | Re-trigger Greptile