Skip to content

Commit 9cf3f3f

Browse files
nihalwasherenihalwashere
committed
Document extract feature
- New concept page: extract primitive overview, schema formats, credits - New guide: structured extraction with code examples (JS, Python, cURL, CLI) - Updated API reference with extract section - Added pages to navigation
1 parent f594b8d commit 9cf3f3f

4 files changed

Lines changed: 385 additions & 1 deletion

File tree

‎api-reference/read.mdx‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,24 @@ Set `maxConcurrency` to limit how many browser slots a batch or crawl job uses.
2121

2222
Check your current usage with [Queue status](/api-reference/account/queue-status).
2323

24+
## Extract
25+
26+
Add the `extract` parameter to pull structured data from the page alongside markdown. Provide a JSON Schema, a shorthand schema, a natural language prompt, or both.
27+
28+
```bash
29+
curl -X POST https://api.reader.dev/v1/read \
30+
-H "x-api-key: $READER_KEY" \
31+
-H "Content-Type: application/json" \
32+
-d '{
33+
"url": "https://example.com/product",
34+
"extract": {
35+
"schema": { "title": "string", "price": "number", "in_stock": "boolean" }
36+
}
37+
}'
38+
```
39+
40+
The response includes an `extracted` field with the structured data and extraction metadata. Adds 2 credits to the scrape cost. Only supported for single-URL scrapes (not batch or crawl). See [Extract](/home/concepts/extract) for details.
41+
2442
## Idempotency
2543

2644
Pass an `x-idempotency-key` header to deduplicate retried POSTs. Reader caches the original response for 24 hours and returns it verbatim on any subsequent request with the same key.

‎home/concepts/extract.mdx‎

Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
---
2+
title: "Extract"
3+
description: "Pull structured data from any web page using an LLM"
4+
---
5+
6+
Extract lets you pull structured data from a web page in a single API call. You provide a URL and a schema (or a natural language prompt), and Reader scrapes the page, converts it to clean markdown, sends it to an LLM, and returns validated structured JSON alongside the usual markdown output.
7+
8+
## How it works
9+
10+
Extract is an optional parameter on `POST /v1/read`. The scrape pipeline runs as usual, then the extracted markdown is sent to an LLM with your schema for structured extraction.
11+
12+
```
13+
URL --> Scrape --> Markdown --> LLM + Schema --> Structured JSON
14+
```
15+
16+
The `extracted` field appears in the response alongside `markdown`, `html`, and other standard fields. If extraction fails, the scrape result is still returned with an error in `metadata.extraction.extractionError`.
17+
18+
## Schema formats
19+
20+
### JSON Schema (full)
21+
22+
Standard JSON Schema with types, descriptions, and nested objects:
23+
24+
```json
25+
{
26+
"extract": {
27+
"schema": {
28+
"type": "object",
29+
"properties": {
30+
"title": { "type": "string", "description": "The product name" },
31+
"price": { "type": "number", "description": "Price without currency symbol" },
32+
"in_stock": { "type": "boolean" }
33+
}
34+
}
35+
}
36+
}
37+
```
38+
39+
### Shorthand
40+
41+
A simpler format where keys map directly to type names:
42+
43+
```json
44+
{
45+
"extract": {
46+
"schema": {
47+
"title": "string",
48+
"price": "number",
49+
"in_stock": "boolean"
50+
}
51+
}
52+
}
53+
```
54+
55+
Shorthand is automatically expanded to full JSON Schema internally. Supported types: `string`, `number`, `integer`, `boolean`, `array`, `object`.
56+
57+
### Prompt-only
58+
59+
No schema, just a natural language instruction. Returns freeform JSON:
60+
61+
```json
62+
{
63+
"extract": {
64+
"prompt": "List the 3 main topics discussed on this page"
65+
}
66+
}
67+
```
68+
69+
### Prompt + Schema
70+
71+
Combine both for maximum control. The prompt guides the LLM while the schema enforces structure:
72+
73+
```json
74+
{
75+
"extract": {
76+
"schema": { "title": "string", "price": "number" },
77+
"prompt": "Focus on the primary product, ignore accessories"
78+
}
79+
}
80+
```
81+
82+
## Response shape
83+
84+
```json
85+
{
86+
"success": true,
87+
"data": {
88+
"url": "https://example.com/product",
89+
"markdown": "# Widget Pro\n\nPrice: $49.99...",
90+
"extracted": {
91+
"title": "Widget Pro",
92+
"price": 49.99,
93+
"in_stock": true
94+
},
95+
"metadata": {
96+
"statusCode": 200,
97+
"duration": 2345,
98+
"scrapedAt": "2026-07-05T12:00:00.000Z"
99+
}
100+
}
101+
}
102+
```
103+
104+
## Null handling
105+
106+
If a field in your schema does not exist on the page, Reader returns `null` for that field. It will never hallucinate values that are not present in the content.
107+
108+
```json
109+
{
110+
"extracted": {
111+
"title": "Widget Pro",
112+
"price": 49.99,
113+
"ceo_name": null,
114+
"ipo_date": null
115+
}
116+
}
117+
```
118+
119+
## Credits
120+
121+
Extract adds 2 credits on top of the scrape cost:
122+
123+
| Operation | Credits |
124+
|-----------|---------|
125+
| Standard scrape | 1 |
126+
| Standard scrape + extract | 3 |
127+
| Premium scrape + extract | 5 |
128+
129+
If the scrape succeeds but extraction fails, the full cost (scrape + extract) is still charged because the LLM call was attempted.
130+
131+
## Limitations
132+
133+
- **Scrape mode only.** Extract is not supported for batch or crawl operations.
134+
- **Token limit.** Page content is capped at 30,000 tokens before being sent to the LLM. Long pages are intelligently truncated (70% from the beginning, 30% from the end).
135+
- **No caching.** Extraction always runs fresh, even if the scrape result is cached. Cached scrape responses do not include `extracted`.
136+
- **Validation retry.** If the LLM output does not match your schema, Reader retries once with additional context about the validation error.
137+
138+
## Error handling
139+
140+
Extract never fails the entire request. If the LLM call fails (timeout, rate limit, invalid response), the scrape result is still returned with `extracted: null` and an error in `metadata.extraction.extractionError`:
141+
142+
```json
143+
{
144+
"extracted": null,
145+
"metadata": {
146+
"extraction": {
147+
"extractionError": "Extraction service rate limited, try again later"
148+
}
149+
}
150+
}
151+
```

‎home/guides/llm/structured-extraction.mdx‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
---
2+
title: "Structured Extraction"
3+
description: "Extract structured data from any web page with schemas or prompts"
4+
---
5+
6+
This guide shows how to use the `extract` parameter on `POST /v1/read` to pull structured data from web pages.
7+
8+
## Product data
9+
10+
Extract product information from an e-commerce page:
11+
12+
<CodeGroup>
13+
14+
```bash cURL
15+
curl -X POST https://api.reader.dev/v1/read \
16+
-H "x-api-key: rdr_YOUR_KEY" \
17+
-H "Content-Type: application/json" \
18+
-d '{
19+
"url": "https://example.com/product/widget-pro",
20+
"extract": {
21+
"schema": {
22+
"name": "string",
23+
"price": "number",
24+
"currency": "string",
25+
"in_stock": "boolean",
26+
"description": "string"
27+
}
28+
}
29+
}'
30+
```
31+
32+
```javascript JavaScript
33+
import { ReaderClient } from "@vakra-dev/reader-js";
34+
35+
const client = new ReaderClient({ apiKey: "rdr_YOUR_KEY" });
36+
37+
const result = await client.read({
38+
url: "https://example.com/product/widget-pro",
39+
extract: {
40+
schema: {
41+
name: "string",
42+
price: "number",
43+
currency: "string",
44+
in_stock: "boolean",
45+
},
46+
},
47+
});
48+
49+
if (result.kind === "scrape") {
50+
console.log(result.data.extracted);
51+
// { name: "Widget Pro", price: 49.99, currency: "USD", in_stock: true }
52+
}
53+
```
54+
55+
```python Python
56+
from reader_py import ReaderClient
57+
58+
client = ReaderClient(api_key="rdr_YOUR_KEY")
59+
60+
result = client.read(
61+
url="https://example.com/product/widget-pro",
62+
extract={"schema": {"name": "string", "price": "number"}},
63+
)
64+
65+
if result.kind == "scrape":
66+
print(result.data.extracted)
67+
```
68+
69+
</CodeGroup>
70+
71+
## Table data
72+
73+
Extract structured data from HTML tables (e.g., a pricing page):
74+
75+
```json
76+
{
77+
"url": "https://example.com/pricing",
78+
"extract": {
79+
"schema": {
80+
"type": "object",
81+
"properties": {
82+
"plans": {
83+
"type": "array",
84+
"items": {
85+
"type": "object",
86+
"properties": {
87+
"name": { "type": "string" },
88+
"price_monthly": { "type": "number" },
89+
"features": { "type": "array", "items": { "type": "string" } }
90+
}
91+
}
92+
}
93+
}
94+
}
95+
}
96+
}
97+
```
98+
99+
## Using field descriptions
100+
101+
Add `description` to schema fields to guide the LLM on what to extract:
102+
103+
```json
104+
{
105+
"url": "https://example.com/article",
106+
"extract": {
107+
"schema": {
108+
"type": "object",
109+
"properties": {
110+
"title": {
111+
"type": "string",
112+
"description": "The main article headline"
113+
},
114+
"author": {
115+
"type": "string",
116+
"description": "The author's full name"
117+
},
118+
"published_date": {
119+
"type": "string",
120+
"description": "Publication date in YYYY-MM-DD format"
121+
},
122+
"summary": {
123+
"type": "string",
124+
"description": "A one-sentence summary of the article"
125+
}
126+
}
127+
}
128+
}
129+
}
130+
```
131+
132+
## Prompt-only mode
133+
134+
When you do not need a fixed schema, use a natural language prompt:
135+
136+
```json
137+
{
138+
"url": "https://example.com/about",
139+
"extract": {
140+
"prompt": "Extract the company name, founding year, and number of employees"
141+
}
142+
}
143+
```
144+
145+
The response will be freeform JSON based on what the LLM finds on the page.
146+
147+
## Combining prompt and schema
148+
149+
Use both together -- the prompt provides context while the schema enforces structure:
150+
151+
```json
152+
{
153+
"url": "https://example.com/product",
154+
"extract": {
155+
"schema": { "name": "string", "price": "number" },
156+
"prompt": "Focus on the main product listing, ignore recommended products in the sidebar"
157+
}
158+
}
159+
```
160+
161+
## Handling missing fields
162+
163+
If a field in your schema does not exist on the page, it comes back as `null`. This is intentional -- Reader will never hallucinate values:
164+
165+
```json
166+
{
167+
"extracted": {
168+
"title": "Widget Pro",
169+
"price": 49.99,
170+
"warranty_years": null
171+
}
172+
}
173+
```
174+
175+
## CLI usage
176+
177+
```bash
178+
# With schema
179+
reader scrape https://example.com/product \
180+
--extract-schema '{"title": "string", "price": "number"}'
181+
182+
# With prompt
183+
reader scrape https://example.com/about \
184+
--extract-prompt "Extract the company name and founding year"
185+
186+
# Both
187+
reader scrape https://example.com/product \
188+
--extract-schema '{"name": "string", "price": "number"}' \
189+
--extract-prompt "Focus on the primary product"
190+
```
191+
192+
## Error handling
193+
194+
Extract never breaks the scrape response. If the LLM fails, you still get the markdown:
195+
196+
```javascript
197+
const result = await client.read({
198+
url: "https://example.com",
199+
extract: { schema: { title: "string" } },
200+
});
201+
202+
if (result.kind === "scrape") {
203+
// Markdown is always available
204+
console.log(result.data.markdown);
205+
206+
// Check if extraction succeeded
207+
if (result.data.extracted) {
208+
console.log("Extracted:", result.data.extracted);
209+
} else {
210+
console.log("Extraction failed:", result.data.metadata.extraction?.extractionError);
211+
}
212+
}
213+
```

0 commit comments

Comments
 (0)