Web Extract API & Scraper
The Web Extract API turns any URL into markdown, text, HTML or structured data as clean JSON.
🤖 Using an AI assistant? Copy this link into ChatGPT / Claude / Cursor — it reads every endpoint and parameter instantly and tells you if this API fits your use case.
The primary scrape endpoint returns a page's final URL, title, metadata (canonical, language, site name, author, publish date, Open Graph) and the extracted content with a confidence signal, and you can map a site's URLs, crawl it, run structured extraction and batch many URLs. It is built for RAG pipelines, AI agents and content tools that need clean, LLM-ready page content without running a headless browser. One ReefAPI key, one shared credit pool, the standard envelope.
Per-action ceilings, the formats vocabulary, and what the extraction block tells you
Every response carries an extraction object saying which parser handled the page and how confident it was, which is the fastest way to tell whether you got the article or the whole page chrome. This table records the hard caps per action, how unknown format names are handled, and what the SSRF guard refuses. All rows come from live calls on 2026-08-27.
| Action or setting | Cap or vocabulary | What comes back |
|---|---|---|
| scrape | 1 URL | title, url, final_url, metadata and extraction{method, rendered, confidence, content_chars}, plus one key per requested format. |
| batch | 10 URLs, per-URL timeout 5-60 s (default 25) | results[] where each entry carries its own ok flag, so one failing URL does not fail the call; meta reports count and ok_count. |
| crawl | 25 pages, depth 0-5 (defaults 10 and 2) | pages[] with depth and page_type on each page, plus pages_visited and stop_reason (complete, max_pages or max_depth). |
| map | 1000 links (default 200) | links[]{url, page_type}, count, a page_types{} tally, sitemaps[], robots_present and sources{sitemap_urls, homepage_links}. |
| extract | 1 URL | jsonld{objects, types}, has_microdata, has_rdfa, tables[]{headers, rows, row_count}, headings[]{level, text}, prices[], emails[], phones[], plus schema_result when you pass a schema. |
| formats | markdown, text, html, rawHtml, metadata, links, images, jsonld | Unknown names are dropped without complaint: ["markdown","pdf","screenshot"] came back with meta.formats ["markdown"]. metadata is returned whether or not you ask for it. |
| extraction.confidence | high, medium, low | "high" with method "trafilatura" on a Wikipedia article (content_chars 26,559); "medium" with "markdownify-fallback" on a client-rendered page; "low" on example.com (content_chars 127). |
| SSRF guard | Private, internal and cloud-metadata targets | http://169.254.169.254/latest/meta-data/ returned error code INVALID_PARAM with "SSRF-blocked: private/internal IP". That is a rejected parameter, not a fetch failure. |
only_main_content trims the content formats, not links. A Wikipedia scrape with only_main_content left at its default still returned 328 entries in links[], including site chrome such as "Main page" and "Contents". Each link carries an internal boolean and a rel field, so filter on those and on the path before treating links[] as article links.
Real request and response JSON
Captured from the indexed primary action, scrape, on .
{
"method": "POST",
"url": "https://api.reefapi.com/web-extract/v1/scrape",
"headers": {
"x-api-key": "$REEF_KEY",
"content-type": "application/json"
},
"body": {
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"formats": [
"markdown",
"metadata"
]
}
}{
"ok": true,
"meta": {
"api": "web-extract",
"endpoint": "scrape",
"mode": "live",
"latency_ms": 1076.4,
"record_count": 1,
"bytes": 231204,
"cache_hit": false,
"method": "bare_dc_curl_cffi+trafilatura",
"browserless": true,
"ssrf_guarded": true,
"final_url": "https://en.wikipedia.org/wiki/Web_scraping",
"formats": [
"markdown",
"metadata"
],
"extraction_method": "trafilatura",
"confidence": "high"
},
"data": {
"final_url": "https://en.wikipedia.org/wiki/Web_scraping",
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"title": "Web scraping - Wikipedia",
"metadata": {
"title": "Web scraping - Wikipedia",
"description": null,
"canonical": "https://en.wikipedia.org/wiki/Web_scraping",
"lang": "en",
"site_name": "Wikimedia Foundation, Inc.",
"author": "Contributors to Wikimedia projects",
"published_at": "[redacted-phone]T18:57:30Z",
"modified_at": null,
"section": null,
"keywords": [],
"favicon": "https://en.wikipedia.org/static/apple-touch/wikipedia.png",
"og": {
"title": "Web scraping - Wikipedia",
"type": "website"
},
"twitter": {},
"hreflang": [],
"feeds": [
"https://en.wikipedia.org/w/index.php?title=Special:RecentChanges&feed=atom"
],
"status_code": 200,
"content_type": "text/html; charset=UTF-8",
"robots_meta": "max-image-preview:standard"
},
"extraction": {
"method": "trafilatura",
"rendered": false,
"confidence": "high",
"content_chars": 25663
},
"markdown": "|  | \n\n**Web scraping**, **web harvesting**, or **web data extraction** is [data scraping](https://en.wikipedia.org/wiki/Data_scraping) used for [extracting data](https://en.wikipedia.org/wiki/Data_extraction) from [websites](https://en.wikipedia.org/wiki/Website).\n\n[World Wide Web](https://en.wikipedia.org/wiki/World_Wide_Web)using the\n\n[Hypertext Transfer Protocol](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol)or a web browser. While web scraping can be done manually "
}
}What the Web Extract API does
| Action | Description | Concrete use case | Key params |
|---|---|---|---|
| scrape | Fetch one URL and return its content in the formats you choose (markdown, plain text, cleaned HTML, raw HTML, metadata, links, images, or structured data). Automatically upgrades thin single-page-app pages to a full browser render when needed. Works on any public page — news articles, product pages, documentation, blogs. | Platform and DevOps teams call scrape to fetch one URL and. | url, formats, render, only_main_content, include_tags, ... |
| map | Discover a site's complete URL surface: reads robots.txt, sitemap.xml (and sitemap indexes), and the homepage's internal links — returns a deduplicated URL list with a page-type label for each (home, pricing, docs, blog, product, legal, contact, about). Use this to index any site or audit its structure. | Security and supply-chain teams call map to discover a site's complete URL surface. | url, search, limit, include_subdomains |
| crawl | Crawl a site starting from a seed URL (up to 25 pages): follows internal links breadth-first with configurable depth, include/exclude URL patterns, and returns every visited page in the formats you choose. Ideal for content indexing, site audits, and building knowledge bases from documentation or blog sites. | Developer-tool builders call crawl to get crawl a site starting from a seed URL (up to 25 pages). | url, max_pages, max_depth, same_domain_only, include_patterns, ... |
| extract | Pull structured data from any public page: returns objects (Product, Article, Organization, etc.), microdata presence, parsed table rows, heading outline, prices, emails, and phone numbers. Supply an optional field-to-path schema to extract specific values directly — great for e-commerce pricing, article metadata, and business listings. | AI-agent developers call extract to pull structured data from any public page. | url, schema, deterministic_only |
| batch | Scrape up to 10 URLs concurrently in one call (each SSRF-guarded). Shared formats. Firecrawl /batch-scrape parity (bounded). | Platform and DevOps teams call batch to get scrape up to 10 URLs concurrently in one call (each SSRF-guarded). | urls, formats, only_main_content, timeout |
Call scrape from your stack
curl -X POST https://api.reefapi.com/web-extract/v1/scrape \
-H "x-api-key: $REEF_KEY" \
-H "content-type: application/json" \
-d '{"url":"https://en.wikipedia.org/wiki/Web_scraping","formats":["markdown","metadata"]}'import requests
r = requests.post(
"https://api.reefapi.com/web-extract/v1/scrape",
headers={"x-api-key": REEF_KEY},
json={
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"formats": [
"markdown",
"metadata"
]
},
)
print(r.json()["data"])const res = await fetch("https://api.reefapi.com/web-extract/v1/scrape", {
method: "POST",
headers: {
"x-api-key": process.env.REEF_KEY,
"content-type": "application/json",
},
body: JSON.stringify({
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"formats": [
"markdown",
"metadata"
]
}),
});
const { ok, data, meta, error } = await res.json();Ask your MCP-connected assistant: call reefapi.web-extract.scrape with {"url":"https://en.wikipedia.org/wiki/Web_scraping","formats":["markdown","metadata"]}.Who uses this API and why
- RAG pipelines call scrape to turn a URL into clean markdown for an AI knowledge base.
- AI agents use extract to pull structured fields from a page without custom parsers.
- Content tools use map and crawl to discover and ingest an entire site's pages.
Questions developers ask before integrating
Does render=force actually run a browser?
Not on this engine right now, and the response says so plainly. Every call came back with meta.browserless true and extraction.rendered false, whether render was auto, force or never. On a client-rendered page the extraction block adds render_note "body is thin/JS-hydrated; render unavailable" and render_available_via "web-capture". So a JavaScript-only page gives you whatever the server HTML contained; for a genuine browser render use the web-capture engine that field points at.
When should I reach for a site-specific engine instead of this one?
Whenever the target defends itself, and whenever you want fields rather than prose. A scrape of an amazon.com product URL returned error code TARGET_BLOCKED. web-extract is built for ordinary public pages such as news articles, documentation, blogs and marketing sites, and it hands you text. A dedicated engine for the same site hands you a parsed price, rating and stock status, which is almost always what you actually wanted.
map returned count 0 for a site that obviously has pages. What happened?
The origin refused both robots.txt and the homepage, and map reports that as an empty surface rather than an error. A live map of stripe.com returned ok:true with count 0, robots_present false and sources {sitemap_urls: 0, homepage_links: 0}. Compare a cooperative site: reefapi.com returned sources {sitemap_urls: 246, homepage_links: 25} and 50 links at limit 50. Read sources before you trust a zero.
How accurate is the page_type label?
It is a URL heuristic, useful for bucketing and not for routing. Labels observed across three sites: home, docs, blog, pricing, about, contact, legal and other. On python.org, /jobs/ was labeled "about" and docs.python.org was labeled "home"; on fastapi.tiangolo.com, 47 of 50 documentation pages landed in "other". The page_types tally in the same response gives you the distribution at a glance.
What does schema_result return when my path does not exist on the page?
Your keys, with null values, so the shape is always safe to destructure. A schema of {"price": "offers.price", "name": "name"} against a product page with no reachable JSON-LD returned {"price": null, "name": null} alongside jsonld.types []. If every key comes back null, check jsonld.types first: an empty types array means there was no structured data to path into at all.
What is the shape of prices[] in an extract response?
Objects of {raw, currency, value} where value is a string and currency is the symbol as printed rather than an ISO code. A product page on books.toscrape.com returned {"raw": "£51.77", "currency": "£", "value": "51.77"}. It is a page-wide sweep, so it also picks up unrelated amounts: that same page returned a second entry of £0.00. Convert value to a number and map the symbol yourself.
Why did crawl stop after one page when I asked for two?
Because it ran out of in-scope links, and it says so. A crawl of example.com with max_pages 2 and max_depth 1 returned pages_visited 1 and stop_reason "complete", since the page's only link goes to iana.org and same_domain_only defaults to true. stop_reason "complete" means the crawl finished naturally, while "max_pages" or "max_depth" mean you hit a ceiling and there is more to fetch.
What is actually inside metadata?
More than the documented handful. Alongside title, canonical, lang, site_name, author and published_at you get status_code, content_type, robots_meta, feeds[], hreflang[], favicon, og{} and twitter{}. Measured: a Wikipedia page returned status_code 200, published_at "2005-09-17T18:57:30Z", modified_at null and feeds carrying the site's Atom URL, while example.com returned status_code 206, meaning the fetch was served as a partial response. published_at present with modified_at null is common, so do not read that as "never updated".
What is the Web Extract API?
Web Extract API is a ReefAPI endpoint group for web extract It returns live JSON through POST requests under /web-extract/v1.
Is the Web Extract API free to try?
Yes. ReefAPI starts with 1,000 free credits, no card required. Web Extract calls use the same shared credit balance as every other ReefAPI engine.
Do I need a Web Extract login or account?
No login to Web Extract is needed for the API response. You call ReefAPI with your x-api-key header, and the playground can run live examples before you create a production key.
How fresh is the Web Extract data?
The page example is captured from a live scrape call, and production requests fetch live data through ReefAPI rather than a static sample.
How many credits does the Web Extract API use?
Web Extract actions currently cost 1-2 credits per successful call. Failed or blocked calls are free, and all APIs draw from one credit pool.
Can I call Web Extract from an AI assistant or MCP client?
Yes. Connect ReefAPI once through MCP and your assistant can call web-extract actions with the same key, credit pool and JSON envelope used by normal REST requests.