Web Extract API API
The Web Extract API turns any URL into markdown, text, HTML or structured data as clean JSON.
🤖 Using an AI assistant? Copy this link into ChatGPT / Claude / Cursor — it reads every endpoint and parameter instantly and tells you if this API fits your use case.
The primary scrape endpoint returns a page's final URL, title, metadata (canonical, language, site name, author, publish date, Open Graph) and the extracted content with a confidence signal, and you can map a site's URLs, crawl it, run structured extraction and batch many URLs. It is built for RAG pipelines, AI agents and content tools that need clean, LLM-ready page content without running a headless browser. One ReefAPI key, one shared credit pool, the standard envelope.
Real request and response JSON
Captured from the indexed primary action, scrape, on .
{
"method": "POST",
"url": "https://api.reefapi.com/web-extract/v1/scrape",
"headers": {
"x-api-key": "$REEF_KEY",
"content-type": "application/json"
},
"body": {
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"formats": [
"markdown",
"metadata"
]
}
}{
"ok": true,
"meta": {
"api": "web-extract",
"endpoint": "scrape",
"mode": "live",
"latency_ms": 1076.4,
"record_count": 1,
"bytes": 231204,
"cache_hit": false,
"method": "bare_dc_curl_cffi+trafilatura",
"browserless": true,
"ssrf_guarded": true,
"final_url": "https://en.wikipedia.org/wiki/Web_scraping",
"formats": [
"markdown",
"metadata"
],
"extraction_method": "trafilatura",
"confidence": "high"
},
"data": {
"final_url": "https://en.wikipedia.org/wiki/Web_scraping",
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"title": "Web scraping - Wikipedia",
"metadata": {
"title": "Web scraping - Wikipedia",
"description": null,
"canonical": "https://en.wikipedia.org/wiki/Web_scraping",
"lang": "en",
"site_name": "Wikimedia Foundation, Inc.",
"author": "Contributors to Wikimedia projects",
"published_at": "[redacted-phone]T18:57:30Z",
"modified_at": null,
"section": null,
"keywords": [],
"favicon": "https://en.wikipedia.org/static/apple-touch/wikipedia.png",
"og": {
"title": "Web scraping - Wikipedia",
"type": "website"
},
"twitter": {},
"hreflang": [],
"feeds": [
"https://en.wikipedia.org/w/index.php?title=Special:RecentChanges&feed=atom"
],
"status_code": 200,
"content_type": "text/html; charset=UTF-8",
"robots_meta": "max-image-preview:standard"
},
"extraction": {
"method": "trafilatura",
"rendered": false,
"confidence": "high",
"content_chars": 25663
},
"markdown": "|  | \n\n**Web scraping**, **web harvesting**, or **web data extraction** is [data scraping](https://en.wikipedia.org/wiki/Data_scraping) used for [extracting data](https://en.wikipedia.org/wiki/Data_extraction) from [websites](https://en.wikipedia.org/wiki/Website).\n\n[World Wide Web](https://en.wikipedia.org/wiki/World_Wide_Web)using the\n\n[Hypertext Transfer Protocol](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol)or a web browser. While web scraping can be done manually "
}
}What the Web Extract API API does
| Action | Description | Concrete use case | Key params |
|---|---|---|---|
| scrape | Fetch one URL and return its content in the formats you choose (markdown, plain text, cleaned HTML, raw HTML, metadata, links, images, or structured data). Automatically upgrades thin single-page-app pages to a full browser render when needed. Works on any public page — news articles, product pages, documentation, blogs. | Ops teams call scrape to fetch one URL and. | url, formats, render, only_main_content, include_tags, ... |
| map | Discover a site's complete URL surface: reads robots.txt, sitemap.xml (and sitemap indexes), and the homepage's internal links — returns a deduplicated URL list with a page-type label for each (home, pricing, docs, blog, product, legal, contact, about). Use this to index any site or audit its structure. | Developer tools call map to discover a site's complete URL surface. | url, search, limit, include_subdomains |
| crawl | Crawl a site starting from a seed URL (up to 25 pages): follows internal links breadth-first with configurable depth, include/exclude URL patterns, and returns every visited page in the formats you choose. Ideal for content indexing, site audits, and building knowledge bases from documentation or blog sites. | Validation workflows call crawl to get crawl a site starting from a seed URL (up to 25 pages). | url, max_pages, max_depth, same_domain_only, include_patterns, ... |
| extract | Pull structured data from any public page: returns objects (Product, Article, Organization, etc.), microdata presence, parsed table rows, heading outline, prices, emails, and phone numbers. Supply an optional field-to-path schema to extract specific values directly — great for e-commerce pricing, article metadata, and business listings. | Data-quality teams call extract to pull structured data from any public page. | url, schema, deterministic_only, timeout |
| batch | Scrape up to 10 URLs concurrently in one call (each SSRF-guarded). Shared formats. Firecrawl /batch-scrape parity (bounded). | Ops teams call batch to get scrape up to 10 URLs concurrently in one call (each SSRF-guarded). | urls, formats, only_main_content, timeout |
Call scrape from your stack
curl -X POST https://api.reefapi.com/web-extract/v1/scrape \
-H "x-api-key: $REEF_KEY" \
-H "content-type: application/json" \
-d '{"url":"https://en.wikipedia.org/wiki/Web_scraping","formats":["markdown","metadata"]}'import requests
r = requests.post(
"https://api.reefapi.com/web-extract/v1/scrape",
headers={"x-api-key": REEF_KEY},
json={
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"formats": [
"markdown",
"metadata"
]
},
)
print(r.json()["data"])const res = await fetch("https://api.reefapi.com/web-extract/v1/scrape", {
method: "POST",
headers: {
"x-api-key": process.env.REEF_KEY,
"content-type": "application/json",
},
body: JSON.stringify({
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"formats": [
"markdown",
"metadata"
]
}),
});
const { ok, data, meta, error } = await res.json();Ask your MCP-connected assistant: call reefapi.web-extract.scrape with {"url":"https://en.wikipedia.org/wiki/Web_scraping","formats":["markdown","metadata"]}.Who uses this API and why
- RAG pipelines call scrape to turn a URL into clean markdown for an AI knowledge base.
- AI agents use extract to pull structured fields from a page without custom parsers.
- Content tools use map and crawl to discover and ingest an entire site's pages.
Questions developers ask before integrating
What is the Web Extract API API?
Web Extract API API is a ReefAPI endpoint group for web extract api It returns live JSON through POST requests under /web-extract/v1.
Is the Web Extract API API free to try?
Yes. ReefAPI starts with 1,000 free credits, no card required. Web Extract API calls use the same shared credit balance as every other ReefAPI engine.
Do I need a Web Extract API login or account?
No login to Web Extract API is needed for the API response. You call ReefAPI with your x-api-key header, and the playground can run live examples before you create a production key.
How fresh is the Web Extract API data?
The page example is captured from a live scrape call, and production requests fetch live data through ReefAPI rather than a static sample.
How many credits does the Web Extract API API use?
Web Extract API actions currently cost 1-2 credits per successful call. Failed or blocked calls are free, and all APIs draw from one credit pool.
Can I call Web Extract API from an AI assistant or MCP client?
Yes. Connect ReefAPI once through MCP and your assistant can call web-extract actions with the same key, credit pool and JSON envelope used by normal REST requests.
Is the Web Extract API API a Web Extract API scraper?
It is the managed alternative to a DIY Web Extract API scraper. Instead of building and maintaining your own scraper — proxies, headless browsers, captcha and constant breakage — you call one ReefAPI endpoint and get the same web extract api back as clean JSON.
Why does my Web Extract API scraper keep getting blocked?
Most Web Extract API scrapers break on anti-bot defenses, rate limits and IP bans that need rotating residential proxies and browser fingerprinting to clear. ReefAPI handles all of that for you — no proxies, no captchas, no maintenance — and returns live JSON. Blocked or failed calls are free.