For the sites we do not have a dedicated engine for
The Web Extract API turns any URL into markdown, text, HTML or structured data as clean JSON.
5 active endpoints, on 1 and 2 credit tiers.
- POST/web-extract/v1/scrape
- POST/web-extract/v1/map
- POST/web-extract/v1/crawl
- POST/web-extract/v1/extract
- POST/web-extract/v1/batch
What Web Extract endpoints does ReefAPI ship?
5 live read endpoints. Read-only data API: no writes, no account actions, no dashboard access on the target site.
Web Extract API
3 of 5 endpoints, ready to run
One page as clean markdown plus its metadata, with an extraction block reporting which method was used, whether the page needed rendering and how confident the result is.
{ "ok": true, "meta": { "api": "web-extract", "endpoint": "scrape", "mode": "live", "latency_ms": 847, "record_count": 1, "cache_hit": false }, "data": { "final_url": "https://example.com", "url": "https://example.com", "title": "Example Domain", "metadata": { "title": "Example Domain", "description": null, "canonical": null, "lang": "en", "site_name": "example.com", "published_at": null, "modified_at": null, "section": null, "keywords": [], "favicon": "https://example.com/favicon.ico", "og": {}, "twitter": {}, "hreflang": [], "feeds": [], "status_code": 206, "content_type": "text/html", "robots_meta": null }, "extraction": { "method": "markdownify-fallback", "rendered": false, "confidence": "low", "content_chars": 127, "render_note": "body is thin/JS-hydrated; render unavailable — see the web-capture engine for browser rendering + screenshots", "render_available_via": "web-capture" }, "markdown": "# Example Domain\n\nThis domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)" } }
How the Web Extract API works
Web Extract is a normal ReefAPI surface — the same four rules that hold for every other engine on the key.
No OAuth app, no request signing, no per-site account. One key covers all 184 engines.
Every route is a POST with a JSON body. Parameters are validated against the published schema before anything is charged.
Credits, not seats. Failed and blocked calls are never charged, and cache hits cost nothing.
One envelope everywhere. meta carries latency_ms, record_count and the endpoint that answered.
Covering the long tail behind a dedicated engine
A product that ingests customer-supplied URLs will always meet sites nobody built an engine for. The fallback needs to be one call, not a scraping project.
{"url": "https://example.com", "limit": 100}Returns the discoverable URLs with each classified by type and the sitemaps it found. One site we mapped came back with two hundred URLs bucketed into home, docs, pricing and about.
{"url": "https://example.com/docs/page"}Then the pages you want, as markdown with the metadata attached — which is the format an index or a model actually consumes.
Map first, scrape second is the pattern: it turns 'crawl this site' into a list you can filter before spending a call per page.
curl -X POST https://api.reefapi.com/web-extract/v1/scrape \
-H "x-api-key: $REEF_KEY" \
-H "content-type: application/json" \
-d '{"url":"https://en.wikipedia.org/wiki/Web_scraping","formats":["markdown","metadata"]}'{
"ok": true,
"data": { … },
"meta": {
"api": "web-extract",
"endpoint": "scrape",
"mode": "live",
"latency_ms": …,
"record_count": …
},
"error": null
}Per-action ceilings, the formats vocabulary, and what the extraction block tells you
Every response carries an extraction object saying which parser handled the page and how confident it was, which is the fastest way to tell whether you got the article or the whole page chrome. This table records the hard caps per action, how unknown format names are handled, and what the SSRF guard refuses. All rows come from live calls on 2026-08-27.
| Action or setting | Cap or vocabulary | What comes back |
|---|---|---|
| scrape | 1 URL | title, url, final_url, metadata and extraction{method, rendered, confidence, content_chars}, plus one key per requested format. |
| batch | 10 URLs, per-URL timeout 5-60 s (default 25) | results[] where each entry carries its own ok flag, so one failing URL does not fail the call; meta reports count and ok_count. |
| crawl | 25 pages, depth 0-5 (defaults 10 and 2) | pages[] with depth and page_type on each page, plus pages_visited and stop_reason (complete, max_pages or max_depth). |
| map | 1000 links (default 200) | links[]{url, page_type}, count, a page_types{} tally, sitemaps[], robots_present and sources{sitemap_urls, homepage_links}. |
| extract | 1 URL | jsonld{objects, types}, has_microdata, has_rdfa, tables[]{headers, rows, row_count}, headings[]{level, text}, prices[], emails[], phones[], plus schema_result when you pass a schema. |
| formats | markdown, text, html, rawHtml, metadata, links, images, jsonld | Unknown names are dropped without complaint: ["markdown","pdf","screenshot"] came back with meta.formats ["markdown"]. metadata is returned whether or not you ask for it. |
| extraction.confidence | high, medium, low | "high" with method "trafilatura" on a Wikipedia article (content_chars 26,559); "medium" with "markdownify-fallback" on a client-rendered page; "low" on example.com (content_chars 127). |
| SSRF guard | Private, internal and cloud-metadata targets | http://169.254.169.254/latest/meta-data/ returned error code INVALID_PARAM with "SSRF-blocked: private/internal IP". That is a rejected parameter, not a fetch failure. |
only_main_content trims the content formats, not links. A Wikipedia scrape with only_main_content left at its default still returned 328 entries in links[], including site chrome such as "Main page" and "Contents". Each link carries an internal boolean and a rel field, so filter on those and on the path before treating links[] as article links.
What it reads, what it renders, and what it refuses to guess
Measured on static pages and on a site map. The extraction row is where expectations usually go wrong.
Every scrape carries an extraction block naming the method used, whether the page had to be rendered, the character count and a confidence label. Content that came from a rendered page and content that came from the raw response are different things, and you are told which you have rather than having to infer it from the result.
Asking for fields by description returns nulls unless a non-deterministic pass is enabled. What the deterministic path does return is everything the page publishes about itself in machine-readable form — JSON-LD objects and their types, tables, headings, prices, emails, phone numbers. That is a narrower promise than 'describe this page for me', and it is deliberately the one we make by default, because a made-up field that looks right is worse than an absent one.
The site map returns each URL with a page type and reports whether they came from a sitemap or from crawling the homepage, along with a count per type. Filtering a classified list is what makes the difference between scraping ten pages and scraping two hundred.
Maximum pages, maximum depth, same-domain restriction and include and exclude patterns are all inputs. A crawl endpoint without hard limits is a way to spend a lot of money by accident, so the limits are required rather than advisory.
Where a dedicated engine exists for a site, that engine returns typed fields — prices as numbers, dates as dates, identifiers that are stable. This returns clean text and whatever structure the page happens to publish. Use it for the long tail; use the specific engine when there is one.
What people build with Web Extract
The jobs this data is most often used for.
endpoints
credits per call
RAG pipelines call scrape to turn a URL into clean markdown for an AI knowledge base.
AI agents use extract to pull structured fields from a page without custom parsers.
Content tools use map and crawl to discover and ingest an entire site's pages.
What Web Extract data costs
The cheapest call here is 1 credit, so $15/mo (Pro) buys 10,000 of them — $1.50 per 1,000 credits. Credits roll over and never expire, and failed or blocked calls are not charged.
Full pricing →- 1,000 free credits on signup, no card
- One key, all 184 APIs, one credit pool
- Failed and blocked calls are never charged
- Credits roll over and never expire
Call it in two lines
Sign up, get 1,000 credits and one key that works on every engine. Then this is the whole protocol.
curl -X POST https://api.reefapi.com/web-extract/v1/scrape \
-H "x-api-key: $REEF_KEY" \
-H "content-type: application/json" \
-d '{"url":"https://en.wikipedia.org/wiki/Web_scraping","formats":["markdown","metadata"]}'import requests
r = requests.post(
"https://api.reefapi.com/web-extract/v1/scrape",
headers={"x-api-key": REEF_KEY},
json={
"url": "https://en.wikipedia.org/wiki/Web_scraping",
"formats": [
"markdown",
"metadata"
]
},
)
print(r.json()["data"])Have a question? We got answers.
The questions people actually ask before wiring up Web Extract.
Get a free key →Does render=force actually run a browser?▾
Not on this engine right now, and the response says so plainly. Every call came back with meta.browserless true and extraction.rendered false, whether render was auto, force or never. On a client-rendered page the extraction block adds render_note "body is thin/JS-hydrated; render unavailable" and render_available_via "web-capture". So a JavaScript-only page gives you whatever the server HTML contained; for a genuine browser render use the web-capture engine that field points at.
When should I reach for a site-specific engine instead of this one?▾
Whenever the target defends itself, and whenever you want fields rather than prose. A scrape of an amazon.com product URL returned error code TARGET_BLOCKED. web-extract is built for ordinary public pages such as news articles, documentation, blogs and marketing sites, and it hands you text. A dedicated engine for the same site hands you a parsed price, rating and stock status, which is almost always what you actually wanted.
map returned count 0 for a site that obviously has pages. What happened?▾
The origin refused both robots.txt and the homepage, and map reports that as an empty surface rather than an error. A live map of stripe.com returned ok:true with count 0, robots_present false and sources {sitemap_urls: 0, homepage_links: 0}. Compare a cooperative site: reefapi.com returned sources {sitemap_urls: 246, homepage_links: 25} and 50 links at limit 50. Read sources before you trust a zero.
How accurate is the page_type label?▾
It is a URL heuristic, useful for bucketing and not for routing. Labels observed across three sites: home, docs, blog, pricing, about, contact, legal and other. On python.org, /jobs/ was labeled "about" and docs.python.org was labeled "home"; on fastapi.tiangolo.com, 47 of 50 documentation pages landed in "other". The page_types tally in the same response gives you the distribution at a glance.
What does schema_result return when my path does not exist on the page?▾
Your keys, with null values, so the shape is always safe to destructure. A schema of {"price": "offers.price", "name": "name"} against a product page with no reachable JSON-LD returned {"price": null, "name": null} alongside jsonld.types []. If every key comes back null, check jsonld.types first: an empty types array means there was no structured data to path into at all.
What is the shape of prices[] in an extract response?▾
Objects of {raw, currency, value} where value is a string and currency is the symbol as printed rather than an ISO code. A product page on books.toscrape.com returned {"raw": "£51.77", "currency": "£", "value": "51.77"}. It is a page-wide sweep, so it also picks up unrelated amounts: that same page returned a second entry of £0.00. Convert value to a number and map the symbol yourself.
Why did crawl stop after one page when I asked for two?▾
Because it ran out of in-scope links, and it says so. A crawl of example.com with max_pages 2 and max_depth 1 returned pages_visited 1 and stop_reason "complete", since the page's only link goes to iana.org and same_domain_only defaults to true. stop_reason "complete" means the crawl finished naturally, while "max_pages" or "max_depth" mean you hit a ceiling and there is more to fetch.
What is actually inside metadata?▾
More than the documented handful. Alongside title, canonical, lang, site_name, author and published_at you get status_code, content_type, robots_meta, feeds[], hreflang[], favicon, og{} and twitter{}. Measured: a Wikipedia page returned status_code 200, published_at "2005-09-17T18:57:30Z", modified_at null and feeds carrying the site's Atom URL, while example.com returned status_code 206, meaning the fetch was served as a partial response. published_at present with modified_at null is common, so do not read that as "never updated".
What is the Web Extract API?▾
Web Extract API is a ReefAPI endpoint group for web extract It returns live JSON through POST requests under /web-extract/v1.
Is the Web Extract API free to try?▾
Yes. ReefAPI starts with 1,000 free credits, no card required. Web Extract calls use the same shared credit balance as every other ReefAPI engine.
Do I need a Web Extract login or account?▾
No login to Web Extract is needed for the API response. You call ReefAPI with your x-api-key header, and the playground can run live examples before you create a production key.
How fresh is the Web Extract data?▾
The page example is captured from a live scrape call, and production requests fetch live data through ReefAPI rather than a static sample.
How many credits does the Web Extract API use?▾
Web Extract actions currently cost 1-2 credits per successful call. Failed or blocked calls are free, and all APIs draw from one credit pool.
Can I call Web Extract from an AI assistant or MCP client?▾
Yes. Connect ReefAPI once through MCP and your assistant can call web-extract actions with the same key, credit pool and JSON envelope used by normal REST requests.
18 Developer Tools APIs on the same key
One key, one credit pool, one response envelope. If you are pulling Web Extract, you are one call away from the rest of the category — no second contract, no second integration.
Need something this API does not do?
Name the endpoint, the field, or a source we do not carry yet. We ship new APIs every week and you would be first to get the key. Real people read every message and reply the same day.
Try it on your own data before you pay anything
The call above is the real endpoint, not a recording. A free key gives you 1,000 credits, the other 183 APIs, and the same envelope everywhere.
Endpoints, parameters and credit costs on this page are read from the live catalog and cannot drift from what the API accepts. Field notes were captured on 2026-08-30.