# Web Extract API — any URL to markdown, text, HTML, metadata, links, structured data (JSON-LD), site map, crawl, and deterministic extract. Security-guarded, no-key.

> Fetch one URL and return its content in the formats you choose (markdown, plain text, cleaned HTML, raw HTML, metadata, links, images, or structured JSON-LD data). Automatically upgrades thin single-page-app pages to a full browser render when needed. Works on any public page — news articles, product pages, documentation, blogs.
> ReefAPI engine `web-extract` · 5 endpoints · clean JSON, no scraping or browsers to manage.

## How to call
- **Endpoint:** `POST https://api.reefapi.com/web-extract/v1/<action>` with a JSON body.
- **Auth:** header `x-api-key: <YOUR_REEFAPI_KEY>` — create one free (1,000 credits, no card): https://reefapi.com/signup
- **Response (every call):** `{ ok: boolean, data: ..., meta: { record_count, credits, ... }, error: { code, message } }` — branch on `ok`. Failed or blocked calls are free.
- **One key + one shared credit pool** across every ReefAPI API. Per-call credits are listed on each endpoint below.
- **Use it from an AI agent (MCP):** connect `https://api.reefapi.com/mcp` (remote streamable-http, `Authorization: Bearer <key>`) and your assistant can call these actions directly.

## Endpoints

### POST /web-extract/v1/scrape — 1 credit
Fetch one URL and return its content in the formats you choose (markdown, plain text, cleaned HTML, raw HTML, metadata, links, images, or structured JSON-LD data). Automatically upgrades thin single-page-app pages to a full browser render when needed. Works on any public page — news articles, product pages, documentation, blogs.

**Parameters:**
- `url` (string, required) — The page URL to extract. Full URL or bare domain (https:// assumed). Only http/https; private/internal/metadata targets are SSRF-blocked.
- `formats` (array, optional, default ["markdown","metadata"]) — Which outputs to return (array or comma-string). Any of: markdown, text, html (cleaned main-content), rawHtml, metadata, links, images, jsonld. Default: markdown+metadata. Unknown values are ignored. For screenshots use the web-capture engine. [one of: markdown, text, html, rawHtml, metadata, links, images, jsonld]
- `render` (enum, optional, default "auto") — When to activate browser rendering for JavaScript-heavy pages. 'auto' (default) detects thin/client-rendered pages and re-fetches them with a real browser automatically — rich server-rendered pages are served instantly without a browser. 'force' always uses a browser; 'never' skips rendering for maximum speed. [one of: auto, never, force]
- `only_main_content` (boolean, optional, default true) — Remove navigation bars, headers, footers, and boilerplate — keep only the main article or content body (default true). Turn off to get the full raw page content.
- `include_tags` (array, optional) — CSS selectors to include — only elements matching these selectors are kept in the extracted content, e.g. ['article', 'main', '.post-body'].
- `exclude_tags` (array, optional) — CSS selectors to remove before extraction — useful for stripping ads, cookie banners, or sidebar widgets, e.g. ['nav', 'footer', '.ads'].
- `target_selector` (string, optional) — Return only the content inside this CSS selector — e.g. 'article.post' to extract a specific section of the page and ignore everything else.
- `timeout` (integer, optional, default 25) — Per-request timeout in seconds (3-60, default 25).

**Returns:** final_url, title, metadata{canonical,lang,site_name,author,published_at,og,...}, extraction{method,rendered,confidence,content_chars,render_source}, + each requested format (markdown/text/html/rawHtml/links[]/images[]/jsonld{}). render=auto transparently upgrades thin/SPA pages to a browser render.

**Example request body:**
```json
{
  "url": "https://en.wikipedia.org/wiki/Web_scraping",
  "formats": [
    "markdown",
    "metadata"
  ]
}
```

### POST /web-extract/v1/map — 2 credits
Discover a site's complete URL surface: reads robots.txt, sitemap.xml (and sitemap indexes), and the homepage's internal links — returns a deduplicated URL list with a page-type label for each (home, pricing, docs, blog, product, legal, contact, about). Use this to index any site or audit its structure.

**Parameters:**
- `url` (string, required) — The page URL to extract. Full URL or bare domain (https:// assumed). Only http/https; private/internal/metadata targets are SSRF-blocked.
- `search` (string, optional) — Filter discovered links by a URL substring — matched URLs are returned first, e.g. '/blog' to prioritize blog pages or '/docs' for documentation.
- `limit` (integer, optional, default 200) — Max links to return (1-1000, default 200).
- `include_subdomains` (boolean, optional, default false) — Include subdomain URLs found in sitemaps (default false).

**Returns:** origin, links[]{url,page_type}, count, page_types{type:count}, sitemaps[], robots_present, sources{sitemap,homepage}

**Example request body:**
```json
{
  "url": "https://stripe.com",
  "limit": 50
}
```

### POST /web-extract/v1/crawl — 1 credit
Crawl a site starting from a seed URL (up to 25 pages): follows internal links breadth-first with configurable depth, include/exclude URL patterns, and returns every visited page in the formats you choose. Ideal for content indexing, site audits, and building knowledge bases from documentation or blog sites.

**Parameters:**
- `url` (string, required) — The page URL to extract. Full URL or bare domain (https:// assumed). Only http/https; private/internal/metadata targets are SSRF-blocked.
- `max_pages` (integer, optional, default 10) — Maximum number of pages to crawl in one call (1–25, default 10). Increase for deeper site coverage.
- `max_depth` (integer, optional, default 2) — Max link depth from the seed URL (0-5, default 2).
- `same_domain_only` (boolean, optional, default true) — Only follow links on the seed's registrable domain (default true).
- `include_patterns` (array, optional) — Regex patterns; only URLs whose path matches are crawled.
- `exclude_patterns` (array, optional) — Regex patterns; URLs whose path matches are skipped.
- `formats` (array, optional, default ["markdown","metadata"]) — Which outputs to return (array or comma-string). Any of: markdown, text, html (cleaned main-content), rawHtml, metadata, links, images, jsonld. Default: markdown+metadata. Unknown values are ignored. For screenshots use the web-capture engine. [one of: markdown, text, html, rawHtml, metadata, links, images, jsonld]
- `timeout` (integer, optional, default 25) — Per-request timeout in seconds (3-60, default 25).

**Returns:** pages[]{url,page_type,...requested formats}, count, stop_reason (complete|max_pages|max_depth), pages_visited

**Example request body:**
```json
{
  "url": "https://example.com",
  "max_pages": 2,
  "max_depth": 1
}
```

### POST /web-extract/v1/extract — 2 credits
Pull structured data from any public page: returns JSON-LD objects (Product, Article, Organization, etc.), microdata presence, parsed table rows, heading outline, prices, emails, and phone numbers. Supply an optional field-to-path schema to extract specific values directly — great for e-commerce pricing, article metadata, and business listings.

**Parameters:**
- `url` (string, required) — The page URL to extract. Full URL or bare domain (https:// assumed). Only http/https; private/internal/metadata targets are SSRF-blocked.
- `schema` (object, optional) — Optional field extraction map — a JSON object where each key is your output field name and the value is a dot-path into the page's structured data, e.g. {'price': 'offers.price', 'name': 'name'}.
- `deterministic_only` (boolean, optional, default true) — Use only rule-based extraction (structured data, microdata, tables, headings, prices) — always-on and fully predictable. Default true; set false to opt into AI-assisted free-form extraction when available.
- `timeout` (integer, optional, default 25) — Per-request timeout in seconds (3-60, default 25).

**Returns:** final_url, jsonld{objects,types}, has_microdata, has_rdfa, tables[], headings[], prices[], emails[], phones[], schema_result{} (if schema given)

**Example request body:**
```json
{
  "url": "https://en.wikipedia.org/wiki/Python_(programming_language)"
}
```

### POST /web-extract/v1/batch — 1 credit
Scrape up to 10 URLs concurrently in one call (each SSRF-guarded). Shared formats. Firecrawl /batch-scrape parity (bounded).

**Parameters:**
- `urls` (array, required) — List of page URLs to scrape in one call (max 10); each is independently SSRF-guarded and fetched concurrently.
- `formats` (array, optional, default ["markdown","metadata"]) — Which outputs to return (array or comma-string). Any of: markdown, text, html (cleaned main-content), rawHtml, metadata, links, images, jsonld. Default: markdown+metadata. Unknown values are ignored. For screenshots use the web-capture engine. [one of: markdown, text, html, rawHtml, metadata, links, images, jsonld]
- `only_main_content` (boolean, optional, default true) — Remove navigation bars, headers, footers, and boilerplate — keep only the main article or content body (default true). Turn off to get the full raw page content.
- `timeout` (integer, optional, default 25) — Per-request timeout in seconds (3-60, default 25).

**Returns:** results[]{ok,url,final_url,...formats | error}, count, ok_count

**Example request body:**
```json
{
  "urls": [
    "https://example.com",
    "https://en.wikipedia.org/wiki/HTML"
  ]
}
```

## More
- Try it live, no code: https://reefapi.com/playground?engine=web-extract
- Human docs page: https://reefapi.com/docs/web-extract
- Every ReefAPI API in one file (for your AI): https://reefapi.com/llms-full.txt