Common Crawl
Look a URL or domain up in a Common Crawl index: status, MIME, language, charset and WARC location, plus every monthly crawl.
/common-crawl/v1/cc_search1 creditCommon Crawl index lookup for a URL — alternate/broader coverage with richer fields (detected language, encoding, WARC offset/filename). Cross-source corroboration.
| Parameter | Allowed / range | Description | |
|---|---|---|---|
| url | required | — | The URL, host, or path to look up in the archive (e.g. 'github.com', 'github.com/torvalds', 'https://example.com/page'). Scheme optional. Use with match=prefix/domain/host to widen. |
| collection | optional | — | Common Crawl index id (e.g. 'CC-MAIN-2026-21'). Omit to use the newest index. List ids via action='cc_indexes'. |
| match = exact | optional | exact · prefix · host · domain | URL match mode for the Common Crawl index query. |
| limit = 100 | optional | 1–1000 | Max captures to return (1-1000, default 100). Page further with meta.resume_key on the snapshots action. |
| from | optional | — | Earliest capture to include. ISO date (YYYY-MM-DD) or a Wayback timestamp (YYYYMMDDhhmmss, any 4-14 digit prefix). Inclusive. |
| to | optional | — | Latest capture to include. ISO date or Wayback timestamp prefix. Inclusive. |
| filter | optional | — | Advanced raw CDX filter expression(s), comma-separated. Format [!]field:regex over urlkey/timestamp/original/mimetype/statuscode/digest/length (e.g. 'original:.*\.pdf$'). Power-user escape hatch. |
/common-crawl/v1/cc_indexes1 creditList the available Common Crawl monthly indexes (id, name, date range) — pick a collection for cc_search.
Try in playground →curl -X POST https://api.reefapi.com/common-crawl/v1/cc_search \
-H "x-api-key: $REEF_KEY" \
-H "content-type: application/json" \
-d '{"url":"github.com","limit":5}'{
"ok": true,
"data": { /* the result */ },
"meta": {
"latency_ms": 240,
"record_count": 12,
"completeness_pct": 100
},
"error": null
}Measured at 60 requests a second across the fleet, with no central bottleneck. Volume pricing is on request, and per-key limits are raised for high-volume accounts.
Tell us a site we do not cover yet and it becomes an engine. A customer asked for bestprice.gr on a Sunday and it was in the catalog the next day.
Median time from a question in the live chat to the first answer, measured across every answered conversation. Setup help included, no support tier to buy.
No per-site plans and no separate subscriptions. One key and one credit pool across the whole catalog, so adding a source costs nothing up front.
Planning something large? Tell us the volume and the sources and we will come back with what it costs and what we would have to build.