Reef AI: ask for any data. Try it
Common Crawl API

Common Crawl index lookups and the list of every monthly crawl, as JSON

endpoints
2
credit per call
1

The Common Crawl API returns lookups in Common Crawl's URL index as JSON: for a URL, path, host or domain, each capture in a monthly crawl with its timestamp, HTTP status, declared and detected MIME type, detected languages, charset, content digest, length, and the WARC file name and offset where the raw record is stored; and the list of every Common Crawl crawl with its date range.

/common-crawl/v1

  • POST/common-crawl/v1/cc_search1 credit
  • POST/common-crawl/v1/cc_indexes1 credit
2 active endpoints. Every call is 1 credit.Every parameter ↓OpenAPI 3.0 spec ↗

1 credit per call≈ $1/1,000 calls1,000 free credits, no card

Get a free API key
Try it live

Run the call yourself, on the real endpoint

Change a parameter and press Try it. The response on the right is what your code will receive.

Common Crawl API

2 of 2 endpoints, ready to run

View docs ↗

Captures of a URL in one Common Crawl crawl

1 credit1 required · 4 optional
POST/common-crawl/v1/cc_search
idle
// Press "Try it" and this pane shows exactly what the
// live site returned this second — including an empty
// result, if that is the truth. No key, no account.

Try every endpoint in the playground

Endpoints

What Common Crawl endpoints does ReefAPI ship?

2 live read endpoints. Read-only data API: no writes, no account actions, no dashboard access on the target site.

  • cc_search

    1 cr

    Common Crawl index lookup for a URL.

    required
    url
    optional
    collection, match, limit, from, to, filter
  • cc_indexes

    1 cr

    List the available Common Crawl monthly indexes (id, name, date range).

    required
    —
    optional
    —

Every parameter, every allowed value →

Missing a Common Crawl endpoint, or need a source we don't have yet?Contact us real people · same-day reply.
How it works

How the Common Crawl API works

Common Crawl is a normal ReefAPI surface — the same four rules that hold for every other engine on the key.

Authenticate01
x-api-key header

No OAuth app, no request signing, no per-site account. One key covers all 746 engines.

Call02
POST /common-crawl/v1/…

Every route is a POST with a JSON body. Parameters are validated against the published schema before anything is charged.

Pay03
1 credit per call

Credits, not seats. Failed and blocked calls are never charged, and nothing is cached: every call reads the source live.

Read04
{ ok, data, meta, error }

One envelope everywhere. meta carries latency_ms, record_count and the endpoint that answered.

Recipe

Find where a page sits in Common Crawl's WARC files

Pick a crawl, look the URL up in its index, and use the WARC file name, offset and length to read that exact record from Common Crawl's public files.

01cc_indexes
POST/common-crawl/v1/cc_indexes
{}

On 2026-11-22: 129 crawls in 1.9 s, newest CC-MAIN-2026-43 (October 2026, 2026-10-05 to 2026-10-18), oldest CC-MAIN-2008-2009. Each with id, name, date range and its index address.

02cc_search
POST/common-crawl/v1/cc_search
{"url": "kernel.org", "limit": 3}

3 captures in CC-MAIN-2026-43, on 2026-10-11, 10-13 and 10-14: status 200, text/html, language eng, about 3,600 bytes each, with warc_filename and warc_offset.

03cc_search
POST/common-crawl/v1/cc_search
{"url": "kernel.org", "match": "domain", "limit": 50}

50 captures across the domain and its subdomains. Old calls to POST /web-archive/v1/cc_search and /cc_indexes keep working unchanged.

For each URL: which monthly crawls captured it, what the crawler got (status, type, language, charset), and where the raw record is stored.

request
curl -X POST https://api.reefapi.com/common-crawl/v1/cc_search \
  -H "x-api-key: $REEF_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"github.com","limit":5}'
response envelope
{
  "ok": true,
  "data": { … },
  "meta": {
    "api": "common-crawl",
    "endpoint": "cc_search",
    "mode": "live",
    "latency_ms": …,
    "record_count": …
  },
  "error": null
}
Fields

What a Common Crawl field contains

The fields cc_search and cc_indexes return. Each row was measured on live Common Crawl data on 2026-11-22.

FieldWhat it holds
id (cc_indexes), collectionThe crawl id, CC-MAIN-YYYY-WW: CC-MAIN-2026-43 is the October 2026 crawl, 2026-10-05 to 2026-10-18. Pass it as collection to cc_search; omit it for the newest crawl.
name, from, toThe crawl's name ("October 2026 Index") and the first and last capture time in it.
cdx_api, timegateThe index address of that crawl on index.commoncrawl.org.
url, urlkeyLookups follow Common Crawl's URL keys: a leading www. is dropped, and a trailing slash is dropped except on the root. urlkey is that key; url is the address the crawler fetched.
matchexact (one URL: kernel.org gave 3 captures with limit 3), prefix (everything under a path: kernel.org/doc gave 20), host (one host: bbc.co.uk with filter statuscode:200 gave 1) or domain (the domain and its subdomains: wikipedia.org gave 100 with limit 100).
timestamp, datetimeCapture time as 14 digits (YYYYMMDDhhmmss, UTC) and as ISO 8601.
mimetype, mime_detectedThe MIME type the server declared and the one Common Crawl detected from the content (text/html on the kernel.org captures).
statuscodeThe HTTP status the crawler got (200 on the kernel.org captures). filter statuscode:200 keeps only those.
languages, encodingThe languages Common Crawl detected in the page (ISO 639-3 codes such as eng) and its character set.
warc_filename, warc_offset, lengthWhere the raw record sits in Common Crawl's public WARC files: file path, byte offset and record length (about 3,600 bytes for kernel.org). Together they let you read that one record without downloading the whole file.
digestHash of the captured content; the same digest in two crawls means the same content.

One cc_search call reads one crawl. Common Crawl does not capture every site in every crawl, so 0 records for a host is a normal answer.

Coverage

Common Crawl coverage, measured

Every number below comes from live Common Crawl calls on 2026-11-22.

Crawls

129 crawls listed by cc_indexes in 1.9 s: 10 of them in 2026, one a month from CC-MAIN-2026-04 (January) to CC-MAIN-2026-43 (October), back to the 2008-2009 crawl.

One URL

kernel.org, match exact, limit 3: 3 captures on 2026-10-11, 10-13 and 10-14, each status 200, text/html, language eng, length about 3,600 bytes, with the WARC filename and offset in CC-MAIN-2026-43. github.com with limit 5: 5 captures. python.org/downloads/: 1.

Prefix, host and domain

kernel.org/doc with match prefix: 20 captures. kernel.org with match domain and limit 50: 50. wikipedia.org with match domain and limit 100: 100. bbc.co.uk with match host and filter statuscode:200: 1.

Older crawls

example.com in CC-MAIN-2026-21: 5 captures. Pass any id from cc_indexes as collection.

Fields per capture

timestamp and datetime, original URL, urlkey, mimetype and mime_detected, statuscode, digest, length, languages, encoding, warc_filename and warc_offset, plus the crawl id in meta.collection.

Speed

2 to 8 s per lookup once warm, 6 to 24 s for a cold lookup. A failed lookup costs 0 credits; a host with no captures returns 0 records, not an error.

Limits

Common Crawl captures each site in periodic bulk crawls, not continuously, so a URL can be absent from one crawl and present in the next. One cc_search reads one crawl; call it once per collection to cover several months. For captures over time at any date, use the Wayback Machine API (/wayback-machine-api).

Use cases

What people build with Common Crawl

The jobs this data is most often used for.

01

Find the WARC file, offset and length of a page in Common Crawl to read the raw record.

02

Check which monthly crawls captured a domain before building a training or research dataset from it.

03

See the detected language and charset Common Crawl recorded for each page of a site.

04

List every Common Crawl crawl with its dates to choose the one to process.

2

endpoints

1

credit per call

Pricing

What Common Crawl data costs

The cheapest call here is 1 credit, so $15/mo (Pro) buys 10,000 of them — $1.50 per 1,000 credits. Credits roll over and never expire, and failed or blocked calls are not charged.

$0.67–$1.50 / 1,000 credits
Full pricing →
  • 1,000 free credits on signup, no card
  • One key, all 746 APIs, one credit pool
  • Failed and blocked calls are never charged
  • Credits roll over and never expire
no credit card1,000 free credits · instant API key · pay by card or crypto
Quickstart

Call it in two lines

Sign up, get 1,000 credits and one key that works on every engine. Then this is the whole protocol.

curl
curl -X POST https://api.reefapi.com/common-crawl/v1/cc_search \
  -H "x-api-key: $REEF_KEY" \
  -H "content-type: application/json" \
  -d '{"url":"github.com","limit":5}'
python
import os
import requests

REEF_KEY = os.environ["REEF_KEY"]

r = requests.post(
    "https://api.reefapi.com/common-crawl/v1/cc_search",
    headers={"x-api-key": REEF_KEY},
    json={
        "url": "github.com",
        "limit": 5
    },
    timeout=90,
)
print(r.json()["data"])
FAQ

Have a question? We got answers.

The questions people actually ask before wiring up Common Crawl.

Is there a Common Crawl API?▾

Yes. The common-crawl API on ReefAPI looks URLs up in Common Crawl's index and lists every Common Crawl crawl, as JSON, no Common Crawl account needed. On 2026-11-22 the crawl list answered in 1.9 seconds and a URL lookup in 2 to 8 seconds once warm.

Which Common Crawl crawls are available?▾

All 129 that Common Crawl lists, measured on 2026-11-22: the newest CC-MAIN-2026-43 (October 2026, 2026-10-05 to 2026-10-18), one a month through 2026 from CC-MAIN-2026-04 (January), back to CC-MAIN-2008-2009. cc_indexes returns each with its date range; older crawls answer too (example.com in CC-MAIN-2026-21: 5 captures).

How do I find the WARC record of a page?▾

Call cc_search with the URL and, optionally, a collection. Each capture carries warc_filename, warc_offset and length, the file, byte offset and size of the raw record in Common Crawl's public WARC files: kernel.org's 3 captures in CC-MAIN-2026-43 were about 3,600 bytes each.

Can I look up a whole domain?▾

Yes. Set match to domain (the domain and its subdomains), host (one host) or prefix (everything under a path), with limit up to 1,000 records per call. On 2026-11-22 kernel.org with match domain and limit 50 returned 50 captures, and wikipedia.org with limit 100 returned 100.

Can I keep only pages that loaded?▾

Yes, with filter statuscode:200. bbc.co.uk with match host and that filter returned 1 capture on 2026-11-22.

Do www. and a trailing slash matter?▾

No. Lookups follow Common Crawl's URL keys: a leading www. is dropped, and a trailing slash is dropped except on the root, so python.org/downloads/ is looked up as python.org/downloads (1 capture on 2026-11-22).

How fast is it?▾

2 to 8 seconds per lookup once warm, 6 to 24 seconds for a cold lookup, 1.9 seconds for the crawl list. A failed lookup costs 0 credits, and a host with no captures returns 0 records rather than an error.

How is this different from the Wayback Machine API?▾

Common Crawl captures the web in periodic bulk crawls and records detected language, charset and WARC location per capture. The Wayback Machine API (/wayback-machine-api) returns the Internet Archive's captures of a URL at any date, with snapshot links to view each archived copy.

Do old web-archive calls still work?▾

Yes. POST /web-archive/v1/cc_search and /cc_indexes keep working unchanged. The call for Common Crawl alone is POST /common-crawl/v1/cc_search and POST /common-crawl/v1/cc_indexes, with the same parameters.

What is the Common Crawl API?▾

Common Crawl API is a ReefAPI endpoint group for look a url or domain up in a common crawl index: status, mime, language, charset and warc location, plus every monthly crawl. It returns live JSON through POST requests under /common-crawl/v1.

Is the Common Crawl API free to try?▾

Yes. ReefAPI starts with 1,000 free credits, no card required. Common Crawl calls use the same shared credit balance as every other ReefAPI engine.

Do I need a Common Crawl login or account?▾

No login to Common Crawl is needed for the API response. You call ReefAPI with your x-api-key header, and the playground can run live examples before you create a production key.

How fresh is the Common Crawl data?▾

The page example is captured from a live cc_search call, and production requests fetch live data through ReefAPI rather than a static sample.

How many credits does the Common Crawl API use?▾

Common Crawl actions currently cost 1 credit per successful call. Failed or blocked calls are free. All APIs draw from one credit pool.

Same key

15 Developer Tools APIs on the same key

One key, one credit pool, one response envelope. If you are pulling Common Crawl, you are one call away from the rest of the category — no second contract, no second integration.

Contact

Need something this API does not do?

Name the endpoint, the field, or a source we do not carry yet. We ship new APIs every week and you would be first to get the key. Real people read every message and reply the same day.

0/4000

No account needed · we reply from [email protected]

Try it on your own data before you pay anything

The call above is the real endpoint, not a recording. A free key gives you 1,000 credits, the other 745 APIs, and the same envelope everywhere.

Endpoints, parameters and credit costs on this page are read from the live catalog and cannot drift from what the API accepts. Field notes were captured on 2026-11-22.