Web Archive API & Scraper
The Web Archive API returns historical website snapshots from the Wayback Machine and Common Crawl as clean JSON.
🤖 Using an AI assistant? Copy this link into ChatGPT / Claude / Cursor — it reads every endpoint and parameter instantly and tells you if this API fits your use case.
The primary snapshots endpoint returns a URL's captures with timestamp, original URL, MIME type, status code, digest and length, and you can find the nearest available capture, pull capture history, list a domain's captures, batch and search Common Crawl indexes. It is built for OSINT, brand-monitoring and research workflows that need a page's history without parsing archive.org's raw CDX API. One ReefAPI key, one shared credit pool, the standard envelope.
Wayback CDX field formats, and how a snapshot URL is built from them
Every capture row is a CDX record, and the two things people get wrong are the timestamp format and the fact that CDX types are strings where you would expect numbers. These are the formats as measured against github.com. The engine returns the raw CDX field plus a derived datetime and a ready-made snapshot_url, so there is nothing to assemble by hand.
| Field | Format as returned | Notes |
|---|---|---|
| timestamp | 14 digits, YYYYMMDDhhmmss, UTC - "20080514210148" | from / to accept an ISO date or any 4-14 digit prefix of this, inclusive |
| datetime | ISO-8601 with Z - "2008-05-14T21:01:48Z" | the same instant as timestamp, derived for convenience |
| snapshot_url | https://web.archive.org/web/<timestamp>/<original> | already assembled per row |
| statuscode | a string, "200" or "404" - never an integer | the status the crawler got at capture time, not the URL's status today |
| digest | base32 SHA-1 of the response body | identical digest means a byte-identical capture; this is what collapse=digest keys on |
| urlkey | SURT form - "com,github)/" | host reversed and comma-separated; this is what collapse=urlkey groups by |
| length | integer, size of the archived record in bytes | the stored record size, not the rendered page weight |
| Common Crawl collection | CC-MAIN-YYYY-WW - "CC-MAIN-2026-34" | 127 indexes were listed on 2026-08-26; 2026-34 covers 2026-08-07 to 2026-08-20 |
collapse is the one parameter that rejects a bad value: collapse "bogus" returns INVALID_PARAM with the nine allowed values in error.detail.allowed. match, status and mime silently ignore anything they do not recognize, so a typo there quietly widens your result set instead of failing.
Real request and response JSON
Captured from the indexed primary action, snapshots, on .
{
"method": "POST",
"url": "https://api.reefapi.com/web-archive/v1/snapshots",
"headers": {
"x-api-key": "$REEF_KEY",
"content-type": "application/json"
},
"body": {
"url": "github.com",
"limit": 10,
"collapse": "digest"
}
}{
"ok": true,
"meta": {
"api": "web-archive",
"endpoint": "snapshots",
"mode": "live",
"latency_ms": 5933.3,
"record_count": 10,
"bytes": 1307,
"cache_hit": false,
"source": "wayback",
"url": "https://web.archive.org/cdx/search/cdx?url=github.com&output=json&fl=urlkey,timestamp,original,mimetype,statuscode,digest,length&limit=10&collapse=digest&showResumeKey=true",
"has_more": true,
"resume_key": "eJxLzs_VSc8syShN0tRXMDIwsDCwNDAzMDIxMDEHAHRmBzE",
"attempts": 1
},
"data": {
"snapshots": [
{
"timestamp": "[redacted-phone]",
"datetime": "[redacted-phone]T21:01:48Z",
"original": "http://github.com/",
"urlkey": "com,github)/",
"mimetype": "text/html",
"statuscode": "200",
"digest": "L4YKKNIYL4FNB5SJWPQ4YII5FJTLWHRF",
"length": 3531,
"snapshot_url": "https://web.archive.org/web/[redacted-phone]/http://github.com/"
},
{
"timestamp": "[redacted-phone]",
"datetime": "[redacted-phone]T20:13:17Z",
"original": "http://github.com:80/?",
"urlkey": "com,github)/",
"mimetype": "text/html",
"statuscode": "200",
"digest": "7HRNG6K4OGFHZZU4KVQO4ZMO7M7LYMLQ",
"length": 3474,
"snapshot_url": "https://web.archive.org/web/[redacted-phone]/http://github.com:80/?"
},
{
"timestamp": "[redacted-phone]",
"datetime": "[redacted-phone]T21:14:38Z",
"original": "http://github.com:80/?",
"urlkey": "com,github)/",
"mimetype": "text/html",
"statuscode": "200",
"digest": "5QK7JOGEJQKSQGH2IVAX7AIT7GTOUU5A",
"length": 4199,
"snapshot_url": "https://web.archive.org/web/[redacted-phone]/http://github.com:80/?"
}
],
"count": 10,
"resume_key": "eJxLzs_VSc8syShN0tRXMDIwsDCwNDAzMDIxMDEHAHRmBzE",
"has_more": true
}
}What the Web Archive API does
| Action | Description | Concrete use case | Key params |
|---|---|---|---|
| snapshots | All Wayback captures for a URL (timestamp, status, mimetype, digest, archival size, snapshot_url), date/status/mime filtered, collapsible, RESUME-KEY PAGINATED. | Platform and DevOps teams call snapshots to get all Wayback captures for a URL (timestamp, status, mimetype, digest, archival size, snapshot_…. | url, from, to, limit, collapse, ... |
| available | Closest single Wayback snapshot to a given date (or the latest). Fast existence check. | Security and supply-chain teams call available to get closest single Wayback snapshot to a given date (or the latest). | url, timestamp |
| history | Lifespan + capture cadence for a URL: EXACT first_seen/last_seen + span, plus per-year capture counts and status/mime breakdown — the SEO/due-diligence summary. | Developer-tool builders call history to get lifespan + capture cadence for a URL. | url, from, to, match |
| domain_captures | All archived URLs under a domain (the domain + its subdomains), one row per unique URL — the 'every page this site ever had' view. Resume-key paginated. | AI-agent developers call domain_captures to get all archived URLs under a domain (the domain + its subdomains), one row per unique URL. | url, from, to, limit, status, ... |
| batch | Closest-snapshot existence + capture-count for up to 20 URLs in one call (bulk archival presence — competitor/portfolio sweeps). | Platform and DevOps teams call batch to get closest-snapshot existence + capture-count for up to 20 URLs in one call (bulk archival presence. | urls, timestamp |
| cc_search | Common Crawl index lookup for a URL — alternate/broader coverage with richer fields (detected language, encoding, WARC offset/filename). Cross-source corroboration. | Security and supply-chain teams call cc_search to get common Crawl index lookup for a URL. | url, collection, match, limit, from, ... |
| cc_indexes | List the available Common Crawl monthly indexes (id, name, date range) — pick a collection for cc_search. | Developer-tool builders call cc_indexes to list the available Common Crawl monthly indexes (id, name, date range). | none |
Call snapshots from your stack
curl -X POST https://api.reefapi.com/web-archive/v1/snapshots \
-H "x-api-key: $REEF_KEY" \
-H "content-type: application/json" \
-d '{"url":"github.com","limit":10,"collapse":"digest"}'import requests
r = requests.post(
"https://api.reefapi.com/web-archive/v1/snapshots",
headers={"x-api-key": REEF_KEY},
json={
"url": "github.com",
"limit": 10,
"collapse": "digest"
},
)
print(r.json()["data"])const res = await fetch("https://api.reefapi.com/web-archive/v1/snapshots", {
method: "POST",
headers: {
"x-api-key": process.env.REEF_KEY,
"content-type": "application/json",
},
body: JSON.stringify({
"url": "github.com",
"limit": 10,
"collapse": "digest"
}),
});
const { ok, data, meta, error } = await res.json();Ask your MCP-connected assistant: call reefapi.web-archive.snapshots with {"url":"github.com","limit":10,"collapse":"digest"}.Who uses this API and why
- OSINT researchers call snapshots to see how a page or claim looked at a point in time.
- Brand-monitoring tools use history to track how a competitor's pricing or messaging changed.
- Compliance teams use available to retrieve the archived version of a page for evidence.
Questions developers ask before integrating
What happens when a URL was never archived?
You get a successful response saying so, not an error. A measured available call on a domain with no captures returned ok:true, meta.record_count 0, available:false and closest:null. batch behaves the same way per row: a three-URL sweep returned archived:true with a closest object for two of them and archived:false with closest:null for the third, plus meta.archived_count 2 against meta.requested 3. Check the boolean, never the HTTP status.
Does "closest" mean the snapshot before my date?
No. Closest means nearest in either direction, and it often lands after the date you asked for. A measured available call on github.com with timestamp 20120601 returned the capture at 20120612150442, eleven days later. requested_timestamp is echoed back so you can measure the gap yourself. If you need a capture that is definitely no later than a date, use snapshots with to set to that date and take the last row.
Why are years, status_codes and mime_types empty on the history action?
Because those three are built from a sampled scan that the engine drops rather than let a call hang. When that happens the response says so: meta.breakdown_partial true, data.partial true, and a note reading "first/last_seen are exact (pinpoint CDX); years/status/mime are a capture-per-day sample (cap 1000) - BREAKDOWN OMITTED". The edges stay exact either way - a measured run on github.com still returned first_seen 20080514210148, last_seen 20260826102004 and span_days 6677 while total_captures read 0. Rely on first_seen, last_seen and span_days; treat the breakdown as best-effort.
How do I page past the first batch of captures?
limit maxes at 1,000 per call and paging is by opaque resume key, not by offset. snapshots and domain_captures return meta.has_more and meta.resume_key; feed that string back as resume_key for the next page. It is a Wayback token rather than a page number, so you cannot jump to page five or compute a total up front. Narrow with from / to, status or mime when you need a bounded set rather than a full walk.
Why is domain_captures full of 404s?
Because it lists every URL the archive ever recorded under the domain, including ones that were already dead when they were crawled. A measured five-row page for github.com contained the 2008 homepage capture at status 200 and https://github.com/!!! at 404 in the same list. Pass status 200 to keep only successful captures, or status "!200" - the leading exclamation mark negates - to inspect only the failures. Rows are already one per unique URL, so the count is pages, not captures.
How is this different from the internet-archive engine?
Different halves of archive.org. This engine reads the Wayback capture index (CDX) plus Common Crawl, and answers "what did this URL look like on this date, and when did it change". The internet-archive engine reads the archive.org item library - books, audio, film, software - and answers "find me this item and its files". They share a hostname and nothing else: there are no identifiers, mediatypes or collections here, and no timestamps or digests there.
When should I use cc_search instead of snapshots?
For corroboration, and for fields Wayback does not carry. Common Crawl rows add languages ("eng"), encoding ("UTF-8"), warc_filename and warc_offset, which let you pull the exact record out of Common Crawl's own WARC files. Coverage differs in kind: Common Crawl runs as periodic bulk crawls, so a measured lookup on github.com returned captures on 2026-08-07 and 2026-08-09 with different digests inside collection CC-MAIN-2026-34. Call cc_indexes first to choose a collection, or omit it to use the newest.
How do I find only the dates a page actually changed?
collapse=digest. It drops consecutive captures whose content hash is identical, so what remains is one row per version rather than one row per crawl. The other collapse values group instead of dedupe: timestamp:4 gives one capture per year, :6 per month, :8 per day and :10 per hour, while urlkey or original give one row per URL, which is what you want with match set to prefix or domain.
What is the Web Archive API?
Web Archive API is a ReefAPI endpoint group for web archive It returns live JSON through POST requests under /web-archive/v1.
Is the Web Archive API free to try?
Yes. ReefAPI starts with 1,000 free credits, no card required. Web Archive calls use the same shared credit balance as every other ReefAPI engine.
Do I need a Web Archive login or account?
No login to Web Archive is needed for the API response. You call ReefAPI with your x-api-key header, and the playground can run live examples before you create a production key.
How fresh is the Web Archive data?
The page example is captured from a live snapshots call, and production requests fetch live data through ReefAPI rather than a static sample.
How many credits does the Web Archive API use?
Web Archive actions currently cost 1 credit per successful call. Failed or blocked calls are free, and all APIs draw from one credit pool.
Can I call Web Archive from an AI assistant or MCP client?
Yes. Connect ReefAPI once through MCP and your assistant can call web-archive actions with the same key, credit pool and JSON envelope used by normal REST requests.