URL Intelligence API & Scraper
The URL Intelligence API cleans and classifies URLs as clean JSON.
🤖 Using an AI assistant? Copy this link into ChatGPT / Claude / Cursor — it reads every endpoint and parameter instantly and tells you if this API fits your use case.
The primary classify endpoint returns the final URL, cleaned and canonical URLs, domain, registrable domain and a short-link flag, and you can extract an entity, batch and resolve a catalog URL. It is built for link analysis, content pipelines and moderation that need normalized, classified URLs. One ReefAPI key, one shared credit pool, the standard envelope.
Five URL fields that look alike and are not
The response carries four different versions of the link plus a host field, and picking the wrong one is the usual cause of duplicate rows in a link table. The measured column is one real classify call on https://youtu.be/dQw4w9WgXcQ?si=abcdef&utm_source=x, a short link that lands on a different host. Note in particular that domain describes the link you sent, not the page you ended up on.
| Field | What it holds | Measured on the youtu.be link |
|---|---|---|
| final_url | Where the link lands, query string untouched | https://www.youtube.com/watch?si=abcdef&utm_source=x&v=dQw4w9WgXcQ&feature=youtu.be |
| cleaned_url | final_url with known tracking params removed | same URL with utm_source dropped, si and feature kept |
| canonical_url | The page's own canonical, the safest dedupe key | https://www.youtube.com/watch?v=dQw4w9WgXcQ |
| domain | Host of the URL you sent, not of final_url | youtu.be |
| registrable_domain | That host reduced to its registrable form | youtu.be |
| identifier | The platform id pulled out of the path, null when no platform matched | dQw4w9WgXcQ |
| recommended | engine, action and ready-made params for the deep fetch | {engine: youtube, action: video, params: {video_id: dQw4w9WgXcQ}} |
redirect.redirected read false on every measured call, including this one where chain_hosts went from youtu.be to www.youtube.com. Use redirect.hops and redirect.chain_hosts to decide whether the link moved, and compare the host inside final_url against domain rather than trusting the flag.
Real request and response JSON
Captured from the indexed primary action, classify, on .
{
"method": "POST",
"url": "https://api.reefapi.com/enrich-url/v1/classify",
"headers": {
"x-api-key": "$REEF_KEY",
"content-type": "application/json"
},
"body": {
"url": "https://www.amazon.com/dp/B08N5WRWNW?tag=aff-20&utm_source=x"
}
}{
"ok": true,
"meta": {
"api": "enrich-url",
"endpoint": "classify",
"mode": "live",
"latency_ms": 1146.7,
"record_count": 1,
"bytes": 0,
"cache_hit": false,
"subcalls": [
{
"api": "url-resolver",
"action": "full",
"status": "ok",
"latency_ms": 779.9
}
],
"entity_type": "product",
"platform": "amazon"
},
"data": {
"input_url": "https://www.amazon.com/dp/B08N5WRWNW?tag=aff-20&utm_source=x",
"final_url": "https://www.amazon.com/dp/B08N5WRWNW?tag=aff-20&utm_source=x",
"cleaned_url": "https://www.amazon.com/dp/B08N5WRWNW?tag=aff-20",
"canonical_url": "https://www.amazon.com/dp/B08N5WRWNW?tag=aff-20",
"domain": "www.amazon.com",
"registrable_domain": "amazon.com",
"is_short_link": false,
"redirect": {
"redirected": false,
"wall_redirect": null,
"hops": 0,
"chain_hosts": [
"www.amazon.com"
]
},
"entity_type": {
"value": "product",
"confidence": 0.9,
"source": "platform-map"
},
"platform": "amazon",
"identifier": null,
"recommended": {
"engine": "amazon",
"action": "product",
"params": {
"product_id": "B08N5WRWNW"
}
},
"tracking": {
"is_affiliate": true,
"affiliate_network": "amazon_associates",
"affiliate_type": "network",
"merchant_hint": "amazon",
"stripped_params": [
"utm_source"
]
},
"preview": {
"title": "Amazon.com",
"description": null,
"image": null,
"site_name": null,
"og_type": null
},
"provenance": {
"resolver": {
"status": "ok",
"engine": "url-resolver"
}
}
}
}What the URL Intelligence API does
| Action | Description | Concrete use case | Key params |
|---|---|---|---|
| classify | Any URL → final_url, cleaned_url (tracking stripped), domain, entity_type{value,confidence,source}, platform, recommended{engine,action,params} (the Reef deep-fetch card; web-extract fallback for unknown hosts), tracking{affiliate,stripped_params}. | Ops teams call classify to get any URL → final_url, cleaned_url (tracking stripped), domain, entity_type{value,confidence,so…. | url, deep |
| extract_entity | Any URL → a lightweight entity card with title, image, description, price, author and publication date — great for link previews and deciding which Reef engine to call next. Use the recommended{} field from classify for the full deep-fetch. | Developer tools call extract_entity to get any URL → a lightweight entity card with title, image, description, price, author and publica…. | url |
| batch | Classify up to 20 URLs in one call (the link-list cleanup + routing use-case): per-URL ok/error, each independently resolved + classified. | Validation workflows call batch to get classify up to 20 URLs in one call (the link-list cleanup + routing use-case). | urls |
| catalog_resolve | META (no network): given a known platform name OR a URL's host, return which Reef engine+action handles it and the param key it needs — the routing table itself (url-resolver marketplace engine_hints + the non-marketplace platform map). For agents discovering 'which engine for X'. | Data-quality teams call catalog_resolve to get mETA (no network). | platform, url |
Call classify from your stack
curl -X POST https://api.reefapi.com/enrich-url/v1/classify \
-H "x-api-key: $REEF_KEY" \
-H "content-type: application/json" \
-d '{"url":"https://www.amazon.com/dp/B08N5WRWNW?tag=aff-20&utm_source=x"}'import requests
r = requests.post(
"https://api.reefapi.com/enrich-url/v1/classify",
headers={"x-api-key": REEF_KEY},
json={
"url": "https://www.amazon.com/dp/B08N5WRWNW?tag=aff-20&utm_source=x"
},
)
print(r.json()["data"])const res = await fetch("https://api.reefapi.com/enrich-url/v1/classify", {
method: "POST",
headers: {
"x-api-key": process.env.REEF_KEY,
"content-type": "application/json",
},
body: JSON.stringify({
"url": "https://www.amazon.com/dp/B08N5WRWNW?tag=aff-20&utm_source=x"
}),
});
const { ok, data, meta, error } = await res.json();Ask your MCP-connected assistant: call reefapi.enrich-url.classify with {"url":"https://www.amazon.com/dp/B08N5WRWNW?tag=aff-20&utm_source=x"}.Who uses this API and why
- Content pipelines call classify to clean and canonicalize URLs before storage.
- Moderation uses classify to flag short links and risky domains.
- Commerce uses catalog_resolve to map a product URL to a known item.
Questions developers ask before integrating
Which links get a real engine and which fall back to web-extract?
The catalog_resolve action returns the routing table itself with no network call. It listed 39 platforms, of which 9 currently carry a null engine: x, facebook, reddit, pinterest, vimeo, gitlab, imdb, walmart and target. Everything else maps to a named engine and action, for example instagram to instagram/profile and amazon to amazon/product. When no platform matches, recommended still comes back, pointing at web-extract with a note that says there is no dedicated engine for that host.
I passed a Zillow property link and got web-extract back. Why?
Real estate hosts are not in the platform table, so the classifier fell through to structured data. The measured call returned entity_type 'product' with source 'jsonld', platform null, and a web-extract card. It did resolve the URL properly on the way, rewriting a placeholder path into the full listing slug. For property links use enrich-property, whose snapshot action takes the same URL and returns status, price, beds, area and photos.
Does a company domain get routed to enrich-company?
No. A measured classify on stripe.com returned entity_type 'company' with confidence 0.6 and source 'og-type', platform null, and a recommended card for web-extract. This engine classifies and routes links, it does not enrich businesses. Take the registrable_domain from here and hand it to enrich-company yourself if you want firmographics, or to enrich-app if the domain is a SaaS product.
Why is the affiliate tag still in cleaned_url?
Because stripping it would break the link's purpose. On a measured Amazon link carrying tag=aff-20 and utm_source=x, stripped_params listed only utm_source, and both cleaned_url and canonical_url kept tag=aff-20. What you get instead is disclosure: tracking.is_affiliate came back true with affiliate_network 'amazon_associates' and merchant_hint 'amazon'. Drop the tag yourself if you need a neutral URL.
The preview title says 'Page Not Found' but the classification looks right. Which do I believe?
The classification. preview is a best-effort read of the page's own metadata and some hosts answer an automated fetch with their error page, which is what produced 'Page Not Found' on an Amazon product link and 'Bitly | Page Not Found | 404' on a dead short link. Those same responses still classified correctly from the platform map, at confidence 0.9 and with a usable recommended card. When the extractor cannot help, provenance.classifier records the status and a missing_reason.
What does a low entity_type confidence actually mean?
It tells you where the answer came from. Values of 0.9 to 0.95 with source 'platform-map' mean the host was recognized outright, which is what amazon, github, instagram and youtube links returned. A value of 0.6 with source 'og-type' means it was read off the page's own og metadata, and 0.35 with source 'unknown' means nothing identified it, which is what an unresolvable short link and a Spotify track both scored.
How does batch handle a bad URL in the list?
It isolates it. A measured batch of four inputs, one of them the string 'not a url', returned ok:true overall with count 4, ok_count 3 and meta.record_count 3. The bad entry appears in results as {ok:false, input_url, error:{code:'INVALID_PARAM'}} while the other three carry full classify payloads. The cap is 20 per call, and 21 urls returns INVALID_PARAM with 'max 20 urls per batch (got 21)'.
Are internal or private addresses safe to send?
They are rejected rather than fetched. A measured classify on http://169.254.169.254/latest/meta-data/, the usual cloud metadata address, returned ok:false with INVALID_PARAM and the message 'SSRF-blocked: private/internal IP'. Non-http schemes such as file:// are refused the same way, so user-submitted links can be passed through without pre-filtering.
What is the URL Intelligence API?
URL Intelligence API is a ReefAPI endpoint group for url intelligence It returns live JSON through POST requests under /enrich-url/v1.
Is the URL Intelligence API free to try?
Yes. ReefAPI starts with 1,000 free credits, no card required. URL Intelligence calls use the same shared credit balance as every other ReefAPI engine.
Do I need an URL Intelligence login or account?
No login to URL Intelligence is needed for the API response. You call ReefAPI with your x-api-key header, and the playground can run live examples before you create a production key.
How fresh is the URL Intelligence data?
The page example is captured from a live classify call, and production requests fetch live data through ReefAPI rather than a static sample.
How many credits does the URL Intelligence API use?
URL Intelligence actions currently cost 1-2 credits per successful call. Failed or blocked calls are free, and all APIs draw from one credit pool.
Can I call URL Intelligence from an AI assistant or MCP client?
Yes. Connect ReefAPI once through MCP and your assistant can call enrich-url actions with the same key, credit pool and JSON envelope used by normal REST requests.