AI Search Visibility API & Scraper
The GEO Intelligence API audits a page's AI-search readiness as clean JSON.
🤖 Using an AI assistant? Copy this link into ChatGPT / Claude / Cursor — it reads every endpoint and parameter instantly and tells you if this API fits your use case.
The primary check endpoint returns a score (grade, breakdown, recommendations), AI-crawler access, llms.txt status and schema, and you can pull crawlers, llms_txt, schema, score and batch. It is built for SEO/GEO tools and site audits that need an AI-visibility signal for a URL. One ReefAPI key, one shared credit pool, the standard envelope.
The five scoring pillars and what each one reads
GEO here means Generative Engine Optimization, not geolocation. The only input is a URL and the output is a 0-100 AI-readiness score built from five pillars whose maximums add to 100. The breakdown block returns each pillar's earned score against its max, and recommendations name the pillar, the priority and the exact points_recoverable, so you can reconstruct the arithmetic yourself.
| Pillar | Max points | What it reads, with a measured example |
|---|---|---|
| crawler_access | 40 | robots.txt verdict for 73 AI bot tokens using Google longest-match; www.nytimes.com blocked 29 of them, 7 of which are flagged critical |
| structured_data | 25 | JSON-LD blocks and @graph types; github.com scored 6.0 because it ships RDFa but no JSON-LD |
| llms_txt | 15 | /llms.txt plus /llms-full.txt with the content type sniffed so an HTML 404 cannot pass; docs.anthropic.com returned a valid 72,234-byte file with 689 links |
| meta_signals | 12 | canonical, OpenGraph and the robots meta tag; github.com scored 11.5 of 12 |
| crawlability | 8 | a declared sitemap.xml, the text ratio and likely_js_app; github.com lost half of it for having no sitemap |
Measured score-to-grade pairs from live runs: 90.0 = A (vercel.com, reefapi.com), 80.5 and 76.5 = B (docs.anthropic.com, github.com), 64.0 = C (developer.mozilla.org), 50.6, 49.0 and 45.0 = D (bbc.com, en.wikipedia.org, stripe.com), 5.0 = F (reddit.com). score is a float, not an integer.
Real request and response JSON
Captured from the indexed primary action, check, on .
{
"method": "POST",
"url": "https://api.reefapi.com/geo-intel/v1/check",
"headers": {
"x-api-key": "$REEF_KEY",
"content-type": "application/json"
},
"body": {
"url": "https://docs.anthropic.com"
}
}{
"ok": true,
"meta": {
"api": "geo-intel",
"endpoint": "check",
"mode": "live",
"latency_ms": 14218.3,
"record_count": 1,
"bytes": 928401,
"cache_hit": false,
"method": "fetch_target_site_bare_dc_curl_cffi",
"ssrf_guarded": true,
"final_url": "https://platform.claude.com/docs/en/home",
"score": 80.5,
"grade": "B",
"critical_blocked": 0
},
"data": {
"url": "https://docs.anthropic.com",
"final_url": "https://platform.claude.com/docs/en/home",
"http_status": 200,
"score": {
"score": 80.5,
"grade": "B",
"breakdown": {
"crawler_access": {
"score": 40,
"max": 40
},
"structured_data": {
"score": 6,
"max": 25
},
"llms_txt": {
"score": 15,
"max": 15
},
"meta_signals": {
"score": 11.5,
"max": 12
},
"crawlability": {
"score": 8,
"max": 8
}
},
"recommendations": [
{
"priority": "high",
"pillar": "structured_data",
"points_recoverable": 19,
"fix": "only microdata/RDFa found — JSON-LD is the AI-preferred format; add it"
}
]
},
"ai_crawlers": {
"bots": [
{
"token": "[redacted-secret]",
"operator": "OpenAI",
"engine": "ChatGPT Search",
"category": "search",
"cites": true,
"critical": true,
"allowed": true,
"matched_rule": null,
"via_ua": "*",
"crawl_delay": null,
"reason": "allowed (no matching disallow)"
},
{
"token": "[redacted-secret]",
"operator": "OpenAI",
"engine": "ChatGPT (user-fetch)",
"category": "agent",
"cites": true,
"critical": true,
"allowed": true,
"matched_rule": null,
"via_ua": "*",
"crawl_delay": null,
"reason": "allowed (no matching disallow)"
},
{
"token": "[redacted-secret]",
"operator": "OpenAI",
"engine": "ChatGPT (training)",
"category": "training",
"cites": false,
"critical": false,
"allowed": true,
"matched_rule": null,
"via_ua": "*",
"crawl_delay": null,
"reason": "allowed (no matching disallow)"
}
],
"summary": {
"total": 73,
"allowed": 73,
"blocked": 0,
"critical_blocked": [],
"citation_bots_blocked": []
},
"blanket_block_all": false,
"robots_present": true
},
"llms_txt": {
"present": true,
"valid": true,
"status": 206,
"content_type": "text/plain; charset=UTF-8",
"looks_like_html": false,
"bytes": 204609,
"title": "Anthropic Developer Documentation",
"h1_count": 1,
"section_count": 3,
"sections": [
"Root URL",
"Available Languages on Website",
"English"
],
"link_count": 1895,
"links_sample": [
{
"text": "Overview",
"url": "https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview.md"
},
{
"text": "Features overview",
"url": "https://platform.claude.com/docs/en/build-with-claude/overview.md"
},
{
"text": "Quickstart",
"url": "https://platform.claude.com/docs/en/agents-and-tools/agent-skills/quickstart.md"
}
],
"findings": [],
"head": "# Anthropic Developer Documentation\n\nThis file provides an overview of the Anthropic API documentation and developer resources.\n\n## Root URL\n\nClaude Developer Platform Console (Requires login)\n\nhttps://platform.claude.com\n\n## Available Languages on Website\n\nThe full documentation is available in the following languages on https://platform.claude.com/docs:\n\n- English (en) - 1894 pages - /docs - Con",
"url": "https://docs.anthropic.com/llms.txt"
},
"llms_full_txt": {
"present": true,
"status": 200,
"content_length_header": null,
"url": "https://docs.anthropic.com/llms-full.txt"
},
"structured_data": {
"jsonld_blocks": 0,
"jsonld_parsed": 0,
"jsonld_unparseable": 0,
"types": [],
"type_counts": {},
"citable_types": [],
"high_value_types": [],
"has_organization": false,
"has_article": false,
"has_faq": false,
"has_breadcrumb": false,
"has_speakable": false,
"has_microdata": false,
"has_rdfa": true,
"has_structured_data": true
},
"meta": {
"title": "Documentation - Claude Platform Docs",
"meta_description": "Claude API Documentation",
"canonical": "https://platform.claude.com/docs/en/home",
"robots_meta": null,
"noindex": false,
"nosnippet": false,
"noai_meta": false,
"ai_meta_directives": null,
"has_opengraph": true,
"og_title": "Documentation",
"og_description": "Claude API Documentation",
"og_type": "article",
"og_site_name": "Claude Platform Docs",
"twitter_card": "summary_large_image",
"has_twitter_card": true,
"html_lang": "en-US",
"hreflang": null,
"feeds": null,
"favicon": "https://platform.claude.com/favicon.ico",
"has_viewport": true
},
"extractability": {
"text_ratio": 0.0048,
"visible_text_chars": 3387,
"script_count": 159,
"script_heavy": true,
"likely_js_app": false
},
"sitemaps": [
"https://platform.claude.com/sitemap.xml"
],
"sitemap_xml_present": true
}
}What the AI Search Visibility API does
| Action | Description | Concrete use case | Key params |
|---|---|---|---|
| check | Full AI-readiness report for a URL: AI-crawler access map + llms.txt + schema + meta + 0-100 score + ranked fix recommendations. One call, everything. | Ops teams call check to get full AI-readiness report for a URL. | url, path, bots, follow_redirects |
| crawlers | AI-crawler access map from robots.txt: per-bot allow/block (Google longest-match), operator, AI engine, purpose (search/training/agent), crawl-delay, sitemaps. | Developer tools call crawlers to get aI-crawler access map from robots.txt. | url, path, bots |
| llms_txt | Detect + parse + validate /llms.txt (and /llms-full.txt) — the emerging AI-context standard. Content-type sniffed so an HTML 404 can't masquerade as a real file. | Validation workflows call llms_txt to detect + parse + validate /llms.txt (and /llms-full.txt). | url |
| schema | Extract and classify schema.org structured data ( and @graph) for AI citability signals: Organization, Article, FAQ, BreadcrumbList, Product, Speakable — plus microdata and RDFa presence. | Data-quality teams call schema to extract and classify schema.org structured data ( and @graph) for AI citability signals. | url, follow_redirects |
| score | Composite AI-readiness score (0-100) + transparent per-pillar breakdown (crawler_access 40 / structured_data 25 / llms_txt 15 / meta 12 / crawlability 8) + ranked recommendations. | Ops teams call score to get composite AI-readiness score (0-100) + transparent per-pillar breakdown (crawler_access 40 /…. | url, path, follow_redirects |
| batch | Score up to 20 sites concurrently in one call (each independently SSRF-guarded). Returns a compact score+grade per URL. | Developer tools call batch to get score up to 20 sites concurrently in one call (each independently SSRF-guarded). | urls |
Call check from your stack
curl -X POST https://api.reefapi.com/geo-intel/v1/check \
-H "x-api-key: $REEF_KEY" \
-H "content-type: application/json" \
-d '{"url":"https://docs.anthropic.com"}'import requests
r = requests.post(
"https://api.reefapi.com/geo-intel/v1/check",
headers={"x-api-key": REEF_KEY},
json={
"url": "https://docs.anthropic.com"
},
)
print(r.json()["data"])const res = await fetch("https://api.reefapi.com/geo-intel/v1/check", {
method: "POST",
headers: {
"x-api-key": process.env.REEF_KEY,
"content-type": "application/json",
},
body: JSON.stringify({
"url": "https://docs.anthropic.com"
}),
});
const { ok, data, meta, error } = await res.json();Ask your MCP-connected assistant: call reefapi.geo-intel.check with {"url":"https://docs.anthropic.com"}.Who uses this API and why
- SEO/GEO tools call check to grade a page's readiness for AI search.
- Agencies use the recommendations to prioritize fixes for a client.
- Monitors use batch to audit a whole site's pages.
Questions developers ask before integrating
Is this a geolocation API?
No. geo-intel never accepts a coordinate, an IP or a postal code, and its only required parameter is a URL. For coordinates and postal codes use geo-postal, and for locating an IP address use ip-geolocation. The name is short for Generative Engine Optimization, which is search-visibility work aimed at ChatGPT, Claude, Perplexity and Google AI Overviews rather than at a map.
Which bots count as critical, and why does blocking GPTBot not hurt my score?
Every bot row carries cites and critical booleans alongside category, which is one of search, agent, assistant, training or scraper. On www.nytimes.com the 7 critical_blocked entries were OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and Google-Extended, all bots that can produce a citation. GPTBot was blocked on the same site but is category training with cites false, so it does not appear in critical_blocked. The score punishes blocking bots that could cite you, not the ones that only train.
The response says has_structured_data true but the schema pillar still lost points. Why?
has_structured_data covers microdata and RDFa as well as JSON-LD, while the pillar rewards JSON-LD specifically. github.com returned has_structured_data true and scored 6.0 of 25, with the recommendation "only microdata/RDFa found - JSON-LD is the AI-preferred format; add it" and points_recoverable 19.0. Read citable_types and high_value_types instead of the boolean: on nytimes.com those were NewsMediaOrganization and WebSite out of four detected types, with type_counts showing 43 ListItem entries.
Does it follow redirects, and can I point it at an internal host?
follow_redirects defaults to true and each hop is validated independently, and the response returns both url and final_url so you can see where the analysis actually landed. Private, internal and cloud-metadata targets are refused: a call for http://169.254.169.254/latest/meta-data/ returned INVALID_PARAM with the message "SSRF-blocked: private/internal IP" and a detail block naming the host and its resolved addresses.
What does batch give up compared with calling check per URL?
batch takes up to 20 URLs and scored three of them concurrently in a single 3.4 second call, but each result carries only url, final_url, score, grade, critical_blocked, llms_txt_present and has_structured_data. There is no per-bot table, no llms.txt parse and no recommendations. Use batch to rank a portfolio, then call crawlers or check on the ones that scored badly.
What is robots_status, and what if robots.txt is unreadable?
robots_status is the literal HTTP status of the robots.txt fetch; www.nytimes.com answered 206 rather than 200 and the parse still succeeded. Alongside it you get robots_present, parse_errors as an array, user_agents_in_robots listing the agent blocks that actually appear in the file, and blanket_block_all for a global Disallow of the site root. If parse_errors is non-empty, treat the per-bot allowed values as unreliable rather than as permission.
Can I test a specific page instead of the homepage?
Yes, with path, which defaults to "/". The check, crawlers and score actions evaluate robots.txt allow and deny rules against that exact path, so passing /blog/some-post tells you whether AI bots may fetch that article rather than whether they may fetch the site root. Sites frequently allow the root and disallow a content directory, and only the path form catches that.
What does llms_txt check that a plain fetch would not?
It sniffs the content type so an HTML 404 page served at /llms.txt cannot be counted as present. A measured docs.anthropic.com response returned present true, valid true, status 200, content_type text/plain, looks_like_html false, bytes 72234, title "Anthropic Developer Documentation", section_count 3, link_count 689 and an empty findings array. llms_full_txt is reported separately with its own status and URL.
What is the AI Search Visibility API?
AI Search Visibility API is a ReefAPI endpoint group for ai search visibility It returns live JSON through POST requests under /geo-intel/v1.
Is the AI Search Visibility API free to try?
Yes. ReefAPI starts with 1,000 free credits, no card required. AI Search Visibility calls use the same shared credit balance as every other ReefAPI engine.
Do I need an AI Search Visibility login or account?
No login to AI Search Visibility is needed for the API response. You call ReefAPI with your x-api-key header, and the playground can run live examples before you create a production key.
How fresh is the AI Search Visibility data?
The page example is captured from a live check call, and production requests fetch live data through ReefAPI rather than a static sample.
How many credits does the AI Search Visibility API use?
AI Search Visibility actions currently cost 1 credit per successful call. Failed or blocked calls are free, and all APIs draw from one credit pool.
Can I call AI Search Visibility from an AI assistant or MCP client?
Yes. Connect ReefAPI once through MCP and your assistant can call geo-intel actions with the same key, credit pool and JSON envelope used by normal REST requests.