Try any of 817 endpoints — live.
Pick an endpoint, load a working example, tweak the params, and send — no signup to try. Results render the way the data deserves; raw JSON, headers & code are one tab away.
AI-crawler access map from robots.txt: per-bot allow/block (Google longest-match), operator, AI engine, purpose (search/training/agent), crawl-delay, sitemaps.
The site to analyse. Accepts a full URL (https://example.com/page) or a bare domain (example.com → https:// assumed). Only http/https; private/internal/metadata targets are SSRF-blocked. Alias: domain.
The path to evaluate robots.txt access against (default '/'). Use a real content path (e.g. /blog/post) to test whether AI bots may fetch THAT page.
Optional subset of AI-bot tokens to report (comma-separated or array). Omit for the full 50+ bot map. Unknown tokens are still checked against robots.txt verbatim.
curl -X POST https://api.reefapi.com/geo-intel/v1/crawlers \
-H "x-api-key: $REEF_KEY" \
-H "content-type: application/json" \
-d '{"url":"https://www.nytimes.com"}'Hit Send to run this endpoint live.