AI Crawler Index — 150 AI crawlers, their robots.txt tokens and IP ranges

# AI Crawler Index > Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. 150 crawlers, 74 operators, 1987 IPv4 and 1062 IPv6 prefixes mirrored from 15 endpoints the operators publish themselves. Static files, no key, no rate limit, CORS open, CC0. Rebuilt 2026-09-06. The question this exists to answer: *which of these do I allow, and what do I lose if I do not?* Operators document that a bot exists. They do not document what blocking it costs you. Every record here carries that, in one field, in plain language. ## Recent changes changed_at: 2026-09-06T10:22:22+00:00 recent_changes: - 2026-09-06T10:22Z - text edited; 278640->278638 bytes - 2026-09-06T10:15Z - 1501->1508 lines; 277892->278640 bytes - 2026-09-06T09:25Z - 1499->1501 lines; 277663->277892 bytes ## Start here - [All 150 crawlers](/data/agents.json): the whole dataset, one file. - [Every page here as markdown](/sitemap.md): the markdown index of this host, and every page also answers as markdown at its own address with `.md` — `/about.html` is [/about.md](/about.md), `/bot/gptbot.html` is [/bot/gptbot.md](/bot/gptbot.md) — and at `.mdx`, `<page>.html.md` and `<page>.html.mdx`, which are the same bytes. `Accept: text/markdown` on the page itself returns the same document. No markup to strip, no JavaScript, no key; the HTML page stays canonical and each mirror says so in a header. - [Tools you can call with a URL](/tools/index.html): the read-only MCP tools of this host, as keyless GET endpoints. No account, no key, no JSON-RPC. - [Who actually crawls this host](/bot/index.html): 976 named clients — bots, registry probes, trust scanners and liveness checkers — one page each, generated from this host's own request log: the exact user-agent, first and last seen, how many addresses it came from, the paths it asked for in order, the status codes it got, and what it asked for that

{ "openapi": "3.1.0", "info": { "title": "AI Crawler Index", "version": "2026-09-06", "summary": "Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex.", "description": "A read-only, static, keyless index of 150 web crawlers operated by 74 companies and projects: what each one is for, the exact robots.txt token and user-agent substring, whether the operator says it obeys robots.txt, how