AI Crawler Index — 150 AI crawlers, their robots.txt tokens and IP ranges

# AI Crawler Index > Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex. 150 crawlers, 74 operators, 1984 IPv4 and 1062 IPv6 prefixes mirrored from 15 endpoints the operators publish themselves. Static files, no key, no rate limit, CORS open, CC0. Rebuilt 2026-09-03. The question this exists to answer: *which of these do I allow, and what do I lose if I do not?* Operators document that a bot exists. They do not document what blocking it costs you. Every record here carries that, in one field, in plain language. ## Start here - [All 150 crawlers](/data/agents.json): the whole dataset, one file. - [Tools you can call with a URL](/tools/index.html): the read-only MCP tools of this host, as keyless GET endpoints. No account, no key, no JSON-RPC. - [Who actually crawls this host](/bot/index.html): 147 named clients — bots, registry probes, trust scanners and liveness checkers — one page each, generated from this host's own request log: the exact user-agent, first and last seen, how many addresses it came from, the paths it asked for in order, the status codes it got, and what it asked for that did not exist. Window 2026-08-31T20:58:11+00:00 to 2026-09-03T00:22:55+00:00 UTC. Most of these names are documented nowhere else on the web. Whole set in one request: [/data/observed-clients.json](/data/observed-clients.json), flat table at [/data/observed-clients.csv](/data/observed-clients.csv). - [Ready-made robots.txt](/policy/index.html): eight policies, each a file you can curl. - [Published IP ranges](/ip-ranges/all.json): every operator prefix list, one schema. - [Upstream status](/status.json): which operator endpoints answered, and when. - [What changed since your last read](/changes.json?since=0): a cursor feed. Read `cursor` from the response, send it back as `since`, and you get only what moved — prefixes added or removed, upstreams that

{ "openapi": "3.1.0", "info": { "title": "AI Crawler Index", "version": "2026-09-03", "summary": "Every AI crawler on the web, what it is for, what blocking it costs you, and the IP ranges its operator publishes — as JSON, CSV, robots.txt and regex.", "description": "A read-only, static, keyless index of 150 web crawlers operated by 74 companies and projects: what each one is for, the exact robots.txt token and user-agent substring, whether the operator says it obeys robots.txt, how