StackFox - Monitor & Control AI Bot Traffic

# StackFox llms.txt # https://stackfox.co ## About StackFox StackFox is the intelligence layer for AI agents. We track who they are, what they can do, and how the web responds. Three databases power everything we build. ## Our Three Databases 1. **Agent Database** - 95+ AI crawlers profiled with user agents, IPs, behaviors, operators, and verification methods 2. **Policy Database** - 1M+ domains analyzed for robots.txt rules, llms.txt adoption, and historical policy changes 3. **Tech Database** - 700+ web technologies detected and cataloged with detection patterns ## Directories - **AI Bot Directory** (`/bots`) - The most comprehensive list of AI crawlers. GPTBot, ClaudeBot, PerplexityBot, and 95+ more with operator info and verification methods. - **Site Database** (`/sites`) - Browse 1M+ analyzed websites. See AI policies, tech stacks, rankings. - **llms.txt Directory** (`/llms-txt`) - Websites that have implemented llms.txt to declare AI permissions. - **Sites with MCP Servers** (`/mcp-servers`) - 100+ verified sites with remote MCP servers. Browse by category, connect AI agents directly. - **Tech Database** (`/technologies`) - 700+ web technologies detected across the web. Frameworks, analytics, AI tools, and more. ## Tools - **Site Analysis** (`/sites/{domain}`) - Full AI posture analysis for any domain. robots.txt rules, llms.txt detection, which bots are blocked/allowed. - **robots.txt Grader** (`/tools/robots-txt-grader`) - Analyze your robots.txt against 95+ AI crawlers. - **robots.txt Generator** (`/tools/robots-txt-generator`) - Create AI-aware robots.txt policies. - **AI Policy Comparison** (`/compare/ai-policy`) - Compare AI policies between multiple sites side-by-side. - **Agent Testing Suite** (`/agent-console`) - Test how AI agents interact with your site. ## Research - **AI Web Index** (`/research/who-blocks-ai`) - Analysis of AI crawler blocking across top 10,000 websites. Industry breakdowns, trends, statistics. - **AI Index Methodology*