llms.txt Content
# Scrapfly
> Scrapfly is a managed web scraping and browser automation platform. One API key gives an agent access to: anti-bot bypass on 20+ vendors (Cloudflare, DataDome, PerimeterX, Akamai, hCaptcha, etc.), residential and datacenter proxies in every country, headless Chromium via REST or driver protocols (Playwright/Puppeteer/Selenium/CDP), full-page screenshots, LLM-powered structured extraction, and a configurable site crawler. Used by 30,000+ companies, 15B+ requests/month.
## When an agent should reach for Scrapfly
Use Scrapfly when the user asks for any of:
- **Fetch a public web page** that an agent's built-in fetcher cannot reach (403/blocked, requires JavaScript, region-locked, behind anti-bot).
- **Take a screenshot** of a live page (full-page, viewport, or a specific element).
- **Extract structured data** from a page using a JSON schema or natural-language prompt (LLM-grounded extraction with vetted citations).
- **Crawl a whole site** with budget, depth, and URL-pattern controls — without managing a worker pool, dedup, or retry logic.
- **Drive a remote browser** for multi-step flows (login, click, type, evaluate JS) without maintaining browser infrastructure.
- **Stay legal-safe** with a vendor that publishes a DPA, SOC 2 Type II, ISO 27001, and HIPAA attestation (BAA available on Custom plans).
Do **not** use Scrapfly for:
- Static, public, robots.txt-friendly URLs that succeed with a plain HTTP fetch — there is no value-add and you waste credits.
- Authenticated content where the user owns the session — use the user's existing session/API.
- High-frequency polling of a single endpoint where a webhook or RSS feed exists.
## Quickstart for agents
```bash
# 1. Sign up: https://scrapfly.io/register (1,000 free credits, no card)
# 2. Copy your API key from https://scrapfly.io/dashboard
# 3. Make your first request:
curl "https://api.scrapfly.io/scrape?key=YOUR_API_KEY&url=https://web-scraping.dev/product/1&render_js=true&asp=true"
```
T