CRW — open-source web scraper & crawler (Firecrawl-compatible)

Docs at docs.fastcrw.com. Hosted API at https://api.fastcrw.com. Self-hosted at http://localhost:3000. Five endpoints: /v1/search, /v1/scrape, /v1/crawl, /v1/map, /v1/extract.

Official SDKs: Python crw, TypeScript crw-sdk

Typed clients for every endpoint. Use them instead of raw HTTP when a task asks for the SDK. Full reference: /sdk-reference.

# pip install crw
from crw import CrwClient

client = CrwClient()  # reads CRW_API_KEY; defaults to https://api.fastcrw.com
page = client.scrape("https://example.com", formats=["markdown", "links"])

# page is a plain dict (no .data wrapper)
print(page["markdown"])           # str: clean page markdown, links kept inline as [text](url)
print(page["links"])              # list[str]: every absolute URL on the page, no anchor text
print(page["metadata"]["title"])  # str
# Run with: python3 scrape.py

Signature: scrape(url, formats=None, only_main_content=None, include_tags=None, exclude_tags=None, *, render_js=None, renderer=None, wait_for=None, json_schema=None, **kwargs) -> dict. Extra keyword arguments such as css_selector are passed to the API as they are.

Titles and URLs from a list page, in one call

css_selector keeps only the matching elements, so the markdown is just their links as [title](url), in page order. No exploratory scrape needed.

import json, re
from crw import CrwClient

page = CrwClient().scrape("https://news.ycombinator.com", formats=["markdown"], css_selector=".titleline > a")
pairs = re.findall(r"\[([^\]]+)\]\((https?://[^)\s]+)\)", page["markdown"])  # [(title, url), ...]
print(json.dumps([{"rank": i, "title": t, "url": u} for i, (t, u) in enumerate(pairs[:5], 1)], indent=2))
// npm install crw-sdk
import { CrwClient } from "crw-sdk";
const crw = new CrwClient(); // reads CRW_API_KEY
const page = await crw.scrape("https://example.com", { formats: ["markdown"] });
console.log(page.markdown);

Other methods: crawl(url), map(url), search(query, limit=5), extract(urls, schema={...}). scrape(url, json_schema={...}) runs managed LLM extraction: on the hosted FREE plan it returns an error, so parse markdown or links there instead.

Quick: copy this and run

Auth (hosted only): Authorization: Bearer $CRW_API_KEY. Self-hosted (http://localhost:3000) needs no header.

Body: { "query": string, "limit": number (default 5, max 20), "lang"?: "en", "tbs"?: "qdr:d|w|m|y", "sources"?: ["web"|"news"|"images"] }

Python (stdlib only — defaults to hosted, fails loudly if key is missing)

#!/usr/bin/env python3
"""Search CRW and print top results as JSON."""
import json, os, sys, urllib.request

API_KEY = os.environ.get("CRW_API_KEY")
# Default to the hosted API. Self-hosting? Set CRW_API_URL=http://localhost:3000
BASE = os.environ.get("CRW_API_URL", "https://api.fastcrw.com")
URL = f"{BASE}/v1/search"

headers = {"Content-Type": "application/json"}
if API_KEY:
    headers["Authorization"] = f"Bearer {API_KEY}"
elif BASE.startswith("https://api.fastcrw.com"):
    sys.exit("error: CRW_API_KEY is required for the hosted API. "
             "Get one at https://fastcrw.com (1000 free credits) "
             "or set CRW_API_URL=http://localhost:3000 to self-host.")

req = urllib.request.Request(
    URL,
    data=json.dumps({"query": "renewable energy trends 2024", "limit": 3}).encode(),
    headers=headers,
    method="POST",
)
with urllib.request.urlopen(req) as r:
    body = json.loads(r.read())  # body == {"success": true, "data": [...]}

# Results live under body["data"]. Each row has `title`, `url`, `snippet`,
# `description`, `position`, `score`, `category` — print the LLM-ready subset.
print(json.dumps(
    [{"title": x["title"], "url": x["url"], "snippet": x["snippet"]}
     for x in body["data"]],
    indent=2,
))
# Run with: python3 search.py  (file is python3-only — no `python` alias needed)

cURL

curl -X POST https://api.fastcrw.com/v1/search \
  -H "Authorization: Bearer $CRW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"renewable energy trends 2024","limit":3}'

CLI (one-shot LLM-ready output)

The crw CLI projects directly to the title/url/snippet schema with --json --fields — no jq reshape needed:

# Install: brew install fastcrw/crw/crw  (or: cargo install crw-cli)
crw search "renewable energy trends 2024" --json --fields title,url,snippet --limit 3

# Available fields: title, url, description, snippet, position, score, category
# --json is shorthand for --format json (also accepts --format text|markdown)
# Self-hosted: no auth. Hosted: export CRW_API_KEY=crw_live_...

Response shape

{
  "success": true,
  "data": [
    {"url": "https://...", "title": "...",
     "description": "...", "snippet": "...",
     "position": 1, "score": 9.5}
  ]
}

Other endpoints

  • POST /v1/scrape {"url":"...","formats":["markdown"]} — one URL → markdown
  • POST /v1/map {"url":"...","limit":50} — discover URLs
  • POST /v1/crawl {"url":"...","maxPages":5} — async multi-page; poll GET /v1/crawl/{id}
  • POST /v1/scrape with formats:["json"] + jsonSchema: structured extract

Install (local setup only)

pip install crw · npx crw-mcp · brew install fastcrw/crw/crw · curl -fsSL https://fastcrw.com/install | sh · docker run -p 3000:3000 ghcr.io/fastcrw/crw:latest

Resources

  • OpenAPI spec
  • llms.txt
  • Context7
  • MCP Server (npm)
  • Typed Python SDK
  • Agent Skills (SKILL.md)
  • CLI (Homebrew tap)
  • GitHub
CRW CRW fastcrw
fastcrw/crw Sign Up
On this page
CRW CRW CRW
Ready to scrape?
Sign up and get 1000 free credits instantly. No credit card required.
Get started, 1000 free credits View on GitHub
CRW CRW CRW
The base layer for agentic web intelligence.
Product
Documentation API Reference MCP Server Changelog
Community
GitHub Issues Discord
Legal
License (AGPL-3.0)
© 2026 CRW. All rights reserved.
All systems operational
esc