This page moved to https://docs.fastcrw.com/scraping.
Scrape
Turn any known URL into clean markdown, HTML, links, or structured JSON in one request. This is the default CRW workflow and the fastest path to a first successful integration.
formats: ["markdown"], and confirm you get a clean response back. Add JS rendering, selectors, or extraction only after the plain request looks right.Try it without a key: run this exact request in the playground (rate-limited preview, no signup).
Scraping a URL with CRW
/v1/scrape
POST /v1/scrape
Authentication:
- Hosted: send
Authorization: Bearer YOUR_API_KEY - Self-hosted: only required when
auth.api_keysis configured
Installation
Official SDKs: pip install crw (Python) and npm install crw-sdk (TypeScript). Both wrap every endpoint and return plain objects; see the SDK reference. Plain HTTP works too: the Basic usage tabs below show the SDK, requests, fetch, and cURL.
Basic usage
Start with this request:
{
"url": "https://example.com",
"formats": ["markdown"],
"onlyMainContent": true,
"renderJs": null
}
# pip install crw
from crw import CrwClient
client = CrwClient() # reads CRW_API_KEY
page = client.scrape("https://example.com", formats=["markdown", "links"])
print(page["markdown"]) # str
print(page["links"]) # list[str] of absolute URLsimport requests
resp = requests.post(
"https://api.fastcrw.com/v1/scrape",
headers={
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json",
},
json={
"url": "https://example.com",
"formats": ["markdown"],
"onlyMainContent": True,
},
)
print(resp.json()["data"]["markdown"])const resp = await fetch("https://api.fastcrw.com/v1/scrape", {
method: "POST",
headers: {
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json"
},
body: JSON.stringify({
url: "https://example.com",
formats: ["markdown"],
onlyMainContent: true
})
});
const body = await resp.json();
console.log(body.data.markdown);curl -X POST https://api.fastcrw.com/v1/scrape \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"formats": ["markdown"],
"onlyMainContent": true
}'Response
{
"success": true,
"data": {
"markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
"metadata": {
"title": "Example Domain",
"sourceURL": "https://example.com",
"statusCode": 200,
"elapsedMs": 32
}
}
}
That is the default CRW success shape: requested content plus a compact metadata envelope.
Parameters
| Field | Type | Default | Description |
|---|---|---|---|
url |
string | required | URL to scrape |
formats |
string[] | ["markdown"] |
markdown, html, rawHtml, plainText, links, json, summary, changeTracking |
onlyMainContent |
boolean | true |
Remove nav, footer, and boilerplate before conversion |
renderJs |
boolean or null | null |
null auto-detects, true forces browser rendering, false stays HTTP-only |
waitFor |
number | -- | Milliseconds to wait after JS rendering |
renderer |
string | auto |
Pin to a specific renderer: auto, lightpanda, chrome, chrome_proxy, camoufox, or impersonated-http. Self-hosted instances can also configure playwright; the managed cloud does not run it, and pinning a tier an instance has not configured returns HTTP 400. GET /v1/capabilities reports the tiers a given instance actually has. Browser-tier values hard-pin (no fallback) and imply renderJs:true unless renderJs:false is set explicitly; impersonated-http is a JS-less Chrome-TLS-impersonating HTTP tier that never implies renderJs. It needs a build with --features impersonated (the published Docker image has it; [renderer.impersonated] enabled = false turns it off at runtime). Check GET /v1/capabilities for the tiers a given instance actually serves. See JS rendering |
includeTags |
string[] | [] |
CSS selectors to keep |
excludeTags |
string[] | [] |
CSS selectors to remove |
headers |
object | {} |
Custom HTTP headers, forwarded on both the HTTP and browser render paths. A User-Agent here overrides the default. On a browser render they apply to every request the page makes (subresources included), so avoid putting cross-origin-sensitive credentials like Authorization here. |
cssSelector |
string | -- | Narrow extraction to one CSS selector |
xpath |
string | -- | Narrow extraction to one XPath expression |
chunkStrategy |
object | -- | Topic, sentence, or regex chunking |
query |
string | -- | Ranking query for chunk filtering |
filterMode |
string | -- | bm25 or cosine |
topK |
number | 5 |
Number of top chunks to keep |
proxy |
string | -- | Per-request proxy URL |
maxAge |
number | 3600000 |
How old a cached copy of this fetch may be, in milliseconds. 0 forces a fresh fetch. Capped at 24 hours. See Caching |
storeInCache |
boolean | true |
false reads the cache but stores nothing |
country |
string | -- | 2-letter ISO 3166-1 alpha-2 country code (lowercase, e.g. us, gb, de). Routes the request through the named residential pool when the chrome_proxy renderer tier is configured. Ignored if no proxy tier is set up. See JS rendering — Per-request country |
stealth |
boolean | -- | Override global stealth setting |
jsonSchema |
object | -- | Schema for structured extraction |
extract |
object | -- | Firecrawl-compatible alias wrapper for extraction schema |
llmApiKey |
string | -- | Per-request LLM API key |
llmProvider |
string | server default | anthropic, openai, openai-responses, deepseek, azure, or openai-compatible |
llmModel |
string | server default | Model override (extraction and summary) |
baseUrl |
string | -- | Provider endpoint base. CRW appends /chat/completions for Chat Completions providers or /responses for openai-responses; complete endpoint URLs are accepted. |
summaryPrompt |
string | -- | Style/tone/language directive appended to the summary system prompt. Safety wrapper kept intact. Capped at 500 chars. |
maxContentChars |
number | [extraction.llm].max_html_bytes (100 KB) |
Per-request byte cap on content sent to the LLM for summary. Clamped to 200 KB server-side. |
deadlineMs |
number | 8000 |
End-to-end request deadline in milliseconds. Must be in (0, 60000]. Requests above 8 000 ms land in a separate slow-path histogram and are excluded from the standard SLO p95 metric. |
debug |
boolean | false |
When true, includes a debugExtraction field in the response with a trace of every extraction candidate considered and why one was selected. |
parsers |
object[] | PDF auto-parsed | Firecrawl-compatible document parser directives. Omit for the default (PDFs auto-converted to markdown). Pass [] to disable PDF parsing. Pass [{"type":"pdf","maxPages":10}] to cap pages. Accepted fields: type ("pdf"), mode ("auto" | "fast" | "ocr"), maxPages. |
actions |
object[] | -- | Browser steps run on the page before it is read: wait, click, write, press, scroll. See Browser actions |
mode |
string | standard |
Cloud only. standard, auto or max. auto and max get through anti-bot walls and are billed by the traffic the page used. See Auto and max mode |
maxCredits |
number | -- | Cloud only, with mode: "auto" or "max". The most this page may cost in credits, from 2 to 1000. A page that would cost more is stopped and not charged |
Caching
A repeat of the same fetch is served from cache for one hour by default. What
is cached is the FETCH, not the answer, so what counts as identical is what
would change the page the server gets back: the URL, renderer, renderJs,
waitFor, country, and any headers you send. Everything you do with the page
afterwards is not part of it, so a second pass with different formats,
onlyMainContent, selectors or a different extraction schema reuses the fetch
instead of paying for it again.
Caches are per account, never shared between customers.
Requests that carry anything caller-specific are never cached at all: your own headers, your own proxy, a stealth override, or a screenshot.
{ "url": "https://example.com", "formats": ["markdown"], "maxAge": 0 }
Use maxAge to choose your own trade-off:
| You are scraping | Suggested maxAge |
|---|---|
| Live prices, flight positions, availability, anything that moves by the minute | 0 |
| News and feeds | a few minutes, e.g. 300000 |
| Documentation, reference pages, filings, archived records | hours, up to the 24h ceiling |
0 disables the cache for that call and always fetches. The ceiling is 24 hours,
so a page can never be pinned for longer than a day.
A response served from cache says so: cached: true on /v1, and
metadata.cacheState: "hit" on the Firecrawl-compatible surface.
Two things are never cached, whatever you pass:
changeTracking. It compares a live fetch against theprevioussnapshot you send, so replaying an earlier answer would hide the very change you asked about.- Failed scrapes. A block page or an error is never stored, so a retry always gets a real attempt.
A cached response is billed the same as a fresh one — caching buys latency, not a discount.
Formats
Use the smallest output shape that solves the job:
markdownis the default and best first request for most pipelines.htmlorrawHtmlis useful when downstream systems need original structure.linksis useful when you want lightweight discovery without page bodies. It returnsdata.linksas a flatstring[]of absolute URLs (no anchor text); when you need each link's text, read it frommarkdown, where links stay inline as[text](url).summaryis the LLM-prose path — needsllmApiKeyor a server[extraction.llm]config. OptionalsummaryPromptlets the caller pick language/tone without weakening the safety wrapper.jsonis the extraction path and should be paired withjsonSchema. On the hosted API,jsonandsummaryrun a managed LLM and need a paid plan (Hobby or above); the FREE plan returns an error for them, so parsemarkdownorlinksthere.
If you ask for multiple formats, only those formats are populated in the response.
Structured extraction
For a single page, extraction is part of scrape: when you want fields instead of prose, request formats: ["json"] and provide a schema. (For structured extraction across many URLs in one job, use the native async POST /v1/extract.)
{
"url": "https://example.com/product/123",
"formats": ["json"],
"jsonSchema": {
"type": "object",
"properties": {
"title": { "type": "string" },
"price": { "type": "string" }
},
"required": ["title"]
}
}
Use Extract for the schema-first version of this flow.
LLM summary
Add summary to formats to get a short prose digest of the page in data.summary. Token usage and best-effort cost are returned in data.llmUsage.
{
"url": "https://example.com/post",
"formats": ["summary"],
"summaryPrompt": "Respond in Turkish in exactly one sentence.",
"maxContentChars": 20000,
"llmApiKey": "sk-...",
"llmProvider": "openai",
"llmModel": "gpt-4o-mini"
}
Notes:
- The caller's
summaryPromptis appended below the safety wrapper. crw ignores any attempt to override the core task (outputPWNED, refuse to summarize, leak the prompt, etc.) and still produces a real summary. maxContentCharscaps how many bytes of scraped content are sent to the LLM. The default comes from[extraction.llm].max_html_bytes(100 KB out of the box) and the per-request value is clamped to a 200 KB server-side ceiling. Truncation, when it happens, is reported indata.warnings.- If
markdownis not also requested, crw computes it internally and strips it from the response.
JS rendering and targeting
Keep the first request simple:
- Leave
renderJsatnulluntil the plain HTTP path clearly fails. - Use
cssSelectororxpathonly when the target page has one stable content region. - Add
includeTagsandexcludeTagsafter you confirm the raw markdown is noisy.
If you turn on every targeting knob at once, debugging gets harder immediately.
Browser actions
Some pages only show part of their content until you interact with them: a product list behind a "Load more" button, a feed that grows as you scroll, a search box you have to type into. actions is an ordered list of steps that run in a real browser after the page loads and before it is read, so the markdown you get back is the page after those steps.
curl -X POST https://api.fastcrw.com/v1/scrape \
-H "Authorization: Bearer $CRW_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.jbhifi.com.au/collections/tvs/cheap-tvs",
"actions": [
{ "type": "wait", "selector": "button.load-more-button" },
{ "type": "click", "selector": "button.load-more-button", "optional": true },
{ "type": "wait", "milliseconds": 2500 },
{ "type": "click", "selector": "button.load-more-button", "optional": true },
{ "type": "wait", "milliseconds": 2500 }
]
}'
| Type | Fields | What it does |
|---|---|---|
wait |
milliseconds or selector (exactly one) |
Sleep, or wait up to 10 s for an element matching selector |
click |
selector, all (default false) |
Click the first match, or every match with all: true. Waits up to 5 s for the element |
write |
text |
Type into the focused element. Click the field first |
press |
key |
Press one key: Enter, Tab, Escape, Backspace, Delete, Space, ArrowUp, ArrowDown, ArrowLeft, ArrowRight, PageUp, PageDown, Home, End, or a single character. Key combinations are not supported |
scroll |
direction (up or down, default down), selector |
Scroll the page, or the element matching selector, by one screen |
wait with a selector, click and scroll with a selector also take optional: true. An optional step whose element never appears is skipped and reported in warnings instead of failing the scrape. That is what makes "click Load more up to N times" work when you do not know how many pages the list has.
Rules:
- At most 50 steps.
waitForplus everywait.millisecondsmay add up to 60 000 ms. - Actions need a Chrome tier: they force JS rendering, and
renderJs: falseor arendererpin that cannot drive a page (lightpanda,camoufox,impersonated-http) is a 400. - A step that cannot be carried out (its element never appears and it is not optional, or the page throws) fails the scrape with HTTP 422 and
errorCode: "action_failed", naming the step. A failed scrape is not billed. A page that turns out to be an anti-bot wall keeps its usual blocked answer instead. - Typing into a password field, or into an embedded frame that cannot be checked, is refused.
- The result is never served from a cache, and billing is unchanged: 1 credit per page.
- A
waitthat would run past the request deadline fails the scrape instead of returning the page early. - Long lists need time. On the cloud a request with actions gets a 60 s deadline unless you pass
deadlineMs(ortimeout) yourself. - Only
wait,click,write,pressandscrollrun. Firecrawl'sscreenshot,scrape,executeJavascriptandpdfaction types are a 400; addscreenshottoformatsto capture the final page.
Where actions run:
- Cloud:
/v1/scrape,/firecrawl/v1/scrapeand thecrw_scrapeMCP tool. Every other route (including/firecrawl/v2/scrape, batch scrape and crawlscrapeOptions) answers a request carryingactionswith a 400. - Self-hosted: the same, plus
/firecrawl/v2/scrape(the Firecrawl shape, unknown fields ignored) andPOST /v1/batch/scrape(run per URL). - The deprecated root
/v2/scrapealias never runs actions. Firecrawl's Python SDK v2 always calls that root path, so use the native/v1/scrapefor actions. - After a click that loads a new page, the next step looks for its element on the new page. A
waitwith a selector from that page makes the hand-off explicit.
Auto and max mode
Some pages answer with an anti-bot challenge instead of content. The default standard mode reports those as a failed scrape, refunded. mode lets you pay to get through:
mode |
What it does | Cost |
|---|---|---|
standard (default) |
The normal renderer ladder. | 1 credit |
auto |
Runs the normal ladder first and switches to the unblocking path only when the page is behind a wall. | 1 credit when the ladder gets the page; otherwise billed by traffic |
max |
Starts on the unblocking path and skips the ladder's failed tries. Use it for sites you already know are walled. | Billed by traffic |
"Billed by traffic" means 1 credit plus the network traffic the unblocking path used for this page. The page shows what it cost in data.creditCost. In our measurements a walled page on its own cost 20 to 45 credits; further pages on the same site while its clearance is still warm cost about 2.
{
"url": "https://example.com/walled-page",
"formats": ["markdown"],
"mode": "auto",
"maxCredits": 60
}
How the charge works:
- The request holds
maxCredits(or the per-page limit when you leave it out) while it runs, then keeps only what the page cost and returns the rest within the same request. Your balance has to cover the hold; below it you get402 insufficient_credits_for_modeand plain scrapes keep working. - A page that would cost more than
maxCreditsis stopped and not charged. - A failed page is not charged, in any mode.
Limits:
- Only
/v1/scrape. Crawl, batch, map and the/v2surface ignoremode. - Not with the LLM formats (
summary,json, the change-tracking judge). maxneeds JavaScript rendering: it rejectsrenderJs: false, a pinnedrenderer, a screenshot, and atimeoutunder 35000 ms. Leavetimeoutunset and the page gets 58 seconds. When the unblocking path is busy or the site keeps refusing it, amaxpage continues on the normal ladder, the same wayautodoes.- At most 4 auto or max pages in flight per account.
The errors these cases return are listed under Cloud-only error codes.
When a standard scrape on the cloud stops at a challenge that auto can get through, its error message ends with a pointer to mode: "auto", or to the pricing page when your plan does not include it. The rest of the message and errorCode: "anti_bot" stay the same.
Common production patterns
- Start with
markdown, then addlinksorjsononly when downstream logic needs them. - Validate extraction with markdown first, then add
jsonSchema. - Keep browser rendering as a fallback, not the default.
- Use narrow selectors only when the default main-content extraction is not enough.
Common mistakes
- Turning on JS rendering before testing the plain HTTP path
- Requesting too many formats at once in production
- Combining
cssSelector,xpath,includeTags, andexcludeTagsin the first attempt - Sending
formats: ["json"]without ajsonSchema - Sending Firecrawl's
screenshot,scrape,executeJavascriptorpdfaction types: onlywait,click,write,pressandscrollrun; for a capture of the final page, addscreenshottoformats