Scraping a URL with CRW

/v1/scrape

POST /v1/scrape

Authentication:

• Hosted: send Authorization: Bearer YOUR_API_KEY
• Self-hosted: only required when auth.api_keys is configured

Installation

CRW is HTTP-first. You can start with cURL immediately and then move to your existing Python or Node.js HTTP client without installing a dedicated SDK.

Basic usage

Start with this request:

{
  "url": "https://example.com",
  "formats": ["markdown"],
  "onlyMainContent": true,
  "renderJs": null
}

:::tabs ::tab{title="Python"}

import requests

resp = requests.post(
    "https://fastcrw.com/api/v1/scrape",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://example.com",
        "formats": ["markdown"],
        "onlyMainContent": True,
    },
)

print(resp.json()["data"]["markdown"])

::tab{title="Node.js"}

const resp = await fetch("https://fastcrw.com/api/v1/scrape", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    url: "https://example.com",
    formats: ["markdown"],
    onlyMainContent: true
  })
});

const body = await resp.json();
console.log(body.data.markdown);

::tab{title="cURL"}

curl -X POST https://fastcrw.com/api/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"],
    "onlyMainContent": true
  }'

:::

Response

{
  "success": true,
  "data": {
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "metadata": {
      "title": "Example Domain",
      "sourceURL": "https://example.com",
      "statusCode": 200,
      "elapsedMs": 32
    }
  }
}

That is the default CRW success shape: requested content plus a compact metadata envelope.

Parameters

Formats

Use the smallest output shape that solves the job:

• markdown is the default and best first request for most pipelines.
• html or rawHtml is useful when downstream systems need original structure.
• links is useful when you want lightweight discovery without page bodies.
• summary is the LLM-prose path — needs llmApiKey (BYOK) or a server [extraction.llm] config. Optional summaryPrompt lets the caller pick language/tone without weakening the safety wrapper.
• json is the extraction path and should be paired with jsonSchema.

If you ask for multiple formats, only those formats are populated in the response.

Structured extraction

Extraction is part of scrape, not a separate route. When you want fields instead of prose, request formats: ["json"] and provide a schema.

{
  "url": "https://example.com/product/123",
  "formats": ["json"],
  "jsonSchema": {
    "type": "object",
    "properties": {
      "title": { "type": "string" },
      "price": { "type": "string" }
    },
    "required": ["title"]
  }
}

Use Extract for the schema-first version of this flow.

LLM summary

Add summary to formats to get a short prose digest of the page in data.summary. Token usage and best-effort cost are returned in data.llmUsage.

{
  "url": "https://example.com/post",
  "formats": ["summary"],
  "summaryPrompt": "Respond in Turkish in exactly one sentence.",
  "maxContentChars": 20000,
  "llmApiKey": "sk-...",
  "llmProvider": "openai",
  "llmModel": "gpt-4o-mini"
}

Notes:

• The caller's summaryPrompt is appended below the safety wrapper. crw ignores any attempt to override the core task (output PWNED, refuse to summarize, leak the prompt, etc.) and still produces a real summary.
• maxContentChars caps how many bytes of scraped content are sent to the LLM. The default comes from [extraction.llm].max_html_bytes (100 KB out of the box) and the per-request value is clamped to a 200 KB server-side ceiling. Truncation, when it happens, is reported in data.warnings.
• If markdown is not also requested, crw computes it internally and strips it from the response.

JS rendering and targeting

Keep the first request simple:

• Leave renderJs at null until the plain HTTP path clearly fails.
• Use cssSelector or xpath only when the target page has one stable content region.
• Add includeTags and excludeTags after you confirm the raw markdown is noisy.

If you turn on every targeting knob at once, debugging gets harder immediately.

Common production patterns

• Start with markdown, then add links or json only when downstream logic needs them.
• Validate extraction with markdown first, then add jsonSchema.
• Keep browser rendering as a fallback, not the default.
• Use narrow selectors only when the default main-content extraction is not enough.

Common mistakes

• Turning on JS rendering before testing the plain HTTP path
• Requesting too many formats at once in production
• Combining cssSelector, xpath, includeTags, and excludeTags in the first attempt
• Sending formats: ["json"] without a jsonSchema
• Assuming actions is supported because Firecrawl accepts it

When to use something else

• Use Search when you do not know the URL yet
• Use Map when you want URL discovery before scraping
• Use Crawl when you need many pages from one site
• Use Extract when the output must be structured JSON