Scraping a URL with CRW

/v1/scrape

POST /v1/scrape

Authentication:

• Hosted: send Authorization: Bearer YOUR_API_KEY
• Self-hosted: only required when auth.api_keys is configured

Installation

CRW is HTTP-first. You can start with cURL immediately and then move to your existing Python or Node.js HTTP client without installing a dedicated SDK. See the Basic usage tabs below for ready-to-run Python (requests), Node.js (fetch), and cURL examples.

Basic usage

Start with this request:

{
  "url": "https://example.com",
  "formats": ["markdown"],
  "onlyMainContent": true,
  "renderJs": null
}

import requests

resp = requests.post(
    "https://api.fastcrw.com/v1/scrape",
    headers={
        "Authorization": "Bearer YOUR_API_KEY",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://example.com",
        "formats": ["markdown"],
        "onlyMainContent": True,
    },
)

print(resp.json()["data"]["markdown"])

const resp = await fetch("https://api.fastcrw.com/v1/scrape", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    url: "https://example.com",
    formats: ["markdown"],
    onlyMainContent: true
  })
});

const body = await resp.json();
console.log(body.data.markdown);

curl -X POST https://api.fastcrw.com/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown"],
    "onlyMainContent": true
  }'

Response

{
  "success": true,
  "data": {
    "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
    "metadata": {
      "title": "Example Domain",
      "sourceURL": "https://example.com",
      "statusCode": 200,
      "elapsedMs": 32
    }
  }
}

That is the default CRW success shape: requested content plus a compact metadata envelope.

Parameters

Formats

Use the smallest output shape that solves the job:

• markdown is the default and best first request for most pipelines.
• html or rawHtml is useful when downstream systems need original structure.
• links is useful when you want lightweight discovery without page bodies.
• summary is the LLM-prose path — needs llmApiKey or a server [extraction.llm] config. Optional summaryPrompt lets the caller pick language/tone without weakening the safety wrapper.
• json is the extraction path and should be paired with jsonSchema.

If you ask for multiple formats, only those formats are populated in the response.

Structured extraction

For a single page, extraction is part of scrape: when you want fields instead of prose, request formats: ["json"] and provide a schema. (For structured extraction across many URLs in one job, use the native async POST /v1/extract.)

{
  "url": "https://example.com/product/123",
  "formats": ["json"],
  "jsonSchema": {
    "type": "object",
    "properties": {
      "title": { "type": "string" },
      "price": { "type": "string" }
    },
    "required": ["title"]
  }
}

Use Extract for the schema-first version of this flow.

LLM summary

Add summary to formats to get a short prose digest of the page in data.summary. Token usage and best-effort cost are returned in data.llmUsage.

{
  "url": "https://example.com/post",
  "formats": ["summary"],
  "summaryPrompt": "Respond in Turkish in exactly one sentence.",
  "maxContentChars": 20000,
  "llmApiKey": "sk-...",
  "llmProvider": "openai",
  "llmModel": "gpt-4o-mini"
}

Notes:

• The caller's summaryPrompt is appended below the safety wrapper. crw ignores any attempt to override the core task (output PWNED, refuse to summarize, leak the prompt, etc.) and still produces a real summary.
• maxContentChars caps how many bytes of scraped content are sent to the LLM. The default comes from [extraction.llm].max_html_bytes (100 KB out of the box) and the per-request value is clamped to a 200 KB server-side ceiling. Truncation, when it happens, is reported in data.warnings.
• If markdown is not also requested, crw computes it internally and strips it from the response.

JS rendering and targeting

Keep the first request simple:

• Leave renderJs at null until the plain HTTP path clearly fails.
• Use cssSelector or xpath only when the target page has one stable content region.
• Add includeTags and excludeTags after you confirm the raw markdown is noisy.

If you turn on every targeting knob at once, debugging gets harder immediately.

Common production patterns

• Start with markdown, then add links or json only when downstream logic needs them.
• Validate extraction with markdown first, then add jsonSchema.
• Keep browser rendering as a fallback, not the default.
• Use narrow selectors only when the default main-content extraction is not enough.

Common mistakes

• Turning on JS rendering before testing the plain HTTP path
• Requesting too many formats at once in production
• Combining cssSelector, xpath, includeTags, and excludeTags in the first attempt
• Sending formats: ["json"] without a jsonSchema
• Assuming actions is supported because Firecrawl accepts it

When to use something else

• Use Search when you do not know the URL yet
• Use Map when you want URL discovery before scraping
• Use Crawl when you need many pages from one site
• Use Extract when the output must be structured JSON