Extracting structured data with CRW

/v1/scrape

POST /v1/scrape

Authentication:

• Hosted: send Authorization: Bearer YOUR_API_KEY
• Self-hosted: only required when auth.api_keys is configured

Installation

Extraction uses the same HTTP route as scrape, so you can use the same client code and only change the payload.

Basic usage

Start with this request:

{
  "url": "https://example.com/product/123",
  "formats": ["json"],
  "jsonSchema": {
    "type": "object",
    "properties": {
      "title": { "type": "string" },
      "price": { "type": "string" },
      "availability": { "type": "string" }
    },
    "required": ["title"]
  }
}

import requests

resp = requests.post(
    "https://api.fastcrw.com/v1/scrape",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "url": "https://example.com/product/123",
        "formats": ["json"],
        "jsonSchema": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "price": {"type": "string"},
                "availability": {"type": "string"},
            },
            "required": ["title"],
        },
    },
)

print(resp.json()["data"]["json"])

const resp = await fetch("https://api.fastcrw.com/v1/scrape", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    url: "https://example.com/product/123",
    formats: ["json"],
    jsonSchema: {
      type: "object",
      properties: {
        title: { type: "string" },
        price: { type: "string" },
        availability: { type: "string" }
      },
      required: ["title"]
    }
  })
});

const body = await resp.json();
console.log(body.data.json);

curl -X POST https://api.fastcrw.com/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url":"https://example.com/product/123",
    "formats":["json"],
    "jsonSchema":{
      "type":"object",
      "properties":{
        "title":{"type":"string"},
        "price":{"type":"string"},
        "availability":{"type":"string"}
      },
      "required":["title"]
    }
  }'

Response

{
  "success": true,
  "data": {
    "json": {
      "title": "Widget",
      "price": "$19.99",
      "availability": "In stock"
    },
    "metadata": {
      "sourceURL": "https://example.com/product/123",
      "statusCode": 200,
      "elapsedMs": 846
    }
  }
}

Parameters

Schema design

The easiest extraction schema is a small one:

• ask for the fields you actually need,
• keep field types simple on the first pass,
• avoid optional fields until the required ones are stable.

Large schemas are harder to debug because you cannot tell whether the issue is the page, the schema, or both.

Extraction quality

Use this workflow:

1. scrape the page as markdown,
2. verify the target content is present,
3. add a minimal schema,
4. expand the schema only after the first JSON result looks right.

If the underlying page scrape is weak, the JSON extraction will also be weak.

Self-hosted LLM and provider control

Self-hosted: set [extraction.llm] in config.toml for a server-wide default, or pass llmApiKey + llmProvider + llmModel per request to override it. Supported providers: anthropic, openai, openai-responses, deepseek, azure, and openai-compatible. Use baseUrl for compatible Chat Completions or Responses endpoints.

Common production patterns

• Validate the target with markdown first, then add extraction.
• Keep the schema as small as possible on the first pass.
• Narrow the page with cssSelector only when the default extraction is too noisy.
• Use a per-request llmApiKey only when you need per-request provider separation.

Common mistakes

• Sending formats: ["json"] without a schema
• Designing a schema that assumes more structure than the page really contains
• Debugging extraction before confirming the underlying scrape succeeded
• Using the async /v1/extract job for a single URL when a synchronous /v1/scrape with formats: ["json"] is simpler

Multi-URL lifecycle and cancellation

POST /v1/extract creates one ordered result slot per requested URL. Poll GET /v1/extract/{id} or cancel idempotently with DELETE /v1/extract/{id}. The lifecycle is:

processing -> completed | failed
processing -> cancelling -> cancelled

cancelling is not terminal: the one URL already claimed by the sequential worker may finish and persist its result and measured usage. No later URL is started. At the terminal barrier, every untouched slot becomes cancelled and omits data, error, llmUsage, and basis. The result array always keeps the original URL count and order. Repeated DELETE returns the persisted state; DELETE after completed, failed, or cancelled does not rewrite it.

curl -X DELETE https://api.fastcrw.com/v1/extract/JOB_ID \
  -H "Authorization: Bearer YOUR_API_KEY"

The TypeScript SDK exposes startExtract, getExtract, and cancelExtract; Python exposes start_extract, get_extract, and cancel_extract. Their convenience extract waiter treats cancelling as non-terminal, raises a typed cancellation error with partial results on cancelled, and performs a best-effort DELETE on timeout. Async starters send Prefer: respond-async for managed and self-hosted parity.

When to use something else

• Use Scrape when prose or markdown is enough
• Use Map when you need discovery before extraction
• Use Crawl when you need schema-based extraction across many pages