Searching the web with CRW

/v1/search

POST http://localhost:3000/v1/search          # self-hosted
POST https://api.fastcrw.com/v1/search        # hosted

Authentication:

• Self-hosted: no auth by default (add a reverse proxy / API key middleware if you expose it publicly)
• Hosted: send Authorization: Bearer YOUR_API_KEY

Installation

Like the rest of the CRW API, search is HTTP-first. Use cURL or your existing HTTP client.

Basic usage

Start with this request:

{
  "query": "web scraping tools",
  "limit": 5
}

import requests

# Self-hosted
resp = requests.post(
    "http://localhost:3000/v1/search",
    json={"query": "web scraping tools", "limit": 5},
)

# Or hosted (with API key)
# resp = requests.post(
#     "https://api.fastcrw.com/v1/search",
#     headers={"Authorization": "Bearer YOUR_API_KEY"},
#     json={"query": "web scraping tools", "limit": 5},
# )

# The results sit directly in `data` on the hosted API; a self-hosted engine
# nests them one level deeper as `data.results`. Read `results` when present,
# fall back to `data` — works on both hosts.
data = resp.json()["data"]
results = data["results"] if isinstance(data, dict) and "results" in data else data

# Each result row has: title, url, snippet, description, position, score.
# `snippet` is the LLM-ready summary line (Firecrawl-compatible name);
# `description` is the same value, kept as an alias.
for item in results:
    print(item["title"], item["url"], item["snippet"])

const resp = await fetch("http://localhost:3000/v1/search", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ query: "web scraping tools", limit: 5 })
});

const body = await resp.json();
// Hosted puts the results directly in `data`; a self-hosted engine nests them
// as `data.results`. Read `results` when present, fall back to `data`.
const { data } = body;
console.log(data?.results ?? data);

# Self-hosted (no auth)
curl -X POST http://localhost:3000/v1/search \
  -H "Content-Type: application/json" \
  -d '{"query": "web scraping tools", "limit": 5}'

# Hosted
curl -X POST https://api.fastcrw.com/v1/search \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "web scraping tools", "limit": 5}'

Response

{
  "success": true,
  "data": [
    {
      "url": "https://example.com/article",
      "title": "Article Title",
      "snippet": "A summary line from the search result...",
      "description": "A summary line from the search result...",
      "position": 1,
      "score": 9.5
    }
  ]
}

snippet is the LLM-ready summary line — name kept identical to Firecrawl so existing pipelines work unchanged. description is the same value under the search-backend-native name. Pick whichever; both are emitted.

That is the flat response shape used when sources is not set.

Parameters

scrapeOptions:

Search result types

Without sources, CRW returns a flat list:

{
  "success": true,
  "data": [
    {
      "url": "https://example.com/article",
      "title": "Article Title",
      "snippet": "Search summary line...",
      "description": "Search summary line...",
      "position": 1,
      "score": 9.5
    }
  ]
}

With sources, CRW returns grouped results:

{
  "success": true,
  "data": {
    "web": [{ "url": "...", "title": "...", "snippet": "...", "description": "..." }],
    "news": [{ "url": "...", "title": "...", "snippet": "...", "description": "...", "publishedDate": "2026-04-02T14:00:00" }],
    "images": [{ "url": "...", "imageUrl": "...", "thumbnailUrl": "..." }]
  }
}

Search with content scraping

When you need more than result snippets, add scrapeOptions:

{
  "query": "web scraping tools",
  "limit": 3,
  "scrapeOptions": {
    "formats": ["markdown"],
    "onlyMainContent": true
  }
}

That enriches eligible results with scraped page content. It is powerful, but it is also the moment search becomes more expensive, so keep it off until you need it.

LLM-assisted search

CRW can turn a search-with-scrape into either per-result summaries or a single synthesized answer (or both).

Per-result summaries (summarizeResults)

{
  "query": "what is tokio rust",
  "limit": 3,
  "scrapeOptions": { "formats": ["markdown"] },
  "summarizeResults": true,
  "summaryPrompt": "Respond in Turkish in one sentence per result.",
  "maxContentChars": 20000,
  "llmApiKey": "sk-...",
  "llmProvider": "openai",
  "llmModel": "gpt-4o-mini"
}

Each scraped result that produced markdown gets a result.summary field. Per-result failures attach a warning on the response but do not fail the whole request. Fan-out is bounded by [extraction.llm].max_concurrency (default 4).

Synthesized answer (answer)

{
  "query": "what is tokio rust",
  "limit": 3,
  "answer": true,
  "answerTopN": 3,
  "answerPrompt": "Respond in Turkish in exactly two sentences.",
  "scrapeOptions": { "formats": ["markdown"] },
  "llmApiKey": "sk-...",
  "llmProvider": "openai",
  "llmModel": "gpt-4o-mini"
}

A self-hosted engine carries these inside data; the hosted API puts answer, citations, llmUsage and warnings at the top level, next to the results. Fields with nothing to report are omitted, not emitted as null or []. Self-hosted:

{
  "success": true,
  "data": {
    "results": [ /* normal flat or grouped search results */ ],
    "answer": "Tokio is a Rust runtime…",
    "citations": [
      { "url": "https://...", "title": "...", "position": 0 }
    ],
    "llmUsage": { "inputTokens": 3420, "outputTokens": 96, "totalTokens": 3516, "estimatedCostUsd": 0.0008, "model": "gpt-4o-mini", "provider": "openai" },
    "warnings": ["answer synthesis unavailable"]
  }
}

Citation discipline:

• source_id returned by the model must map to a source actually in the input list. Fabricated ids are dropped.
• position is clamped to [0, sources.len()).
• The list is deduped on (source_id, position) and capped at 20 entries.

If the answer call fails (rate limit, network error, etc.), answer is null, any successful per-result summaries are still returned, and warnings explains what went wrong. CRW does not throw away partial work.

Caller-supplied directives

summaryPrompt and answerPrompt let you steer language/tone/format without weakening the safety wrapper:

• They are appended below the hardcoded system prompt, not in place of it.
• The wrapper explicitly tells the model to ignore directive contents that try to replace the task (fixed-string outputs, refusals, citation-skip, prompt leaks).
• Each directive is truncated to 500 chars server-side.

Where the key comes from

Same per-request key pattern as /v1/scrape: send llmApiKey / llmProvider / llmModel / baseUrl in the request body, or configure [extraction.llm] in config.toml.

Freshness, sources, and categories

• Use tbs when freshness matters more than broad recall.
• Use sources when you want different result groups such as web, news, or images.
• Use categories to narrow the query domain without rewriting the query itself.

Good default: add one narrowing control at a time so you can see which one actually improved the results.

Curated vs. passthrough categories

categories accepts two kinds of values, and you can mix them freely (up to 5 entries):

The curated names (github/research/pdf) are Firecrawl-compatible and behave exactly as before. Passthrough values are the additive part: CRW does not maintain its own engine list for them — it hands the category string to the search backend, which already knows the engine→category routing from its own settings.yml. That means new categories work without any CRW code or config change, and your self-hosted backend governs exactly which engines each category hits.

{
  "query": "crispr base editing",
  "categories": ["science"],
  "limit": 5
}

{
  "query": "rust async runtime",
  "categories": ["research", "it"]
}

In the second example, research still drives CRW's curated academic engines while it is forwarded to the search backend as a native category — both apply to the same query.

Backend query parameters CRW sends

For reference (and when debugging a self-hosted backend directly), this is how the public request fields map onto the /search query parameters CRW emits:

safesearch is available on the low-level crw search CLI but is not exposed on /v1/search.

Self-hosting the search sidecar

The default docker-compose.yml ships a hardened search container:

• Read-only root filesystem with sized tmpfs scratch
• All Linux capabilities dropped, no-new-privileges
• mem_limit, pids_limit set
• Pinned upstream image tag (we never run :latest)
• Config mounted read-only from config/searxng/settings.yml

It is mere-aggregation under AGPL — you are running an unmodified upstream image with config mounted at runtime, so no §13 corresponding-source obligations attach to the image itself. If you redistribute your CRW deployment publicly, AGPL §13 still requires you to offer the corresponding source of CRW (which is already on GitHub) to your users.

Common production patterns

• Start with search only, then add scrapeOptions after you verify result quality.
• Use sources: ["news"] or tbs when freshness matters more than broad recall.
• Use categories: ["github"] or ["research"] to narrow noisy queries.
• Keep limit low on the first pass so the result quality is easy to inspect.

Common mistakes

• Adding scrapeOptions to every search before you know you need page content
• Confusing sources with categories
• Treating qdr:h as truly hourly precision; the backend collapses it to day
• Sending plainText or json in scrapeOptions.formats — use /v1/scrape for those

When to use something else

• Use Scrape when you already know the exact URL
• Use Map for site-specific discovery
• Use Crawl when you need a bounded multi-page job on one site