JS Rendering

crw supports JavaScript rendering for single-page applications (SPAs) and JS-heavy sites via the Chrome DevTools Protocol (CDP).

Rendering Modes

Set the rendering mode in your config:

[renderer]
mode = "auto"          # auto | lightpanda | playwright | chrome | none
page_timeout_ms = 30000
pool_size = 4

Per-request control

Override the rendering mode per request using renderJs:

{
  "url": "https://spa-app.com",
  "renderJs": true,
  "waitFor": 3000
}

Global default

To force JS rendering for every request that doesn't specify renderJs explicitly, set render_js_default in your config:

[renderer]
mode = "chrome"
render_js_default = true   # alias: force_js = true

# Or via environment variables
CRW_RENDERER__MODE=chrome
CRW_RENDERER__RENDER_JS_DEFAULT=true
# Backward-compat alias:
CRW_RENDERER__FORCE_JS=true

Precedence: a per-request renderJs always wins over the global default. Unset (null) on the request falls back to the default; if the default is also unset, the auto-detection heuristics below apply.

Per-request renderer override

When mode = "auto" and you have multiple renderers configured (e.g., LightPanda + Chrome), the auto-detect chain decides which one to use. Sometimes you already know that a specific site needs Chrome (Cloudflare-protected SPAs) or LightPanda (fast static-JS sites). Pin the renderer per request with the renderer field:

{
  "url": "https://x.com/elonmusk",
  "renderer": "chrome"
}

Pinned implies JS

A non-auto renderer value implies renderJs:true. If you set renderJs:false explicitly, the request stays HTTP-only and the pin is silently ignored — renderJs:false always wins. This means the availability check is also skipped when renderJs:false is set, so combinations like {"mode":"none","renderJs":false,"renderer":"chrome"} are accepted.

Errors and validation

If the named renderer isn't available in the server's pool, the request returns HTTP 400 immediately with errorCode: "invalid_request" and a message listing the configured renderers:

{
  "success": false,
  "error": "renderer 'chrome' not available; configured renderers: [lightpanda]. Update server config or omit the 'renderer' field.",
  "errorCode": "invalid_request"
}

For /v1/crawl, this validation runs once at job acceptance — bad combinations return 400 before the job is queued.

Pinning reduces resilience

Hard-pinning a renderer means transient failures of that renderer surface as errors instead of silently falling back to HTTP. If you need maximum resilience, omit renderer (or set it to auto) so the auto-detect chain can fall back. If you need determinism — "I know this site needs Chrome and a LightPanda success would be a wrong answer" — pin it.

For crawls, per-page failures of a pinned renderer are still logged and skipped; the rest of the crawl continues. Pages that fail will not appear in the results.

Per-request country (residential proxy tier)

When the operator configures a chrome_proxy tier backed by a residential proxy provider (e.g. DataImpulse), each request can route through a specific country by setting country in the request body:

{
  "url": "https://geo-restricted.example.com",
  "country": "us"
}

The country selector is composed at request time and supplied to Chrome via the CDP Fetch.authRequired event — no per-country container restart needed, and concurrent requests can target different countries without cross-contamination.

This field has no effect when the chrome_proxy tier is not configured. It is a renderer-tier routing hint, not a content filter — the engine does not translate or localize the response.

See Configuration — Residential proxy tier for the operator-side env vars (CRW_RENDERER__PROXY_BASE_USER, CRW_RENDERER__PROXY_BASE_PASS, CRW_RENDERER__PROXY_DEFAULT_COUNTRY).

When To Turn It On

Enable JS rendering when the page content is not present in the initial HTML response. Typical examples:

• single-page applications,
• pages that fetch content after hydration,
• and sites where the meaningful body is assembled client-side.

Do not enable it blindly for every request. HTTP-only fetches are faster and cheaper.

Choosing a waitFor Value

Start with the smallest value that works:

• 500 to 1000 for lightly hydrated pages,
• 2000 for typical JS-heavy pages,
• 3000 to 5000 only when you have confirmed the target hydrates slowly.

:::warning Long waits are not automatically safer. They increase latency and can hide the fact that the page is blocked rather than merely slow. :::

Auto-detection

When renderJs is null, crw fetches the page via HTTP first, then checks for SPA signals:

Triggered when body text < 200 chars and:

• Contains id="root", id="app", id="__next", id="__nuxt", id="__gatsby", id="svelte"
• Contains ng-app, data-reactroot
• Contains window.__initial_state__, __next_data__, window.__remixcontext, window.__astro

Triggered when:

• <noscript> contains "enable javascript"
• Body text < 500 chars and URL matches Framer, Webflow, Wix, or Squarespace domains

CDP Backends

LightPanda (Recommended)

Fastest option. Lightweight browser engine purpose-built for scraping.

# Auto-install
crw-server setup

# Manual start
lightpanda serve --host 127.0.0.1 --port 9222 &

[renderer.lightpanda]
ws_url = "ws://127.0.0.1:9222/"

Playwright

[renderer.playwright]
ws_url = "ws://playwright:9222"

Chrome / Chromium

[renderer.chrome]
ws_url = "ws://chrome:9222"

How CDP rendering works

1. Open a new browser tab via Target.createTarget
2. Navigate to the URL
3. Attach to the target via Target.attachToTarget
4. Wait for the configured time (waitFor or default 2000ms)
5. Execute Runtime.evaluate("document.documentElement.outerHTML") to get rendered HTML
6. Close the tab via Target.closeTarget

Cloud vs Self-hosted

• Cloud: JS rendering is always available. The managed infrastructure runs a LightPanda sidecar alongside the engine. Available on fastcrw.com (cloud).
• Self-hosted: You must run crw-server setup or configure a CDP browser (LightPanda, Chrome, or Playwright) in your config.toml under [renderer]. If no JS renderer is configured, requests with renderJs: true will fall back to HTTP-only fetching and include a warning.

What To Inspect in the Response

When rendered output looks wrong, check:

• metadata.renderedWith to verify a browser was actually used,
• metadata.elapsedMs to understand the cost of the request,
• and warning to catch anti-bot or fallback situations.

Troubleshooting

• Empty content from JS-heavy sites: Increase waitFor (e.g., 3000-5000). Some SPAs need extra time to hydrate.
• renderedWith: "http_only_fallback" in metadata: JS rendering was requested but no renderer is available. Check your deployment configuration.
• Internal error on renderJs: true: Verify the LightPanda sidecar is running and reachable. Check /health for renderer status.
• Still poor output after increasing waitFor: The issue may be anti-bot protection or authentication flow, not rendering delay.

Docker Compose

The included docker-compose.yml runs crw with a LightPanda sidecar:

services:
  crw:
    image: ghcr.io/us/crw:latest
    ports:
      - "3000:3000"
    environment:
      - CRW_RENDERER__LIGHTPANDA__WS_URL=ws://lightpanda:9222
  lightpanda:
    image: lightpanda/lightpanda:latest
    command: ["serve", "--host", "0.0.0.0", "--port", "9222"]