How do I tell if a page is JavaScript-rendered?

Curl it and inspect the body. If the text you see in the browser is missing from curl -s URL | grep, the page is client-rendered and needs a headless browser.

Why not always use a headless browser?

Browser rendering is roughly an order of magnitude more expensive in CPU and RAM than plain HTTP. Auto-detecting per page keeps cost down — over half of public pages still render server-side and do not need a browser at all.

What does waitFor actually wait for?

It is a millisecond budget after navigation, applied only to browser renderers. Use it when content arrives via a delayed XHR that you cannot bind to a specific selector.

How do you scrape JavaScript-rendered pages?

Short answer: JavaScript-rendered pages send an almost-empty HTML shell that the browser fills in by executing scripts and fetching JSON. A plain HTTP scraper sees the shell and gives up. To scrape the real content you need a headless browser that runs the scripts, waits for the DOM to settle, then serialises the final HTML.

Why plain HTTP scrapers fail

Single-page applications (React, Vue, Svelte, Next.js client routes) ship a minimal HTML document — often just a root div and a bundle script. The visible text only appears after the JavaScript runs in a browser. A request library like curl or Python requests never executes that script, so it captures the empty shell. The same failure mode happens with infinite-scroll listings, modal-loaded detail panes, and client-side filters.

The headless-browser approach

A correct JS-aware scraper does this:

Launches a headless browser (Chromium, Firefox, or a leaner alternative).
Navigates to the URL and waits for either networkidle, a DOM selector, or a fixed millisecond budget.
Optionally executes user-provided JavaScript (click "load more", scroll, dismiss a cookie banner).
Serialises document.documentElement.outerHTML as the rendered HTML.
Feeds the rendered HTML through a Markdown converter and link extractor.

The cost is RAM and CPU per page. A full Chromium instance is typically several hundred MB resident — this is why headless-browser-first scrapers are expensive at scale.

How fastCRW does it

fastCRW auto-detects whether a page needs a browser. Static pages go through the HTTP fast path; pages that require JavaScript route to a headless renderer. Instead of full Chromium, the default browser path is LightPanda, a Rust-native headless browser, keeping the whole binary at ~50 MB RAM idle. Full Chromium is still available when a page needs it. Set "renderer": "lightpanda" or "chromium" on POST /v1/scrape — see the scrape endpoint reference for the request body. waitFor takes a millisecond budget for browser renderers.

Frequently asked

How do I tell if a page is JavaScript-rendered?: Curl it and inspect the body. If the text you see in the browser is missing from curl -s URL | grep, the page is client-rendered and needs a headless browser.
Why not always use a headless browser?: Browser rendering is roughly an order of magnitude more expensive in CPU and RAM than plain HTTP. Auto-detecting per page keeps cost down — over half of public pages still render server-side and do not need a browser at all.
What does waitFor actually wait for?: It is a millisecond budget after navigation, applied only to browser renderers. Use it when content arrives via a delayed XHR that you cannot bind to a specific selector.

← CRW docs home