More API

Map

Discover the URLs a site exposes before you scrape or crawl it. Map is the lightest way to understand scope without paying the cost of a full multi-page extraction job.

Best for: site discovery
Returns: links only
Start with: sitemap on, low depth
View Crawl View Scrape
Try it in the Playground
Inspect reachability before you recurse
Use one root URL, keep maxDepth small, and inspect the discovered links. If the map is wrong, the crawl will be wrong too.
Open Playground Jump to Crawl

Mapping a site with CRW

/v1/map

POST /v1/map

Authentication:

  • Hosted: send Authorization: Bearer YOUR_API_KEY
  • Self-hosted: only required when auth.api_keys is configured

Installation

Map is also a plain HTTP route. No dedicated SDK is required.

Basic usage

Start with this request:

{
  "url": "https://example.com",
  "maxDepth": 1,
  "useSitemap": true
}

:::tabs ::tab{title="Python"}

import requests

resp = requests.post(
    "https://fastcrw.com/api/v1/map",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "url": "https://example.com",
        "maxDepth": 1,
        "useSitemap": True,
    },
)

print(resp.json()["data"]["links"])

::tab{title="Node.js"}

const resp = await fetch("https://fastcrw.com/api/v1/map", {
  method: "POST",
  headers: {
    "Authorization": "Bearer YOUR_API_KEY",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    url: "https://example.com",
    maxDepth: 1,
    useSitemap: true
  })
});

const body = await resp.json();
console.log(body.data.links);

::tab{title="cURL"}

curl -X POST https://fastcrw.com/api/v1/map \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "maxDepth": 1,
    "useSitemap": true
  }'

:::

Response

{
  "success": true,
  "data": {
    "links": [
      "https://example.com",
      "https://example.com/about"
    ]
  }
}

Parameters

Field Type Default Description
url string required Site or page URL to start discovery from
maxDepth number 2 Maximum discovery depth
useSitemap boolean true Read sitemap hints when available
timeout number server default Custom timeout in seconds

Sitemap behavior

With useSitemap: true, CRW uses sitemap hints when they are available. That usually makes the first discovery pass faster and more complete on structured sites.

Good default:

  • keep sitemap on,
  • keep depth low,
  • inspect the discovered links,
  • then decide whether crawl is worth it.

When map is better than crawl

Use map when:

  • you need to understand a site's shape before extracting any content,
  • you want a cheap first pass over a large site,
  • or you are deciding which section is worth crawling.

Use crawl only after you already trust the section you want to recurse through.

Common production patterns

  • Run map before crawl when you do not yet trust the start URL scope.
  • Keep maxDepth low first so you can inspect the discovered section.
  • Use sitemap hints when you want a faster first pass over structured sites.

Common mistakes

  • Using map when you already know the exact page and only need its content
  • Expecting map to return page bodies; it only returns discovered links
  • Letting depth grow before inspecting whether the discovered section is useful

When to use something else

  • Use Scrape when you want the content of a known page
  • Use Crawl when you are ready to recurse through a bounded section
  • Use Search when you do not know the site or page set yet