NewIntroducing semantic snapshotsPair every capture with structured DOM data →
domscout

Crawl & batch

Durable jobs across an entire site

Fire a batch or a crawl, get a job ID, and let it run — bounded by hard limits, respectful of robots.txt, and reported through signed webhooks.

Get your free API key Read the API docs
Batch (cURL)
curl -X POST https://api.domscout.io/batch \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{ "requests": [{ "url": "https://example.com/p/1" }, { "url": "https://example.com/p/2" }] }'
Node.js
const start = await fetch("https://api.domscout.io/crawl", {
  method: "POST",
  headers: { "Content-Type": "application/json", "x-api-key": "YOUR_API_KEY" },
  body: JSON.stringify({
    seedUrl: "https://example.com/docs",
    allowedOrigins: ["https://example.com"],
    maxPages: 200,
    maxDepth: 3,
    callbackUrl: "https://hooks.example.com/domscout",
  }),
});
const { jobId } = await start.json();

// Poll, or just wait for the signed callback above.
const status = await fetch(`https://api.domscout.io/crawl/${jobId}`, {
  headers: { "x-api-key": "YOUR_API_KEY" },
});
console.log(await status.json());
Python
import requests

headers = {"Content-Type": "application/json", "x-api-key": "YOUR_API_KEY"}
job = requests.post("https://api.domscout.io/crawl", headers=headers, json={
    "seedUrl": "https://example.com/docs",
    "allowedOrigins": ["https://example.com"],
    "maxPages": 200,
    "maxDepth": 3,
    "callbackUrl": "https://hooks.example.com/domscout",
}).json()

status = requests.get(f"https://api.domscout.io/crawl/{job['jobId']}", headers=headers)
print(status.json())
Crawl body
{
  "seedUrl": "https://example.com/docs",
  "allowedOrigins": ["https://example.com"],
  "maxPages": 500,
  "maxDepth": 5,
  "capture": { "responseType": "json", "semanticSnapshot": { "detail": "summary" } },
  "callbackUrl": "https://hooks.example.com/domscout"
}

Plan access

Batch and crawl are Business+ workflows. The semantic snapshot in this sample is included with Pro+. A deployment-wide emergency pause is reported as FEATURE_NOT_ENABLED with a platform reason.

Crawl capabilities

Batches of up to 100 pages

Submit a list of URLs as one job and get per-child results as they complete. Quota is charged only when a child starts — never for creating the job.

Bounded, polite crawls

Crawls require an explicit HTTPS allowlist, always respect robots.txt with no bypass, and cap at 500 pages and depth 5. No unbounded traversal, ever.

Cooperative cancellation

Cancel a parent job and unstarted children stop cleanly; browser work already in flight is allowed to finish and report, so nothing is left mid-write.

Signed delivery, on schedule

Results arrive as signed webhooks with retries at 1, 5, 15, and 60 minutes. Miss a delivery? Replay it idempotently without paying for another capture.

Screenshot API

The same capture engine, one page at a time.

Semantic snapshots

Per-page Markdown; Pro+ accounts can add DOM intelligence.

Web monitors

Recurring captures for the pages you watch.

How crawls and batches run

Submit once, then either poll or be told. The job outlives the worker that started it, which is the only property that matters over a run long enough to be interrupted.

  1. Step 1

    Submit a batch or a crawl

    A batch takes a list of up to 100 URLs. A crawl takes a seed URL, an explicit HTTPS origin allowlist, and its own page and depth caps. Either way the response is immediate: a jobId, a pollUrl, and a status.

  2. Step 2

    The job survives the worker

    Children are processed durably, so a recycled worker, a timeout, or a deploy landing mid-run does not lose the pages already done. Quota is charged when a child starts, never when the parent job is created.

  3. Step 3

    Poll, or take the signed callback

    GET the poll URL for aggregate counts and per-child results with status, canonical URL, depth, timings, and errors. Or give the job a callbackUrl and the finished result is delivered as a signed webhook, retried at 1, 5, 15, and 60 minutes.

Crawl and batch questions, answered

Batch versus crawl, robots.txt, webhook delivery, cancellation, and plan access.

A batch is a list of URLs you already have - up to 100 per job, captured independently. A crawl starts from one seed URL and discovers pages by following links inside an origin allowlist you specify. Both are durable parent jobs with per-child results and the same webhook contract.

Give your product a browser.

Get clean web content and visual proof into your workflow in minutes.

Get your free API key Read the docs