Developer guide · API v1

From a URL to usable data.

Learn the shortest path from your first page scrape to a bounded site crawl. Start with Markdown, add selector-based fields when you need structured data, then move to durable jobs when the work gets bigger.

Start in 3 minutes Open playground

cURL, Node, Python JSON-first responses robots-aware crawling

Request anatomyPOST /screenshot

Authenticate

x-api-key header

Choose a source

url or inline html

Choose an output

markdown, fields, image, or job

response → markdown + analysis + metadata

Before you run: the public production base is https://api.domscout.io; create a key in the dashboard and keep it server-side. Browser execution can also be disabled by a platform safety hold; if the API returns BROWSER_ISOLATION_REQUIRED, treat that 403 as terminal instead of retrying.

01 · Quickstart

Make your first scrape

You do not need an SDK. Create an API key, set the base URL, send JSON to POST /screenshot, and read the returned markdown. The same request works from cURL, Node.js, or Python.

Create a key

Use the dashboard's API Keys page. Keep the key server-side and send it only as x-api-key.

Continue

Set your base

Save the public production base and key in environment variables so they never land in source control.

Continue

Send JSON

Ask for JSON when you want Markdown, fields, diagnostics, or action results in the response.

Continue

Keep credentials out of code

Use local environment variables or your deployment secret manager. The API key belongs in the request header, never in the URL or a committed example.

environment

export DOMSCOUT_API_BASE="https://api.domscout.io"
export DOMSCOUT_API_KEY="YOUR_API_KEY"

cURL

curl -X POST "$DOMSCOUT_API_BASE/screenshot" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -d '{
    "url": "https://example.com/article",
    "responseType": "json",
    "extractMarkdown": true
  }'

Node.js

const response = await fetch(`${process.env.DOMSCOUT_API_BASE}/screenshot`, {
  method: "POST",
  headers: { "content-type": "application/json", "x-api-key": process.env.DOMSCOUT_API_KEY },
  body: JSON.stringify({
    url: "https://example.com/article",
    responseType: "json",
    extractMarkdown: true,
  }),
});
const data = await response.json();
console.log(data.markdown ?? data.pollUrl);

Python

import os
import requests

response = requests.post(
    f"{os.environ['DOMSCOUT_API_BASE']}/screenshot",
    headers={"x-api-key": os.environ["DOMSCOUT_API_KEY"]},
    json={"url": "https://example.com/article", "responseType": "json", "extractMarkdown": True},
)
data = response.json()
print(data.get("markdown") or data.get("pollUrl"))

Choose a workflow

Four ways to use domscout

Pick the smallest workflow that matches the job. A one-page scrape is the fastest way to get clean content; a map helps you understand a site before you spend normal capture quota; a crawl is for repeatable multi-page collection.

Scrape one page

Render a page and return agent-readable Markdown, a screenshot, or both. Best for articles, product pages, and research.

Read the guide

Extract fields

Use CSS selectors to return predictable text, numbers, URLs, attributes, HTML, booleans, or lists.

Read the guide

Map a site

Discover sitemap and same-site links first. Mapping is bounded and does not run page JavaScript.

Read the guide

Crawl a site

Queue a bounded crawl with an explicit origin allowlist, filters, Markdown output, and durable progress.

Read the guide

02 · Scrape

Scrape a page into Markdown

POST /scrape returns a page as readable Markdown with its title, links, and metadata — useful for RAG, summarization, search indexing, and downstream agents. It runs the same engine as POST /screenshot and costs the same, but it extracts Markdown by default and leaves the image out of the response unless you ask for it with screenshot: true. Add cleanup when consent banners, ads, or sticky chrome are getting in the way.

Cleanup availability: Cleanup and other DOM intelligence are included on Pro+ plans. A deployment-wide emergency pause returns 403 FEATURE_NOT_ENABLED with a platform reason; it is not cleared by upgrading or retrying.

POST /scrape

curl -X POST "$DOMSCOUT_API_BASE/scrape" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -d '{
    "url": "https://example.com/products/42"
  }'

Read the response

The response has status: "success" with url, title, markdown, links, and metadata at the top level. If the request becomes durable, the API returns 202 with a jobId and pollUrl instead.

Good defaultStart with Markdown. Ask for a screenshot or rich DOM analysis only when your workflow needs it; richer output can be larger and may move to a durable job automatically.

Fast mode — no browser

curl -X POST "$DOMSCOUT_API_BASE/scrape" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -d '{
    "url": "https://docs.example.com/guide",
    "renderJs": false
  }'

Skip the browser

Send renderJs: false (or fast: true) to answer from a single guarded HTTP request instead of a browser. Same response shape, same credit cost, typically under 200 ms. Ideal for documentation, blogs, news, and changelogs.

Fast mode does not run JavaScript. A client-rendered page returns whatever its server-side HTML holds, which can be an empty shell. Every response reports metadata.renderJs and metadata.contentQuality — read them before concluding a page is empty, and retry with renderJs: true when they say the page needed a browser.

03 · Structured data

Extract fields without an LLM

Add extract.fields to the same screenshot request. Each field names a selector and a type. The result is deterministic and reports whether every field was found, missing, or invalid. Set strict: true when a missing required field should fail the request.

selector-based extraction

{
  "url": "https://example.com/products/42",
  "responseType": "json",
  "extract": {
    "strict": false,
    "fields": {
      "name": { "selector": "h1", "type": "text", "required": true },
      "price": { "selector": "[data-price]", "type": "number" },
      "image": { "selector": "img.product", "type": "attribute", "attribute": "src" },
      "tags": { "selector": ".tag", "type": "list", "all": true }
    }
  }
}

Supported field types

textnumberbooleanattributehtmlurllist

Where to look

Fields are returned under analysis.extraction.fields. Set all: true for multiple matches; list fields always return every match. If selectors are unstable, use semantic snapshots to inspect the rendered DOM and selector candidates first.

04 · Discover

Map before you crawl

Use POST /map when you want a URL inventory first. It combines sitemap locations and same-site links, applies path filters, removes fragments and common tracking parameters, and returns a source for each URL. It is discovery only—it does not execute JavaScript.

POST /map

curl -X POST "$DOMSCOUT_API_BASE/map" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -d '{
    "url": "https://docs.example.com",
    "maxPages": 25,
    "maxDepth": 2,
    "includePaths": ["/docs/"],
    "discoverLinks": true,
    "includeSitemap": true
  }'

Use the map to

Preview how many URLs are discoverable before a crawl.
Keep only a documentation or product path with includePaths.
Feed a curated URL list into POST /batch instead of crawling everything.
Check counts.urls and truncated before choosing larger limits.

Map limits: up to 100 fetched pages, depth 3, and 500 returned URLs. Robots rules still apply.

Map, LLM extraction, search, and semantic indexing are Pro-or-higher routes. Plan, provider, and platform checks are evaluated before work starts.

05 · Crawl

Crawl a site safely and predictably

A crawl is a durable parent job with one child job per page. It requires an HTTPS seedUrl and at least one HTTPS allowedOrigins entry. Set page capture options under capture; do not put url, async, or callbacks inside that template. Crawls always respect robots.txt—sending respectRobots: false is rejected. Every started child follows the normal included-quota → prepaid-credit → permitted-overage waterfall; there is no separate crawl quota, and the parent terminalizes promptly when its created children are terminal.

POST /crawl

curl -X POST "$DOMSCOUT_API_BASE/crawl" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -H "Idempotency-Key: YOUR_UNIQUE_RUN_ID" \
  -d '{
    "seedUrl": "https://docs.example.com/start",
    "allowedOrigins": ["https://docs.example.com"],
    "includePatterns": ["/docs/.*"],
    "excludePatterns": ["/docs/private/.*"],
    "maxDepth": 2,
    "maxPages": 50,
    "capture": { "extractMarkdown": true, "format": "png" }
  }'

Allowlist

Every page must remain inside an explicitly allowed HTTPS origin.

Bound it

Use maxPages, maxDepth, includePatterns, and excludePatterns to define the frontier.

Make it safe to retry

Send a unique Idempotency-Key so a network retry does not create duplicate work.

06 · Durable work

Handle jobs, polling, and cancellation

Always support both outcomes from a capture request: 200 for an inline result and 202 for accepted durable work. Batch and crawl submissions return a parent jobId; poll the returned pollUrl until the status is done, error, or cancelled.

pending

Accepted but not started.

processing

A worker is running the capture or child page.

retrying

The workflow is retrying a recoverable failure.

done

Read result for a capture, or items/counts for a parent job.

error / cancelled

Stop polling and inspect error or cancellation metadata.

pollUrl

const job = await response.json();
let status = job;

while (["pending", "processing", "retrying"].includes(status.status)) {
  await new Promise((resolve) => setTimeout(resolve, 2000));
  const next = await fetch(job.pollUrl, { headers: { "x-api-key": process.env.DOMSCOUT_API_KEY } });
  status = await next.json();
}
if (status.status === "done") console.log(status.result ?? status.items);

Cancellation is cooperative. Deleting a job stops unstarted work; a page already in the browser may finish and report its result. Polling endpoints and cancellation are not capture-producing calls, so they do not consume capture quota.

07 · Provider routes

Use AI extraction and semantic indexing

These synchronous routes do not launch Chromium and require Pro or higher. They are useful after a scrape: use /extract for schema-validated JSON, and /index plus /semantic-search for an account-scoped retrieval layer.

Web search is not currently offered. POST /search is withheld: no provider is configured, and it answers 503 SEARCH_NOT_CONFIGURED on every call regardless of plan. Do not build against it. Separately, a Pro plan permits the routes below but does not make an upstream provider healthy — the August 11 audit found /extract timing out, so check capabilities from GET /credits before depending on it.

Schema-validated extraction

Send one of url, content, or html, plus a prompt and object-shaped JSON Schema. The configured model must return JSON that passes validation; invalid model output is a 502, not silently accepted data. If a URL request reaches the 18-second intelligence deadline, retry with a bounded content or HTML excerpt rather than fewer pages.

POST /extract

{
  "content": "Product: Example, price: 42",
  "prompt": "Extract the product name and price.",
  "schema": {
    "type": "object",
    "required": ["name", "price"],
    "properties": {
      "name": { "type": "string" },
      "price": { "type": "number" }
    }
  }
}

Search and scrape

Withheld

POST /search is not currently offered. No search provider is configured, so every call answers 503 SEARCH_NOT_CONFIGURED on any plan, and GET /credits reports it unavailable. It is listed here because the route exists and the contract describes it — not because it can be used. No sample is shown, deliberately.

Index content

Index a URL, text content, or HTML. domscout chunks it with bounded overlap, creates embeddings, and stores the document in your account-scoped semantic index. Add a title and metadata to make results easier to use.

POST /index

{
  "content": "Markdown from a prior screenshot...",
  "sourceUrl": "https://example.com/article",
  "title": "Example article",
  "metadata": { "section": "research" },
  "chunkSize": 4000,
  "chunkOverlap": 400,
  "maxChunks": 500
}

Search your index

Query only documents owned by the authenticated account. Results include content, source URL, title, chunk index, metadata, and cosine similarity. Limit results to 1–20 or scope to a document ID.

POST /semantic-search

{
  "query": "How does cancellation work?",
  "limit": 10
}

JavaScript-rendered contentThe provider /extract route fetches URL text directly. For JavaScript-rendered pages, first call /screenshot with extractMarkdown: true, then send the returned Markdown as content.

08 · Account & support

Budget requests and report problems

GET /credits is the safest way to price a request before running it. POST /feedback is a non-billable channel for documentation gaps, errors, and feature requests.

GET /credits

curl "$DOMSCOUT_API_BASE/credits" \
  -H "x-api-key: $DOMSCOUT_API_KEY"

POST /feedback

curl -X POST "$DOMSCOUT_API_BASE/feedback" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -d '{
    "type": "docs_gap",
    "severity": "minor",
    "message": "The example did not explain how to poll a crawl.",
    "endpoint": "POST /crawl"
  }'

GET /credits returns prepaid balance, month-to-date spend, plan quota, overage ceiling, rate limit, and the current surcharge table. It consumes no credits or quota. Feedback requires a non-empty message, is never billed, and can include requestId so the server attaches the original request record.

Help us make this better

This endpoint is the fastest route into the backlog, and it is free on every plan, including the free one. A report with the four things below can usually be reproduced the same day; one without them usually cannot be reproduced at all.

requestId
The request id, whenever there was one
Every response carries X-Request-Id. Send it and the server attaches its own record of that request — status, error, latency, format, the target URL — so you do not have to restate any of it, and we are looking at the same capture you were.
expected / actual
What you expected, and what you got
Two short sentences. Most reports describe only the second, and the difference between “the Markdown was wrong” and “I expected the pricing table as a Markdown table; I got the prices as separate paragraphs” is the difference between a triage question and a fix.
severity
Whether you were blocked
“blocking” means you could not complete the task at all, and it is read differently from the rest. Using it accurately is what keeps it useful — for you and for everyone whose blocking report is queued behind it.
type
What kind of thing it is
bug, error_report, docs_gap, feature_request, praise, or other. docs_gap is the one that is chronically under-used: if a page on this site did not answer your question, that is a defect in the page and we would like to know which page and which question.

You do not need permission to send one, and there is no wrong volume. If something is slower, uglier or more surprising than you expected, that is worth a report even when nothing failed — those are the reports that change the product, and the ones we get fewest of.

What a request costs. A viewport capture is 1 credit; options that force another render or another whole-DOM pass add to it. **No single request exceeds 10 credits**, so you can bound your worst case before sending.
Request feature	Credits
base capture (png/jpeg/webp, viewport), or probe: true	1
fullPage or lazyScroll	+1
format: "pdf"	+1
captureMode: "skeleton" (two renders)	+1
captureMode: "grid" (three renders)	+2
semanticSnapshot / diagnostics / extract / cleanup	+1 each
injectJS / captureTimeline	+2 each
more than 10 actions	+1
POST /map	+1
POST /extract	+3
POST /index	+2
POST /semantic-search	+1
maximum for any single request	10

Route surcharges are additions to the capture they ride on, so POST /index with a URL costs 3 and each POST /semantic-search query costs 2 — worth pricing into any search feature you expose to end users. extractMarkdown, semanticNodes,simplifyDom and render add nothing. Submitting a batch or a crawl is free; the children are charged individually. Every response carries the actual figure inX-Domscout-Credits-Cost.

Check availability before you spend: the same response carries a capabilities report saying which gated routes and capture options this key can actually use, and why not when it cannot. reason: "plan" is cleared by an upgrade; "account_rollout" (trusted JavaScript), "deployment", and "platform" (including the product emergency pause) are not. It reports configuration, not upstream health.

Reference map

Find the right endpoint

The OpenAPI document is the source of truth for every field, default, enum, response, plan gate, and limit. Use this table to orient yourself, then open the contract for exact wire shapes. For non-JavaScript readers, the full text reference includes every endpoint and parameter.

Open contract

Capture & extract

POST/screenshot

Render a URL or HTML and optionally return Markdown, fields, DOM analysis, or a binary artifact.

GET/capture/{captureId}

Read an owned retained saved capture's finite expiry or legacy permanent state and short-lived signed downloads; absent, foreign, and finite-expired IDs all return 404.

POST/extract

Turn a URL, HTML, or text into JSON validated against your schema.

POST/search

Withheld — no provider is configured; answers 503 SEARCH_NOT_CONFIGURED.

Discover & crawl

POST/map

Discover bounded URLs from sitemaps and page links; it does not execute JavaScript.

POST/crawl

Queue a bounded, robots-aware crawl using an explicit HTTPS origin allowlist.

GET / DELETE/crawl/{jobId}

Read progress and results, or cooperatively cancel a crawl.

Durable work

POST/batch

Submit up to 100 independent capture requests as one parent workflow.

GET / DELETE/batch/{jobId}

Read child progress and results, or cooperatively cancel a batch.

GET / DELETE/job/{jobId}

Read a job or request cancellation.

GET/job/{jobId}/metadata

Read timing, safe result summaries, and counts without rich results or saved-artifact descriptors.

POST/webhooks/deliveries/{deliveryId}/replay

Replay a stored signed delivery without launching a browser.

Account & support

GET/credits

Read quota, prepaid balance, rate limits, and the live credit table; never billed.

POST/index

Chunk, embed, and store account-owned content for semantic retrieval.

POST/semantic-search

Query your account-owned semantic index.

POST/feedback

Report a problem or documentation gap; never billed.

Boundaries

Know the limits before you scale

These are the practical boundaries most first integrations hit. The contract remains authoritative, and plan gates can restrict capabilities even when a field is valid.

Area	Default boundary
One-page URL	HTTPS public URL, up to 4,096 characters
Batch	1–100 capture requests
Crawl	1–500 pages, depth 0–5
Map	Up to 100 fetched pages, depth 0–3, 500 returned URLs
Actions	Up to 25 actions per capture
Robots	Always respected for map, search scraping, and crawls

Scraping checklist

Use HTTPS targets and send authentication in scoped headers or cookies, never in a URL.
Start with a small maxPages and maxDepth, then increase only when the result is useful.
Keep robots.txt compliance on; domscout does not provide a bypass.
Use an Idempotency-Key for durable submissions and handle 202 before parsing a result.
Never log or commit API keys, cookies, or injected JavaScript containing secrets.

Keep going

Ready to inspect a real page?

Use the playground to build a request visually, see the JSON response, and copy the generated cURL, Node.js, or Python integration.

Open playground

Rich capture Automation Jobs & webhooks OpenAPI