Run work that outlives one HTTP request

Set async: true on POST /screenshot to force worker execution. You should also handle automatic 202 responses: large rich JSON, multiple whole-DOM passes, or slow combinations such as rich analysis plus fullPage can be promoted even without the flag.

curl -X POST "$DOMSCOUT_API_BASE/screenshot" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -H "Idempotency-Key: capture-UNIQUE_RUN_ID" \
  -d '{
    "url": "https://example.com/article",
    "async": true,
    "responseType": "json",
    "extractMarkdown": true,
    "save": true,
    "captureName": "article-snapshot",
    "retentionDays": 30
  }'

A batch is a parent job with one child job per request. Each item may use a URL or inline HTML and the normal screenshot options, but async, save, callbackUrl, and callbackSecret belong to the parent and are rejected inside an item.

curl -X POST "$DOMSCOUT_API_BASE/batch" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -H "Idempotency-Key: batch-products-UNIQUE_RUN_ID" \
  -d '{
    "requests": [
      { "url": "https://example.com/products/1", "responseType": "json", "extractMarkdown": true },
      { "url": "https://example.com/products/2", "responseType": "json", "extractMarkdown": true }
    ],
    "metadata": { "source": "catalog-import" },
    "callbackUrl": "https://YOUR_PUBLIC_HTTPS_ENDPOINT/domscout"
  }'

Crawls create one queued child per discovered page. They require seedUrl and a non-empty list of HTTPS allowedOrigins. The seed and every discovered URL must pass the service SSRF guard and the explicit allowlist.

curl -X POST "$DOMSCOUT_API_BASE/crawl" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -H "Idempotency-Key: docs-crawl-UNIQUE_RUN_ID" \
  -d '{
    "seedUrl": "https://docs.example.com/start",
    "allowedOrigins": ["https://docs.example.com"],
    "includePatterns": ["/docs/.*"],
    "excludePatterns": ["/docs/private/.*"],
    "maxDepth": 3,
    "maxPages": 100,
    "capture": {
      "format": "png",
      "extractMarkdown": true,
      "cleanup": true
    },
    "metadata": { "source": "docs-ingest" },
    "callbackUrl": "https://YOUR_PUBLIC_HTTPS_ENDPOINT/domscout"
  }'

Crawl patterns are bounded for safety. Catastrophic backtracking patterns such as nested quantifiers, overlapping repeated alternatives, or more than 40 quantifiers are rejected; a CPU budget can also stop admitting remaining candidates and report patternWarning.

async function waitForJob(pollUrl, apiKey) {
  for (;;) {
    const response = await fetch(pollUrl, { headers: { "x-api-key": apiKey } });
    const job = await response.json();
    if (["done", "error", "cancelled"].includes(job.status)) return job;
    await new Promise((resolve) => setTimeout(resolve, 2000));
  }
}

curl "$DOMSCOUT_API_BASE/job/JOB_ID/metadata" \
  -H "x-api-key: $DOMSCOUT_API_KEY"

curl -X DELETE "$DOMSCOUT_API_BASE/crawl/JOB_ID" \
  -H "x-api-key: $DOMSCOUT_API_KEY"

The metadata endpoint returns state, timing breakdowns, result summaries, and aggregate counts without echoing job input. It does not expose S3 keys or promise saved-artifact metadata. Use GET /capture/{captureId} with the owning account's API key for a retained saved capture's finite expiry or legacy permanent state and short-lived signed downloads. That route has no separate S3-key field, but its URL can contain storage-path information and is a secret capability. Cancelling an already-terminal owned job returns safe terminal metadata; unknown or foreign jobs return 404.

Batch and crawl requests may supply a public HTTPS callbackUrl. Legacy per-job callbacks remain supported. Callback destinations must pass private-address, metadata-address, DNS, and redirect checks at admission and delivery.

x-domscout-id: DELIVERY_ID
x-domscout-timestamp: ISO_8601_UTC
x-domscout-signature: HMAC_SHA256(secret, timestamp + "." + JSON_PAYLOAD)

curl -X POST "$DOMSCOUT_API_BASE/webhooks/deliveries/DELIVERY_ID/replay" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -H "Idempotency-Key: replay-UNIQUE_RUN_ID"

Only capture-producing work consumes quota: direct captures, started batch children, and started crawl children. A crawl child uses the same included-quota → prepaid-credit → permitted-overage waterfall as a direct capture; it has no separate crawl quota. Job reads, metadata reads, cancellation, webhook replay, GET /credits, and POST /feedback are control-plane operations. Credits and plans are separate: credits pay for captures but do not unlock plan capabilities.