For teams testing extraction and AI pipelines

Build a test set of real pages, each with its text and a picture

Crawl a site you have the right to use into a fixed set of real pages, each with its Markdown and a screenshot to check labels against.

Get your free API key Try the playground

The problem

An extraction prompt works on the five pages it was written against and fails on the sixth. Testing it properly takes a few hundred real pages, kept fixed so results compare from run to run, with a picture of each so a person can check what the right answer was.

How domscout handles it

A crawl from a seed URL, limited to the origins you allow, returns each page's Markdown and a screenshot from one durable job, and always obeys robots.txt. Store the results as your fixed set, label the right answer for each page with the screenshot beside it, and run every prompt, model or rule change against the same pages. Use sites you own or have permission to use this way.

Where it shows up

A regression set for an extraction prompt or schema.
A labelled sample for comparing models on your own content.
Fixed fixtures for a scraper's test suite.

The request

POST /crawl. Crawl starts at Business. A crawl stops at 500 pages and depth 5, needs an explicit HTTPS origin allowlist, and always obeys robots.txt. Each page costs what the same capture costs on its own, 1 credit here.

cURL

curl -X POST https://api.domscout.io/crawl \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "seedUrl": "https://docs.example.com",
    "allowedOrigins": ["https://docs.example.com"],
    "maxPages": 300,
    "capture": {"extractMarkdown": true, "format": "webp"}
  }'

Node.js

const response = await fetch("https://api.domscout.io/crawl", {
  method: "POST",
  headers: { "Content-Type": "application/json", "x-api-key": "YOUR_API_KEY" },
  body: JSON.stringify({
    seedUrl: "https://docs.example.com",
    allowedOrigins: ["https://docs.example.com"],
    maxPages: 300,
    capture: { extractMarkdown: true, format: "webp" }
  }),
});
console.log(await response.json());

Python

import requests

response = requests.post(
    "https://api.domscout.io/crawl",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={
        "seedUrl": "https://docs.example.com",
        "allowedOrigins": ["https://docs.example.com"],
        "maxPages": 300,
        "capture": {"extractMarkdown": True, "format": "webp"}
    },
)
print(response.json())

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
)

func main() {
	body, _ := json.Marshal(map[string]any{
		"seedUrl": "https://docs.example.com",
		"allowedOrigins": []any{"https://docs.example.com"},
		"maxPages": 300,
		"capture": map[string]any{"extractMarkdown": true, "format": "webp"},
	})
	req, _ := http.NewRequest("POST", "https://api.domscout.io/crawl", bytes.NewReader(body))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("x-api-key", "YOUR_API_KEY")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	raw, _ := io.ReadAll(res.Body)
	fmt.Println(string(raw))
}

PHP

<?php
$ch = curl_init('https://api.domscout.io/crawl');
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => ['Content-Type: application/json', 'x-api-key: YOUR_API_KEY'],
    CURLOPT_POSTFIELDS => json_encode([
        'seedUrl' => 'https://docs.example.com',
        'allowedOrigins' => ['https://docs.example.com'],
        'maxPages' => 300,
        'capture' => ['extractMarkdown' => true, 'format' => 'webp']
    ]),
]);
$response = curl_exec($ch);
if ($response === false) {
    throw new RuntimeException(curl_error($ch));
}
echo $response;

Ruby

require "json"
require "net/http"

response = Net::HTTP.post(
  URI("https://api.domscout.io/crawl"),
  {
    seedUrl: "https://docs.example.com",
    allowedOrigins: ["https://docs.example.com"],
    maxPages: 300,
    capture: { extractMarkdown: true, format: "webp" }
  }.to_json,
  "Content-Type" => "application/json",
  "x-api-key" => "YOUR_API_KEY"
)
puts response.body

Java

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Capture {
    public static void main(String[] args) throws Exception {
        String body = """
            {
                "seedUrl": "https://docs.example.com",
                "allowedOrigins": ["https://docs.example.com"],
                "maxPages": 300,
                "capture": {"extractMarkdown": true, "format": "webp"}
            }
            """;
        HttpRequest request = HttpRequest.newBuilder(URI.create("https://api.domscout.io/crawl"))
            .header("Content-Type", "application/json")
            .header("x-api-key", "YOUR_API_KEY")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();
        HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println(response.body());
    }
}

using System.Net.Http.Json;

using var client = new HttpClient();
client.DefaultRequestHeaders.Add("x-api-key", "YOUR_API_KEY");

var response = await client.PostAsJsonAsync("https://api.domscout.io/crawl", new {
    seedUrl = "https://docs.example.com",
    allowedOrigins = new[] { "https://docs.example.com" },
    maxPages = 300,
    capture = new { extractMarkdown = true, format = "webp" }
});
Console.WriteLine(await response.Content.ReadAsStringAsync());

Go further

Crawling a whole site

Crawl & batch

AI extraction to a JSON Schema

More in Data extraction and search: Crawling a whole site · Structured data from web pages · AI extraction to a JSON Schema · Semantic search over your content · Content behind clicks · Company research from websites · All use cases

Give your product a browser.

Get clean web content and visual proof into your workflow in minutes.

Get your free API key Read the docs