For audits, migrations and sitemaps

List a site's pages from its sitemaps and its links

Combine a site's sitemaps with the links on its pages, and see where each URL was found.

Get your free API key Try the playground

The problem

Before an audit, a migration or an ingestion job, you need the list of pages. The sitemap is out of date, the CMS export misses the landing pages, and the pages nobody links to are exactly the ones worth knowing about.

How domscout handles it

POST /map reads the sitemaps a site declares, including any named in robots.txt, follows links across its pages, and returns one deduplicated list with fragments and common tracking parameters removed. Each URL carries its source, sitemap or link, and its depth. To find the gaps, run it twice, once with includeSitemap: false and once with discoverLinks: false, and compare: a URL only the sitemap lists is a candidate for a page nothing links to, and one only the links reach is a page the sitemap forgot. robots.txt is always obeyed and no JavaScript runs.

Where it shows up

The URL inventory before a migration or a redesign.
Pages the sitemap misses, and sitemap entries nothing links to.
The list of pages to feed a batch, a crawl or an index.

The request

POST /map. /map starts at Pro and costs 2 credits a request. One request reads at most 100 pages, to a depth of 3, and returns at most 500 URLs; truncated is true when there were more to find.

cURL

curl -X POST https://api.domscout.io/map \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.example.com", "maxUrls": 500}'

Node.js

const response = await fetch("https://api.domscout.io/map", {
  method: "POST",
  headers: { "Content-Type": "application/json", "x-api-key": "YOUR_API_KEY" },
  body: JSON.stringify({ url: "https://www.example.com", maxUrls: 500 }),
});
const result = await response.json();
console.log(result.urls);

Python

import requests

response = requests.post(
    "https://api.domscout.io/map",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"url": "https://www.example.com", "maxUrls": 500},
)
print(response.json()["urls"])

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
)

func main() {
	body, _ := json.Marshal(map[string]any{"url": "https://www.example.com", "maxUrls": 500})
	req, _ := http.NewRequest("POST", "https://api.domscout.io/map", bytes.NewReader(body))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("x-api-key", "YOUR_API_KEY")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	var result map[string]any
	json.NewDecoder(res.Body).Decode(&result)
	fmt.Println(result["urls"])
}

PHP

<?php
$ch = curl_init('https://api.domscout.io/map');
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => ['Content-Type: application/json', 'x-api-key: YOUR_API_KEY'],
    CURLOPT_POSTFIELDS => json_encode(['url' => 'https://www.example.com', 'maxUrls' => 500]),
]);
$response = curl_exec($ch);
if ($response === false) {
    throw new RuntimeException(curl_error($ch));
}
$result = json_decode($response, true);
print_r($result['urls']);

Ruby

require "json"
require "net/http"

response = Net::HTTP.post(
  URI("https://api.domscout.io/map"),
  { url: "https://www.example.com", maxUrls: 500 }.to_json,
  "Content-Type" => "application/json",
  "x-api-key" => "YOUR_API_KEY"
)
puts JSON.parse(response.body)["urls"]

Java

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Capture {
    public static void main(String[] args) throws Exception {
        String body = """
            {"url": "https://www.example.com", "maxUrls": 500}
            """;
        HttpRequest request = HttpRequest.newBuilder(URI.create("https://api.domscout.io/map"))
            .header("Content-Type", "application/json")
            .header("x-api-key", "YOUR_API_KEY")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();
        HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println(response.body());
    }
}

using System.Net.Http.Json;
using System.Text.Json;

using var client = new HttpClient();
client.DefaultRequestHeaders.Add("x-api-key", "YOUR_API_KEY");

var response = await client.PostAsJsonAsync("https://api.domscout.io/map", new { url = "https://www.example.com", maxUrls = 500 });
var result = await response.Content.ReadFromJsonAsync<JsonElement>();
Console.WriteLine(result.GetProperty("urls"));

Go further

URL mapping docs

Website migration checks

Crawling a whole site

More in SEO and site health: SEO regression checks · Link preview checks · Website migration checks · JavaScript SEO checks · Page weight and load metrics · Backlink checks · JSON-LD structured data checks · hreflang checks · Website audits for prospects · All use cases

Give your product a browser.

Get clean web content and visual proof into your workflow in minutes.

Get your free API key Read the docs