NewIntroducing semantic snapshotsPair every capture with structured DOM data →
domscout

Free tools · no sign-up

Find all pages on a website

Enter a website and get a list of its pages, taken from its sitemaps and from the links between its pages, with each one marked by where it was found.

3 free runs an hour, shared with the other free tools. No account needed.

Four ways to see every page by hand

  1. Open the sitemap. Most sites publish a sitemap listing the pages they want found. Try /sitemap.xml first. A sitemap index links to further sitemaps, often one per section.https://example.com/sitemap.xml
  2. Read robots.txt. When the sitemap is not at /sitemap.xml, robots.txt often says where it is, on a line starting with Sitemap:.https://example.com/robots.txt
  3. Search Google with site:. The site: operator lists pages Google has indexed. The result count is a rough estimate, and pages Google has not indexed do not appear.site:example.com
  4. Use Search Console, if it is your site. The Pages report in Google Search Console lists every URL Google knows about, including the ones it chose not to index and why.

Each method sees a different set. Comparing the sitemap with the pages links reach is how you find the pages nothing links to, and the ones the sitemap forgot. See how to compare them.

The same list from your code

With an API key it is one POST to /map. Replace YOUR_API_KEY with a key from the dashboard.

cURL
curl -X POST https://api.domscout.io/map \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.example.com", "maxUrls": 500}'
Node.js
const response = await fetch("https://api.domscout.io/map", {
  method: "POST",
  headers: { "Content-Type": "application/json", "x-api-key": "YOUR_API_KEY" },
  body: JSON.stringify({ url: "https://www.example.com", maxUrls: 500 }),
});
const result = await response.json();
console.log(result.urls);
Python
import requests

response = requests.post(
    "https://api.domscout.io/map",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"url": "https://www.example.com", "maxUrls": 500},
)
print(response.json()["urls"])
Go
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
)

func main() {
	body, _ := json.Marshal(map[string]any{"url": "https://www.example.com", "maxUrls": 500})
	req, _ := http.NewRequest("POST", "https://api.domscout.io/map", bytes.NewReader(body))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("x-api-key", "YOUR_API_KEY")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	var result map[string]any
	json.NewDecoder(res.Body).Decode(&result)
	fmt.Println(result["urls"])
}
PHP
<?php
$ch = curl_init('https://api.domscout.io/map');
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => ['Content-Type: application/json', 'x-api-key: YOUR_API_KEY'],
    CURLOPT_POSTFIELDS => json_encode(['url' => 'https://www.example.com', 'maxUrls' => 500]),
]);
$response = curl_exec($ch);
if ($response === false) {
    throw new RuntimeException(curl_error($ch));
}
$result = json_decode($response, true);
print_r($result['urls']);
Ruby
require "json"
require "net/http"

response = Net::HTTP.post(
  URI("https://api.domscout.io/map"),
  { url: "https://www.example.com", maxUrls: 500 }.to_json,
  "Content-Type" => "application/json",
  "x-api-key" => "YOUR_API_KEY"
)
puts JSON.parse(response.body)["urls"]
Java
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Capture {
    public static void main(String[] args) throws Exception {
        String body = """
            {"url": "https://www.example.com", "maxUrls": 500}
            """;
        HttpRequest request = HttpRequest.newBuilder(URI.create("https://api.domscout.io/map"))
            .header("Content-Type", "application/json")
            .header("x-api-key", "YOUR_API_KEY")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();
        HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println(response.body());
    }
}
C#
using System.Net.Http.Json;
using System.Text.Json;

using var client = new HttpClient();
client.DefaultRequestHeaders.Add("x-api-key", "YOUR_API_KEY");

var response = await client.PostAsJsonAsync("https://api.domscout.io/map", new { url = "https://www.example.com", maxUrls = 500 });
var result = await response.Content.ReadFromJsonAsync<JsonElement>();
Console.WriteLine(result.GetProperty("urls"));

Questions

It reads the sitemaps the site declares, including any named in robots.txt, then follows links from the address you enter, up to two clicks deep, reading at most 25 pages. Duplicates, fragments and common tracking parameters are removed, and each URL is marked with where it was found.

More free tools

Website screenshot

Capture any public page as PNG, JPEG or WebP, at desktop or phone size.

URL to Markdown

Turn a web page into clean Markdown with a token estimate, ready to hand to an LLM.

Meta tag and Open Graph preview

See the title, description, og: tags and canonical URL a page publishes after its JavaScript runs.

JavaScript rendering checker

See whether a page's text is in the HTML the server sends or only appears after JavaScript runs.

llms.txt generator

Draft an llms.txt for a site from its homepage and sitemap, grouped into sections and ready to edit.

XML sitemap checker

Paste a sitemap and see which entries Google would refuse or ignore, and on which line.

Hreflang checker

Paste a page's HTML and check its hreflang codes, URLs, self-reference and canonical.

robots.txt tester

Paste a robots.txt and see whether a crawler may fetch a URL, and which rule decides it.

Give your product a browser.

Get clean web content and visual proof into your workflow in minutes.

Get your free API key Read the docs
Find All Pages on a Website - Free Page Finder | domscout