For sites that want to be read by AI

Build llms.txt and llms-full.txt from the pages you already have

List your documentation's pages, turn each into Markdown, and assemble the two files AI assistants look for.

Get your free API key Try the playground

The problem

llms.txt is a short index of your important pages written for language models, and llms-full.txt is the content itself in Markdown. Written by hand, both are stale by the next release, and a generic HTML-to-text converter keeps your navigation menus and loses the structure that made the docs readable.

How domscout handles it

POST /map lists the URLs under your docs path, from the sitemap and from links. POST /scrape with renderJs: false then turns each page into Markdown from one fast HTTP request, keeping headings, lists, links and code blocks; tables come through as plain rows, so capture table-heavy pages with the browser instead. Check metadata.contentQuality and retry with the browser for any page that came back thin. Write each page's title and a one-line description into llms.txt, join the Markdown into llms-full.txt, and regenerate both on every docs deploy.

Where it shows up

llms.txt and llms-full.txt regenerated by the docs build.
A Markdown copy of each docs page, served next to the HTML.
The same Markdown fed into your own assistant's index.

The request

POST /scrape. /scrape is on every plan at 1 credit a page. /map, for the list of URLs, starts at Pro and costs 2 credits; if you already have a sitemap, you can read the URLs from it instead.

cURL

curl -X POST https://api.domscout.io/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com/getting-started", "renderJs": false}'

Node.js

const response = await fetch("https://api.domscout.io/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json", "x-api-key": "YOUR_API_KEY" },
  body: JSON.stringify({ url: "https://docs.example.com/getting-started", renderJs: false }),
});
const result = await response.json();
console.log(result.markdown);

Python

import requests

response = requests.post(
    "https://api.domscout.io/scrape",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={
        "url": "https://docs.example.com/getting-started",
        "renderJs": False
    },
)
print(response.json()["markdown"])

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
)

func main() {
	body, _ := json.Marshal(map[string]any{
		"url": "https://docs.example.com/getting-started",
		"renderJs": false,
	})
	req, _ := http.NewRequest("POST", "https://api.domscout.io/scrape", bytes.NewReader(body))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("x-api-key", "YOUR_API_KEY")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	var result map[string]any
	json.NewDecoder(res.Body).Decode(&result)
	fmt.Println(result["markdown"])
}

PHP

<?php
$ch = curl_init('https://api.domscout.io/scrape');
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => ['Content-Type: application/json', 'x-api-key: YOUR_API_KEY'],
    CURLOPT_POSTFIELDS => json_encode([
        'url' => 'https://docs.example.com/getting-started',
        'renderJs' => false
    ]),
]);
$response = curl_exec($ch);
if ($response === false) {
    throw new RuntimeException(curl_error($ch));
}
$result = json_decode($response, true);
print_r($result['markdown']);

Ruby

require "json"
require "net/http"

response = Net::HTTP.post(
  URI("https://api.domscout.io/scrape"),
  { url: "https://docs.example.com/getting-started", renderJs: false }.to_json,
  "Content-Type" => "application/json",
  "x-api-key" => "YOUR_API_KEY"
)
puts JSON.parse(response.body)["markdown"]

Java

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Capture {
    public static void main(String[] args) throws Exception {
        String body = """
            {
                "url": "https://docs.example.com/getting-started",
                "renderJs": false
            }
            """;
        HttpRequest request = HttpRequest.newBuilder(URI.create("https://api.domscout.io/scrape"))
            .header("Content-Type", "application/json")
            .header("x-api-key", "YOUR_API_KEY")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();
        HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println(response.body());
    }
}

using System.Net.Http.Json;
using System.Text.Json;

using var client = new HttpClient();
client.DefaultRequestHeaders.Add("x-api-key", "YOUR_API_KEY");

var response = await client.PostAsJsonAsync("https://api.domscout.io/scrape", new {
    url = "https://docs.example.com/getting-started",
    renderJs = false
});
var result = await response.Content.ReadFromJsonAsync<JsonElement>();
Console.WriteLine(result.GetProperty("markdown"));

Go further

Find every page on a site

Guide: web pages to Markdown for an LLM

Web access for AI agents

More in AI and agents: Web access for AI agents · Screenshots for vision models · Rendering generated HTML · Coding agents checking their work · Newsletter curation · FAQ drafts from documentation · All use cases

Give your product a browser.

Get clean web content and visual proof into your workflow in minutes.

Get your free API key Read the docs