For teams that ship websites

Catch a noindex or a wrong canonical before search engines do

Read the head tags a search engine reads and fail the build when a deploy ships a noindex or a wrong canonical.

Get your free API key Try the playground

The problem

The worst SEO bugs are silent. A staging flag ships a noindex to production, a canonical keeps pointing at the preview domain, a template change drops the meta description from every product page. The page still renders, still returns 200, still looks right in a browser, and almost nothing in a normal test suite looks inside the head.

How domscout handles it

One request reads the tags you name by CSS selector from the rendered page and reports each as found or missing, so an absent tag cannot pass for an empty one. To see the head as your server sends it, before any JavaScript runs, a renderJs: false request returns the served title, description, Open Graph tags and canonical URL; named fields like robots need the rendered page. Run the check in CI against a preview URL and fail the build on a problem.

Where it shows up

A post-deploy check on the pages that carry your organic traffic.
A pre-merge check against preview deployments.
A scheduled comparison of the served and the rendered title, description and canonical.

The request

POST /scrape. extract starts at Pro and adds 1 credit to the capture, so each page checked costs 2. Monitors can collect these fields on a schedule, but their alerts compare the title, status code, text, links and DOM, not extracted fields.

cURL

curl -X POST https://api.domscout.io/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.example.com/",
    "extract": {
      "fields": {
        "canonical": {
          "selector": "head link[rel='\''canonical'\'']",
          "type": "attribute",
          "attribute": "href"
        },
        "robots": {
          "selector": "head meta[name='\''robots'\'']",
          "type": "attribute",
          "attribute": "content"
        }
      }
    }
  }'

Node.js

const response = await fetch("https://api.domscout.io/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json", "x-api-key": "YOUR_API_KEY" },
  body: JSON.stringify({
    url: "https://www.example.com/",
    extract: {
      fields: {
        canonical: {
          selector: "head link[rel='canonical']",
          type: "attribute",
          attribute: "href"
        },
        robots: {
          selector: "head meta[name='robots']",
          type: "attribute",
          attribute: "content"
        }
      }
    }
  }),
});
console.log(await response.json());

Python

import requests

response = requests.post(
    "https://api.domscout.io/scrape",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={
        "url": "https://www.example.com/",
        "extract": {
            "fields": {
                "canonical": {
                    "selector": "head link[rel='canonical']",
                    "type": "attribute",
                    "attribute": "href"
                },
                "robots": {
                    "selector": "head meta[name='robots']",
                    "type": "attribute",
                    "attribute": "content"
                }
            }
        }
    },
)
print(response.json())

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io"
	"net/http"
)

func main() {
	body, _ := json.Marshal(map[string]any{
		"url": "https://www.example.com/",
		"extract": map[string]any{
			"fields": map[string]any{
				"canonical": map[string]any{
					"selector": "head link[rel='canonical']",
					"type": "attribute",
					"attribute": "href",
				},
				"robots": map[string]any{
					"selector": "head meta[name='robots']",
					"type": "attribute",
					"attribute": "content",
				},
			},
		},
	})
	req, _ := http.NewRequest("POST", "https://api.domscout.io/scrape", bytes.NewReader(body))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("x-api-key", "YOUR_API_KEY")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	raw, _ := io.ReadAll(res.Body)
	fmt.Println(string(raw))
}

PHP

<?php
$ch = curl_init('https://api.domscout.io/scrape');
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => ['Content-Type: application/json', 'x-api-key: YOUR_API_KEY'],
    CURLOPT_POSTFIELDS => json_encode([
        'url' => 'https://www.example.com/',
        'extract' => [
            'fields' => [
                'canonical' => [
                    'selector' => 'head link[rel=\'canonical\']',
                    'type' => 'attribute',
                    'attribute' => 'href'
                ],
                'robots' => [
                    'selector' => 'head meta[name=\'robots\']',
                    'type' => 'attribute',
                    'attribute' => 'content'
                ]
            ]
        ]
    ]),
]);
$response = curl_exec($ch);
if ($response === false) {
    throw new RuntimeException(curl_error($ch));
}
echo $response;

Ruby

require "json"
require "net/http"

response = Net::HTTP.post(
  URI("https://api.domscout.io/scrape"),
  {
    url: "https://www.example.com/",
    extract: {
      fields: {
        canonical: {
          selector: "head link[rel='canonical']",
          type: "attribute",
          attribute: "href"
        },
        robots: {
          selector: "head meta[name='robots']",
          type: "attribute",
          attribute: "content"
        }
      }
    }
  }.to_json,
  "Content-Type" => "application/json",
  "x-api-key" => "YOUR_API_KEY"
)
puts response.body

Java

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Capture {
    public static void main(String[] args) throws Exception {
        String body = """
            {
                "url": "https://www.example.com/",
                "extract": {
                    "fields": {
                        "canonical": {
                            "selector": "head link[rel='canonical']",
                            "type": "attribute",
                            "attribute": "href"
                        },
                        "robots": {
                            "selector": "head meta[name='robots']",
                            "type": "attribute",
                            "attribute": "content"
                        }
                    }
                }
            }
            """;
        HttpRequest request = HttpRequest.newBuilder(URI.create("https://api.domscout.io/scrape"))
            .header("Content-Type", "application/json")
            .header("x-api-key", "YOUR_API_KEY")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();
        HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println(response.body());
    }
}

using System.Net.Http.Json;

using var client = new HttpClient();
client.DefaultRequestHeaders.Add("x-api-key", "YOUR_API_KEY");

var response = await client.PostAsJsonAsync("https://api.domscout.io/scrape", new {
    url = "https://www.example.com/",
    extract = new {
        fields = new {
            canonical = new {
                selector = "head link[rel='canonical']",
                type = "attribute",
                attribute = "href"
            },
            robots = new {
                selector = "head meta[name='robots']",
                type = "attribute",
                attribute = "content"
            }
        }
    }
});
Console.WriteLine(await response.Content.ReadAsStringAsync());

Go further

Guide: catch a noindex on deploy

Saved SEO head tags payload

Try the meta tag preview

Other use cases: Web access for AI agents · Website change monitoring · Web archiving and evidence · Crawling a whole site · Thumbnails and link previews · PDFs from your own HTML · Structured data from web pages · Link preview checks · Responsive layout checks

Give your product a browser.

Get clean web content and visual proof into your workflow in minutes.

Get your free API key Read the docs