Catch a noindex or canonical change on deploy

September 17, 2026guidesseomonitoring

The worst SEO bugs are silent. A staging flag ships a noindex to production, a canonical keeps pointing at the preview domain, a template change drops the meta description from every product page. Nothing breaks. No test fails. The page still renders, still returns 200, still looks right in a browser. Weeks later the traffic is gone and nobody can say which deploy did it.

The reason these survive is that almost nothing in a normal test suite looks inside <head>. A snapshot test renders a component; an end-to-end test clicks through a flow. Neither reads the tags that decide whether the page is indexed at all.

This guide wires that check into a request you can run in CI, and shows what the answer looks like when the page disagrees with itself.

What a search engine reads before it reads your page

`<title>` and the meta description, which are what a result shows.
The canonical link, which says which URL owns this content. Pointing it at a staging host removes the real page from consideration.
Meta robots, where noindex means exactly what it says.
hreflang links, if you serve more than one language.
OpenGraph and Twitter tags, which decide what a shared link looks like.
JSON-LD structured data, which is what gets quoted back in a rich result.
The `h1`, and specifically whether there is more than one.

One request that returns all of them

extract takes a map of field names to CSS selectors. Each field says what to read: the element text, or a named attribute.

bash

curl -sX POST https://api.domscout.io/scrape \
  -H "Content-Type: application/json" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -d '{
    "url": "https://example.com/",
    "extract": {
      "fields": {
        "title": { "selector": "head title", "type": "text" },
        "description": { "selector": "head meta[name=\"description\"]", "type": "attribute", "attribute": "content" },
        "canonical": { "selector": "head link[rel=\"canonical\"]", "type": "attribute", "attribute": "href" },
        "robots": { "selector": "head meta[name=\"robots\"]", "type": "attribute", "attribute": "content" },
        "hreflang": { "selector": "head link[rel=\"alternate\"][hreflang]", "type": "attribute", "attribute": "hreflang", "all": true },
        "jsonLd": { "selector": "head script[type=\"application/ld+json\"]", "type": "html", "all": true },
        "h1": { "selector": "h1", "type": "text", "all": true }
      }
    }
  }'

The same body is documented as a saved payload on the capture templates page, so you can keep it in the dashboard and load it into the playground instead of pasting it each time.

What comes back

Every field reports a status rather than an empty string, which is the part that makes this usable in a test:

bash

{
  "analysis": {
    "extraction": {
      "fields": {
        "title": { "status": "found", "value": "Example Domain" },
        "description": { "status": "missing" },
        "canonical": { "status": "found", "value": "https://staging.example.com/" },
        "robots": { "status": "found", "value": "noindex, nofollow" },
        "h1": { "status": "found", "value": ["Example Domain"] }
      }
    }
  }
}

status is one of found, missing or invalid_selector. A missing description is reported as missing, not as "", so your assertion can tell the difference between a tag that is absent and a tag that is present and empty. invalid_selector means the selector itself was wrong, which stops a typo in your check from reading as a clean pass.

Two bugs are visible in that response: the canonical points at staging, and the page is telling search engines not to index it.

Check the served HTML, not just the rendered page

Send renderJs: false and the answer comes from one guarded HTTP GET instead of a browser, so you see the tags as they arrive from your server, before any JavaScript runs.

That matters because a crawler does not always run your JavaScript before deciding what a page is. A canonical injected client-side is a canonical a crawler may never see. Running the same extraction twice, once with renderJs: true and once with false, tells you whether your head tags depend on hydration. The response reports metadata.renderJs so you can tell the two answers apart after the fact.

Both cost the same, so the only price of checking both is a second request.

Fail the build instead of reading a report

The check is worth more in CI than in a dashboard, because a deploy that never ships is cheaper than one you roll back:

bash

const EXPECTED_CANONICAL = "https://www.example.com/";

const res = await fetch("https://api.domscout.io/scrape", {
  method: "POST",
  headers: {
    "Content-Type": "application/json",
    "x-api-key": process.env.DOMSCOUT_API_KEY,
  },
  body: JSON.stringify({
    url: "https://www.example.com/",
    renderJs: false,
    extract: {
      fields: {
        canonical: { selector: "head link[rel='canonical']", type: "attribute", attribute: "href" },
        robots: { selector: "head meta[name='robots']", type: "attribute", attribute: "content" },
        h1: { selector: "h1", type: "text", all: true },
      },
    },
  }),
});

const { fields } = (await res.json()).analysis.extraction;
const problems = [];

if (fields.canonical.value !== EXPECTED_CANONICAL) {
  problems.push(`canonical is ${fields.canonical.value ?? "missing"}`);
}
if (fields.robots.status === "found" && /noindex/i.test(fields.robots.value)) {
  problems.push("page is noindex");
}
if (Array.isArray(fields.h1.value) && fields.h1.value.length > 1) {
  problems.push(`${fields.h1.value.length} h1 elements`);
}

if (problems.length) {
  console.error(`SEO check failed: ${problems.join(", ")}`);
  process.exit(1);
}

Run it against the deployed URL after a deploy, or against a preview URL before one. For a whole site rather than one page, a crawl runs the same capture options across every page it discovers, and each child result carries its own extraction.

What a monitor does and does not do today

A monitor can carry extract in its captureParams, so every scheduled run collects these tags. But monitor alerts currently compare the title, the status code, the Markdown, the links and the DOM snapshot. Extracted fields are collected and returned, not diffed. If you want an alert the moment a canonical changes, compare the values yourself between runs, or run the CI check above on a schedule.

Saying this plainly is the point: a guide that implied the alert already exists would waste an afternoon of yours proving it does not.

What it costs

A capture is one credit, and extract adds one, so each page checked this way costs two credits whatever renderJs you use. extract is available on Pro and above; the free Hobby plan does not include it. Checking twenty key pages on every deploy, twice a day, is well inside the Pro plan's monthly allowance.

Next steps

Load the saved payload into the playground and run it against your own site.
For the same check across a whole site, see crawl and batch.
To give an assistant this as a tool instead of calling it yourself, see giving an MCP client a real browser.