How to convert a web page to Markdown for an LLM

Paste a web page's HTML into a language model and most of what you pay for is not the page. It is framework payloads, inline CSS, three copies of the navigation, a cookie banner and a footer with sixty links. Markdown keeps what a reader sees — headings, paragraphs, lists, tables, links — and drops the rest, which is why it has become the default input for retrieval pipelines and agents.

This guide is the shortest path from a URL to Markdown with the domscout API, in curl, Node and Python, followed by the two decisions that matter in production: whether the page needs a real browser, and what to do when the answer comes back thin.

Why not fetch the HTML and convert it yourself?

For a static documentation page, that works. It stops working in three common situations:

• The content is rendered by JavaScript. A single-page app answers a plain HTTP request with an empty shell.
• The content arrives late. Prices, search results and review counts often load after the first paint.
• The markup is mostly not content. Even when the text is present, an HTML-to-Markdown library cannot tell which of a page's four link lists is the article.

A renderer solves the first two by loading the page in Chromium before extracting anything. Extracting on the server solves the third once, without a parser per site. The longer version is in what a browser sees that a fetch does not.

One request: URL in, Markdown out

POST /scrape renders the page and returns it as a document. Create an API key in the dashboard and keep it in an environment variable, never in source code.

curl -sX POST https://api.domscout.io/scrape \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -H "content-type: application/json" \
  -d '{"url": "https://example.com"}'

The response:

{
  "status": "success",
  "url": "https://example.com/",
  "title": "Example Domain",
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples...",
  "links": [{ "href": "https://www.iana.org/domains/example", "text": "More information..." }],
  "metadata": {
    "statusCode": 200,
    "renderJs": true,
    "wordCount": 28,
    "estimatedTokens": 43,
    "contentQuality": "ok"
  }
}

markdown, links and metadata are always present. A page that yielded no text returns an empty string and an empty array rather than leaving the fields out, so your code never has to check whether they exist. (metadata carries more fields than shown here, including the credit cost of the call.)

Node.js

const res = await fetch("https://api.domscout.io/scrape", {
  method: "POST",
  headers: {
    "x-api-key": process.env.DOMSCOUT_API_KEY,
    "content-type": "application/json",
  },
  body: JSON.stringify({ url: "https://example.com" }),
});
if (!res.ok) throw new Error(`scrape failed: ${res.status} ${await res.text()}`);

const page = await res.json();
console.log(page.title, page.metadata.estimatedTokens);
console.log(page.markdown);

Python

import os
import requests

res = requests.post(
    "https://api.domscout.io/scrape",
    headers={"x-api-key": os.environ["DOMSCOUT_API_KEY"]},
    json={"url": "https://example.com"},
    timeout=60,
)
res.raise_for_status()

page = res.json()
print(page["title"], page["metadata"]["estimatedTokens"])
print(page["markdown"])

Two responses that are not a document

• 202 Accepted. On Pro plans and above, a capture expected to be slow can be moved to a durable job before any browser work starts. The body carries a jobId; poll GET /job/{jobId} until status is done, then read result.
• 504 Gateway Timeout. The code is RENDER_DEADLINE_EXCEEDED: the page did not finish inside the synchronous window. Send the same request with "async": true (Pro and above) and poll the job instead.

Handle both from the start. A pipeline that assumes every answer is a 200 works on your test URL and fails on the slowest page in production.

Wait for the part you actually need

If the content you care about loads after the first render, tell the browser what to wait for:

{
  "url": "https://example.com/search?q=widgets",
  "waitForSelector": ".results-loaded"
}

If the selector never appears, the request is refused with 400 WAIT_FOR_SELECTOR_TIMEOUT instead of returning a half-loaded page. That is the failure you want: loud and specific.

Skip the browser when the page does not need one

Documentation, blogs, news articles and changelogs already have their content in the HTML. For those, send renderJs: false (or its alias fast: true). The API answers from one guarded HTTP request instead of starting Chromium: the same response shape and the same credit cost, roughly an order of magnitude faster.

{ "url": "https://example.com/docs/getting-started", "renderJs": false }

The catch is that fast mode does not run JavaScript, so a client-rendered page comes back as whatever its server HTML contained, which is often nothing. The API does not guess on your behalf. It tells you what happened:

• metadata.renderJs is false on every fast response.
• metadata.contentQuality is ok, low_text (1 to 149 words), empty, likely_blocked or too_large.
• metadata.advice appears when the answer looks thin, and suggests the retry.

That makes "fast first, browser only when needed" a few lines:

async function scrape(url, renderJs) {
  const res = await fetch("https://api.domscout.io/scrape", {
    method: "POST",
    headers: {
      "x-api-key": process.env.DOMSCOUT_API_KEY,
      "content-type": "application/json",
    },
    body: JSON.stringify({ url, renderJs }),
  });
  if (!res.ok) throw new Error(`scrape failed: ${res.status} ${await res.text()}`);
  return res.json();
}

let page = await scrape("https://example.com/docs", false);
if (["empty", "low_text"].includes(page.metadata.contentQuality)) {
  page = await scrape("https://example.com/docs", true);
}

A short page can be low_text legitimately, so decide that threshold per source. If the fast path answers 415 UNSUPPORTED_CONTENT_TYPE, the URL returned a PDF or another binary file, and only the browser path can read it.

What it costs

A Markdown scrape costs 1 credit whether or not the browser runs; Markdown extraction adds nothing to a capture. The free Hobby plan includes 100 a month and paid plans start at 5,000 a month — see pricing. GET /credits returns your balance and the full price list without spending anything.

Before you hand it to a model

• Check the token estimate first. metadata.estimatedTokens tells you whether the page fits your context window before you spend a model call finding out.
• Keep the links. The links array holds the page's outbound links with their anchor text, which is what an agent needs to decide where to go next.
• Chunk on headings, not on character counts. Headings are natural boundaries, and splitting on them keeps each section's title attached to its text.
• Do not treat a 200 from the target as success. A consent wall or a bot challenge is a 200 too. contentQuality: "likely_blocked" is the hint, and some heavily protected sites refuse automated traffic however it arrives.

Next steps

• Try a URL in the playground and copy the request it builds.
• For many pages at once, batches and crawls run the same capture as durable jobs.
• To give an AI assistant this as a tool rather than calling it yourself, see giving an MCP client a real browser.