NewIntroducing semantic snapshotsPair every capture with structured DOM data →

Blog

What a browser sees that a fetch does not

All posts
engineeringextraction

Every project that reads the web starts the same way: someone writes a fetch, prints the response, and it works. Then it works less. Then a specific site returns a page with no content in it, and the project acquires a headless browser, a queue, and a person whose week is now about Chromium.

The gap between those two moments is worth understanding before you land in it, because it is not one problem. It is four, and they fail differently.

1. The content was never in the HTML

The obvious one. A single-page application ships a shell and fills it in from JavaScript, so a fetch returns a <div id="root"> and nothing else. This failure is at least honest — you look at the response and there is visibly nothing there.

What makes it awkward is that it is rarely all-or-nothing. Plenty of pages are server-rendered for the parts a search engine cares about and client-rendered for the parts you care about: the price, the stock status, the review count, the "in your area" block. The fetch returns a page that looks complete and is missing exactly the field you came for, and nothing in the response says so.

2. The content arrived after you stopped listening

A browser does not finish loading a page at one moment. It finishes several times: when the document parses, when the deferred scripts run, when the fonts land, when the third XHR resolves, when the intersection observer notices you scrolled.

A fetch has no equivalent of "wait until the thing I want exists". A renderer does, and it is the single most useful control in the API:

json
{
  "url": "https://example.com/search?q=widgets",
  "waitForSelector": ".results-loaded",
  "extractMarkdown": true
}

Without something like that, a capture of a page that loads in two stages is a race, and races produce intermittent bugs that are extremely expensive to diagnose because they are not reproducible on the machine of the person diagnosing them.

3. The page behaves differently for you than for a person

Viewport size changes what renders — not just how it looks. Responsive sites routinely omit entire sections below a breakpoint, load different images, and collapse a table into a list. Locale and timezone change dates, currencies, and sometimes which content is shown at all. A device with touch gets a different navigation.

None of this is deception; it is just a page doing its job. But it means "what does this page say" is an incomplete question. The complete one is "what does this page say to a 1440-wide desktop in Europe/Berlin", and a fetch cannot ask it. A capture can:

json
{
  "url": "https://example.com/pricing",
  "width": 1440,
  "height": 900,
  "render": { "colorScheme": "dark", "reducedMotion": true }
}

4. The markup is not the content

This one survives even when the fetch works perfectly. A modern page is mostly not content: framework hydration payloads, inline critical CSS, analytics, three navigation menus, a cookie banner, a newsletter modal, a footer with sixty links. All of it is real markup and none of it is what you wanted.

If the consumer is a language model, this is not merely wasteful — it is actively harmful. Every token of navigation is a token of budget spent, and worse, it is *plausible context*. A model reading a page where the same six phrases appear in the header, the sidebar and the footer will weight them, because repetition is a signal. The noise does not just cost money; it changes the answer.

Turning the rendered DOM into Markdown solves that once, on the server, in a way that does not need a per-site parser. Headings, tables, lists, links and alt text survive. Script soup does not.

Which side is your problem on?

A useful test: fetch the page, search the response for a string you can see in your browser, and note what happens.

  • The string is there and the response is huge — you have problem 4 only. You need extraction, not necessarily rendering.
  • The string is missing — you have problem 1 or 2, and you need a browser.
  • The string is there but different — you have problem 3, and you need control over the viewport and locale, not just a renderer.
  • It works on some pages and not others, unpredictably — that is problem 2 wearing a disguise, and it will not get better on its own.

The part nobody budgets for

Deciding you need a browser is the easy half. The hard half is that a browser is not a library, it is an operational surface: a process that leaks memory, crashes on certain pages, needs a version pinned, needs a sandbox because it is running code from strangers, and needs somewhere to run that is neither always-on nor cold-starting for eight seconds.

That is the part this product exists to be, and the reason it is priced against a measured cost per capture rather than a guess. If you want to see the difference on a page you care about, the playground will render one without an account, and the docs have the full request shape.