For pages that hide content until asked

Read the content that only appears after a click

Click the tab, accordion or load-more button first, then read the page, in one request.

Get your free API key Try the playground

The problem

The specifications are on the second tab, the answers are inside collapsed accordions, and the rest of the reviews arrive when someone presses "Show more". A request that reads the page as it first loads gets the headings and none of what is under them.

How domscout handles it

actions run before the page is read: click a button by its CSS selector, wait for what it loads, scroll to bring in the next set of results. The Markdown, links and any extracted fields come from the page as it stands after the last step, and actionResults reports each step, so a button that was not there fails with its own error code instead of returning the unexpanded page. Clicking, typing, scrolling and waiting by CSS selector work on every plan; finding elements by role, label or text, and waiting for text to appear, start at Pro.

Where it shows up

Product specifications held behind tabs.
FAQ answers inside collapsed accordions.
Listings that load the next set when a button is pressed.

The request

POST /scrape. Available on every plan, including the free Hobby plan, at 1 credit. A request runs up to 25 steps, and more than 10 adds 1 credit.

cURL

curl -X POST https://api.domscout.io/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/42",
    "actions": [
      {
        "action": "click",
        "selector": "button[data-tab='\''specifications'\'']"
      },
      {"action": "wait", "time": 1000}
    ]
  }'

Node.js

const response = await fetch("https://api.domscout.io/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json", "x-api-key": "YOUR_API_KEY" },
  body: JSON.stringify({
    url: "https://example.com/products/42",
    actions: [
      { action: "click", selector: "button[data-tab='specifications']" },
      { action: "wait", time: 1000 }
    ]
  }),
});
const result = await response.json();
console.log(result.markdown);

Python

import requests

response = requests.post(
    "https://api.domscout.io/scrape",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={
        "url": "https://example.com/products/42",
        "actions": [
            {
                "action": "click",
                "selector": "button[data-tab='specifications']"
            },
            {"action": "wait", "time": 1000}
        ]
    },
)
print(response.json()["markdown"])

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
)

func main() {
	body, _ := json.Marshal(map[string]any{
		"url": "https://example.com/products/42",
		"actions": []any{
			map[string]any{
				"action": "click",
				"selector": "button[data-tab='specifications']",
			},
			map[string]any{"action": "wait", "time": 1000},
		},
	})
	req, _ := http.NewRequest("POST", "https://api.domscout.io/scrape", bytes.NewReader(body))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("x-api-key", "YOUR_API_KEY")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	var result map[string]any
	json.NewDecoder(res.Body).Decode(&result)
	fmt.Println(result["markdown"])
}

PHP

<?php
$ch = curl_init('https://api.domscout.io/scrape');
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => ['Content-Type: application/json', 'x-api-key: YOUR_API_KEY'],
    CURLOPT_POSTFIELDS => json_encode([
        'url' => 'https://example.com/products/42',
        'actions' => [
            [
                'action' => 'click',
                'selector' => 'button[data-tab=\'specifications\']'
            ],
            ['action' => 'wait', 'time' => 1000]
        ]
    ]),
]);
$response = curl_exec($ch);
if ($response === false) {
    throw new RuntimeException(curl_error($ch));
}
$result = json_decode($response, true);
print_r($result['markdown']);

Ruby

require "json"
require "net/http"

response = Net::HTTP.post(
  URI("https://api.domscout.io/scrape"),
  {
    url: "https://example.com/products/42",
    actions: [
      { action: "click", selector: "button[data-tab='specifications']" },
      { action: "wait", time: 1000 }
    ]
  }.to_json,
  "Content-Type" => "application/json",
  "x-api-key" => "YOUR_API_KEY"
)
puts JSON.parse(response.body)["markdown"]

Java

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Capture {
    public static void main(String[] args) throws Exception {
        String body = """
            {
                "url": "https://example.com/products/42",
                "actions": [
                    {
                        "action": "click",
                        "selector": "button[data-tab='specifications']"
                    },
                    {"action": "wait", "time": 1000}
                ]
            }
            """;
        HttpRequest request = HttpRequest.newBuilder(URI.create("https://api.domscout.io/scrape"))
            .header("Content-Type", "application/json")
            .header("x-api-key", "YOUR_API_KEY")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();
        HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println(response.body());
    }
}

using System.Net.Http.Json;
using System.Text.Json;

using var client = new HttpClient();
client.DefaultRequestHeaders.Add("x-api-key", "YOUR_API_KEY");

var response = await client.PostAsJsonAsync("https://api.domscout.io/scrape", new {
    url = "https://example.com/products/42",
    actions = new[] {
        new {
            action = "click",
            selector = "button[data-tab='specifications']"
        },
        new { action = "wait", time = 1000 }
    }
});
var result = await response.Content.ReadFromJsonAsync<JsonElement>();
Console.WriteLine(result.GetProperty("markdown"));

Go further

The action reference

Structured data from web pages

Signup and checkout flow checks

More in Data extraction and search: Crawling a whole site · Structured data from web pages · AI extraction to a JSON Schema · Semantic search over your content · Company research from websites · Evaluation sets from real pages · All use cases

Give your product a browser.

Get clean web content and visual proof into your workflow in minutes.

Get your free API key Read the docs