For readers and newsletters

Follow a site that has no RSS feed

Read the site's index page on a schedule, keep the links you have seen, and turn the new ones into feed items.

Get your free API key Try the playground

The problem

The company blog, the regulator's news page, the open-source project's announcements: all published regularly, none with a feed. Checking them by hand works for about a week.

How domscout handles it

POST /scrape returns the links on a page alongside its Markdown. Request the index page on your schedule, keep the addresses you have already seen, and every new address under the posts path is a new item: its link text makes a usable title, and a second request for the post gives you the summary. On static pages renderJs: false answers from one fast HTTP request and returns up to 100 links; a page that builds its list in the browser needs the rendered request. On Pro, a monitor with checkLinks can do the checking and send a webhook naming the two captures to compare.

Where it shows up

Feeds for blogs and newsrooms that never published one.
New-post alerts routed into a newsletter or a research channel.
A reading list across sources with no common format.

The request

POST /scrape. Available on every plan, including the free Hobby plan, at 1 credit a check. Monitors, which run the check on a schedule for you, start at Pro.

cURL

curl -X POST https://api.domscout.io/scrape \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/news", "renderJs": false}'

Node.js

const response = await fetch("https://api.domscout.io/scrape", {
  method: "POST",
  headers: { "Content-Type": "application/json", "x-api-key": "YOUR_API_KEY" },
  body: JSON.stringify({ url: "https://example.com/news", renderJs: false }),
});
const result = await response.json();
console.log(result.links);

Python

import requests

response = requests.post(
    "https://api.domscout.io/scrape",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"url": "https://example.com/news", "renderJs": False},
)
print(response.json()["links"])

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"net/http"
)

func main() {
	body, _ := json.Marshal(map[string]any{"url": "https://example.com/news", "renderJs": false})
	req, _ := http.NewRequest("POST", "https://api.domscout.io/scrape", bytes.NewReader(body))
	req.Header.Set("Content-Type", "application/json")
	req.Header.Set("x-api-key", "YOUR_API_KEY")

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	var result map[string]any
	json.NewDecoder(res.Body).Decode(&result)
	fmt.Println(result["links"])
}

PHP

<?php
$ch = curl_init('https://api.domscout.io/scrape');
curl_setopt_array($ch, [
    CURLOPT_POST => true,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_HTTPHEADER => ['Content-Type: application/json', 'x-api-key: YOUR_API_KEY'],
    CURLOPT_POSTFIELDS => json_encode(['url' => 'https://example.com/news', 'renderJs' => false]),
]);
$response = curl_exec($ch);
if ($response === false) {
    throw new RuntimeException(curl_error($ch));
}
$result = json_decode($response, true);
print_r($result['links']);

Ruby

require "json"
require "net/http"

response = Net::HTTP.post(
  URI("https://api.domscout.io/scrape"),
  { url: "https://example.com/news", renderJs: false }.to_json,
  "Content-Type" => "application/json",
  "x-api-key" => "YOUR_API_KEY"
)
puts JSON.parse(response.body)["links"]

Java

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Capture {
    public static void main(String[] args) throws Exception {
        String body = """
            {"url": "https://example.com/news", "renderJs": false}
            """;
        HttpRequest request = HttpRequest.newBuilder(URI.create("https://api.domscout.io/scrape"))
            .header("Content-Type", "application/json")
            .header("x-api-key", "YOUR_API_KEY")
            .POST(HttpRequest.BodyPublishers.ofString(body))
            .build();
        HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println(response.body());
    }
}

using System.Net.Http.Json;
using System.Text.Json;

using var client = new HttpClient();
client.DefaultRequestHeaders.Add("x-api-key", "YOUR_API_KEY");

var response = await client.PostAsJsonAsync("https://api.domscout.io/scrape", new { url = "https://example.com/news", renderJs = false });
var result = await response.Content.ReadFromJsonAsync<JsonElement>();
Console.WriteLine(result.GetProperty("links"));

Go further

Website change monitoring

Newsletter curation

Web monitors

More in Monitoring and alerts: Website change monitoring · Competitor price tracking · Minimum advertised price checks · New job posting alerts · Vendor changelog watching · Tender, grant and notice alerts · All use cases

Give your product a browser.

Get clean web content and visual proof into your workflow in minutes.

Get your free API key Read the docs