What happens when the worker dies mid-crawl

One page is a request. Nine hundred pages is a distributed system, and the thing that makes it one is not the scale — it is the certainty of interruption. Over a run long enough to matter, a worker will be recycled, a function will time out, a deploy will land, a target site will rate-limit you halfway through. None of that is exceptional. It is the normal operating condition.

So the question a batch API has to answer is not "how do we avoid failure". It is: when the run is interrupted, what do you still have?

The three answers, from worst to best

Nothing. The run is a single long-lived process holding results in memory. It dies, you get an error, you start again from page one. This is what a naive implementation does, and it is fine right up until the first run that takes longer than the thing running it.

A number. The run reports "failed at page 617". You now know where to resume, but you have to write the resumption yourself, and you have to store the first 616 results somewhere the failure did not take with it.

The work. Every completed child is durable the moment it completes, independent of the parent. The parent is a record of intent, not a container of results. An interruption costs you the pages that were in flight and nothing else.

The third is what durable workflows mean here, and almost every other property below follows from it.

Why the parent is not where results live

If children write their own results, several awkward problems disappear at once.

You can read partial results while the run is still going, which matters because a crawl of a large site is not an operation you want to wait on blindly. You can retry a single failed child without re-running its siblings. And a parent that crashes can be repaired by its own retry — it re-reads which children are done and picks up from there — instead of needing a human to notice, diagnose, and restart it.

That last one is the difference between a system that needs an operator and one that does not.

Charging at the right moment

Quota is charged when a child starts, never when a parent is created.

This sounds like an accounting detail and it is really a correctness property. If a parent were charged up front for the work it intends to do, then every cancellation, every crash before the first child, and every run that turns out to have fewer pages than expected would produce a refund path — and refund paths are where billing bugs live. Charging at the point work actually begins means a job that never ran costs nothing, and there is no reconciliation to get wrong.

It also gives cancellation a clean meaning. Cancel a crawl at page 200 and you have paid for 200 pages, not 900, and not "900 minus a credit we will process later".

Idempotency, and the retry you did not intend

The most expensive mistake available to a client of a batch API is submitting the same run twice. It is also the easiest mistake to make, because it does not require anyone to do anything wrong: a request times out at the client while it succeeds at the server, the client retries, and now there are two crawls of the same site running concurrently against a target that is about to start rate-limiting both.

An idempotency key makes the retry safe:

curl -X POST "$DOMSCOUT_API_BASE/crawl" \
  -H "Content-Type: application/json" \
  -H "x-api-key: $DOMSCOUT_API_KEY" \
  -H "Idempotency-Key: nightly-docs-2026-08-08" \
  -d '{
    "seedUrl": "https://docs.example.com/start",
    "allowedOrigins": ["https://docs.example.com"],
    "maxPages": 500
  }'

Send that twice and you get one run, twice. The key should be derived from what makes the run unique to *you* — the date, the target, the schedule slot — not generated fresh per attempt, because a fresh key per attempt is exactly the thing it exists to prevent.

Knowing what is happening without polling yourself to death

A long job needs a way to be watched that does not require a tight loop. Two of them, really:

• `GET /job/{id}` returns the current state, for when you want to ask. Poll it on a back-off, not on a timer.
• A webhook fires when the run reaches a terminal state, for when you would rather not ask at all.

Deliveries are signed and stored, and a stored delivery can be replayed — which is the part that matters at 3am, when the endpoint that was supposed to receive the callback was the thing that was down. A webhook you cannot replay is a webhook that has to be right the first time, and nothing is right the first time.

The boring conclusion

None of this is clever. It is the set of decisions that stop a long-running job from being a source of operational surprise: results that outlive their parent, charges that follow real work, retries that cannot duplicate, and completion you can be told about rather than having to discover.

The jobs and webhooks guide has the full contract, and the crawl page covers what the seed, origin and pattern controls actually do.