Broken link checking

The page is fetched, its links are extracted and de-duplicated, and each one is requested in turn. Internal links are prioritised over external ones, so if the run hits its budget the links you can actually fix are the ones already checked.

Links per run

50 by default

Overall concurrency

5

Per-host concurrency

2

Total time budget

45 seconds

Requests ask for the first byte rather than the whole document. That is polite and it is fast — and it is better than the common approach of trying a HEAD request first and falling back to a GET, which doubles the load against every server that rejects HEAD.

A dead link on your own domain is your bug. The run fails.

A dead link to an external site is worth knowing about and worth fixing eventually, but it is not a defect in your deployment — somebody else deleted a page. It is reported without failing the run.

Rate-limiting is handled the same way. A 429 from an external host means we asked too quickly, not that the link is broken, and reporting that as a broken link would be simply wrong.

Concurrency is capped per host as well as overall. Fifty links to one domain would otherwise mean fifty requests to that origin inside a few seconds, which is indistinguishable from a small denial-of-service and is a good way to get blocked.

There is also a wall-clock budget for the whole run, not just a per-request timeout. If the run runs out of time, the result reports exactly how many links were checked and how many were skipped, with the reason. Silently truncating at fifty links and reporting “no broken links found” would be a lie, and the sort of lie you only discover months later.

It is the clearest case in the product. External links break because of decisions other people make, on their timetable. Scheduled hourly, this check would eventually wake somebody at 3am because a blog you linked to two years ago reorganised its URLs.

Run it when you publish, or when you are cleaning up. Every run is kept, so you can see what changed since the last one.