Auditing a Site by What Actually Responds

I ran an audit on one of my blogs that came back clean. Every article present, full count, exactly the number of posts I expected. A chunk of those articles were returning 404 to anyone who clicked through to them.

Both things were true at once, and that is the part worth understanding. The audit was not lying. It answered a different question from the one I thought I had asked. I asked whether my content existed and it told me the files were on disk. I wanted to know whether the pages worked, and it had no opinion about that, because it never made a single request to the live site.

Files on disk and URLs that serve are different claims

A content site is rarely files served straight off a disk anymore. You write markdown, it lives in a repository, a build step turns it into HTML, and the result gets pushed to a host or CDN. Three stages, and a file can be perfectly healthy at stage one while simply not existing at stage three.

Every audit script written in a hurry conflates the two, because listing files in a folder is trivial and fetching a few hundred URLs takes slightly more effort. The easy version gets written, it produces a green number, and everyone believes it.

The ways the two drift apart are more numerous than you would expect.

A build can fail on one page and succeed on the rest. Malformed frontmatter, a broken date, a character the generator is strict about, and some generators skip that file and carry on, logging a warning nobody reads. Build passes, site deploys, one page silently missing.

A slug can change without the file changing. If the URL is generated from a title field rather than the filename, editing the title moves the URL while the file stays put. The old address 404s, along with every link anyone built to it.

Trailing slashes cause trouble. Some hosts treat /article and /article/ as identical, some redirect between them, some serve one and 404 the other. That is host configuration, not something your content controls, and it can change under you during a migration.

Case sensitivity behaves the same way. A capital letter in a filename works locally on macOS or Windows and breaks on a Linux host where case is part of the identity.

Redirect rules eat paths. One overly broad rule written to fix a single URL can swallow a whole directory, and since redirects return 3xx rather than an error, nothing looks broken unless you are watching for it.

And a sitemap will happily list URLs that never built. Sitemaps are usually generated from your content inventory, which is the same source the naive audit reads. Your sitemap says the page exists, your audit agrees, and both are wrong in the same direction because they share an origin. Two sources agreeing means nothing when they read from the same place.

What the audit should actually be

A list of every URL you believe you publish, fetched over HTTP, with the status code recorded. That is the whole idea. You are not inspecting your files. You are asking the live site, from outside, the question a visitor or a crawler would ask.

The moment you do that, the number stops being reassuring and starts being useful. Mine went from a clean full count to a specific list of URLs returning 404, which is actionable in a way a green tick never is.

It is a small piece of work. Read your list of URLs, request each one, record the status code and the final URL after redirects, write it to a file. A dozen lines in whatever language you already use.

Two details are worth getting right. Follow redirects but record that one happened and where it landed, because a 200 reached via three hops is a different situation from a direct 200. And cap your concurrency or add a small delay, because hammering your own CDN with hundreds of simultaneous requests can trip rate limiting and hand you a pile of 429s that look like genuine failures.

Run it from outside your own network, or at least know if you are not. A machine with a stale DNS entry, or a network with a different route to your host, can return answers a normal visitor would never get.

Get your URL list from more than one place

The disagreement between sources is itself the finding.

Your content inventory is one source: whatever you consider the canonical set of published things. Your sitemap is a second, though often derived from the first, so treat it as partly redundant. Search Console gives you a third and genuinely independent one, because it reports what actually got crawled and indexed, which is a fact about the outside world rather than about your repository.

Where those three disagree is where the problems live. A URL in your inventory that Search Console has never seen is probably a page that never built. A URL indexed but absent from your inventory is one you forgot, an old address still circulating, or a parameter variant creating a duplicate. An indexed URL now returning 404 is lost content, and that is the worst category, because it worked well enough to get indexed and then stopped.

This matters more if your site is fully static with no origin server behind it. The HTML is built and pushed to a CDN, and no application is running anywhere that could answer for a page that was never built. On a traditional server with a database, a missing template or slug mismatch often still produces something you can spot in the logs. On a static host, an unbuilt page is simply absent, with no process to notice. You find out by asking the URL, or a visitor finds out first.

Check the links between pages too

Take every internal link across your published pages, resolve it, record what comes back.

Internal links to dead pages are common after any restructure and they are doubly costly: you spend crawl attention pointing at nothing, and you send real readers into a dead end from inside your own content. An audit that checks pages but not the links between them does half the job, and the link half is usually where the rot appears first.

Soft 404s defeat all of the above

A soft 404 returns a perfectly good 200 while containing nothing useful. The build produced the shell of the page, header and navigation and footer, with the article body missing. Or your host serves a friendly not-found page at every unknown address and returns 200 with it, which some setups do by default and which is worse than a plain 404.

Status checking passes all of those with a clean green result. That is the same trap as the file listing, one level up: you moved from checking files to checking responses, a real improvement, but you are still checking a proxy for the thing you care about.

The cheap defence is content length. Record bytes returned alongside the status code, and open anything dramatically shorter than a typical article by hand. Crude, occasionally noisy on genuinely short pages, and it catches the empty shell for almost no extra effort.

What a 404 costs, and what to do about it

Worth being precise rather than dramatic. A page returning 404 eventually drops out of the index, so its traffic goes. External links pointing at it stop passing value, which is the expensive part if you paid for or earned those links. Internal links to it waste crawl.

What it does not do is trigger sitewide punishment. A handful of 404s is normal on any site. The problem is a 404 sitting on a URL that used to be worth something while nobody notices for six months.

When you find one, the decision is short. If the content still exists somewhere, restore it at the original address, since that preserves everything pointing at it. If it is genuinely gone but something close covers the same ground, redirect there. If there is no equivalent and nothing links to it, let it 404 deliberately rather than by neglect. Do not reflexively redirect everything to the homepage; an irrelevant redirect gets treated roughly like a 404 anyway, and you have hidden the problem behind a 3xx.

Run this after any migration, host change, or section rename, because those break URLs in bulk. Otherwise on a schedule, and the schedule matters less than it running without you remembering to trigger it.

One limit worth stating: this tells you a URL responds and roughly how much came back. It does not tell you the page is correct or complete. Status codes are the floor, not the ceiling.

But start with the floor. A list of URLs and their status codes, fetched from outside, compared against what you think you publish. It will tell you something your file listing never can.

If you want more of this kind of breakdown, on keyword research, link building, and the technical side of SEO, from someone running the sites and eating the mistakes, check out the rest of what we’ve put together here.