Crawl budget SEO: who actually has one and who's wasting time on it

What crawl budget actually is

Crawl budget is the number of URLs on your site that a search engine’s crawler is willing and able to fetch in a given period. It’s made of two things pushed together. Crawl rate limit is how fast Googlebot will hit your server without knocking it over, based on how your server responds. Crawl demand is how much Google actually wants your pages, based on how useful and how fresh it thinks they are. Multiply those two and you get a rough budget.

The term gets thrown around like it’s a universal SEO metric everyone should be tracking, right next to domain authority and keyword density. It isn’t. Most sites never bump into a crawl budget ceiling in their entire lifetime. I run a handful of content sites and affiliate properties, and crawl budget has been a real constraint on exactly one of them, a site with north of 200,000 URLs from user-generated pages and a sprawling filter system. Every other site I manage, crawl budget was never the bottleneck, even when rankings were bad for other reasons.

Where the number comes from, mechanically

Googlebot doesn’t request a fixed quota and stop. It adjusts based on signals it gets back from your server in real time. If your server starts responding slowly or throwing 5xx errors, the crawler backs off, because hammering a struggling server serves nobody. If your server responds fast and clean, the crawler can afford to request more without host load becoming a problem.

Crawl demand works differently. It’s shaped by things like how often your content changes, how many pages you have that look worth revisiting, and how the crawler’s own scheduling decides your site fits into its priorities relative to everything else it has to crawl. This is where I stop speculating. Nobody outside Google can tell you the exact formula, and any post that claims to know the weighting is guessing. What’s observable from the outside is the effect: pages that change often and get linked to internally tend to get recrawled more; pages that sit static and orphaned get visited less.

Who actually runs into this

Size and page-generation pattern are the two things that put crawl budget on the table. Google’s own guidance, which matches what shows up in the Crawl Stats report for large sites, points at this being a concern mainly for sites with a very large number of URLs, roughly the low millions and up, or sites that generate URLs faster than they can be meaningfully crawled, like faceted navigation, infinite calendar pages, or user-generated content with parameter combinations that multiply endlessly.

If you run a site with a few hundred or a few thousand pages, you almost certainly don’t have a crawl budget problem. You might have a crawl budget symptom that’s actually something else: bad internal linking, a sitemap that’s stale, pages that return soft 404s, or content that just isn’t good enough to earn a revisit. It’s easy to reach for “crawl budget” as an explanation because it sounds technical and out of your hands. Most of the time the real cause is a lot more boring and a lot more fixable.

What burns crawl budget on sites that do have one

On the one site of mine where this mattered, the waste wasn’t exotic. It was:

Faceted navigation URLs. Every combination of filter and sort parameter generated a unique, crawlable URL. Color plus size plus price range plus sort order, multiplied out, turned a few thousand real product pages into hundreds of thousands of near-duplicate URL variants, all technically crawlable, almost none of them worth a separate crawl.

Infinite or near-infinite pagination. Paginated archives that went on for hundreds of pages, most of them thin and rarely visited by real users, still sat there as fetchable URLs competing for crawl attention against pages that actually mattered.

Redirect chains. Old URLs redirecting to newer URLs that had themselves been redirected again. Each hop is a separate request. A two or three hop chain on a small site is a rounding error. Multiplied across tens of thousands of legacy URLs, it’s real overhead.

Slow server response times under load. This is the crawl rate limit side, not the demand side. A server that’s slow to respond, especially during the hours when the crawler happens to be active, gets throttled by the crawler as a matter of self-preservation on Google’s part, not punishment.

None of that is unique to any one platform or CMS. It’s what happens when a site’s URL structure grows without anyone deciding on purpose how many crawlable variants should exist.

What doesn’t matter if your site is small

If you’re running a site with a normal number of pages, a handful of blog categories, a product catalog that doesn’t explode into filter combinations, you don’t need a crawl budget strategy. You need your sitemap to be accurate, your internal links to reach every page you care about, and your server to respond quickly and consistently. That’s it. Chasing crawl budget optimization on a small site is time spent on a problem that doesn’t exist yet, at the cost of time not spent on content or links, which are far more likely to be your actual constraint.

How I actually check whether it’s an issue

Search Console’s Crawl Stats report, under Settings, is the first stop. It shows total crawl requests over time, broken down by response code and by file type. If total crawl requests are flat or declining while your page count is growing, that’s a signal worth investigating. If most of your crawl requests are hitting 404s or redirects instead of 200s, that tells you the crawler is spending its attention on dead weight instead of live content.

Past that, server log files are the real source of truth, because they show exactly which URLs Googlebot requested and how often, not a sampled summary. Filtering raw or CDN logs for the Googlebot user agent (verified by reverse DNS lookup, not just the string, since that string is trivially spoofable) shows you what’s actually being crawled versus what you assume is being crawled. On the one site where I did this exercise, the log pull was the thing that made the faceted navigation waste visible. Nothing else did.

If you don’t have log access or the traffic volume to make log analysis worthwhile, Crawl Stats plus a straightforward XML sitemap coverage check in Search Console gets you most of the way there for free.

What to actually do about it, if it’s real

Fixing crawl budget waste isn’t glamorous and it isn’t a growth hack. It’s mostly subtraction. Block or noindex the low-value URL variants you don’t need crawled, using robots.txt for parameters that should never be fetched and noindex for pages that should be fetched but not indexed. Collapse redirect chains down to a single hop. Fix or remove pages that consistently 404 instead of leaving them in the crawl path. Keep your XML sitemap limited to canonical URLs you actually want indexed, not every URL that technically exists. And keep server response times fast and consistent, because a slow server caps your crawl rate no matter how clean your URL structure is.

None of this changes how fast a page ranks once it’s crawled, and nobody can promise you a timeline for how quickly a fix shows up in the crawl data, because that depends on factors outside any one site owner’s visibility. What it does is stop wasting the crawler’s attention on pages that were never going to earn traffic anyway, which frees that attention up for the pages that might.

If your site is small, skip this whole exercise and go work on content or links instead. If your site is large enough that this genuinely applies to you, the fix is unglamorous log analysis and URL discipline, not a plugin or a setting you flip once.

If you want help figuring out whether crawl budget is actually your bottleneck or something else wearing its name, that’s the kind of technical SEO work we do at The SEO Desk.