Technical2 min read · Reviewed February 2026

Crawl Budget

Crawl budget is the number of pages a search engine will crawl on your site in a given period. It is set by how fast your server responds and how much Google wants your content. When it runs out, new or updated pages sit unseen and unranked.

Why it matters for your rankings

A page that never gets crawled can never rank. That is the whole argument. On a small brochure site this rarely bites, because Google can cover a few hundred URLs easily. The problem shows up once faceted navigation, filter parameters, session IDs, paginated archives or thin tag pages inflate your URL count into the tens of thousands.

Google then spends its visits on near-duplicate junk while your money pages and fresh posts wait days or weeks for attention. You feel it as slow indexing after a launch, stale titles in the SERPs and updated content that will not refresh. Crawl budget is also a quality signal in practice: sites that waste it usually have the redirect chains, soft 404s and duplicate content that drag rankings down for their own reasons.

How to check it on your site

Open the Crawl stats report

In Google Search Console go to Settings then Crawl stats. Read total crawl requests per day alongside average response time. Sustained response times above 600ms mean Google will deliberately slow down and request fewer pages.

Check what the crawls returned

In the same report open By response. Healthy sites are dominated by 200 and 304. Large blocks of 404, 301 or 5xx mean a real share of your budget is being spent on URLs that return nothing useful.

Compare discovery against refresh

Still in Crawl stats, open By purpose. If Refresh dwarfs Discovery while you are publishing every week, Googlebot is recycling old URLs instead of finding your new ones.

Read the Pages report

Open Indexing then Pages. 'Discovered - currently not indexed' is the signature crawl budget symptom: Google knows the URL exists and has chosen not to spend a crawl on it.

Crawl the site yourself

Run Screaming Frog or Sitebulb as a spider and compare the crawlable URL count with the URL count in your XML sitemap. A gap of five or ten times usually means faceted navigation or tracking parameters are generating URLs.

Confirm against server logs

Pull 30 days of server or CDN logs, filter to verified Googlebot and group by path. This is the only report that shows what Google actually spent your budget on, rather than what it might have.