SEO

Crawl Budget

The finite attention a search engine spends crawling your site. Wasted on duplicates, redirect chains, or junk URLs, it delays the pages that matter.

Crawl budget is the number of pages a search engine is willing and able to fetch from your site in a given window. Googlebot does not crawl infinitely; it allocates attention based on your site’s health, speed, and perceived importance, then spends that allocation on whatever URLs it discovers.

Small sites rarely exhaust their budget. The problem arrives with scale and with mess: faceted URLs that generate thousands of near-duplicates, redirect chains, soft-404s, orphaned pages, and parameter variations that all compete for the same finite attention. When the crawler spends its visit on junk, your new and updated pages wait longer to be seen.

The remedy is architectural. A clean URL structure, one canonical form per page, a sitemap that lists only indexable destinations, fast responses, and intentional internal linking all steer the crawler toward what deserves ranking. This is a large part of what technical SEO actually buys: not magic, just a site that spends the crawler’s time on the pages that earn revenue.