How it works
Search engines cannot visit every page on the web every day. For each site, Google decides how much crawling to do. Two things shape that amount.
- How much your server can handle. Google tries not to slow your site down. If the server answers quickly and without errors, Google can crawl more. If the server struggles, Google backs off.
- How much Google wants to crawl. Popular pages and pages that change often get visited more. Pages that look like duplicates or offer little value get visited less.
Together, these two set your crawl budget.
Who needs to think about it
Most site owners do not. Crawl budget mostly matters for very large sites, such as big online stores, or sites that create endless address variations through filters and sorting options. On a site with a few hundred or a few thousand pages, Google can normally crawl everything it wants to.
If a page on a small site is missing from Google, the cause is almost always something else. The lesson crawling and indexing, explained simply walks through the usual reasons.
Why it matters on large sites
- Visits spent on useless addresses are visits not spent on the pages you care about.
- New and updated pages can take longer to be picked up.
- Long redirect chains and error pages use up visits.
GOOD TO KNOWCrawl budget is not a ranking factor. Being crawled more often does not make a page rank higher. It only affects how soon Google sees new and changed pages.
How to check yours
- Open Google Search Console, go to Settings and open the Crawl stats report.
- Look at the number of crawl requests over time and at how many of the responses are errors.
- Use robots.txt to keep crawlers out of areas with no search value, such as cart pages and internal search results.
- Keep your XML sitemap limited to the pages you want in search.