Most site owners never open the Crawl stats report, because it is tucked away under Settings. That is a shame, because it is the only place Google shows you how often it visits your site, how fast your server answers and what goes wrong along the way. This lesson walks through the report and the few patterns worth acting on.
If the words crawling and crawl budget are new to you, start with crawling and indexing, explained simply.
Where to find it
- Open your property in Google Search Console.
- In the menu on the left, click Settings, near the bottom.
- Under Crawling, open the Crawl stats report.
You need a domain property or a property at the root of your site to see the full report. It covers the last 90 days.
The three numbers at the top
The chart at the top shows three lines. Click each one to switch it on.
- Total crawl requests. How many times Google asked your server for something: pages, images, scripts and style files. Daily swings are normal.
- Total download size. How much data Google downloaded. A sudden jump often means pages got much heavier, such as images that are no longer compressed.
- Average response time. How long your server took to answer, in milliseconds. This is the one to watch. A slow answer limits how much Google crawls, and usually means slow pages for visitors too.
Host status: can Google reach you?
Next to the chart, Host status says whether Google had trouble with your site in the last 90 days. It checks three things:
- robots.txt fetch: whether Google could read your robots.txt file. If it fails often, Google may stop crawling until it can read it again.
- DNS resolution: whether your domain name pointed to a server.
- Server connectivity: whether the server answered at all.
A green tick on all three means there is nothing to do. A warning is worth opening, because it lists the days things failed.
The full on-page checklist we use on every page, free by email.
The breakdowns: where the visits go
Below the chart, four tables break down every request:
- By response. The status codes Google got back. Most should be 200 OK. A large share of 404s means links pointing at deleted pages. Many redirects mean Google keeps asking for old addresses. Server errors (5xx) need attention straight away.
- By file type. HTML pages, images, scripts and so on.
- By purpose. Discovery means new addresses, refresh means pages Google knew already. A site that publishes often should see a healthy share of discovery.
- By Googlebot type. Mostly the smartphone crawler, since Google indexes the mobile version of pages.
Click any row to see example addresses. That is how you find the actual pages behind a problem.
Three patterns worth acting on
Response time keeps climbing. Check your hosting, caching and any heavy plugins. A faster server helps Google and your visitors alike, and it shows up in Core Web Vitals too.
Many 404s or redirects. Open the examples. Fix your own links to point at live pages, and redirect old addresses that still get visits. The redirect checker shows whether each redirect reaches a live page in one hop.
Lots of crawling on addresses you do not want. Filter and sort combinations, search results pages, endless calendars. Block them in robots.txt so the visits go to pages that matter, and check your XML sitemap lists only real pages with the sitemap checker.
What a healthy report looks like
For a typical small or medium site, a healthy report has a few things in common:
- Host status shows three green ticks.
- Most responses, well over nine in ten, are 200 OK.
- Average response time stays steady, ideally under about 600 milliseconds.
- Crawl requests rise a little after you publish or update pages, then settle again.
If you recently moved pages, expect a short rise in redirects while Google catches up. The lesson how to change a URL without losing traffic shows how to keep that tidy. If you added a sitemap, expect more discovery requests for a while; how to create an XML sitemap and submit it covers that step.
On a small site, one look a month is enough. If Host status is green and most responses are 200, you can close the report and get back to writing.
This is lesson 5 of the Technical SEO series. Want to check a single page instead? Run the free audit.
