Optimizing Crawl Budgets: What’s New

Google’s newly updated “Crawling infrastructure” documentation includes “Optimize your crawl budget,” a section with pointers on how websites can expose more pages to the search giant’s bots. The topic is important, as pages must be crawled to appear in Google’s index and search results.

The documentation addresses the need for a “budget” given the sheer number of global URLs. A poorly organized website will likely take longer to crawl and thus have fewer indexed pages.

Here are takeaways from the new guidelines.

Google Guidelines

Every new site starts with a “conservative” crawl budget. When launching a site, start with a modest number of pages to ensure that “Google can cover all the important content without overloading your servers.”

Over time, the crawl budget increases based on the site’s size and updates. Regular updates (especially if automatically included in its XML sitemaps) encourage the Googlebot to return and likely increase the crawl budget over time.

Update your product inventory and your blog regularly and gradually. For example, instead of launching 100 products in one day, consider three per day for a month, roughly.

Improving your site’s load time may increase its crawl budget. We’ve long known that faster websites are better crawled. Monitor your site’s performance, and strive to improve its speed.

Popular pages receive more frequent crawls. Googlebot crawls pages with many inbound links (external and internal) more frequently.

Other ways to improve crawl frequency and speed:

• Use the disallow command in robots.txt to block pages you don’t want indexed or in SERPs. Examples could be pages for private logins, shopping carts, and those dynamically generated.

• Remove or consolidate duplicate content.

• Avoid soft 404 pages (deleted pages that return a 200 code) as Googlebot will continue to crawl them. Return 404 or 410 responses if the pages no longer exist.

• Avoid redirect chains (multiple redirects for a single page)

• Use Search Console’s crawl stats report (Settings > Crawl stats) to monitor the crawl rate and problems encountered by Googlebot.

Search Console’s “Crawl stats” report reveals crawl frequency and problems, if any. Click image to enlarge.

Beware ‘URL Caching’

Google has added a new “URL caching” recommendation to the guidelines:

Support 304 (Not Modified) HTTP status codes. If a page hasn’t changed since Google last crawled it, returning a 304 code tells Google to reuse the cached version, saving your server bandwidth and resources.

My experience is to follow this recommendation only for unimportant pages, such as expired products and policy pages. Telling Google via a 304 code that a page is stale could affect its visibility in SERPs or even its index status.