Technical SEO

SEO: Google Improves Crawling for Geotargeted Sites

Ecommerce sites that use geotargeting to serve different content to customers in different countries have faced challenges with organic search. On Wednesday, Google announced its efforts to overcome these challenges, enabling more multinational sites to rank in Google’s search results.

Geotargeting is a widely accepted feature that improves customer experience on a site. Unfortunately, some of the ways in which geotargeting features are implemented can actually act as a barrier to search engines, preventing them from crawling and indexing content not intended for visitors physically located in the United States.

The problem has been that Google looks American. Googlebot — the crawler that Google uses to index the web — crawls from an IP address based in California.

The problem has been that Google looks American. Googlebot — the crawler that Google uses to index the web — crawls from an IP address based in California.

As a result, some sites that employ geotargeting serve only Googlebot content intended for U.S. customers. Content intended for customers in other countries did not get crawled and had no chance of being included in Google’s search results.

If you use geotargeting on your site and you’ve been concerned about low levels of organic search visits to non-U.S. pages, this may be part of the problem.

Google’s announcement indicates its desire to resolve this issue. Google has created new “locale-aware crawl configurations” for Googlebot. When Google algorithmically detects that a site is delivering different country- or language-based content, these new configurations may be employed to crawl and index that content.

According to Google’s announcement, the new configurations include the following.

  • Geo-distributed crawling, where Googlebot would start to use IP addresses that appear to be coming from outside the U.S., in addition to the current IP addresses that appear to be from the U.S. that Googlebot currently uses.
  • Language-dependent crawling, where Googlebot would start to crawl with an “Accept-Language” HTTP header in the request.

If Google’s algorithm detects that your site requires one of these new crawler configurations to more thoroughly index it, the new configurations will be deployed automatically. No changes or notifications are required on your part. The flip side of that, however, is that there is no way to request that Googlebot engage one of its locale-aware crawl configurations on your site, and no way to determine if one has already being used.

Optimize for Location and Language

Google’s recent changes do not eliminate the need to optimize your site with location and language information. The update marks a beginning, but Google’s start isn’t something to be relied upon to solve search-engine-optimization issues with geotargeting.

To start with, only Google has made this announcement, which means that only Google’s search results have the opportunity to be changed by the new algorithm and configurations. Other search engines have yet to announce that they’re working on this issue.

In addition, the nature of algorithmic detection requires that the algorithm be able to accurately detect the need on your site to engage one of these new Googlebot configurations. The algorithm and configurations are new and will be tested and fine-tuned over time as more combinations of countries and languages are supported.

Even Pierre Far, Google U.K.’s webmaster trends analyst, indicated that these are Google’s “first small steps,” and that Google “still (very strongly) recommend[s] having separate URLs for different locales and using rel-alternate-hreflang annotation for them.”

Use the following guidelines to ensure that the searchers in each country, speaking each language you support, land on the correct pages on your site to buy the products they want.

  • Separate URLs. Place content intended for separate countries and languages in separate directories so that the content is actually on separate pages with separate URLs. For example, a product available in the U.S. and Australia may both be in English, but require separate content for pricing, promotions, descriptions, and availability. Rather than using geotargeting to modify the content of a single URL ( to display U.S. or Australian information, create two separate URLs to display the two different pages ( and
  • ccTLDs. Hosting content on the country code top-level domain (ccTLD) for the targeted country sends a clear geographic signal. For example, content hosted on the .de ccTLD is presumed by search engines and customers alike to host content targeted for German customers. This option has its SEO drawbacks as well, though, including the lost ability to centralize the authority of all of your country sites into a single stronger domain.
  • Hreflang. Use Google’s preferred method of identifying country and language: hreflang tags embedded in the head of each page. Hreflang tags are visible only to crawlers and are supported only by Google. Bing and other engines support the meta language tag, a different type.
  • Targeting in Webmaster Tools. Both Google and Bing offer geographic targeting in their webmaster tools. It’s a quick and easy way to send another corroborating signal.
Jill Kocher Brown
Jill Kocher Brown
Bio   •   RSS Feed