Search Engines And URL Redirects
Consider the consequences
We recently came across an interesting issue with the search engines during our recent launch, which was caused by something that I had never considered about URL redirects. The issue was that when we launched our new site, we removed large sections of our old site which left many of our indexed URLs orphaned. We did the best that we could to map over our old, legacy URLs to relevant sections of our new site, but in many cases there was now apparent place to redirect them to.
So our solution was to help people out by re-directing any URL that did not map over to a search results page. Rather than get a boring 404 "page not found" response, we had a 301 "permanent redirect" in place mapping to a search results page showing results on our site based on the URL that someone had tried. It seemed like a pretty smart thing to do, at least from the standpoint of human visitors.
However, shortly after our launch it seemed that we were being penalized by the search engines more than would normally be expected. Of course, and large change in a website will throw the search engines for a loop, and they will need to couple of weeks to recover and get it all sorted out. In our case, it was taking too long and we were definitely not showing up for keywords that we used to show up prominently for. Something was amiss.
Enter the master, Stephan Spencer, who was the one that identified a very viable reason for this, which I had never even considered. Stephan managed to follow the trail of a request made to our website, which broke down something like this:
A user makes a request for a URL that no longer exists on our site. In this case, the "user" is a search engine robot.
Our server doesn't find that URL, so instead responds with a 301 "permanently moved" response, telling the search engine that the URL it has tried is no longer valid, and that it needs to instead look somewhere else from now on. At the same time, a redirect to the search results page is sent.
The robot then makes a request at the search results pages, where it receives a 200 "all is well" response, and the page is rendered out.
From my side, this seemed to be ideal because rather than not finding a page that it is expecting, the search engines would get a page full of good links and content.
However, there is an evil side-effect here that if it weren't for Stephan, I'm not sure I would have figured out. That side effect comes from a behavior that Google (and probably other search engines) has that is meant, I assume, to diagnose whether or not sites are trying to "play the system". It turns out that Google will routinely try garbage URLs on a website, such as www.practicalecommerce.com/adlsjfhfdsgkjh, just to verify that the proper 404 "page not found" response is delivered back. The more I think about it, the more it makes sense. If a website is responding to garbage URLs with 200 "all is well" responses, odds are that they are trying to play the system with regard to search engines indexing their site.
The lessons learned:
- Stephan is really good at diagnosing SEO problems.
- Search results pages are great (as outlined in many of our articles) as long as they are linked to internally and the search being done is intentional.
- 404 pages are not a bad thing, as they communicate to users and robots alike that something is no longer available.
- The slightest little mistake, no matter how well intentioned, can have serious consequences with search engines.
The issue has since been resolved, and hopefully we will see the search engines become more friendly to us. We still do quite well, but we can definitely do better.
