Practical Ecommerce

SEO: Try Surfing Like a Search Engine Spider

I’m a visual learner. I need to see and interact with things to understand them. It therefore helps me to mimic the behavior of search engine spiders when I’m digging into a site for the first time to review its search engine optimization issues. I like to think of it as “being the bot.”

The key thing to remember is that search engines crawlers don’t have JavaScript, CSS or cookies enabled. They can’t “see” content embedded in media files such as images, Flash, video or audio. You take all this away, and what are you left with? Plain HTML, text and links. In many cases, the entire focus of the page is rendered indecipherable, and in some cases the site isn’t even navigable. It’s an incredible illustration of the structural challenges bots (i.e. search engine robots) face when they’re crawling a site.

Emulate a Search Engine Spider

Firefox’s Web Developer Toolbar by Chris Pederick will make surfing like a search spider much easier. For this exercise, focus on the yellow highlighted sections of the toolbar and select the following features:

  1. Disable > Disable JavaScript > All JavaScript
  2. Cookies > Disable Cookies > All Cookies
  3. CSS > Disable Styles > All Styles
  4. Images > Disable Images > All Images
  5. Visit any page and you’re surfing approximately like a bot. If you’re already at the page you want to test, reload the page to get the full experience.

Another way to confirm what Googlebot is seeing is to visit Google’s text-only cache with JavaScript, cookies and CSS disabled. It should look the same as the live page with all those things disabled. If it doesn’t, it’s possible there is some form of user agent or bot detection in play on the site to deliver a different experience to search engine crawlers.

Let’s take a test drive. It’s very easy to see the difference on many major consumer brands, because their sites tend to be dependent on Flash and JavaScript to deliver a glossy, engaging, interactive brand experience. But what does this cost organic search traffic and sales from this traffic?

Peek at Victoria’s Secret

We’ll start with, clearly a very brand-conscious site.

For starters, the header and footer navigation require JavaScript to load, effectively removing the navigation the site for search engine spider and for visitors without JavaScript enabled.

The primary content is image-based. The only text on the page for search engine spiders to crawl is the alternative attributes in the seven feature images on the page. This handful of alt attribute text, like the images, is entirely focused on enticing humans to click. No argument, that’s an important function of a home page. However, Victoria’s Secret is keeping its keyword theme of its home page a secret from the search engines. The title tag sends the sole focused keyword signal on the page.

If we were to peek at Victoria Secrets’ web analytics, I’ll bet dollars to doughnuts that they receive the vast majority of their natural search traffic and sales for branded search phrases (such as “victorias secret”), and a very small percentage for non-branded terms (such as “lingerie”). Despite Victoria Secret’s efforts to feature dresses, swimwear and sandals on the home page, it’s highly unlikely that it’s winning any non-branded searches for those very popular summer fashion search phrases. In short, creating strong textual signals for and ensuring that the navigation is crawlable would be the first step to winning non-branded traffic and sales.

What About Adidas?

Now let’s review another example,

Adidas is using a tactic known as graceful degradation to provide a crawlable text-based experience when the user/agent visiting doesn’t support JavaScript or CSS. It’s certainly not pretty to human eyes, but it’s not supposed to be – that’s what the whiz-bang Flash version is for. To a bot or a human on a text reader it’s perfectly navigable.

That said, Adidas should pay more attention to the kinds of text it’s providing. While the amount of text and links is strong, the template and the words used don’t send a coherent keyword theme. The home page lacks a strong title tag and H1 heading, though there are several H3 headings. The text in the title tag and the H3 headings aren’t sending a signal that Adidas is a power in shoes, athletics, sports or any other relevant keyword market that Adidas should legitimately be able to capture.


Removing the signals that we humans are trained to pay attention to – the shiny, glossy images and animations – forces us to consider the textual signals that a site is really sending on each page. If those textual signals are weak, unfocused or nonexistent, search engines will have a harder time matching your site with popular search phrases when real customers are searching for your products.

Jill Kocher Brown

Jill Kocher Brown

Bio   •   RSS Feed


Sign up for our email newsletter

  1. Jill Kocher April 16, 2009 Reply

    BTW: Matt Cutts (Google’s head of webspam) prefers to use Lynx to navigate like a bot. Lynx’s textual output format is hard for me to decipher, but that’s definitely another good option. Try the Yellowpipe Lynx Viewer (

  2. kesabsy April 21, 2009 Reply

    If you try this with our site ( you will see that not much changes, because we have done a lot of modifications that leave the page fairly intact if you come to it with a browser that doesn’t have CSS or Javascript enbled. Take a look, and I welcome comments.

  3. SEOTheOtherPointOfView April 21, 2009 Reply

    I like the article alot and has some deliberate and profound points to be made but misses the other point of view.. Although it’s extremely important for the search engines to rank you so you can get traffic.

    But once the traffic/visitor arrives at your site, you’d best be prepared to capture them visually and have a "Call To Action" for them so they can actually request contact and/or buy a product or service.

    Branded companies like Victorias Secret and Adidas are not really concerned too much about SEO as they’re at the top of the Branded Chain and people know and search for them by brand name and not a product or service search term phrase.

    I know that even if a site has good content and prices, if they don’t instill integrity, trust and a professional appearance in the way their site looks, I am still hesitant.

    If we did not pay attention to packaging, then why not just have all text websites with no visual appeal whatsoever and then we’d all be on a true level playing field.

    There is a happy medium and only the truely experienced web companies ( can help you out on both front and backends.

    But Jill made some excellent points.. probably just limited to time and space..haha

  4. Jill Kocher April 24, 2009 Reply

    Howdy kesabsy — yup — the homepage has incorporated the principles of sending JS-less & CSS-less signals to search engines. However, a homepage is typically the strongest page for SEO. Have a look at the next layers down. The category, subcat & product pages need to pull their weight, also.

  5. Jill Kocher April 24, 2009 Reply

    Agreed, "SEOTheOtherPointOfView," it’s critical that a major etailer (as well as not so major etailers) sends equally strong signals to search engines and humans alike. Walking that fine line between brand prominence and SEO prominence is an art that few SEO folk (or marketing folk) understand. Utilizing graceful degradation and CSS-based design principles ETHICALLY is the answer to having your brand/SEO cake and eating it too. It’s very achievable.

  6. D Lajoie July 3, 2014 Reply

    I’m working on a website right now that is built in DNN (Dot Net Nuke). When I use the tool you describe, on many pages there is nothing! Does that mean the content won’t be crawled?

    Here is the website

    Is there a fix for DNN to have the content appear?