Practical Ecommerce

Ecommerce Know-How: Bot Herding, A SEO Internal Linking Tactic

A relatively simple SEO linking strategy can improve how well and how often your store’s product pages are indexed and displayed in organic search engine results.

This linking technique goes by several names. Bruce Clay Inc., an Internet business consultancy, refers to it as “siloing.” SEOmoz uses the term “PageRank sculpting.” And the Search Engine Roundtable called it “bot herding” in a recent discussion. While these names and practitioners claim subtle differences in meaning and application, the general technique seeks to concentrate PageRank on a site’s most important landing and product pages, improving how those pages perform in search engine results.

Ecommerce Know-How: Bot Herding

Bot herding or siloing can be divided into two categories. First, you have structural siloing.

“Siloing is creating solid themes on your site based on linking together all the relevant pages,” said Virginia Nussey, Associate Media Writer and Blogger for Bruce Clay Inc. “A site that has a number of pages on a subject is viewed by the search engine as a relevant result to serve to a user looking for information on that subject. Creating well-supported themes helps increase your site’s subject relevance in the eyes of the search engines.”

Second, there is a sort of virtual bot herding that simply seeks to encourage or discourage search spiders and bots to follow one link or another. In this “eCommerce Know-How,” I will focus on this second—virtual—aspect of bot herding.

The nofollow Attribute

In its simplest form, bot herding, uses the rel=” nofollow” attribute to prevent search spiders and bots from following some links, effectively herding those bots to other links.

<code>&lt;a href="" rel="nofollow"&gt;World's Best Source for Ecommerce Information&lt;/a&gt;

When Googlebot, MSNbot, Yahoo! Slurp (yes that is the bot’s name), and other search spiders encounter this link they will effectively not follow it, instead focusing on other available links. More importantly, these bots won’t transfer any of a page’s ranking power to nofollow links.

Figure showing Ranking Power Distribution to three pages

Don’t Dilute Ranking Power

Exactly how Google, Yahoo!, or MSN rank a page is a trade secret, but it is assumed that each search engine looks at least two aspects of every web page: (1) relevancy and (2) importance.

Relevancy describes whether or not a particular page’s content matches a given search query. If I search for “ecommerce tips,” returning pages from the Practical eCommerce site would be relevant. But if I searched for “Russian restaurant in San Francisco,” those same Practical eCommerce pages would likely be irrelevant.

Importance seeks to measure the value of a page’s content. If, for example, there were two web pages with a similar word count and both included the keyword phrase “ecommerce tips,” how would a search engine know which page was a better result for the search query? The answer: links. Search engines count incoming links to determine how important a web page is. And it is for this reason that bot herding matters.

Imagine that there is a page that has ten points of ranking power and ten links. Search engines will divide those ten points of ranking power evenly across all ten links, distributing one vote of importance to each destination page. If we added ten more links, the total ranking power (number of votes) would stay the same, but each destination page would only get a half (0.5) vote. If we added 20 more links, each vote would be worth a quarter (0.25) vote of importance. And remember importance may determine how high a page shows up in search engine results.

Conversely, if we had a page with ten links, and we “fenced off” five of those links using the nofollow attribute, we would pass two votes to each of the open links and no votes to the nofollow/fenced off links. We would be focusing a page’s ranking power on those links we deemed important, herding the spiders and bots in the direction we want them to go.

“Yahoo! Slurp obeys the rel=’nofollow’ attribute for links,” Yahoo! said in its webcrawler FAQs. “Web publishers and blog owners can apply a rel=’nofollow’ attribute to any hyperlink on their page to indicate that the link may not be an approved or trusted link. Note: Slurp may use a ‘nofollow’ link for discovering content, but the link will not be considered an ‘approved’ link for consideration for ranking of the target page.”

Notice what Yahoo! says above, essentially nofollow links won’t boost a target page’s rank.

Putting Up Fences

By focusing ranking power on some links, we hope to improve a target page’s importance and thereby improve how well it performs in search engine results.

So now it is time to put up fences on your site, herding search engine spiders and bots—or more importantly ranking power—to links that matter. For example, if you have a product category page with several links to product detail pages, you want push as much ranking power to those product detail pages as you possibly can. But I will bet that those same product category pages all have links to your store’s privacy policy, terms and conditions, contact us page, about us page, shopping cart, etc. None of these links need to perform well in search engines and, therefore, not an ounce of ranking power should be headed their way.

Put a nofollow fence on all of these sorts of links:

  • Privacy policies
  • Terms and conditions
  • Customer service
  • About us
  • Contact us
  • Newsletter registration
  • Shopping cart
  • Contests
  • Links in product reviews

Focus Ranking Power

Use the rel=”nofollow” attribute on links to any pages that don’t need to perform well in search engines and focus ranking power on your product pages. The technique is simple and the results can be huge.

Bot Herding Video


Armando Roggio

Armando Roggio

Bio   •   RSS Feed


Sign up for our email newsletter

Get the Practical Ecommerce RSS feed

Comments ( 7 )

  1. jordonm December 30, 2008 Reply

    I don’t fully agree with the "nofollow fence". Some of the examples listed actually have very good page rank on my site. My homepage is a 5, my about us, contact, and shipping terms are 4’s. Why would I want a nofollow on those? Please explain –

  2. Armando Roggio December 30, 2008 Reply


    Do you really care if your shipping terms are showing up in Google search results?

    How much better would it be if you could focus that PageRank on a product page and sell something? Or what if by adding rel="nofollow" to every link to your shipping terms page, you could make your home page a 6? Wouldn’t that have a greater effect on your bottom line than having the highest ranked shipping terms page in the world?

    The point here is to decide which pages are the most important (which pages earn you money) and concentrate your ranking power on those pages so that they perform better in search engines.

    In my opinion, a shipping terms page with a PageRank of 4 represents a lot of wasted ranking power.

    Does that make sense?

  3. jordonm December 30, 2008 Reply

    @ Armando

    Yes it does make sense. While shipping terms ranking is not important to me, I’m content with my contact page to have a ranking of 4. Wouldn’t some of this "wasted ranking" trickle down anyways?

    My site has over 40,000 skus…so it’s difficult to focus on single product pages. However I do understand the point you are making and thanks for the article!

  4. Armando Roggio December 30, 2008 Reply


    No. It will not trickle down–if I understand what you mean.

    If you find it hard to focus on a single product page, consider focusing on category pages.

    Also, is it safe to assume that you have a link to your about page, contact us page, shipping terms page, and home page on every other page of your site? (Are these links in your global navigation?) If so, add rel="nofollow" to the about page, contact us page, and shipping terms page links and focus much of that ranking power on your home page. It should see a strong increase in PageRank.

    Finally, it sounds like you are using the Google tool bar to measure PageRank. Keep in mind that tool produces a representative rank, not an actual PageRank score, so you can significantly improve your ranking power—even doubling it—without changing your PageRank on the Google toolbar.

  5. larsomatic December 31, 2008 Reply

    It seems to me that using nofollow links in an artificial way like this is a blatant attempt to control your own PageRank or search engine ranking, and that the search engines will eventually be able to spot it as a technique and either discount it or punish it.

    When you start building your site for search engines and not for end users, that goes down the path of potentially ending up with some kind of a search engine ban.

    From what I see, Bruce Clay’s use of "Siloing" still applies to end users, because it’s a way of organizing your site so that you can see related stuff. I don’t see where he mentions anything about nofollow links.

    I agree with (and use) Bruce’s method all the time.

    But I would hesitate to start messing with a bunch of complicated nofollow stuff that goes against my own rule of creating content and navigation for people, not search engine spiders.

  6. Armando Roggio December 31, 2008 Reply


    Using the _rel="nofollow"_ link is a way to control PageRank, but search engines actually encourage it.

    Note the quote from Yahoo! in the article. If Yahoo! did not want users to control their links, why would they support the _nofollow_ attribute?

    And, actually, if you don’t put a _nofollow_ on ads or other paid links, Google gets really mad. So again, I think this practice actually improves search results.

    Next, I also think highly of Bruce Clay’s methods, but he does in fact suggest using the _nofollow_ attribute. The _nofollow_ attribute is key to what he calls virtual silos–which happen to be the same basic tactic described above. If you need proof, try reading [Hey Rand, Do You Mean Siloing?]( on Bruce Clay’s website.

    Finally, Bruce Clay’s method actually resemble something that information architects have been doing with document development for years—yes, even years before Bruce. I am currently working on an article about the technique, which should be published here on Practical eCommerce in the next couple of weeks.

    You may also want to check out the Practical eCommerce, _Everything SEO_ guide that we will be publishing in January.

    Thank you for reading and commenting.

  7. Carlos Rivera July 21, 2009 Reply

    What is your point of view in adding nofollow to global "category trees" with drop down navigation? Wouldn’t that affect the ability for the bots to completely and regularly spider my entire site?

    What about adding nofollow to a product page’s "breadcrumbs" and "related category" links?

    Should I heard all SEO juice to the homepage and selectively distribute all of my SEO juice from there?

    Thanks, I love your articles.