Blocking Search Engine Spiders?

Most websites can’t resist the traffic that Google, MSN and Yahoo provide, but there seems to be a new trend developing where large sites of destination are denying the Search Engines (‘SE’) indexing access.

A month or so ago, Brett Tabke of Webmasterworld (by far, the largest webmaster forum on the internet) pulled access to all SE spider bots. There was much speculation on why Brett had done this, and more specifically, many were wondering if he was testing his website’s dependency on the SEs. If any content based website could pull this off, this was one of them. Tabke’s explanation, as it turns out, was that he was trying to get rid of the rogue spiders, rather than those of the major SEs.

And now as of yesterday, we have Craigslist doing the same thing.

Over the years, the SEs have been the major brokers of web content and traffic. Whether they are saying so or not, the owners of these major websites of destination are testing the waters. With dedicated communities behind them, and word-of-mouth viral growth on their side, these sites are in a position to limit scraper sites from stealing their content, and reduce their dependency on SE traffic.

Finding content on the internet has become simplified with the Googles of the world, but could become much more complicated if every SE wasn’t indexing every website!

Also, I have to wonder if we’re seeing a bit of a power shift? Wouldn’t it be interesting if some of these websites of destination started auctioning off spider access to the highest bidder? Websites and SE’s have traditionally exchanged free indexing access for free search traffic, but what if Yahoo or MSN cherry picked 50 top websites of destination and licensed exclusive indexing rights. Would that shift the search market away from Google?

A very interesting trend indeed.