Blocking Search Engine Spiders?
Most websites can’t resist the traffic that Google, MSN and Yahoo provide, but there seems to be a new trend developing where large sites of destination are denying the Search Engines (‘SE’) indexing access.
A month or so ago, Brett Tabke of Webmasterworld (by far, the largest webmaster forum on the internet) pulled access to all SE spider bots. There was much speculation on why Brett had done this, and more specifically, many were wondering if he was testing his website’s dependency on the SEs. If any content based website could pull this off, this was one of them. . . . [more]


