Search Engine Spiders Lost Without Guidance - Post This Sign!
Monday, July 7th, 2008The robots.txt file is an exclusion standard required by all
web crawlers/robots to tell them what files and directories
that you want them to stay OUT of on your site. Not all
crawlers/bots follow the exclusion standard and will continue
crawling your site anyway. I like to call them “Bad Bots” or
trespassers. We block them by IP exclusion which […]