Robots.txt: Do All Bots Comply with E...July 26, 2012 / Search Engines Unfortunately, not all search bots and spiders comply with robots exclusion rules; nor do they have to either. While ...