I'm with you on this, not for huge sites but if even 1% of sites used this behav...

cmdrfred on May 18, 2015 | parent | context | favorite | on: What one may find in robots.txt

I'm with you on this, not for huge sites but if even 1% of sites used this behavior malicious bots would be forced to honor robots.txt