# Robots.txt file from http://www.choding.com/ # # All robots will spider the domain. User-agent: * Disallow: / User-agent: AboutUsBot Disallow: / User-agent: ia_archiver Disallow: / User-agent: Browsershots Disallow: