# A robots.txt file on a website functions as a request that specified robots # ignore specified files or directories in their search. # See the defintion on http://en.wikipedia.org/wiki/Robots.txt User-agent: * # for all bots Disallow: # allow all robots to visit all files