User-agent: * Disallow: /
disallows any robot to retrieve any page from your site. The "Disallow:" field can be used to prevent robots from retrieving pages under the specified directory path. For example:
User-agent: * Disallow: /webstat/
disallows accesses to pages under "/webstat/" directory. Multiple "Disallow:" fields is permitted. Empty "Disallow:" means any request is allowed. The "User-agent:" field can be used to specify a particular Web robot, by naming the robot's User-Agent identification name. If no robots.txt file is found, by convention, a robot can assume that any access request is allowed.