Blocking robots and spiders from crawling your site 

  
To disallow the spiders and block all robots from visiting your site, add the following in your robots.txt

User-agent: *
Disallow: /