Remove entire website

If you wish to exclude your entire website from Google’s index

Remove site from search engines and prevent all robots from crawling it in the future

User-agent: *
Disallow: /

Note: Please note that Googlebot does not interpret a 401/403 response (”Unauthorized”/”Forbidden”) to a robots.txt fetch as a request not to crawl any pages on the site.

To remove your site from Google only and prevent just Googlebot from crawling your site in the future

User-agent: Googlebot
Disallow: /

Allow Googlebot to index all http pages but no https pages

Each port must have its own robots.txt file. In particular, if you serve content via both http and https, you’ll need a separate robots.txt file for each of these protocols.
For your http protocol (http://yourserver.com/robots.txt)

User-agent: *
Allow: /

For the https protocol (https://yourserver.com/robots.txt)

User-agent: *
Disallow: /

Tags: , , , ,

Bookmark it & read later!
  • Digg
  • del.icio.us
  • Netvouz
  • DZone
  • ThisNext
  • MisterWong
  • Wists
  • StumbleUpon
  • De.lirio.us
  • Fleck
  • Linkter
  • Reddit
  • Slashdot
  • Smarking
  • YahooMyWeb
  • BlinkList
  • co.mments
  • SphereIt
  • blinkbits

Back to TOP | SEO101 Homepage  

SEO102

Sponsores



TeloTechSEOSEO Web Design PortfolioTeloTech SEO Web Design Portfolio Club Cafe CafeCoffee