UPDATED ROBOTS.TXT FOR BLOGGER.



blogger custom robots.txt file has allowed us to block or diallow any folders or pages on our website or blog by our own selves.we just have to edit custom robots.txt file and rest of work will do crawlers.



by default,custom settings for robots.txt for blogger is as follows.



User-agent: Mediapartners-Google
Disallow: 


user-agent: *

disallow: /search

allow: /

Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml redirect=false&start-index=1&max-results=500



where as the the terms are explained as;

user-agent is for all robots.do not change this code or you will have troubles and troubles.

google mediapartners-google: is a google adsense crawler from google.its crawles your website for matching the ads that fits best with your content.changing this will destroy your adsense earning.


disallow: you can use this file to allow or block any pages or folders on your website.by default.blog labels are not indexed as it is blocked by robots.txt.


latest update:

after google pagination update,we use the following configuration file to block google pagination page.




User-agent: Mediapartners-Google
Disallow: 

User-agent: *
Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=
Allow: /

Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml redirect=false&start-index=1&max-results=500





Share your views...

1 Respones to "UPDATED ROBOTS.TXT FOR BLOGGER."

Admin said...

Hi, may I request for the recommended values for the ffg:

Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=

Thanks


8 April 2013 at 04:12

Post a Comment

 

Contributors

Our Partners

© 2010 BLOGSPOT TUTORIALS ONLINE| USMAN MOHIUDDIN 123 All Rights Reserved Thesis WordPress Theme Converted into Blogger Template by Hack Tutors.info