UPDATED ROBOTS.TXT FOR BLOGGER.
blogger custom robots.txt file has allowed us to block or diallow any folders or pages on our website or blog by our own selves.we just have to edit custom robots.txt file and rest of work will do crawlers.
by default,custom settings for robots.txt for blogger is as follows.
User-agent: Mediapartners-Google
Disallow:
user-agent: *
disallow: /search
allow: /
Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml redirect=false&start-index=1&max-results=500
where as the the terms are explained as;
user-agent is for all robots.do not change this code or you will have troubles and troubles.
google mediapartners-google: is a google adsense crawler from google.its crawles your website for matching the ads that fits best with your content.changing this will destroy your adsense earning.
disallow: you can use this file to allow or block any pages or folders on your website.by default.blog labels are not indexed as it is blocked by robots.txt.
latest update:
after google pagination update,we use the following configuration file to block google pagination page.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=
Allow: /
Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml redirect=false&start-index=1&max-results=500
by default,custom settings for robots.txt for blogger is as follows.
User-agent: Mediapartners-Google
Disallow:
user-agent: *
disallow: /search
allow: /
Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml redirect=false&start-index=1&max-results=500
where as the the terms are explained as;
user-agent is for all robots.do not change this code or you will have troubles and troubles.
google mediapartners-google: is a google adsense crawler from google.its crawles your website for matching the ads that fits best with your content.changing this will destroy your adsense earning.
disallow: you can use this file to allow or block any pages or folders on your website.by default.blog labels are not indexed as it is blocked by robots.txt.
latest update:
after google pagination update,we use the following configuration file to block google pagination page.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=
Allow: /
Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml redirect=false&start-index=1&max-results=500
Tags: blogspot tutorial
Previous Article
Subscribe to:
Post Comments (Atom)
Share your views...
1 Respones to "UPDATED ROBOTS.TXT FOR BLOGGER."
Hi, may I request for the recommended values for the ffg:
Disallow: /search?updated-min=
Disallow: /search?updated-max=
Disallow: /search/label/*?updated-min=
Disallow: /search/label/*?updated-max=
Thanks
8 April 2013 at 04:12
Post a Comment