HOW TO SET CUSTOM ROBOT.TXT FOR BLOGGER BLOG




what is a robots.txt?


a robots.txt file is a file that tells search engines weather what pages to crawl and what not to crawl or block access to some specific pages and directory.this is very helpful for websites that want to block access to some important pages on thier websites.simply you are telling a search engine not to index those pages.

default robots.txt file for blogpsot blogs:


the default robots.txt file for blogspot blog looks just like this one;

Sitemap: http://www.handmadereviews.net/feeds/posts/default?orderby=UPDATED

this file has been causing errors as reported by some bloggers.so google introduced a new custom robots.txt feature for blogspot blog where you can set the robots.txt file confiqured by yourself.


how to set custom robots.txt for blogspot blogs.

there are three main components of blogger robots.txt file.

1-google adsense crawler

2-sitemap file

3-user agent(search engines)


User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /


Sitemap: http://usmanmohiuddin123.blogspot.com/atom.xml?redirect=false&start-index=1&max-results=500


this is the complete and best confiquaration file for custom robots.txt for blogspot blogs.it allows all pages to be index expect search labels.you will have better crawling and fast indexing for blogspot blogs.





Share your views...

2 Respones to "HOW TO SET CUSTOM ROBOT.TXT FOR BLOGGER BLOG"

Anonymous said...

Thanks for this article. Traffic on my blog www.funWithPuzzles.com was going down and wrong values in robots.txt was the reason for it.

I have updated now robots.txt file now.


11 February 2013 at 13:39
Kratos Olympian said...

Is that configuration for those with adsense?


5 April 2013 at 12:51

Post a Comment

 

Contributors

Our Partners

© 2010 BLOGSPOT TUTORIALS ONLINE| USMAN MOHIUDDIN 123 All Rights Reserved Thesis WordPress Theme Converted into Blogger Template by Hack Tutors.info