what is a robots.txt?
a robots.txt file is a file that tells search engines weather what pages to crawl and what not to crawl or block access to some specific pages and directory.this is very helpful for websites that want to block access to some important pages on thier websites.simply you are telling a search engine not to index those pages.
default robots.txt file for blogpsot blogs:
the default robots.txt file for blogspot blog looks just like this one;
this file has been causing errors as reported by some bloggers.so google introduced a new custom robots.txt feature for blogspot blog where you can set the robots.txt file confiqured by yourself.
how to set custom robots.txt for blogspot blogs.
there are three main components of blogger robots.txt file.
1-google adsense crawler
3-user agent(search engines)
this is the complete and best confiquaration file for custom robots.txt for blogspot blogs.it allows all pages to be index expect search labels.you will have better crawling and fast indexing for blogspot blogs.
Tags: blogspot tutorial