When search engine crawlers (robots) look at a website, the first file they will look at is not your index.html page. It is your robots.txt file.
This little file that sits in the root “/” of your website contains instructions on what files the robot can and cannot look at within the website.
Read More:
http://www.hallaminternet.com/2014/the-importance-of-a-robots-txt-file/
Comments
Post a Comment