How to Create Robots.txt File?
Sometimes, you have directories or files that want not to be crawled by search engines. With creating robots.txt file, you can control access of search engine spiders to your web site.
The first file that spiders search in a website is Robots.txt file.
Use Notepad to Create robots file
Creating Robots.txt file is very simple. The extension of file is TXT, so you can use Notepad to create it.
If you want to allow search engines crawl entire of your website only type the below words:
User-agent: *
Disallow:
Then save the file as TXT format and upload it to the root directory of your site.
If you want to disallow some specific files or directories to be crawled act as below: This allows all robots to crawl all files except the images directory and the stats directory.
User-agent: *
Disallow: /images/
Disallow: /stats/
If you want to disallow to a particular robot, you should write the name of robot:
This disallow Google robot to images directory.
User-agent: Googlebot-Image
Disallow: /images/
Do you want to Save your Time & Learn SEO Sooner?! Read:
Aaron Wall's SEO Book
Related Articles
SEO Home > SEO > How to create Robots.txt file? |
