robots.txt file for SEO

Get To Know About robots.txt-Best SEO Practices 2020

Last Updated on by Sumi

What is robots.txt? 

Generally, a robots.txt file tells search engines about your website’s format of engagement. The main thing about SEO is to send the right signals to search engines and this robots.txt, is one of the ways to communicate your preferences to search engines. 

In other words, this prevents web crawlers from accessing part of a website. It contains commands to search engines to tell any of the web pages can be indexed or not. Talking about the main aim of this manages the crawl time of the robot by forbidding with low added value. 

Why does robots file matter? 

Well, having robots.txt file isn’t that important for many websites, especially small ones. 

 Robots.txt File Is Important For SEO
Importance of robots.txt

It is said that there’s no good reason not to have one. This will give you more control to where search engines can and can’t go through your website, and can help with: 

  • Prevents the crawling of duplicate content
  • Keeps sections of a website private 
  • Prevents server overload 
  • Prevents images and other files appearing in search results. 

Note: Google doesn’t index web pages that are blocked in robots.txt. 

You should be careful while making changes to your robots file. Because this file has the ability to make big parts of your website inaccessible for search engines. 

Where to put the robots file? 

You don’t know if you have a robots.txt file or not? Then: 

  1. Type your root domain and, 
  1. Add ‘/robots.txt’ at the end of the URL. 

 For instance, the robots file for “Panorabanks” can be found at https://www.panorabanques.com

How to create a robots file? 

Basically, its file contains some rules i.e. the format, syntax, and location. Talking about, format, and location, you can use any editor to create its file. However, the editor should be able to create ASCII or UTF-8 files. 

Format and rules 

  • The robots.txt file must be placed at the root of the server/site. 
  • It cannot be placed in a subdirectory but can be applied to subdomains. 
  • Its file name must be in lower case. 
  • You can have only one file. 

Good practices for SEO

  • Be sure that not to block the content of your website that you wish to explore. 
  • Links on pages blocked by robots.txt will not be followed. 
  • Don’t use it to prevent sensitive data from being displayed in search engine results pages. If you are sure to block your page from search results, you can use a different method, like password protection. 
  • Sometimes search engines have multiple users. Like, Google uses Googlebot for natural search and similarly, Googlebot-Image for image search. And most user agents of the same search engine follow the same rules. So, it is not necessary to specify rules for the different robots in a search engine. 

Keep on learning

Pooja Karki

Pooja is Web Developer and Content Writer. She is fond of anything that is related to writing and coding and really passionate about her work. Her aim is to reach the goals and doing everything with a smile. Besides that, she loves traveling and reading novels.

More From Author

nepal telecom

Nepal Telecom services will be affected tonight

Oppo Watch

Oppo Watch with Wear OS and Snapdragon Wear 3100 launching soon in Nepal