Quote Originally Posted by Katren View Post



Hello Bhavya,


Okay, sure, Let me explain it simply




Actually, The Robots.txt means, it tells to the search engine which page needs to crawl and which page no need to crawl in the particular site. If you don't want to crawl any particular page in your web, you simply define it with disallow.


This is a format of Creating robots.txt file:


Basic format:
---------------
User-agent: [user-agent name]
Disallow: [URL string not to be crawled]


And also we should focus on the following method as well.



  • The robots.txt file must be placed in a websiteÂ’s top-level directory (root).
  • Robots.txt is case sensitive: the file should be named “robots.txt” (not Robots.txt, robots.TXT).
  • If you have any subdomain, you need to create the robots.txt file separately. Like this: (at blog.abcd.com/robots.txt and abcd.com/robots.txt)


This is the way to check the robots. txt file: https://domain.com/robots.txt

Hope you can understand
Thanks so much for this simple and brief explanation, yeah I understood it, I think you forgot to tell about index page.