Quote Originally Posted by Bhavya View Post
Hi
Quote Originally Posted by Bhavya View Post
Katern
Can you give me a little deep explanation about Index page andRobots.txt ? Thanks in advance.



Hello Bhavya,


Okay, sure, Let me explain it simply




Actually, The Robots.txt means, it tells to the search engine which page needs to crawl and which page no need to crawl in the particular site. If you don't want to crawl any particular page in your web, you simply define it with disallow.


This is a format of Creating robots.txt file:


Basic format:
---------------
User-agent: [user-agent name]
Disallow: [URL string not to be crawled]


And also we should focus on the following method as well.



  • The robots.txt file must be placed in a websiteÂ’s top-level directory (root).
  • Robots.txt is case sensitive: the file should be named “robots.txt” (not Robots.txt, robots.TXT).
  • If you have any subdomain, you need to create the robots.txt file separately. Like this: (at blog.abcd.com/robots.txt and abcd.com/robots.txt)


This is the way to check the robots. txt file: https://domain.com/robots.txt

Hope you can understand