Thread: Important Technical SEO Checklist in 2018

Originally Posted by Bhavya

Thanks for the reply katern, Yeah I knew about these factors and you also gave me a brief explanation, once again thank you

I'm so happy to explain to you regarding the Tech SEO If you need any clarification regarding this, Don't hesitate and get back to me! I'm here to help you always

Originally Posted by Bhavya

Hi

Originally Posted by Bhavya

Katern
Can you give me a little deep explanation about Index page andRobots.txt ? Thanks in advance.

Hello Bhavya,


Okay, sure, Let me explain it simply




Actually, The Robots.txt means, it tells to the search engine which page needs to crawl and which page no need to crawl in the particular site. If you don't want to crawl any particular page in your web, you simply define it with disallow.


This is a format of Creating robots.txt file:


Basic format:
---------------
User-agent: [user-agent name]
Disallow: [URL string not to be crawled]


And also we should focus on the following method as well.

• The robots.txt file must be placed in a websiteÂ’s top-level directory (root).
• Robots.txt is case sensitive: the file should be named “robots.txt” (not Robots.txt, robots.TXT).
• If you have any subdomain, you need to create the robots.txt file separately. Like this: (at blog.abcd.com/robots.txt and abcd.com/robots.txt)

This is the way to check the robots. txt file: https://domain.com/robots.txt

Hope you can understand