Using the Meta Robots Tag for SEO
What Is The Robots Meta Tag and Does It Matter?
The robots meta tag is an HTML tag that goes the
<head> of a page and provides instructions to bots. Like the robots.txt file, it tells search engine crawlers whether or not they are allowed to index a page. You can also use it to tell search engines whether or not to follow links on the page. Unlike the robots.txt file, the meta robots tag only applies to the page it’s on, so you can’t use it to disallow entire directories or file types.
The robots meta tag matters because it adds an extra layer of protection to the robots.txt file. When a crawler follows an external link and lands on one of your pages, it can still crawl and index that page because it hasn’t seen the robots.txt file. The robots meta tag prevents this crawling and indexing from happening.