These weeks the micro-world of SEO has been convulsive, although it is always. It is especially due to the fact that Google has announced “modifications” relevant in Robots.txt. The two most significant changes are born through the company’s idea to create a true standard with this file and the way in which the bots understand it. That added to the confirmation that some directives do not have a real impact on Googlebot.
One of these directives is the noindex within the robots file. I do not know why someone would still use this in 2019, but there are still sites that implement it. Additionally, noindex through robots.txt is not the only directive that will disappear.
Since these rules have never been documented by Google, their use and how they affect Googlebot is very low. Using them creates errors that damage the presence of sites in search results in a way that we believe is not intentional by webmasters.
Contenidos
What are the options to not index content in robots.txt
It’s not that this has changed much, but with the announcement of Google many false myths about the use of this file have fallen like a house of cards. The current rules for not indexing content or removing it from search results remain the same:
Using NoIndex in headers and HTML
Generally and assuming that today the vast majority of users are using a CMS like WordPress, Joomla, PrestaShop or Magento, this is done easily through plugins and extensions like Yoast SEO, All In One SEO Pack and their counterparts for the rest of content managers.
In the case of not having SEO plugins or having a custom project, you can always resort to modifying the code of each URL to send the signal to Google.
Noindex through htaccess
You have the option to block certain URLs or directories to bots through the htaccess file of your server. Perhaps this is one of the best solutions, but the least flexible for my personal taste. If you definitely want Google not to index a URL in any way, this is the best option.
Other alternatives
You can also use the 410 codes or use the Disavow option through Search Console in case the content is already indexed. Use the latter option with caution since it is not very recommended to abuse it, besides it is always better to apply noindex from the headers than a Disavow.
The company from Mountain View assures that, unless we indicate otherwise, it is still effective to protect content with a password that we do not want our bot to crawl. Logically, this last point depends a lot on the context of the content behind.
The standard for robots.txt
As I said just a few lines above, the big G has the intention of making its way of understanding the robots.txt file a standard for the web. So much so that they have released part of the documentation so that webmasters can learn something better about how Googlebot behaves when using the file, with it you can even make simulations on your local server.
I leave you access also to this link where you can find the official documentation of Robots.txt, but I anticipate that only the following elements are allowed:
- Allow.
- Disallow.
- User-Agent.
- Sitemap.
If you have never done it before, it is a recommended reading especially to understand the levels of priority by groups and how to add directories and URLs.