Google had internal discussions about allowing the noindex
Posted: Sun Apr 06, 2025 5:04 am
Google's John Mueller revealed in a recent LinkedIn post that Google has been having internal discussions about implementing a noindex directive in robots.txt . This directive, which is currently non-standard, would allow publishers to block both crawling and indexing of content.
Mueller explained that the idea of introducing the noindex directive was considered 10 years ago. The reason was to make it easier for publishers to block content from being indexed without having to use a robots.txt file and robots meta tags.
Ultimately, Google decided not to support this dentist data directive, fearing that it could easily lead to critical parts of a website being accidentally removed from the search engine index. Many people copy and paste robots.txt files without careful review, potentially inadvertently blocking important content.
Mueller's robots.txt file has been causing a stir over the past week due to its unusual use of non-standard directives. Some SEOs believe that Mueller's robots.txt file is serving as a testing ground for various experiments. Others believe that Mueller is simply making mistakes.
Whatever the truth, this episode highlights the importance of following robots.txt standards. Using non-standard directives can lead to unpredictable results and make the job of search engine crawlers more difficult.
Mueller explained that the idea of introducing the noindex directive was considered 10 years ago. The reason was to make it easier for publishers to block content from being indexed without having to use a robots.txt file and robots meta tags.
Ultimately, Google decided not to support this dentist data directive, fearing that it could easily lead to critical parts of a website being accidentally removed from the search engine index. Many people copy and paste robots.txt files without careful review, potentially inadvertently blocking important content.
Mueller's robots.txt file has been causing a stir over the past week due to its unusual use of non-standard directives. Some SEOs believe that Mueller's robots.txt file is serving as a testing ground for various experiments. Others believe that Mueller is simply making mistakes.
Whatever the truth, this episode highlights the importance of following robots.txt standards. Using non-standard directives can lead to unpredictable results and make the job of search engine crawlers more difficult.