.Google.com's John Mueller addressed a concern concerning why Google marks web pages that are actually prohibited coming from crawling through robots.txt and also why the it is actually risk-free to ignore the associated Search Console records concerning those crawls.Bot Website Traffic To Query Parameter URLs.The individual inquiring the inquiry recorded that crawlers were producing web links to non-existent concern guideline Links (? q= xyz) to web pages along with noindex meta tags that are additionally blocked out in robots.txt. What caused the concern is actually that Google.com is crawling the hyperlinks to those webpages, getting shut out by robots.txt (without watching a noindex robotics meta tag) then obtaining reported in Google Search Console as "Indexed, though blocked by robots.txt.".The person asked the following question:." However here is actually the significant concern: why would Google mark pages when they can't even observe the web content? What's the perk because?".Google's John Mueller validated that if they can't creep the web page they can not view the noindex meta tag. He also creates a fascinating acknowledgment of the internet site: hunt operator, recommending to neglect the results because the "common" users won't find those results.He composed:." Yes, you're appropriate: if our company can't creep the web page, our team can not view the noindex. That claimed, if our team can't creep the pages, after that there is actually not a lot for us to index. So while you may find several of those webpages along with a targeted website:- query, the average user will not see them, so I wouldn't fuss over it. Noindex is actually likewise alright (without robots.txt disallow), it simply indicates the Links will definitely find yourself being crept (as well as wind up in the Explore Console record for crawled/not listed-- neither of these conditions cause problems to the remainder of the site). The fundamental part is that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the constraints in operation the Website: hunt evolved search operator for diagnostic reasons. One of those explanations is because it's not linked to the normal search mark, it's a different trait entirely.Google's John Mueller commented on the site hunt driver in 2021:." The short answer is actually that a website: inquiry is actually certainly not indicated to be total, nor made use of for diagnostics purposes.An internet site question is actually a particular kind of search that confines the end results to a specific website. It is actually generally simply the word website, a colon, and after that the site's domain.This inquiry restricts the end results to a certain site. It is actually certainly not implied to be an extensive selection of all the webpages from that internet site.".2. Noindex tag without making use of a robots.txt is actually fine for these kinds of conditions where a crawler is actually connecting to non-existent web pages that are acquiring uncovered by Googlebot.3. Links with the noindex tag are going to generate a "crawled/not catalogued" item in Explore Console and that those will not have a damaging effect on the rest of the website.Go through the question and respond to on LinkedIn:.Why will Google mark web pages when they can't even observe the material?Featured Photo by Shutterstock/Krakenimages. com.