Google announced that website owners can use the robots.txt protocol to prevent its Googlebot crawler from indexing age-verification popups, even when those popups are visible to users.
This updated guidance offers site administrators more control over which resources Googlebot processes during content rendering, particularly for elements like age gates that are not core content.
John Mueller, a search advocate at Google, confirmed that disallowing the crawling of age-verification popups via robots.txt is an acceptable practice. This can be achieved by either blocking the external domain hosting the popup or by disallowing the specific JavaScript file responsible for loading it.
Previously, Google generally mandated that Googlebot crawl all resources visible to users to accurately render a webpage. The company’s stance on age-verification popups marks a degree of flexibility in this policy for specific scenarios.
Mueller emphasized that the website must still function correctly even if Googlebot does not fetch the blocked resources. This requirement is typically not an issue for age-verification popups, which often overlay content without disrupting core site functionality.
The change provides website owners with a mechanism to manage how Googlebot interacts with certain non-essential, user-facing elements, potentially streamlining the indexing process and avoiding unintended indexing of temporary or interstitial content.
Source: Search Engine Journal
