Claude AI Chats Indexed by Google Despite Noindex Directives

Image credit: Search Engine Journal

Shared conversations from Anthropic‘s Claude AI unexpectedly appeared in Google search results over the weekend, raising privacy concerns despite the company’s use of noindex directives.

The indexing occurred because Google’s crawlers were blocked by robots.txt from accessing pages containing noindex tags, preventing the directives from being processed, according to Google guidance.

TechCrunch reported discovering sensitive material, including medical information and internal company documents, among the indexed Claude share pages.

Anthropic stated that share links only appear in search results if they have been publicly posted and are not otherwise guessable or discoverable unless explicitly shared.

Google’s guidance specifies that a noindex tag is only effective if the crawler can access the page; if a page is blocked by robots.txt, it can still be indexed if linked from other accessible pages.

John Mueller, a Search Advocate at Google, clarified on social media that disallowing a page via robots.txt does not equate to a noindex directive.

Martin Splitt, a Developer Advocate at Google, further explained that if Googlebot is blocked by robots.txt, it cannot see the noindex tag and therefore may index the URL if it discovers it through other means.

The issue is not exclusive to Anthropic, as customer chats from other AI platforms, including OpenAI and Google Bard, have previously appeared in search results, according to reports from 404 Media.

Daniel J. Glover, a Google spokesperson, told The Decoder that the company provides clear documentation on how to prevent content from appearing in search results.

Amie Rotherham, a spokesperson for Anthropic, told TechCrunch that the company was investigating the reports and taking steps to address the issue.

Ned Adriance, another Google spokesperson, stated that Google’s systems accurately reflect the directives provided by websites and that webmasters have tools to manage their content’s visibility.

Google’s official documentation advises webmasters that robots.txt disallow rules prevent crawling but do not guarantee exclusion from the index, recommending the use of noindex tags or password protection for sensitive content.

Source: Search Engine Journal