Google cautions against separate AI markdown for websites

Image credit: Search Engine Journal

Google’s John Mueller advised against developing separate markdown versions of websites specifically for artificial intelligence agents, stating that a well-designed site should inherently serve both human users and AI.

Mueller said a website built with proper accessibility and user experience in mind will function effectively for search engines, large language models (LLMs) and people.

Creating distinct “agent-friendly” versions to address underlying accessibility issues introduces technical debt and necessitates repeated reworks, according to Mueller.

The guidance followed a February announcement by Cloudflare that its infrastructure could automatically generate markdown files. Cloudflare indicated these files could help developers reduce LLM token consumption and noted that OpenAI‘s search crawler, OAI-SearchBot, was consuming markdown pages.

OpenAI’s official documentation for OAI-SearchBot describes its function as crawling websites for search results within ChatGPT, but it does not specifically recommend the use of markdown files.

Separately, Stephanie Walter highlighted a concern that some websites are prioritizing AI accessibility over human needs. Walter noted that certain sites offer text versions for LLMs while neglecting fundamental accessibility requirements for human users.

Industry experts have long pointed to user behavior signals as a consistent driver of search engine rankings. This makes universal website accessibility a logical and beneficial strategy for search engine optimization (SEO) that serves all audiences.

Source: Search Engine Journal