NEW YORK — Jeremy Howard of Answer.AI released version two of the llms.txt specification on August 10, introducing formal Markdown linking to enhance capabilities for AI agents and documentation platforms.
The update standardizes how AI agents discover and utilize Markdown versions of web pages, building on two years of practical experience with the file format.
llms.txt V2 supports two primary URL patterns for Markdown pages: appending .md to an existing filename, such as /page.html.md, or replacing the original extension with .md, as in /page.md.
New link relations were introduced to facilitate this discovery, according to Howard. Agents can now use rel=”alternate” type=”text/markdown” to find Markdown versions of content and rel=”describedby” to locate the covering llms.txt file.
These link relations can be implemented either through HTML <link> elements within a web page or via HTTP Link response headers, with the latter offering greater versatility for web developers.
Thousands of websites currently publish llms.txt files, and platforms like Mintlify have adopted the specification, indicating its growing acceptance within the developer community, Answer.AI reported.
Despite the update, Google’s Search team has consistently stated that it does not use llms.txt for search visibility or ranking purposes. However, Google’s Chrome Lighthouse tool includes an Agentic Browsing check that specifically looks for the presence of the llms.txt file.
The V2 update clarifies the format’s intended purpose, focusing on its utility for coding agents, integrated development environment (IDE) tools, and documentation platforms, rather than making a new argument for its relevance to search engine optimization.
Organizations like Anthropic and OpenAI, which develop large language models, could potentially leverage these formalized Markdown links to improve how their AI agents process and understand web content.
Source: Search Engine Journal
