Cloudflare and OpenAI significantly expanded ChatGPT’s search capabilities by indexing hundreds of millions of new web pages through a collaborative research pilot, the companies announced Tuesday.
The pilot program leveraged approximately 1 percent of Cloudflare’s extensive network to assist OpenAI’s search crawler in discovering and integrating a vast number of previously unindexed web pages, enhancing the breadth of information available through ChatGPT.
Cloudflare’s network signals played a key role in guiding OpenAI’s search crawler to valuable web pages, including those with limited external links, according to the companies.
These signals also helped crawlers avoid unnecessary requests and identify potentially harmful content, such as fake login forms, improving overall crawling efficiency and safety.
OpenAI operates its crawler with distinct directives for search (OAI-SearchBot) and model training (GPTBot), allowing website operators to manage how their content is utilized, the company stated.
Websites can further control AI service access through Cloudflare’s AI Crawl Control service, which provides tools to manage which artificial intelligence services can access their sites.
The next phase of development for the pilot includes improving AI page selection algorithms and testing indicators for rapid content changes.
Cloudflare also plans to measure the results of new content discovery and aims to provide broad access to these improved content discovery capabilities by the end of 2026.
The collaboration underscores an ongoing effort to enhance the foundational data available to large language models and improve the accuracy and comprehensiveness of AI-powered search functions globally.
Source: Search Engine Journal
