ChatGPT develops ‘Labrador’ search index to reduce Google reliance

Image credit: Abondance

ChatGPT developed a proprietary family of search indexes, named Labrador, to reduce its reliance on external search providers and enhance the accuracy of its query responses, the company announced.

The initiative, which began in 2023, represents a significant strategic shift for OpenAI‘s conversational AI, aiming to process a substantial majority of user queries through its own indexing system.

Nick Turley, head of ChatGPT, testified during Google‘s antitrust trial that the project’s goal was to answer 80 percent of queries using its internal index by the end of 2023.

Labrador encompasses a wide array of specialized sub-indexes, covering general web content, PDF documents, YouTube videos, news articles, arXiv papers, Wikipedia entries, local information, finance, legal, medical, shopping and images.

OpenAI is currently conducting A/B testing of Labrador against results obtained through web scraping, particularly for shopping-related queries, with these experiments affecting up to 20 percent of user conversations, according to Turley.

Unlike a single monolithic index, Labrador operates as a family of distinct indexes, storing comprehensive page content, crawl dates and publication dates, similar to the architecture of traditional search engines.

The development of an exabyte-scale indexing system is further corroborated by OpenAI job postings for roles such as ‘Software Engineer, Foundations Search’ and ‘Engineering Manager, Online Data Systems.’

The decision to build an internal solution followed Google’s refusal to grant OpenAI full access to its extensive search index data, prompting OpenAI to create its own independent infrastructure.

This move positions ChatGPT to potentially challenge established search engine models by integrating its own indexed content directly into AI responses, rather than relying solely on third-party search results.

Source: Abondance