OpenAI uses three-tier system for ChatGPT information retrieval

Image credit: Abondance

OpenAI employs a previously undocumented three-tier system for ChatGPT’s information retrieval, utilizing an internal web index, a reading cache, and live-opened pages, according to new research released by digital marketing firm RESONEO and technical SEO platform Oncrawl.

The findings reveal a complex, cost-driven architecture that prioritizes OpenAI‘s own resources before accessing external search results.

RESONEO’s analysis, based on 1,200 ChatGPT responses and 88,000 associated search results, identified internal pipelines named ‘labrador,’ ‘bright,’ and ‘oxylabs’ that manage the sourcing of information.

OpenAI maintains its own proprietary web index, dubbed ‘labrador’ by the researchers, which is distinct from Microsoft’s Bing search engine, according to the report. This internal index reportedly supplies nearly all citations for ChatGPT’s instant-mode searches.

The ‘labrador’ index provides full page titles and query-independent snippets, typically around 200 characters, often extracted from H1 headings and surrounding text while disregarding meta descriptions, the research indicated.

For queries demanding near real-time data or when the internal ‘labrador’ index proves insufficient, ChatGPT reportedly leverages scraped Google results via pipelines identified as ‘bright’ and ‘oxylabs.’

Jérôme Salomon, CEO of RESONEO, stated that ChatGPT’s retrieval strategy is primarily cost-driven, utilizing its internal ‘labrador’ hub for free, instant responses and only resorting to external sources when necessary.

The research also noted that specialized feeds handle shopping and local results, operating independently of the general web search mechanisms. These include dedicated feeds like P1, P2, and P3 for shopping, and B1, B3, Yelp, and Tripadvisor for local information.

These specialized feeds do not interact with the web search component of ChatGPT’s retrieval stack, according to the findings.

Source: Abondance