Fintech Websites Often Inaccessible to AI Crawlers, Study Finds

Image credit: Search Engine Journal

LISBON, Portugal — More than one-third of leading fintech websites are largely inaccessible to artificial intelligence crawlers due to their reliance on client-side JavaScript for content rendering, according to a new study.

The research, conducted on May 25, 2026, found that 36 percent of top fintech homepages deliver less than 80 percent of their content in raw HTML, a format most AI agents cannot process.

Most AI crawlers, including GPTBot, ClaudeBot, and PerplexityBot, typically process only the raw HTTP response of a webpage and do not render JavaScript, a common method for dynamically loading web content, the study reported.

The analysis by Web Performance Tools examined 274 fintech homepages from the CNBC World’s Top Fintech Companies 2025 list, comparing content fetched via raw HTTP requests with content rendered by a full browser.

Fifty-five websites, representing 20 percent of the sample, returned less than 30 percent of their content without JavaScript. Among these, 47 websites delivered zero content in their raw HTML, making them effectively blank pages to non-rendering AI agents.

This widespread dependency on client-side JavaScript for essential content poses a significant challenge for AI visibility and content discoverability within the fintech sector, according to Web Performance Tools.

The findings underscore the importance of adhering to the ‘Structure’ pillar of Machine-First Architecture, which mandates that critical information should not rely on client-side JavaScript to ensure AI visibility, Web Performance Tools stated.

Google’s search engine, for instance, has advanced JavaScript rendering capabilities, but many newer AI crawlers lack this functionality, creating a disparity in how web content is indexed and understood by different AI systems.

The research suggests that fintech companies may inadvertently be limiting their reach and discoverability by AI agents that power various applications, from search to data analysis, if they do not optimize their content delivery methods.

Ensuring that key information is available in the initial HTTP response is important for broader AI engagement, Web Performance Tools said, highlighting the need for developers to consider AI crawler limitations.

Source: Search Engine Journal