The SEO Insights Blog

AI robot connected to a network of server nodes

OpenAI says robots.txt may not apply to ChatGPT’s fetch bot

OpenAI argues robots.txt rules may not apply to ChatGPT’s fetch bot for user-initiated requests, leading to widespread d...

Palumbo Angela

August 14, 2026

Common Crawl Published a Manual for Being Visible to AI. I Automated It.

Automated Tool Simplifies AI Visibility Audit for Websites

Discover the new automated tool that simplifies Common Crawl’s AI Visibility Audit, helping websites understand and mana...

Joyce de Castro

August 10, 2026

What Top Stories Inside AI Overviews Means For Publishers And Brands In 2026 And Beyond

Google embeds Top Stories in AI Overviews, impacting publishers

Google is integrating Top Stories into AI Overviews. Discover the impact on publishers, ineffective opt-out methods, and...

Saeed Ashif Ahmed

August 4, 2026

Google's John Mueller and Martin Splitt said that spammed search box pages may be treated as hacked and become a quality issue.

Google warns webmasters of search box spam, ‘hacked’ site flags

Google’s John Mueller and Martin Splitt explained how website search box spam can cause site quality issues and ‘hacked’...

Saeed Ashif Ahmed

July 31, 2026

Google's John Mueller explains the SEO impact of random URLs injected into web pages by CMS platforms.

Google: CMS-generated internal URLs do not harm SEO rankings

John Mueller of Google clarifies that automatically generated internal URLs from CMS platforms like Squarespace do not a...

Saeed Ashif Ahmed

July 29, 2026

Googlebot may ignore general robots.txt rules for specific ones

Googlebot may ignore general robots.txt rules for specific ones

Google’s John Mueller clarified that Googlebot prioritizes specific user-agent rules in robots.txt, potentially leading...

Joyce de Castro

July 28, 2026

A woman using Claude on a laptop and phone

Claude AI Chats Indexed by Google Despite Noindex Directives

Sensitive Claude AI chats appeared in Google search results due to conflicting robots.txt and noindex directives, prompt...

Palumbo Angela

July 28, 2026

Google's John Mueller explains a robots.txt quirk that can cause Googlebot to ignore robots.txt rules.

Google identifies robots.txt misconfiguration allowing spam indexing

Google’s John Mueller explains a common robots.txt error causing search box spam to be indexed. Learn how to fix it for...

Joyce de Castro

July 25, 2026