2 min read
Googlebot may ignore general robots.txt rules for specific ones
Google’s John Mueller clarified that Googlebot prioritizes specific user-agent rules in robots.txt, potentially leading...
Joyce de Castro
July 28, 2026
Our Blog
The SEO Insights Blog
Filtering by Tag: User Agent
Clear filter
Showing recently cached posts. We're having trouble reaching our content server, so this list may be slightly out of date. It'll refresh automatically once the connection is back.
2 min read
Google rebrands NotebookLM to Gemini Notebook, raising scraping concerns
Google’s NotebookLM rebrands to Gemini Notebook, raising concerns for website owners about AI scraping. Learn how to blo...
Palumbo Angela
July 18, 2026
2 min read
Google AI agent ignores robots.txt, uses crypto ID
Google introduces Google-Agent, a new AI browsing tool that ignores robots.txt and uses cryptographic identity. Learn ho...
Joyce de Castro
May 17, 2026