Google issued clarifications on search engine optimization guidelines regarding 503 status codes, AI crawler behavior, URL handling, and image format support for webmasters.
The guidance aims to help site administrators manage content visibility and indexing as AI models increasingly interact with web data, alongside news of OpenAI’s expanded search indexing efforts.
For temporary website downtime, such as during PHP patch installations, Google recommends using a 503 HTTP status code for a maximum duration of one to two days to ensure indexed URLs are preserved, according to Google.
Gary Illyes, a Google analyst, stated that current robots.txt rules, defined by RFC 9309, exclusively govern crawling access and do not dictate how content is used after it has been crawled. Illyes indicated that a draft proposal is underway for a new ‘Content-Usage’ rule.
Google also confirmed that its ‘Google-Extended’ token allows webmasters to control whether content crawled by Google can be utilized for training its Gemini AI models and for grounding information in Gemini Apps and Vertex AI.
In a related development, OpenAI, in collaboration with Cloudflare, conducted a pilot program that enabled OpenAI to discover and index hundreds of millions of new web pages for its ChatGPT search capabilities.
Regarding URL structures, Google’s John Mueller confirmed that Google treats raw bracket URLs and their percent-encoded counterparts as distinct Uniform Resource Locators. Mueller advised webmasters to consistently use one form to avoid potential indexing issues.
Additionally, Google clarified its position on JPEG XL images, stating that the format is not currently supported in Google Images or Discover previews. This lack of support persists despite both Chrome and Firefox web browsers now offering native support for JPEG XL.
Source: Search Engine Journal
