Google considers URLs containing raw square brackets and their percent-encoded equivalents as separate addresses, potentially impacting how web content is crawled and indexed, a company official confirmed.
This distinction means web developers and SEO professionals must consistently use one format for URLs to ensure optimal indexing and avoid potential issues with duplicate content, according to Google Search Advocate John Mueller.
Mueller explained the search engine’s policy aligns with RFC 3986, an internet standard that classifies brackets as reserved characters. Under this standard, the raw form of a reserved character and its percent-encoded version are not deemed equivalent.
Google’s systems will crawl both versions of a URL if they exist on a website. If these distinct URLs serve identical content, Google will engage its deduplication process, selecting one version to index and display in search results, Mueller said.
To prevent confusion and ensure consistent indexing, Mueller recommended webmasters consistently use a single URL format across all web elements. This includes links, canonical tags, sitemaps, and structured data.
He specifically advised using percent-encoded brackets (%5B and %5D) for new website implementations. While existing sites that currently use raw brackets might continue to be indexed without immediate issues, Google discourages this practice for new developments.
The search advocate noted similar rules regarding duplicate formats also apply to other characters such as spaces, plus signs, and ampersands, which also have specific encoding requirements.
Vanessa Fox, a prominent figure in the SEO community and former Google employee, has previously highlighted the complexities of URL handling and its implications for search engine optimization.
The clarification from Google underscores the importance of adhering to web standards and best practices for URL construction to maintain search visibility and efficient content management.
Source: Search Engine Journal
