Google’s Illyes Undecided on Dynamic ‘unavailable_after’ Dates

Image credit: Search Engine Journal

Google spokesperson Gary Illyes expressed uncertainty Thursday regarding the implications of dynamically shifting ‘unavailable_after’ dates for frequently renewed content, leaving search engine optimization professionals without clear guidance.

The ambiguity centers on how Google‘s systems would manage crawl budget and index selection when a page’s expiration date is repeatedly extended, a common scenario for online classifieds or listings.

SEO consultant Javier Lorente Murillo had inquired about using the ‘unavailable_after’ meta tag for classified advertisements with listings that are frequently renewed. Murillo sought to manage crawl budget effectively and prevent expired listings from generating 404 errors.

Illyes stated his “gut feeling” was that advancing the ‘unavailable_after’ date forward would be acceptable. However, he acknowledged the need to verify whether Google’s systems could reliably trust the tag for pages that expire and are subsequently renewed, according to his remarks.

Google’s documentation specifies that ‘unavailable_after’ instructs the search engine not to display a page after a specified date and to crawl it less frequently thereafter. This directive is intended to signal content obsolescence.

Illyes clarified that Google would need to recrawl a page to detect any updated ‘unavailable_after’ date. He added that, in his view, this tag primarily influences index selection, serving as a “drop this now” signal for content removal.

Official Google documentation confirms that both meta tags and X-Robots-Tag headers are only detected when a URL is crawled. This means any changes to the ‘unavailable_after’ date necessitate a new fetch by Googlebot to be recognized.

The specific scenario of an ‘unavailable_after’ date that changes with every content renewal is not explicitly addressed in Google’s current documentation, creating a gap in official guidance for webmasters and SEOs.

Illyes’s assertion that the rule impacts nothing beyond index selection is also not explicitly detailed within Google’s publicly available documentation on the ‘unavailable_after’ directive.

Source: Search Engine Journal