Google has added a brand new part to the crawl price range administration assist doc. The brand new part is for the If-Modified-Since request header. The part is titled “Specify content material adjustments with HTTP standing codes.”
Google wrote, “Google typically helps the If-Modified-Since and If-None-Match HTTP request headers for crawling. Google’s crawlers do not ship the headers with all crawl makes an attempt; it will depend on the use case of the request (for instance, AdsBot is extra more likely to set the If-Modified-Since and If-None-Match HTTP request headers).”
“If our crawlers ship the If-Modified-Since header, the header’s worth is the date and time the content material was final crawled. Primarily based on that worth, the server could select to return a 304 (Not Modified) HTTP standing code with no response physique, during which case Google will reuse the content material model it crawled the final time. If the content material is newer than the date specified by the crawler within the If-Modified-Since header, the server can return a 200 (OK) HTTP standing code with the response physique.”
“Independently of the request headers, you’ll be able to ship a 304 (Not Modified) HTTP standing code and no response physique for any Googlebot request if the content material hasn’t modified since Googlebot final visited the URL. This can save your server processing time and assets, which can not directly enhance crawl effectivity.”
I’m not certain if I’ve any commentary on this, exterior of the truth that this new part was added to this web page.
Discussion board dialogue at Twitter.
