Gary Illyes from Google posted a brand new PSA on LinkedIn saying that the most typical motive a website unexpectedly blocks Googlebot from crawling is because of a misconfiguration of a firewall or CDN.
Gary wrote, “test what visitors your firewalls and CDN are blocking.” “By far the most typical subject in my inbox is expounded to firewalls or CDNs blocking googlebot visitors. If I attain out to the blocking website, within the overwhelming majority of the circumstances the blockage is unintended.”
So what are you able to do? Gary stated, “I’ve stated this earlier than, however wish to emphasize it once more: make a behavior of checking your block guidelines. We publish our IP ranges so it ought to be very straightforward to run an automation that checks the block guidelines towards the googlebot subnets.”
Gary linked to this assist doc for extra particulars.
Briefly, do what you possibly can to check to see in case your website is accessible to Googlebot. You need to use the URL inspection device in Google Search Console, as one technique. Additionally, affirm together with your CDN or firewall firm that they’re permitting Googlebot and ask them to show it.
Discussion board dialogue at on LinkedIn.
