Right here is one other PSA from Gary Illyes of Google. Briefly, should you serve a 4xx standing code together with your robots.txt file, then Google will ignore the principles you’ve gotten laid out in that file.
Why? Properly, 4xx standing codes means the doc is just not obtainable, so Google will not verify it as a result of the server says it’s not obtainable. Gary mentioned this as a result of he obtained a grievance or two about Google not respecting the robots.txt guidelines.
Gary wrote on LinkedIn, “PSA from my inbox: should you serve your robotstxt with a 403 HTTP standing code, all guidelines within the file might be ignored by Googlebot. Consumer errors (4xx, besides 429) imply unavailable robotstxt, as in, a 404 and a 403 are equal on this case.”
Briefly, be sure your robots.txt file serves a 200 standing code and Google can entry it.
Discussion board dialogue at LinkedIn.
