One of our clients uses a SPIP website and began to lose pages from Google’s index. The Google Search Console reported 500 errors on many of the pages, but the pages were visible to users. When investigating the logs or simulating a Googlebot browser-agent, I found that SPIP was sending HTTP error 429 (too many requests) to Googlebot and other search engine crawlers. This was because SPIP uses the server load average function to determine if it should allow bots’ crawling, however shared hosting providers have usually ultra performant CPUs with a high load value.

Continue reading