If you've wondered why the CPAN Testers has been having problems lately, then look no further than the various web crawler bots, particularly Yahoo! Slurp, which appears to have gone into overkill mode .. literally .. in recent days. As a consequence I've started looking at making the robots.txt a little more restrictive to stop them taking down the site.

At one point I figured that I could harness the web crawlers to help create the pages for the dynamic and static sites, but they're proved more of a hindrance than a help. So I've been looking at ways to slow down their site crawling for CPAN Testers. As such I came across a forum post that was very helpful in explaining the Crawl-Delay feature of the robots.txt. Yahoo! themselves try and explain it, but exceedly badly in my book, as they failed to mention the basic part of the value .. it's in seconds! Thankfully one of the forum posters mentions this and even provides a useful example (Yahoo! please take note).

I've now updated the robots.txt, so hopefully that will take affect soon.