Over the weekend, I put together the evidence for the Microsoft Bing team, to help them understand why their bots are hitting the site so aggressively. In addition I put several questions to them as to how their bot clusters work, and why they don't follow some very simple rules.

To give you an understanding of the problem, the following are some basic stats gleaned from the logs, for those requests identifying themselves as 'msnbot'.

First: 09/May/2010:02:26:22 +0200

Last:  15/May/2010:01:21:42 +0200

Total Hits:      82130

Total Requests:  79999

Total IPs:       161

Status Code 200: 8473

Status Code 301: 83

Status Code 403: 73572

Status Code 500: 2

Max host per second:  74 [09/May/2010:10:42:06 +0200] 65.55.104.39

Max hosts per second: 15 [12/May/2010:16:19:42 +0200]

Max hits per second:  77 [14/May/2010:15:03:15 +0200]

The First/Last timestamps are the first and last entries in the logs that were processed. A period of roughly 6 days. Total hits is the number of entries processed, and total requests is the number of non 'robots,txt' requests, followed by a breakdown of the count for each status code returned.

However, the next 3 are the areas that are most questionable. Max host per second, is the greatest number of hits in a single second by single IP address. The Max hosts per second, is the greatest number of unique IPs submitting a request in a single second. Lastly, Max hits per second is the greatest number of requests seen in any single second.

So 15 unique IP address hitting the site in a single second, with a maximum rate of 77 requests a second. Does that strike you as reasonable?

In my response to Microsoft I have pointed out that when a bot receives a 403 for every request (for the last 4 months), the bot should be intelligent enough to back off the requests and default to the robots.txt. As it turned out the Apache config was also returning a 403 for the blocked IPs (which has since been fixed), though I maintain that if a bot receives a 403 for a request for robots.txt, it should equate to being disallowed to search and index the site.

However, the attack on Wednesday was from a fresh set of IPs that weren't blocked, and were able to read the robots.txt, which explicitly disallows msnbot. All these new IPs completely disregarded robots.txt and searched and indexed the site anyway. I had I not been on the machine at the time, they would have knocked the machine offline.

Another further point I made, was that CPAN Testers is a non-commercial, non-High Availability site, funded by the Perl community and administered by volunteers. Sending 77 requests a second is exactly the same as a denial of service attack for us. We don't have funds and resources to managed a fleet of web and database servers with a load balancer calmly managing requests for us. Treating us (or any other non-commercial site) like a major news site is just plain irresponsible.

One link that was sent to me, highlighted that this aggressive behaviour has, at least in one case, cost a site owner money in excessive bandwidth charges. If Microsoft do not put some reasonable controls on their bots then it is only a matter of time before someone sues or contacts the FBI for computer use.

I have provided the logs to Microsoft, so we'll see what their response is. As of Saturday, they obviously changed something, as all hits to the www.cpantesters.org domain all switched to just robots.txt requests, seeming to honour the request not to index the site. However, I was still seeing 4 requests a second for a single IP, so it seems they have still some work to do.