CPAN Testers is only made possible with the support of our sponsors.
For more information on sponsoring, please visit the I ♥ CPAN Testers website.

Upgrade Notice

The CPAN Testers Blog site has been upgraded since you last accessed the site. Please press the F5 key or CTRL-R to refresh your browser cache to use the latest javascript and CSS files.

News & Views

Posted by Barbie
on 13th May 2010

Back In January, I reported how Microsoft had launched what amounted to a denial of service attack on the CPAN Testers server. It seems that 4 months later, we have yet again been targeted for attack from Microsoft. After the last attack, any IP address matching '65.55.*.*', hitting the main CPAN Testers website, was blocked (returning a 403 code). Every few weeks I check to see whether Microsoft have actually learnt and calmed down their attack on the server. So far, disappointingly, despite an alleged Microsoft developer saying they would look into it, the attack on the server has continue with little alteration to their frequency and numbers. Had they changed and been considerably less aggressive I would have lifted the ban.

Yesterday, Microsoft launch a further attack on the server using a complete new set of IP addresses. Now, just to clarify, this wasn't just a complete new set of IP addresses, but a completely new set PLUS the original set, thus effective doubling the attack on the server. Now you could claim stupidity or ignorance on behalf of the msnbot/Bing developers, but after being warned last time, and receiving 403s from their existing bots, by adding in a whole new set of IPs, I consider this latest attack nothing short of malicious.

These new IP address have now been added to the blocklist, and I'm now writing a script to alert me should any new IP address from Microsoft be added to their attack formation. Thankfully, I happened to be on the server at the time as both attacks hit, and managed to catch the IPs before they took out the server completely.

With my last post about this, I was accused of doing a disservice to Perl. Had I not been furious at the time, and written about the incident, I wouldn't have learnt that this was a Microsoft tactic that had infuriate a lot of people, and discovered that I wasn't the only sysadmin or website administrator around the world that had chosen to block Microsoft from their websites and servers. If Microsoft think thuggery is the way to improve their search content, then they are very sadly mistaken.

Update: Microsoft have now been in touch, and again apologised. We'll have to wait and see whether this can be resolved.

File Under: server
NO COMMENTS

Comments

Write a letter to their ISP.

Posted by Hater on Thursday, 13th May 2010


MSNBot DOS

Yah. I had them hitting a CGI script here dozens of times per second a month or so ago. I finally modified the script to specifically refuse to serve anything to MSNBot as well as adding a access rule to block the subnet hitting me.

Posted by Benjamin Franz on Thursday, 13th May 2010


FBI?

Why not report them to the FBI ?

Posted by Andreas on Thursday, 13th May 2010


robots.txt

You do know that you need a separate robots.txt file for every subdomain, do you? For example the robots.txt file for devel.cpantesters.org is:

User-agent: *
Disallow: /private/

You disallow the private subdir, but this seems to be a default value. Otherwise you set no limitations at all.

From http://www.robotstxt.org/robotstxt.html:

When a robot looks for the "/robots.txt" file for URL, it strips the path component from the URL (everything from the first single slash), and puts "/robots.txt" in its place.

For example, for "<tt>http://www.example.com/shop/index.html</tt>, it will remove the "<tt>/shop/index.html</tt>", and replace it with "<tt>/robots.txt</tt>", and will end up with "http://www.example.com/robots.txt".

Posted by Thijs Blaauw on Friday, 14th May 2010


re: robots.txt

Thijs I am well aware that a robots.txt applies to the specific domain or sub-domain. MSNBot has been aggressively hitting just www.cpantesters.org, and as such have been blocked from access that domain only.

For all other sub-domains they have not been hitting those so aggressively, so I have allowed them to crawl those sub-domains. Should that change they will be blocked from those domains too.

Posted by Barbie on Sunday, 16th May 2010


Add A Comment

Comments now have limited rich text support thanks to TinyMCE.

*
*