Last month CPAN Testers managed to clock up 60 million reports.

Congratulations to Chris Williams (BINGOS) for submitting the 60 millionth CPAN Testers Report. The report itself was a PASS for BenchmarkAnything-Storage-Frontend-Lib and was submitted on 20th September 2015.

Back in March 2014 I predicted the 50 millionth report would arrive in February 2015. I wasn"t too far off, as Chris Williams (again) submitted a PASS report for Map-Tube on 9th January 2015. As such I predict we"ll see the 70 millionth in June 2016. Time will tell.

For over a month the CPAN Testers server has been having problems. In most cases the server locked up for no apparent reason. Sometimes it would work for a few days after a reboot, other times it would bearly last 5 minutes.

With the help of a few people, notably Matt S Trout and Doug Bell, we tried to figure out what was going on. I noted that the 'release' table and SQLite database that is created from it, seemed to be the most fragile for the server, as most times when it was touched, the server seemed to lock up within a few seconds. The table and database itself were used for a project Adam Kennedy created several years agao. These days only the bots and crawlers seemed to be interested. As such, I disabled to see whether that would help. It didn't.

We were getting stuck at every corner it seemed. I got Bytemark, our hosts involved to try and check the disks, but all appeared to be running fine. The SATA disks did seem to be very active, but that's normal for us, as there are alot of disk writes to create the sites. As we've had the current server for nearly two years, Bytemark suggest moving to a new server, which has better CPU and RAM capacity, which we've now completed. The server has now been up for over a week, the sites are building and running without a problem, and best yet the reports site is building faster than it has been for a long time.

Although we still have no clear idea why the cpantesters2 kept falling over (it was even falling over when we were rsync-ing the data to cpantesters3, when nothing else was running on either server!), we suspect there is a faulty CPU or RAM module. There isn't much we can do about that, as there isn't a disk check type feature that can find bad blocks and disable them for CPUs and RAM, at least that I'm aware of. So a move to a new server is about all we could have done.

As a result of all this, we have some ideas to help improve the sites and any future rebuilds, so if we do suffer again, we can recover much quicker. I'm currently documenting how the current systems work, and we have at least 5 projects planned to help improve various parts of the eco-system. Some have been long overdue, others are new and fresh ideas to help take CPAN Testers further into the future. There is a lot to learn about scaling with this project ;)

My sincere thanks to MST, Doug, and Chris at Bytemark, for helping out and getting us back online. It's been a frustrating month or so, and I'm just relieved to have the sites back and stable again. My thanks to everyone who stepped up and offered help and support for the project, it really was very much appreciated. Sorry if I didn't get back to you, or weren't able to take advantage of your assistance. Some of the offers of help were taken advantage of, and a couple are planned for future projects to help grow the project. We'll hopefully have more news about them as we start to design and implement them in the coming months. The future looks promising, and it wouldn't be without everyone's support. Thank you.

It's good to be back!

To begin with, a huge thank you to Nestoria, who selected CPAN Testers as their module of the month. Although we kind of break the mould of a module, we are exceptionally grateful for the donation via Gratipay, and for the promotion. As mentioned several times over the last few months, CPAN Testers are looking towards companies to help fund the servers we currently run, so donations from companies such as Nestoria are very gratefully received, and hopefully will help us to continue for many years to come. It is also nice to hear how CPAN Testers helped them too, when releasing Number-Format-SouthAsian.

We have been in discussions with a few other people and will have further announcements about sponsors and donations soon, but in the meantime, if you or your company would be willing to make a donation to CPAN Testers, you can do so via Gratipay, The CPAN Testers Fund, and/or by contacting me (Barbie), and we'll make sure you get suitable recognition for your contribution.

During December some initial thoughts were put together in preparation for the QA Hackathon 2015. The event in Berlin brings together several key people within CPAN Testers, as well as CPAN and Test communities, and will be an ideal opportunity to look at how we are processing the reports. In the first instance, I want to re-engineer the way the feeds are read, and how pages are built. Working with MessageQueues and Enterprise Service Buses recently, has hightlighted that we can make use of these technologies to speed processing up. As a consequence, I now have some areas of coding that I plan to start before the Hackathon, and hopefully provide some key parties a chance to review and improve the designs.

Although happening in January, I did want to pick up on an outage that happened earlier this month. My thanks to Bytemark for helping to resurrect the server for us. Thankfully, only one index table was knocked out, but after rebuilding, everything does appear to be back on track. Another task for the QA Hackathon is to look at a better way to reduce the risk for these indices. We also want to hopefully plan a move to MongoDB for the Metabase, which will stablise things considerably.

Sadly once again a bit delayed this month, but I do promise to improve in the coming months.

So last month saw the London Perl Workshop take place. Several talks related to testing, and plenty of interest in CPAN Testers. Mark Keating and crew were videoing many of the talks. Sadly the microphone position meant the volume is quite low in places, but they are still worth watching. You can see testing talks from myself (slides), DrForr, and additionally my lightning talk (slides). If you missed the event, there are plenty of great talks you can now catch on the Shadowcat channel.

Also during last month there have been several fixes to the Reports Builder. Also following some conversations at LPW, I came away with a few ideas to improve the performance of the Feed and Builder components so that we can get almost real time production of reports. Over the coming months and likely during the 2015 QA Hackathon, I aim to work more on this and hopoefully have a solution working around the hackathon. Karen Etheridge also help to spot a fault with the Admin site, in that if a tester hasn't allocated a contact email address, any report that has been flagged isn't allocated to them, but to the Admin user. It does mean there is a bit of extra work for me at the moment, but ultimately I need to rework the way testers are notified, particularly in the event of the system using an old email address. If you've flagged any reports and haven't seen them removed as yet, please give me a nudge and I'll approve their removal.

One person I would to single out for making a great effort to promote testing in general, is Sinan Unur, who has been writing some great blog posts. TDD is all well and good, but who's testing the tests?, You've gotta quotemeta! and Tests should not fail due to EOL differences across platforms to mention a few. He has some great in depth Perl posts, and all are well worth reading.

A colleague of mine, Matt Wheatley, recently had a blog post published about how he works in development. Except the post isn't so much about how he works, but more about the value of a QA Department. Our QA team is awesome, and we perhaps rely on them a bit too much sometimes, but I imagine there are many awesome QA teams out there, and this is just one tribute to them all. I especially liked the reference to Bill Sempf's tweat.

We had another Metabase overflow last month, which means we've filled another bucket in AWS. David Golden has created a new one, but it does mean we need to think about moving to our newer architechure sooner rather than later. David, Neil Bowers and I have been in discussions recently, and I hope we will get some movement towards this in the coming year. There will be a lot of changes needed, both to code and routing, so it won't be a simple change. However, it will be a change that will help us grow more productively in the future.

More new coming next month. And my New Year's Resolution is to get these summaries out a bit more on time!

Sorry October's summary is a bit late, as I had rather a busy few months recently, and finding time to finish this one prove more tricky than normal.

However, first off, a thanks to Gabor Szabo, as one of his screencast featured a brief explanation of how CPAN Testers are intergrated into MetaCPAN. Although anyone familiar with the CPAN Testers Report site, will probably have understood what the numbers mean, the screencast is a handy help to anyone new to MetaCPAN or CPAN to learn more about using MetaCPAN. If anyone else has similar screencasts about CPAN Testers, or wishes to create one, please let me know.

My thanks to Pete Houston for notifying me early in the month that the feed to CPAN Testers had got stuck. Another reason why we really want to move away from Amazon and SimpleDB. This is still being investigated, and hopefully will be a project during the QA Hackathon next year. It also gave me pause to think of better ways the feed and builder could be processing. Thet are both mostly serial process, although there are now 2 builders (distros and authors). However, having worked a lot with MessageQueues over the last year, I'm looking at rewriting both the feed and builders, and again is likely to be my main project during the QA Hackathon in 2015. The ultimate aim, as has been asked on many many occassions, is to try and get as close to real time as we possibly can. On a good day we are less than an hour behind, but being a minute of two behind would be even better.

Karen Etheridge highlighted some problems with the CPAN Testers Admin site. Firstly the login had broken again. I've now permanently fixed this, and uploaded the code to CPAN so the old version doesn't overwrite the new version again. However, Karen also spotted a few other problems. Next up was the find a report by GUID. I'm not quite sure what I tested before, but I suspect I overwrote the working code again :( Again this is now fixed and the code is on CPAN :)

The current CPAN Admin problem is figuring out why, when marking reports for removal, they aren't getting alerted to the tester. They were getting added to the Admin user though, so they weren't lost, but it has highlighted a few cross-reference points that are not being made. Connecting a report to tester, or more imporantly to the tester's current email address is not always straight forward, so the fallback is to email the Admin user. I'm working on the code to reconcile these cross-references better, and in the coming weeks I should get that working properly again. In the meantime if you have marked reports, please bear with me, and I'll get to them as soon as I can. In the process of digging through the report removal process, I also discovered that the Builder mechanism for this was flawed, at least from an Author perspective. I've since corrected this too.

Andreas König noted that several reports from one tester were essentially the same. Breno G. de Oliveira helped to identify the problem, and the reasoning behind it. The reporter for cpanminus, doesn't currently record history as the other smokers, and consequently doesn't filter out repeated attempts by the author to install the same distribution. Breno is planning to work more on the Common Client, and integrate this into the cpanminus Reporter. So if you get similar repeated reports, please be gentle with the tester :)

Subramanian Karunanithi was the casualty of version numbering with CGI-Test. Thanks to some investigation work from Jonas B. Nielsen, he highlighted the latest version was 0.52, but the installed version was 0.104. The problem with version numbering within the installers is that they see 0.104 as Zero Point One Hundred and Four, which is greater than Zero Point Fifty Two. Karen posted a bug report, and Alex Tokarev has now released version 1.000, so all good. However, it does bear repeating that Version Numbers Should be Boring.

Next month we'll have a few notes about LPW, more fixes to the toolchain and some ideas for the future.