One of the first offers of help for getting the CPAN Testers server fixed, was from a guy called Doug Bell. Doug impressed me a lot with that first email, and was the only person who I felt really understood what I was asking for in my post about wanting a successor. Not that others didn't grasp that, but the fact Doug was the only person to send me a CV, proved that he got that this wasn't just a small project, or that I wanted to hand over the keys to someone who wasn't going to make a commitment to the project. Doug credentials impressed me, and his ideas and help since being invited to be involved, only proved he wasn't in this for the short-haul.

Many may know Doug as PREACTION, both on CPAN and IRC. Doug is the current leader of Chicago Perl Mongers, which also amused me, particularly once you know I'm based in Birmingham (UK) and Chicago is our twin city in America, and that I'm the leader of Birmingham Perl Mongers. Cue Twilight Zone theme tune! Doug has worked in web development, systems development and understands SysAdmin responsibilities. An ideal person to look after CPAN Testers, as you really do need a bit of everything. He also, perhaps most importantly, understands scale. CPAN Testers is a huge project, not just in terms of the number of reports (nearly 60 million), but the whole eco-system that is driven across the whole project.

I really needed a successor, as my home life and work life have been taking priority in recent years, and I don't want CPAN Testers to suffer because of that. I believe Doug is ideal to take over the project, and I hope you can give Doug the help and support that most have given me over the past 10 years or so. I'm not walking away completely, and will still be around to offer advice and help where needed, but Doug will be the one to drive the project forward from now on. He has some great ideas to help grow and improve the project, and I for one am really looking forward to seeing them come to fruition.

Doug and I have yet to meet, but hopefully the QA Hackathon 2016 will be the first time that I, along with all the other QA enthusiasts, will be able to sit down and discuss/design/plan his ideas for the future.

Welcome Doug, and thank you.

For over a month the CPAN Testers server has been having problems. In most cases the server locked up for no apparent reason. Sometimes it would work for a few days after a reboot, other times it would bearly last 5 minutes.

With the help of a few people, notably Matt S Trout and Doug Bell, we tried to figure out what was going on. I noted that the 'release' table and SQLite database that is created from it, seemed to be the most fragile for the server, as most times when it was touched, the server seemed to lock up within a few seconds. The table and database itself were used for a project Adam Kennedy created several years agao. These days only the bots and crawlers seemed to be interested. As such, I disabled to see whether that would help. It didn't.

We were getting stuck at every corner it seemed. I got Bytemark, our hosts involved to try and check the disks, but all appeared to be running fine. The SATA disks did seem to be very active, but that's normal for us, as there are alot of disk writes to create the sites. As we've had the current server for nearly two years, Bytemark suggest moving to a new server, which has better CPU and RAM capacity, which we've now completed. The server has now been up for over a week, the sites are building and running without a problem, and best yet the reports site is building faster than it has been for a long time.

Although we still have no clear idea why the cpantesters2 kept falling over (it was even falling over when we were rsync-ing the data to cpantesters3, when nothing else was running on either server!), we suspect there is a faulty CPU or RAM module. There isn't much we can do about that, as there isn't a disk check type feature that can find bad blocks and disable them for CPUs and RAM, at least that I'm aware of. So a move to a new server is about all we could have done.

As a result of all this, we have some ideas to help improve the sites and any future rebuilds, so if we do suffer again, we can recover much quicker. I'm currently documenting how the current systems work, and we have at least 5 projects planned to help improve various parts of the eco-system. Some have been long overdue, others are new and fresh ideas to help take CPAN Testers further into the future. There is a lot to learn about scaling with this project ;)

My sincere thanks to MST, Doug, and Chris at Bytemark, for helping out and getting us back online. It's been a frustrating month or so, and I'm just relieved to have the sites back and stable again. My thanks to everyone who stepped up and offered help and support for the project, it really was very much appreciated. Sorry if I didn't get back to you, or weren't able to take advantage of your assistance. Some of the offers of help were taken advantage of, and a couple are planned for future projects to help grow the project. We'll hopefully have more news about them as we start to design and implement them in the coming months. The future looks promising, and it wouldn't be without everyone's support. Thank you.

It's good to be back!

A very late summary, and that dispite one of my aims for this year was to get better at posing these on time :( As a consequence, if anyone is willing to help out with posting summaries, please get in touch.

For this post, I want to concentrate on promotion of CPAN Testers. Mark Keating wrote a very eloquent post regarding how testers themselves can promote CPAN Testers, with his post Smoke me an Onion baby. A few years ago, BinGOs and myself toyed with the idea of getting a T-shirt sorted to give out at YAPCs, promoting CPAN Testers. Unfortunately, we couldn't think of a suitable tag line and we're not graphic artists, so we never got anything sorted. In Mark's post he came up with some wonderful graphics to be used as badges for testers to promote their own involvement. If you like Mark's creations and wish to use them on your website, please get in touch and he'll provide copies for you to use. In the meantime watch out for stickers and badges at various future events.

Over the past few months, we have some interest from a few companies that have expressed a desire to support CPAN Testers. Both via monetary funding and spare capacity on servers. We are still in discussion with these companies, so hopefully we will have some new sponsors in the near future. We are always will to have more companies support us, as alhough individual funding is very much appreciated, we would like to see more corporate sponsorship to help raise our profile. If your company uses Perl, and makes use of CPAN Testers either directly or inderectly, and would be willing to help support the long term future of CPAN Testers, please get in touch.

Individual funding, as mentioned, is always very much appreciated, and if you would like to contribute in any way, you can make donations via the CPAN Testers Fund, or Gratipay. All proceeds are managed for us by the Enlightened Perl Organisation, who are a not-fot-profit organisation.

The reason for promotion, and requests for sponsorship, has come about as the Birmingham Perl Mongers' funding of the CPAN Testers servers comes to an end in September 2015, when we'll need to find other sources of funding for the current server. On top of this we have the Metabase server, which also requires funding. Sadly dedicated servers don't come cheap these days, so it roughly costs around £4200 (US$6280 or €5800) per year. Over the next six months we'll be doing as much as we can to raise the funding, but if you can help promote this to your employers that would be wonderful.

If you have any ideas to help promote CPAN Testers in other ways, please let me know and we'll see what we can do.

To begin with, a huge thank you to Nestoria, who selected CPAN Testers as their module of the month. Although we kind of break the mould of a module, we are exceptionally grateful for the donation via Gratipay, and for the promotion. As mentioned several times over the last few months, CPAN Testers are looking towards companies to help fund the servers we currently run, so donations from companies such as Nestoria are very gratefully received, and hopefully will help us to continue for many years to come. It is also nice to hear how CPAN Testers helped them too, when releasing Number-Format-SouthAsian.

We have been in discussions with a few other people and will have further announcements about sponsors and donations soon, but in the meantime, if you or your company would be willing to make a donation to CPAN Testers, you can do so via Gratipay, The CPAN Testers Fund, and/or by contacting me (Barbie), and we'll make sure you get suitable recognition for your contribution.

During December some initial thoughts were put together in preparation for the QA Hackathon 2015. The event in Berlin brings together several key people within CPAN Testers, as well as CPAN and Test communities, and will be an ideal opportunity to look at how we are processing the reports. In the first instance, I want to re-engineer the way the feeds are read, and how pages are built. Working with MessageQueues and Enterprise Service Buses recently, has hightlighted that we can make use of these technologies to speed processing up. As a consequence, I now have some areas of coding that I plan to start before the Hackathon, and hopefully provide some key parties a chance to review and improve the designs.

Although happening in January, I did want to pick up on an outage that happened earlier this month. My thanks to Bytemark for helping to resurrect the server for us. Thankfully, only one index table was knocked out, but after rebuilding, everything does appear to be back on track. Another task for the QA Hackathon is to look at a better way to reduce the risk for these indices. We also want to hopefully plan a move to MongoDB for the Metabase, which will stablise things considerably.

Sadly once again a bit delayed this month, but I do promise to improve in the coming months.

So last month saw the London Perl Workshop take place. Several talks related to testing, and plenty of interest in CPAN Testers. Mark Keating and crew were videoing many of the talks. Sadly the microphone position meant the volume is quite low in places, but they are still worth watching. You can see testing talks from myself (slides), DrForr, and additionally my lightning talk (slides). If you missed the event, there are plenty of great talks you can now catch on the Shadowcat channel.

Also during last month there have been several fixes to the Reports Builder. Also following some conversations at LPW, I came away with a few ideas to improve the performance of the Feed and Builder components so that we can get almost real time production of reports. Over the coming months and likely during the 2015 QA Hackathon, I aim to work more on this and hopoefully have a solution working around the hackathon. Karen Etheridge also help to spot a fault with the Admin site, in that if a tester hasn't allocated a contact email address, any report that has been flagged isn't allocated to them, but to the Admin user. It does mean there is a bit of extra work for me at the moment, but ultimately I need to rework the way testers are notified, particularly in the event of the system using an old email address. If you've flagged any reports and haven't seen them removed as yet, please give me a nudge and I'll approve their removal.

One person I would to single out for making a great effort to promote testing in general, is Sinan Unur, who has been writing some great blog posts. TDD is all well and good, but who's testing the tests?, You've gotta quotemeta! and Tests should not fail due to EOL differences across platforms to mention a few. He has some great in depth Perl posts, and all are well worth reading.

A colleague of mine, Matt Wheatley, recently had a blog post published about how he works in development. Except the post isn't so much about how he works, but more about the value of a QA Department. Our QA team is awesome, and we perhaps rely on them a bit too much sometimes, but I imagine there are many awesome QA teams out there, and this is just one tribute to them all. I especially liked the reference to Bill Sempf's tweat.

We had another Metabase overflow last month, which means we've filled another bucket in AWS. David Golden has created a new one, but it does mean we need to think about moving to our newer architechure sooner rather than later. David, Neil Bowers and I have been in discussions recently, and I hope we will get some movement towards this in the coming year. There will be a lot of changes needed, both to code and routing, so it won't be a simple change. However, it will be a change that will help us grow more productively in the future.

More new coming next month. And my New Year's Resolution is to get these summaries out a bit more on time!