CPAN Testers Blog

60 Million Reports

2015-10-12T23:49:15Z

Last month CPAN Testers managed to clock up 60 million reports.

Congratulations to Chris Williams (BINGOS) for submitting the 60 millionth CPAN Testers Report. The report itself was a PASS for BenchmarkAnything-Storage-Frontend-Lib and was submitted on 20th September 2015.

Back in March 2014 I predicted the 50 millionth report would arrive in February 2015. I wasn"t too far off, as Chris Williams (again) submitted a PASS report for Map-Tube on 9th January 2015. As such I predict we"ll see the 70 millionth in June 2016. Time will tell.

Introducing Doug Bell

2015-09-19T08:28:16Z

One of the first offers of help for getting the CPAN Testers server fixed, was from a guy called Doug Bell. Doug impressed me a lot with that first email, and was the only person who I felt really understood what I was asking for in my post about wanting a successor. Not that others didn't grasp that, but the fact Doug was the only person to send me a CV, proved that he got that this wasn't just a small project, or that I wanted to hand over the keys to someone who wasn't going to make a commitment to the project. Doug credentials impressed me, and his ideas and help since being invited to be involved, only proved he wasn't in this for the short-haul.

Many may know Doug as PREACTION, both on CPAN and IRC. Doug is the current leader of Chicago Perl Mongers, which also amused me, particularly once you know I'm based in Birmingham (UK) and Chicago is our twin city in America, and that I'm the leader of Birmingham Perl Mongers. Cue Twilight Zone theme tune! Doug has worked in web development, systems development and understands SysAdmin responsibilities. An ideal person to look after CPAN Testers, as you really do need a bit of everything. He also, perhaps most importantly, understands scale. CPAN Testers is a huge project, not just in terms of the number of reports (nearly 60 million), but the whole eco-system that is driven across the whole project.

I really needed a successor, as my home life and work life have been taking priority in recent years, and I don't want CPAN Testers to suffer because of that. I believe Doug is ideal to take over the project, and I hope you can give Doug the help and support that most have given me over the past 10 years or so. I'm not walking away completely, and will still be around to offer advice and help where needed, but Doug will be the one to drive the project forward from now on. He has some great ideas to help grow and improve the project, and I for one am really looking forward to seeing them come to fruition.

Doug and I have yet to meet, but hopefully the QA Hackathon 2016 will be the first time that I, along with all the other QA enthusiasts, will be able to sit down and discuss/design/plan his ideas for the future.

Welcome Doug, and thank you.

CPAN Testers Is Back

2015-09-18T08:01:05Z

For over a month the CPAN Testers server has been having problems. In most cases the server locked up for no apparent reason. Sometimes it would work for a few days after a reboot, other times it would bearly last 5 minutes.

With the help of a few people, notably Matt S Trout and Doug Bell, we tried to figure out what was going on. I noted that the 'release' table and SQLite database that is created from it, seemed to be the most fragile for the server, as most times when it was touched, the server seemed to lock up within a few seconds. The table and database itself were used for a project Adam Kennedy created several years agao. These days only the bots and crawlers seemed to be interested. As such, I disabled to see whether that would help. It didn't.

We were getting stuck at every corner it seemed. I got Bytemark, our hosts involved to try and check the disks, but all appeared to be running fine. The SATA disks did seem to be very active, but that's normal for us, as there are alot of disk writes to create the sites. As we've had the current server for nearly two years, Bytemark suggest moving to a new server, which has better CPU and RAM capacity, which we've now completed. The server has now been up for over a week, the sites are building and running without a problem, and best yet the reports site is building faster than it has been for a long time.

Although we still have no clear idea why the cpantesters2 kept falling over (it was even falling over when we were rsync-ing the data to cpantesters3, when nothing else was running on either server!), we suspect there is a faulty CPU or RAM module. There isn't much we can do about that, as there isn't a disk check type feature that can find bad blocks and disable them for CPUs and RAM, at least that I'm aware of. So a move to a new server is about all we could have done.

As a result of all this, we have some ideas to help improve the sites and any future rebuilds, so if we do suffer again, we can recover much quicker. I'm currently documenting how the current systems work, and we have at least 5 projects planned to help improve various parts of the eco-system. Some have been long overdue, others are new and fresh ideas to help take CPAN Testers further into the future. There is a lot to learn about scaling with this project ;)

My sincere thanks to MST, Doug, and Chris at Bytemark, for helping out and getting us back online. It's been a frustrating month or so, and I'm just relieved to have the sites back and stable again. My thanks to everyone who stepped up and offered help and support for the project, it really was very much appreciated. Sorry if I didn't get back to you, or weren't able to take advantage of your assistance. Some of the offers of help were taken advantage of, and a couple are planned for future projects to help grow the project. We'll hopefully have more news about them as we start to design and implement them in the coming months. The future looks promising, and it wouldn't be without everyone's support. Thank you.

It's good to be back!

CPAN Testers Site Update & Successor Wanted

2015-08-30T12:38:52Z

As some may have noticed the CPAN Testers websites are not well. The underlying server is locking up, and as yet the cause is undetermined. I had thought I had cracked it, as the Release Data information that was provided for Adam Kennedy several years ago, seemed to be a key factor in the processes running when the server locked up. As a consequence I disabled the table from all the processes and yesterday all seemed well, with the server responding for nearly 30 hours in a row. Sadly, this faltered this morning at around 02:30 UTC. At the moment I'm checking tables and gettting things back into a usable state again. However, the current situation is not sustainable.

Unfortunately, the server has been unwell for the past month, and as I currently don't have a lot of free time to keep putting it back online, I need someone who is willing to spend time helping to diagnosing the problems, and secondly to take on a lot of the responsibility of CPAN Testers. I'm not disappearing completely, and hope to write more documentation to explain the whole eco-system, so anyone taking on the project in the future has a better idea of what it entails. Most of the work in recent times has been SysAdmin type maintenance work to help keep things ticking over, but occasionally there is coding work to rewrite older parts of the system, and add new features.

Due to the size of the CPAN Testers project, I can appreciate why it might seem a bit daunting, which is why I'm not planning to disappear completely. I'd like help to begin with, particularly for the current problems, but longer term I'd like to pass on the knowledge I have, to someone who wants to improve and develop the sites further. If you feel up for such a challenge, please get in touch.

I have now been involved with CPAN Testers for over 10 years and it is definitely time for some fresh eyes. If you want to know more, please contact me at barbie@cpan.org.

My thanks to Matt S Trout over the past couple of weeks, who has helped to give me some pointers and help identify the problem with the Release Data.

UPDATE 2015-08-31: Many thanks to all those who have emailed offering help. It is great to see several people stepping forward to volunteer their time. At the moment, one person with some great credentials, and experience of sites like CPAN Testers, has stepped forward and is taking a look over the server for me. I don't want to have too many cooks in the mix, so I'm letting him have first crack. More news when we've had a chance to do some more detailed investigation.

Our Latest Sponsor: ABC Startsiden

2015-08-17T07:59:49Z

It is with great pleasure that we officially announce a new sponsor for the CPAN Testers Project, ABC Startsiden.

ABC Startsiden AS owns, publishes and manages some of the most visited Norwegian web sites including an Internet portal (www.startsiden.no), an online newspaper (www.abcnyheter.no) and a TV-guide (www.meetv.no). ABC Startsiden AS is also Google"s largest search partner in Scandinavia.

ABC Startsiden have long been a long time supporter of Perl, CPAN and CPAN Testers, and we are delighted to have them as one our sponsors.

If you or your company would like to be a sponsor of the project, please get in touch. Alternatively, you can make donations via the CPAN Testers Fund, or Gratipay.

Uncommon OS Platforms Wanted for Testing

2015-04-01T22:37:15Z

Below is a post to the CPAN Testers mailing list by Mark Overmeer. Mark asked me earlier in the week about promoting under-tested OSes, which has been something I've been meaning to do for awhile. Alhough, I planned to write a blog post, however as Mark's post to the list covers it all so well, with Mark's permission I am posting it in full. If you have an under-CPAN tested OS, please consider running a smoke tester to help both Mark and CPAN testers. Over to you Mark ....

Dear testers,

CPAN Testers is a unique infrastructure. I (ab)use it to collect platform specific information around POSIX compliancy, which I cannot find anywhere else. I need more uncommon platforms to run my tests... Please cpantest my POSIX::1003 module.

* Let me explain.
The POSIX.pm module offers a few extra OS calls, which are not in core (why is getpwnam() still build-in?) But the POSIX standards are much larger. See where the 1200+ functions of POSIX* are: Overview - Functions

POSIX::1003 extends POSIX.pm, firstly breaking its exported info into groups with some documentation. Most importantly, the module autodetects constants and adds more functions. There are many many more constants than core modules define.

* Approach.
I have manually grabbed constant names from the manual pages of many operating systems. There are collections of manual pages which are helpful. I have found close to 1700 constants this way. During installation of the module, XS builds tables with the names which are found. Those names are also added to the manual pages. Example: Errno - Constants

I add new functions very fast, but need to see which platforms support which extensions. Also, I need to spot compilation problems for platforms which I do not have.

Worth to notice: I go to great pain *not* to hide platform differences! People may want to produce abstractions which do hide differences, but to do that correctly, we need the pure native interface first. For instance, I offer a pure getuid() and geteuid() and getreuid() and getresuid() (if platforms offer them), without an attempt to unify them.* New project. Ok, to support my own development, I was looking for the reverse mapping: we do have a "platform to constants" map via manual-pages, but it is not clear when constants and functions got added to some OS. Now, I use cpantesters to get the reverse mapping! When tests are run on cpantesters, they dump the discovered tables. It will benefit Perl's connection to the OS, but this data is useful for any OS programmer.

Have a look at posix.cpan6.net what I do with the information collected via cpantesters. It's a new project, so far from optimal output. When I get more cpantest results, I will (have to) be smarter with the browsing.

* Please help
Please run tests for POSIX::1003 -- especially when you run something else than Mac, Linux, FreeBSD, Windows. At least, I would like to get results from aix, hpux, solaris, and openbsd. Thanks in advance!

CPAN Testers Speakers Wanted

2015-03-22T13:36:14Z

Are you attending a YAPC or other large technical event this year? Willing to be a speaker at such an event? Are you familiar with CPAN Testers, even as a new tester?

As I haven't been able to attend YAPCs for a few years, I am looking for someone to help keep the CPAN Testers project fresh in people's minds. As such, if you've thought you'd like to do a talk at a YAPC, but haven't had the idea for a topic, or may be you've previously done a presentaion, and would like to do another, and you would be happy to do something that covers some aspect of CPAN Testers, please get in touch. It's an ideal opportunity if you are a new speaker for your first presentation at a large event.

There are plenty of topics around CPAN Testers you could promote, particularly about getting involved with CPAN Testers, either as just a tester, as a toolchain developer or even as someone that likes to analyse common errors. Your perspective is also likely to be very different from others and can be a great source of inspiration for new testers.

The subject of a CPAN Testers talk came up in a discussion, as YAPC::NA is coming soon, and having a presentation on CPAN Testers would be very welcome. Their deadline for talk submissions has sadly past, and I had wanted to post this a few weeks ago, but there might still be the opportunity to present as part of the beginners track. If you are interested, and can put something together quickly, we can see whether they have a spare slot, or a the very least would be able to keep you in reserve should anyone drop out.

Long term, I would be delighted to have others speak at any future YAPC::Europe, YAPC::NA, YAPC::Asia, YAPC::Brazil, YAPC::Russia, Perl Workshops and other large technical events. If you need resources, please feel free use already existing presentations, stats from the CPAN Testers Statistics site or contact me, and I can hopefully point you in the right direction. Sadly, I may not have time available to help write the presentation, but I am willing to proof-read anything, should you need me to.

YAPC::Europe is happening in Granada this year, and it would be great to finally have another CPAN Testers presentation after several years. If you can submit a talk, please do. Let me know and I'll make sure we promote it too, both here in the blog and on the twitter feed. That goes for any CPAN Testers presentation at any technical event.

CPAN Testers needs you :)

CPAN Testers Summary - January 2015 - Permanent Vacation

2015-03-22T13:00:09Z

A very late summary, and that dispite one of my aims for this year was to get better at posing these on time :( As a consequence, if anyone is willing to help out with posting summaries, please get in touch.

For this post, I want to concentrate on promotion of CPAN Testers. Mark Keating wrote a very eloquent post regarding how testers themselves can promote CPAN Testers, with his post Smoke me an Onion baby. A few years ago, BinGOs and myself toyed with the idea of getting a T-shirt sorted to give out at YAPCs, promoting CPAN Testers. Unfortunately, we couldn't think of a suitable tag line and we're not graphic artists, so we never got anything sorted. In Mark's post he came up with some wonderful graphics to be used as badges for testers to promote their own involvement. If you like Mark's creations and wish to use them on your website, please get in touch and he'll provide copies for you to use. In the meantime watch out for stickers and badges at various future events.

Over the past few months, we have some interest from a few companies that have expressed a desire to support CPAN Testers. Both via monetary funding and spare capacity on servers. We are still in discussion with these companies, so hopefully we will have some new sponsors in the near future. We are always will to have more companies support us, as alhough individual funding is very much appreciated, we would like to see more corporate sponsorship to help raise our profile. If your company uses Perl, and makes use of CPAN Testers either directly or inderectly, and would be willing to help support the long term future of CPAN Testers, please get in touch.

Individual funding, as mentioned, is always very much appreciated, and if you would like to contribute in any way, you can make donations via the CPAN Testers Fund, or Gratipay. All proceeds are managed for us by the Enlightened Perl Organisation, who are a not-fot-profit organisation.

The reason for promotion, and requests for sponsorship, has come about as the Birmingham Perl Mongers' funding of the CPAN Testers servers comes to an end in September 2015, when we'll need to find other sources of funding for the current server. On top of this we have the Metabase server, which also requires funding. Sadly dedicated servers don't come cheap these days, so it roughly costs around £4200 (US$6280 or €5800) per year. Over the next six months we'll be doing as much as we can to raise the funding, but if you can help promote this to your employers that would be wonderful.

If you have any ideas to help promote CPAN Testers in other ways, please let me know and we'll see what we can do.

CPAN Testers Summary - December 2014 - The House of Blue Light

2015-01-26T22:39:01Z

To begin with, a huge thank you to Nestoria, who selected CPAN Testers as their module of the month. Although we kind of break the mould of a module, we are exceptionally grateful for the donation via Gratipay, and for the promotion. As mentioned several times over the last few months, CPAN Testers are looking towards companies to help fund the servers we currently run, so donations from companies such as Nestoria are very gratefully received, and hopefully will help us to continue for many years to come. It is also nice to hear how CPAN Testers helped them too, when releasing Number-Format-SouthAsian.

We have been in discussions with a few other people and will have further announcements about sponsors and donations soon, but in the meantime, if you or your company would be willing to make a donation to CPAN Testers, you can do so via Gratipay, The CPAN Testers Fund, and/or by contacting me (Barbie), and we'll make sure you get suitable recognition for your contribution.

During December some initial thoughts were put together in preparation for the QA Hackathon 2015. The event in Berlin brings together several key people within CPAN Testers, as well as CPAN and Test communities, and will be an ideal opportunity to look at how we are processing the reports. In the first instance, I want to re-engineer the way the feeds are read, and how pages are built. Working with MessageQueues and Enterprise Service Buses recently, has hightlighted that we can make use of these technologies to speed processing up. As a consequence, I now have some areas of coding that I plan to start before the Hackathon, and hopefully provide some key parties a chance to review and improve the designs.

Although happening in January, I did want to pick up on an outage that happened earlier this month. My thanks to Bytemark for helping to resurrect the server for us. Thankfully, only one index table was knocked out, but after rebuilding, everything does appear to be back on track. Another task for the QA Hackathon is to look at a better way to reduce the risk for these indices. We also want to hopefully plan a move to MongoDB for the Metabase, which will stablise things considerably.

Sadly once again a bit delayed this month, but I do promise to improve in the coming months.

CPAN Testers Summary - November 2014 - Who Do We Think We Are

2014-12-23T08:04:30Z

So last month saw the London Perl Workshop take place. Several talks related to testing, and plenty of interest in CPAN Testers. Mark Keating and crew were videoing many of the talks. Sadly the microphone position meant the volume is quite low in places, but they are still worth watching. You can see testing talks from myself (slides), DrForr, and additionally my lightning talk (slides). If you missed the event, there are plenty of great talks you can now catch on the Shadowcat channel.

Also during last month there have been several fixes to the Reports Builder. Also following some conversations at LPW, I came away with a few ideas to improve the performance of the Feed and Builder components so that we can get almost real time production of reports. Over the coming months and likely during the 2015 QA Hackathon, I aim to work more on this and hopoefully have a solution working around the hackathon. Karen Etheridge also help to spot a fault with the Admin site, in that if a tester hasn't allocated a contact email address, any report that has been flagged isn't allocated to them, but to the Admin user. It does mean there is a bit of extra work for me at the moment, but ultimately I need to rework the way testers are notified, particularly in the event of the system using an old email address. If you've flagged any reports and haven't seen them removed as yet, please give me a nudge and I'll approve their removal.

One person I would to single out for making a great effort to promote testing in general, is Sinan Unur, who has been writing some great blog posts. TDD is all well and good, but who's testing the tests?, You've gotta quotemeta! and Tests should not fail due to EOL differences across platforms to mention a few. He has some great in depth Perl posts, and all are well worth reading.

A colleague of mine, Matt Wheatley, recently had a blog post published about how he works in development. Except the post isn't so much about how he works, but more about the value of a QA Department. Our QA team is awesome, and we perhaps rely on them a bit too much sometimes, but I imagine there are many awesome QA teams out there, and this is just one tribute to them all. I especially liked the reference to Bill Sempf's tweat.

We had another Metabase overflow last month, which means we've filled another bucket in AWS. David Golden has created a new one, but it does mean we need to think about moving to our newer architechure sooner rather than later. David, Neil Bowers and I have been in discussions recently, and I hope we will get some movement towards this in the coming year. There will be a lot of changes needed, both to code and routing, so it won't be a simple change. However, it will be a change that will help us grow more productively in the future.

More new coming next month. And my New Year's Resolution is to get these summaries out a bit more on time!