To begin with, a huge thank you to Nestoria, who selected CPAN Testers as their module of the month. Although we kind of break the mould of a module, we are exceptionally grateful for the donation via Gratipay, and for the promotion. As mentioned several times over the last few months, CPAN Testers are looking towards companies to help fund the servers we currently run, so donations from companies such as Nestoria are very gratefully received, and hopefully will help us to continue for many years to come. It is also nice to hear how CPAN Testers helped them too, when releasing Number-Format-SouthAsian.

We have been in discussions with a few other people and will have further announcements about sponsors and donations soon, but in the meantime, if you or your company would be willing to make a donation to CPAN Testers, you can do so via Gratipay, The CPAN Testers Fund, and/or by contacting me (Barbie), and we'll make sure you get suitable recognition for your contribution.

During December some initial thoughts were put together in preparation for the QA Hackathon 2015. The event in Berlin brings together several key people within CPAN Testers, as well as CPAN and Test communities, and will be an ideal opportunity to look at how we are processing the reports. In the first instance, I want to re-engineer the way the feeds are read, and how pages are built. Working with MessageQueues and Enterprise Service Buses recently, has hightlighted that we can make use of these technologies to speed processing up. As a consequence, I now have some areas of coding that I plan to start before the Hackathon, and hopefully provide some key parties a chance to review and improve the designs.

Although happening in January, I did want to pick up on an outage that happened earlier this month. My thanks to Bytemark for helping to resurrect the server for us. Thankfully, only one index table was knocked out, but after rebuilding, everything does appear to be back on track. Another task for the QA Hackathon is to look at a better way to reduce the risk for these indices. We also want to hopefully plan a move to MongoDB for the Metabase, which will stablise things considerably.

Sadly once again a bit delayed this month, but I do promise to improve in the coming months.

Sorry October's summary is a bit late, as I had rather a busy few months recently, and finding time to finish this one prove more tricky than normal.

However, first off, a thanks to Gabor Szabo, as one of his screencast featured a brief explanation of how CPAN Testers are intergrated into MetaCPAN. Although anyone familiar with the CPAN Testers Report site, will probably have understood what the numbers mean, the screencast is a handy help to anyone new to MetaCPAN or CPAN to learn more about using MetaCPAN. If anyone else has similar screencasts about CPAN Testers, or wishes to create one, please let me know.

My thanks to Pete Houston for notifying me early in the month that the feed to CPAN Testers had got stuck. Another reason why we really want to move away from Amazon and SimpleDB. This is still being investigated, and hopefully will be a project during the QA Hackathon next year. It also gave me pause to think of better ways the feed and builder could be processing. Thet are both mostly serial process, although there are now 2 builders (distros and authors). However, having worked a lot with MessageQueues over the last year, I'm looking at rewriting both the feed and builders, and again is likely to be my main project during the QA Hackathon in 2015. The ultimate aim, as has been asked on many many occassions, is to try and get as close to real time as we possibly can. On a good day we are less than an hour behind, but being a minute of two behind would be even better.

Karen Etheridge highlighted some problems with the CPAN Testers Admin site. Firstly the login had broken again. I've now permanently fixed this, and uploaded the code to CPAN so the old version doesn't overwrite the new version again. However, Karen also spotted a few other problems. Next up was the find a report by GUID. I'm not quite sure what I tested before, but I suspect I overwrote the working code again :( Again this is now fixed and the code is on CPAN :)

The current CPAN Admin problem is figuring out why, when marking reports for removal, they aren't getting alerted to the tester. They were getting added to the Admin user though, so they weren't lost, but it has highlighted a few cross-reference points that are not being made. Connecting a report to tester, or more imporantly to the tester's current email address is not always straight forward, so the fallback is to email the Admin user. I'm working on the code to reconcile these cross-references better, and in the coming weeks I should get that working properly again. In the meantime if you have marked reports, please bear with me, and I'll get to them as soon as I can. In the process of digging through the report removal process, I also discovered that the Builder mechanism for this was flawed, at least from an Author perspective. I've since corrected this too.

Andreas König noted that several reports from one tester were essentially the same. Breno G. de Oliveira helped to identify the problem, and the reasoning behind it. The reporter for cpanminus, doesn't currently record history as the other smokers, and consequently doesn't filter out repeated attempts by the author to install the same distribution. Breno is planning to work more on the Common Client, and integrate this into the cpanminus Reporter. So if you get similar repeated reports, please be gentle with the tester :)

Subramanian Karunanithi was the casualty of version numbering with CGI-Test. Thanks to some investigation work from Jonas B. Nielsen, he highlighted the latest version was 0.52, but the installed version was 0.104. The problem with version numbering within the installers is that they see 0.104 as Zero Point One Hundred and Four, which is greater than Zero Point Fifty Two. Karen posted a bug report, and Alex Tokarev has now released version 1.000, so all good. However, it does bear repeating that Version Numbers Should be Boring.

Next month we'll have a few notes about LPW, more fixes to the toolchain and some ideas for the future.

First big thing of July was our announcement for the new CPAN Testers Admin website. Sadly there are still someproblems for authors logging in, but we have been trying to get to the bottom if it. I've also been working on an alternative login mechanism, which will also be used by the CPAN Testers Preferences site too. We'll keep you posted when we have a fix. In the meantime, as a tester please feel free tosign up and claim your email addresses.

Renée Bäcker tweeted a solution to get your FAIL reports. The script, get_fail_reports.pl, retrieves the RSS feed for a specific author. Ideal if you want to filter and categorise FAIL reports toyour own requirements.

In another tweet, Stanislaw Pusep asked whether there was a problem with the CPAN Testers, as he was getting several submission failures. Thanks to Breno G. de Oliveira and Kent Fredric, it was identified as a problem with IO::Socket::IP. Specifically the smoke environment needed to downgrade to version 0.29. Unusual that a downgrade fixes the problem, but it seems a recent release, version 0.30, has had some undesired consequences.

And in yet another tweet, Ashley Willis pointed out that there is no link to 'how to be a tester' on the CPAN Testers Reports site, as well as most of the other sites. Although there is the Wiki page, the QuickStart page, it is a little too verbose, and probably jumps in a little too quickly, particularly for someone new to CPAN Testers. As such, I'd like to write an introduction page, that starts with some explanations for a completely fresh install of each client, then links to the QuickStart page. If anyone whats tomake a start before I get to it, please feel free.

Christian Walde asked on the mailing list about the defaults for the summary reports that get sent out. It seems BooK's settings had somehow become set to everything, and having spoken to others, it seems several authors are ignoring these mails as they don'tknow how tochange the settings. The defaults currently are set to only send FAIL reports, where the mix of perl and platform have not been sent previously. It was questioned whether this should change, but I believe adding more would not go down well with authors, and if authors really want additional reports they can update their Preferences, or look on the Reports site. However, as Christian highlighted, the emails themselves didn't make the link clear to change preferences, so this has now changed, and the link now appears earlier in the email.

Phillip Moore highlighted a problem with the way indexing happens in PAUSE. After David Golden forwarded to the modules mailing list, Grahan Barr replied and explained why this happened. As of version 2.00, FindBin::libs no longer has this "trick", but it does highlight that if you do have problems finding modules that are not indexed, it may have been purposefully hidden by the author. Dave Horner asked about the full name used in the Metabase secret file. Somehow Dave's secret file was missing the fullname field. Unfortunately, after the initial submission, the Metabase stores the communication using a Metabase GUID. However, all is not lost. If you have now submitted a secret and want to be able to associate your name and/or PAUSE id to your submission email,you can now do so via the new Admin site.

Sadly I missed posting about CPAN Day, but Neil Bowers did admirably posting various hints and tips of how to improve your modules and distributions. I'll post a little about this in next month's summary.

Lastly I would like to thank John Anderson for his keynote talk at YAPC::NA 2014. His commendation of CPAN Testers was very much appreciated. I only got to watch it recently, along with several others in the YAPC::NA feed. For those heading for Sofia this week for YAPC::Europe 2014, you have just as many great talks to look forward to. Sadly no CPAN Testers talks this year, but may be next year.

As part of the QA Hackathon in Lyon, one of my aims was to produce an API to the CPAN Testers reports. More specifically, provide the ability to retrieve a copy of a report as it was stored in the Metabase.

Although reports are available on the Reports site, they are a reconstruction of the report, as if it was an email. This was how reports were originally sent, but since 2010, the Metabase recieves each report as a JSON object, which it then saves with further references to other facts. When the Generator retrieves the JSON object it stores a copy before parsing the contents for the cpanstats database.

The reconstructed report pages might be enough to glean enough information to figure out the immediate problem, but sometimes it requires more long term analysis, which is where the Analysis site comes in. Currently Andreas scrapes the reports pages, which although mostly works, in the future, when we start to include much more metadata, this isn't going to be suitable. As such, returning a hash or JSON object is needed.

During the QA Hackathon I did manage to write the server side software to provide the API needed. However, I didn't get to finish the client side software. Over the following weeks I cleaned up the code, added some tests and finally CPAN-Testers-WWW-Reports-Query-Report was released to the world.

The module does a little more than I first planned, but that is mainly because I wanted people to be able to use the report returned in any way the see fit. So you can retrieve a Metabase::Fact object (the default), a Perl hash representation or the original JSON object. All that is required is the report Metabase GUID or the cpanstats ID. So how does it work?

The Options

You don't need to provide any options, as the module will use some defaults. Below is an example of the options, which incidentally happen to be the default options:

my %options = (
    as_json => 0,
    as_hash => 0,
    host    => 'http://cpantesters.org'
);

I allow you to defined the host, just in case we supply a similar API from the Metabase directly, or a new mirror (possibly even a local one) can support the same API.

The Object

Now we establish the object. If an options hash isn't provided, the defaults will be used.

my $query = CPAN::Testers::WWW::Reports::Query::Report->new( %options );

The Report

# get by cpanstats ID
my $result = $query->report( report => 40000000 );

# get by Metabase GUID
$result = $query->report( report => '0b3fd09a-7e50-11e3-9609-5744ee331862' );

The report method also allows you to override the as_json and as_hash attributes, should you need to retrieve the report slightly differently in some cases.

# force return as JSON
my $result = $query->report( report => 40000000, as_json => 1 );

The Last Error

In the event something goes wrong, you can hopefully get some information about what the problem was via the error() method. I can't guarantee it will be the whole picture, but hopefully it will give you some indication of what the problem might be.

my $error = $query->error;

The Conclusion

If you have suggestions for improvements, or spot problems in the current code, please let me know. If you do use the code, whether for mash-ups or for personal analysis, it would be nice to hear you've found the module useful. You could even blog about it :)

So the 2014 QA Hackathon has drawn to a close, but it is far from the end of the work, particularly for CPAN Testers. You can read several blog posts detailing many aspects of the QA and testing community work done during the hackathon, as well as several aspects of the toolchain, including PAUSE (which saw a lot of collaboration). It does get said often, and it bears repeating, that the QA Hackathons are a vaulable part of the Perl community, and help to drive many projects. Without them it is likely that key elements of the infrastructure we have come to rely on (PAUSE, CPAN, BACKPAN, MetaCPAN, CPAN Testers, CPANTS) would be a long way from being the resourceful, stable and continually developing components we have come to accept. Next year's hackathon will be in Berlin, and I for one am very much looking forward to it.

Aside from the work on the database for CPAN Testers during the hackathon, I did get to make a few releases, but several elements I started during the hackathon, were only completed during the following weeks. One of these CPAN-Testers-WWW-Reports-Query-Report, now enables you to retrieve the Metabase Fact object, JSON or hash representation of a specific report. For those looking at analysing the similarities and differencies between reports, this may make things a little easier, particularly when we start introducing more element facts into the report fact. Currently this only works for reports stored in the Metabase, so those early reports are not currently retrievable .. yet.

I discussed a new command line tool to submit reports with H.Merijn "Tux" Brand, who is keen to run reports in standalone instances. I see this working similar to Garu's cpanm-reporter, and with the common client that Garu has been working on, this could be a nice addition to the submission options. Liz Mattijsen, Tobias Leich (who easily has the coolest gravatar on CPAN) and I talked about how Perl6 distributions could be incorporated into CPAN Testers. There are some subtle differences, but there are also many common factors too. It was interesting to read the Rakudo blog post about Perl 6 and CPAN, as overcoming some of the hurdles potentially facing Perl6 developers are likely to help make CPAN a better place for all of us. The currently proposed solution is a similar approach to how different namespaces are stored in the Metabase. For the time being though, Perl6 distributions are excluded from CPAN Testers, but once we have a Perl6 smoker there is no reason not to include them. I'm not sure how soon that will be, but watch this space.

Andreas König and I once again looked at the way the reports are stored in the Metabase. Andreas had already highlighted the updated date, which is meant to be the date it entered the Metabase, was in actual fact the created date (the date of the testers platform). Along with David Golden, we looked at the code used by the Metabase, but failed to find anything wrong with it. It's hopefully something we can take more time over in the future, however the next priority for the Metabase is getting it moved onto MongoDB and away from SimpleDB. In the meantime, the Generation code, due to time constraints and the lack thereof, has been running using the 2010 version of the Metabase/AWS interface. During the hackathon and the following weeks, I finally upgraded to use the revamped version released in 2012. Although still troublesome to find all the reports, the search interface has been much improved, and now we have a much more reliable feed from the Metabase. This is also in part due to the rewrite of the internals of the Generation code to be more pro-active in find unusual gaps between reports.

I spoke with Neil Bowers during the hackthon too. Neil had suggested some ideas that I'd wanted to included into the CPAN Testers Statistics site. We discussed others during the hackathon, and although I have several notes to work from, it will be a little while yet before I can put aside some time to implement them. Neil has no end to the ideas to help improve CPAN, and I hope he'll be a good sounding board for ideas to incorporate into the site in the future. On the first day of the hackathon he posted about how quickly CPAN Testers test a distribution, after being uploaded to PAUSE. He was surprised to see some reports posted almost instantly. This is largely thanks to Andreas' work to bring the Tier 1 mirrors up to date within 10 seconds of a distribution being successfully uploaded to PAUSE. People such as Chris and Andreas use their own T1 mirrors to feed into their smokers, so as a consequence the reports can appear on the Reports site within one hour of the distribution hitting PAUSE!

I had intended to launch the CPAN Testers Admin site during the hackathon, but didn't get the chance to prepare a launch statement. I did wonder how a launch on 1st April would go down, but elected to wait a little longer. So expect some news on that front very soon.

Just before the hackathon we reach 40 million report submissions. An impressive number by any standards, and gives a compelling argument that Perl & CPAN are still being actively maintained and developed.

John Scoles made a post related to CPAN Testers, highlighting how CPAN Testers showed why a particular test needed a little more thought. Thanks to the wide variety of perls and platforms tested by the CPAN Testers, it provided John with some intriguing failures, and made for an informative look at how hashes called in scalar context are handled differently between versions of Perl and platforms. Prior to the QA Hackathon Neil Bowers posted about his new idea for An author's CPAN dashboard, which he continued to develop during the hackathon too. We even discussed, together with Olaf Alders, whether this was something that should feature in MetaCPAN. For now it won't, but hopefully it can develop further and w'll see what people make of it, once more APIs are fed into the dashboard.

Another month wrapped up, and another QA Hackathon over. Lots to do and plenty of fresh ideas coming. Happy testing.