The MySQL databases have now been rebuilt and correctly synced between each other. The SQLite databases are now being updated and these should be completed by the end of the week. This has enabled me to write some simple scripts to create and repair the databases, which I'll now be including in a separate distribution to be released on GitHub. It will also include all the apache, mysql, logrotate, cron and other config and script files, so that if we ever have to rebuild again, getting started is a lot easier.

The websites are now rebuilding too. The statistics site will be complete shortly, as it takes a while to rerun all the statistical analysis. Trying to analyse everything all at once tends to grind the server to halt, whereas analysing bitesize chunks, although slightly slower, uses less memory and saves progress to disk, so we don't have to start again from scratch if anything falls over.

The reports site still has 27,000 entries to process, so I'm not expecting that to be down to a more reasonable number until the weekend. Turning the builder back on has highlighted some of the tweaks that are now missing from the work I did in the few weeks prior to the disk crash. In time I hope to address these performance tweaks again, but for now I'm just letting it build in peace :)

I'm now preparing to turn back on the feeds, to catch up on all the reports submitted in the last few weeks. I have run a couple of requests to fill in some of the gaps, and all seems to be fine, so I hope that a few days of dedicate feed processing should get through the bulk of reports from the last 3 weeks.

I haven't switched apache back on yet, as one thing that I've noticed is that server load is getting unnecessarily high. I know there is an intense amount of disk IO happening at the moment, but there appears to far too much disk access that there was previously. As a consequence I've started to reduce both the number of logs and log output. In most cases the output was only for information or debugging, so isn't necessary except in unusual circumstances. Hopefully after the weekend I'll get to switch Apache back on and can start some performance tuning.

If anyone has knowledge of performance tuning Debian Squeeze system processes, please get in touch. Trying to work out what tasks are essential and what can be run less frequent is getting tricky.