It looks like we started getting some database timeouts starting about 4 hours ago. Both the usual suspects were asleep at this time, so it wasn't something we actively changed. My working theory is squirrels in the data-centre.
We've sent our crack team of squirrel hunters to chase the rodents. We have sourced the finest hazelnuts as bait, and we'll all be donning faux-fur coats while we hunt them (although there is a minor disagreement over red vs grey, which turned ugly).
Once we've exhausted all squirrel-based theories (we have many - one involves using disk platters for an ad-hoc party), we'll give up and try profiling the database server. It might be that after all.
But I still think it is squirrels.
well, the good news is that we found some squirrels. Or at least, I think they were squirrels. You can get black squirrels, right? With thin wiry tails about the length of their body? Whatever; it turns out that black-squirrel is delicious when grilled. With ketchup, of course - you wouldn't eat black-squirrel without ketchup.
When we'd finished the impromptu bbq, we did some work to kick the misbehaving database server into shape. We think it is scared enough to behave now.
By the way, does anyone know the best way to clear charcoal smoke out of a server room?