Database Corruption
Sunday, July 31st, 2005The Webmaster-Talk database went boom this morning. I wake up to find a PM in my inbox on WT, from Tim, saying that the database is screwed up and he really needed to leave and if I could take care of it.
At that point, we had corrupted thread and post tables so no threads or posts were showing up. Not such a big deal, we should be able to revert to a backup that was taken the day before or maybe even repair the database. So after I read the PM, I go and run our repair scripts just in case it might fix the problem. It doesn’t. Not 5 minutes after I read the PM from Tim, the entire database goes kaput. You couldn’t even see our ‘forum closed’ error message.
So now I’m going into the Kualo support desk to take over where Tim left off. The guy there says he will restore the backup that we had from 11 hours prior. Thats great, I’m thinking. Not the best, but 11 hours isn’t too much. Then he comes back and says the backup from 11 hours ago was too corrupt to be restored! So we had to go all the way back to a backup made on the 26th! 11 hours not so bad, 5 days? Bad!
Alas, we are up and running (though not under the best circumstances). I just made an announcement explaining why everything is 5 days old. I wonder if any members are going to explode. I know how they’d feel (well, I know how I feel), but I hope they can understand it was unforeseen and out of our control.
Update 1
And just as I typed this up, our post table (amongst a couple others) is corrupt again! I think it’s time to move our mySQL server to another box.
Update 2 (8:00PM)
So everything should be fine now. The same backup from the 26th was repaired and restored. I’ve opened the forums and and I’m hoping nothing bad will happen. It’s been open for about 10 minutes and I don’t see any corruption, so I hope that means we’re home free.