6:24 pm
I had a wonderful birthday. I really did. Jaspenelle pampered me all day (not that she doesn’t pamper me all the time anyways, it was just more so that day). I relaxed, played with the cats, we went to a movie (The Reaping - great movie BTW, you should see it it you haven’t) at a new movie theater in town. All in all I had a good time.
It was just when I was about to go to bed that the disaster struck. I was trying to upload a video of Aos to youtube (check back friday for the feline friday if you want to watch it) when suddenly the internet stopped working. I came into the studio where phoenix, our beloved [web server|file server|router|desktop|beast of a machine] is kept. To my horror, 4 of the hard drives in the RAID array were showing Orange - they had died. Unfortunetly, that was just one too many. To the credit of linux, it was still running - just not very well. The filesystem was giving I/O errors for most operations, and had moved itself into read-only mode. The routing part of it wasn’t really down, just the transparent proxy I had setup that needed to be able to write to disk.
I gave phoenix CPR for a few hours and decided that it was a goner. I shut it off, reconfigured the network so that phoenix wasn’t required as a router, and started to come up with a plan to get our personal websites back online, and allow both myself and Jaspenelle to work on the computer at once (with phoenix gone, we were down to a single desktop: mooncougar).
From 2am to 8am, I pointed the DNS entry for our local computer (local.garjasp.com) to my hosted server. Because all of our personal sites hosted locally used a CNAME to that one entry (actually required because my dynamic DNS can only update one entry - local.garjasp.com), I was easily able to put up a message letting people know what happened:
Server Crash
Our beloved server, phoenix, has had another massive hard drive crash. This time, he signed a DNR, so we are letting him die the death he wishes. Our personal websites will be down until we get another server set up, and our online time will be limited, as we now have to share a single laptop.
Michael and Jaspenelle Stewart, 21-Apr-2007
My plan of action was simple:
- Buy a new computer (woohoo! birthday present), which would be the desktop that I would work on.
- Repurpose mooncougar to be the local file server/router
- The laptop would become Jaspenelle’s “desktop” as that is what she uses most of the time anyways
- Expedite the setup of a second hosted server that will now run all of our personal sites as well as other sites I host. (In the meantime our websites are hosted on mooncougar, which is why they are quite slow)
We ended up buying a Compaq Presario SR5010NX with Windows Vista Home Basic. (Yes, I’ve returned to the Beast — Windows) for a little under $450. It took me the better part of 8 hours to remove all the crap on it I don’t need, put all the crap on it I do need (Firefox, Thunderbird, Gaim/Pidgin, Gimp, gVim, PuTTY, TortoiseSVN, OpenOffice, and Avast Antivirus — I may have returned to the beast, but I still love open source)
While installers were running and files were downloading, I set to repurposing mooncouger. I had originally planned to just wipe it and start fresh, but when I tried to boot from the Ubuntu CD-ROM, it wouldn’t load (it booted, just wouldn’t come up). It turns out that the CD-ROM in mooncougar is on it’s last legs. So instead, I fired up aptitude and put a minus on the x11 category. Then promptly spent 3 hours resolving the broken packages
During all this (still multitasking), I started unpacking the backups. This is the second time I’ve had to restore from backups this year, and it was a breeze. My backup system Just Works.
This morning, in the hour I had before I had to go to work, I got Apache, MySQL, and PHP configured the same way phoenix was, and reset the DNS for local.garjasp.com to point back to mooncougar. I had to leave for work before I could test as well as I would have liked. But most the sites were working fine. A few tweaks and cache rebuilds here and there when I got home today, and everything seems to be working well.
I cannot stress how important it is to have a daily backup routine. Mine is automated, and well tested. If I did not have it, I would have lost everything - websites I’ve designed, jaspenelle’s art, software that is a work in progress (including the entire SVN history), and much much more. If you don’t have a backup routine, and have anything that you can’t lose, then you are playing with fire. It will burn you one day. Hard drives crash. Technology fails. Be prepared. I was, and because of it, I have everything still.