The Fun Begins
So on Thursday 12/30/2011 I was in the basement and I heard a pretty loud noise coming from the server rack. I thought I was just losing another fan, so I didn’t pay it much attention.
Then I realized that my server was not behaving as it should, so I looked at the console. Lots of drive errors. Great.
I rebooted it and it refused to come back up. Oh boy, this is going to be fun.
I do run backups every night of the home directories and the mysql databases, and they get copied to my NAS device, so I should be in pretty good shape.
I also had another server built (that I built more than a year ago, maybe two) that I was planning on migrating to anyway, so this forced my hand.
I run a handful of email lists for some friends and a handful of websites for myself and some friends. How bad can this be?
This is long, click continue if you want the gory details.
Server OS
Well, to start it off the version of Ubuntu on the new server was so old that it was unsupported and I couldn’t install the packages I was missing. So I got to figure out how to upgrade from 9.0.4 (Jaunty) to 10.04 LTS (Lucid). It involved downloading the CD ISO images, mounting them with loopback and running the installer. Details here: Upgrades via alternate CD
I had to do that twice – once to get from 9.0.4 to 9.10 and again to get from 9.10 to 10.04 LTS. That was a good time. It’s not really recommended to go from 9.04 to 10.04 without reinstalling, but I did it, and it worked out okay.
Email and Mailing Lists
The old server was running Sendmail. Which I know how to configure pretty well. I decided to use Postfix on the new server. It’s supposed to be easier to configure than Sendmail (and it is, once you know how) but I was not as familiar with it.
I host 5 domains, so I needed virtual email hosting, so I had to figure that out. I also run Spamassassin, SqlGrey and Procmail, so there is some fun interaction to figure out there. Luckily all these packages are part of the Ubuntu repo, so installing them is easy, and the config files are mostly ready to work together. But the config files are in different places compared to my old server, so it was fun to chase them down.
So it took me about 1/2 a day to get that all sorted out so that I could receive mail for all my domains.
For the mailing lists I was running Majordomo on the old server. The new server is using Mailman. Mailman is much more friendly with a web interface for administration. But there is extra work to do to make it use virtual domains and that took some figuring.
Then I ran into my first real snag. My backups did not include the email list configs. I did not have a list of all the members of the lists.
D’oh.
So I built them as best I could by harvesting addresses from my emails from the lists. Two of the lists have additional administrators and they had both recently grabbed a list of members, so I was golden there, but the other lists were not complete. Oh well. I harvested a few more addresses from the moderation emails I got when non-members posted, but I knew I was still short.
And then the firewall crashes…
As if I wasn’t having enough fun, my firewall started acting flaky when I was trying to change the SNAT for the server to solve an email spam blacklisting issue caused by the fact that the forward and reverse DNS didn’t match.
So because it was being flaky, I rebooted it. Remotely. And what do you think happened?
It refused to come back up.
Luckily Liz was at home so she could be my eyes and hands. I called her and she went down and got on the console. It will full of hard drive errors and the OS was asking for a manual FSCK run. Oh great. So I talked her though firing FSCK and it started complaining about tons of bad sectors. Fabulous.
So we let it run. It took about an hour I guess. Then I had her boot it up again. It came up but then it hung when it was trying to bring up the external interface. I had seen this before, so I had her power cycle the box again. It came back up and then hung again. So I told her to just let it sit. After about an hour it just started working, but the web interface did not come back up. It was passing traffic, so I was happy for the time being.
When I got home I took a newer hard drive, the chassis of the server that had blown up it’s hard drive and a new copy of IPCop and built a new firewall. It only took an hour and a half and a little googling to figure out how to do SNAT in the new version (hint: manually add IPTables commands to the /etc/rc.d/rc.firewall.local file.)
I swapped it in and away we went.
But now the fans in the box are making lots of noise, so I’ll be swapping them out too. I love old hardware…
BTW the uptime on the old firewall was 422 days before I rebooted it. Not too shabby.
And back to getting the server rebuilt.
Web Sites, PHP and MySql
More fun occurred when I started to restore the websites and mysql databases. I copied all the home directories over and that was fine. Then I restored the mysql database for my primary website.
The old server was running PHP 4.x and MySql 4.x. The new server is running PHP 5.3.2 and MySql 5.0.17. So is this going to go smoothly? What do you guess?
MySql made no complaints about the database restore. Just slurped it right up.
My server hosts several different PHP applications – Mambo CMS, Joomla CMS, Serendipity for blogging (like this blog) and a PHPBB Forum.
Not surprisingly, none of them are the current version.
So I point my browser to the front page of my site. And I get an error message. Turns out the permissions are all messed up in the web root. Not a difficult fix. Then I hit reload and I get my web site – sort of. The framework and menus are there, but no content. That’s really weird.
I have to do a lot of searching with google to figure out that Mambo doesn’t play well with PHP 5.x. Oh great. More searching tells me that the fix is to upgrade Mambo, then hand edit a couple of files.
The upgrade path from my current version (4.5.3h) is to apply all the patches, in order. One of the patches has you run a SQL script to update the database. Two of the patches have you delete some files and hand edit some others. Oh fun. Did I mention that I have more than one Mambo installation?
In order I installed:
- Mambo_Patch_4.5.3h_to_4.5.4.zip <- requires file deletion
- MamboV454_Security_Patch1.zip
- MamboV454_Security_Patch2.zip
- Mambov454_Security_Patch3.zip
- Mambo_Patch_4.5.4SP3_to_4.5.5.zip
- Mambo_Patch_4.5.5_to_4.6.2.zip <- requires file deletion, SQL run and file editing
- Mambo_Patch_4.6.2Lite_to_4.6.5.zip <- requires file editing
You can download all these patches here. Then you need to hand edit a couple of files so they support PHP 5.x. Those hand edits are here.
In addition you need to turn off strict mode for MySql 5.x. Those instructions are here.
Wonder of wonders, when I got done with all that messing about, my web site came up. 🙂
So I steeled my nerves and clicked on the link for my blog. And it came right up. Seriously? That’s a win!
I restored the rest of the MySql databases and did the Mambo patch dance for another site. And it came right up! This site was the one that also had the PHPBB forum on it. So I clicked on that link and it worked! I logged in and created a test posting – and it worked! Awesome!
But let me back up here a little. Remember when I said I didn’t have the config files backed up? That includes the Apache config files for all my web sites. Now those shouldn’t be hard to recreate but it would be nice to have them.
So I borrowed a IDE to USB interface and plugged in the hard drive for the server. Wonder of wonders it came up and I could copy data from it. So I grabbed the Apache configs and the email list member lists. Whoo-hoo. It’s getting better.
And then I transferred my domains…
As if I wasn’t having enough fun, I decided I would transfer my 5 domains away from GoDaddy. GoDaddy is supporting the SOPA bill, and that’s a bad deal.
I had a domain coming up for renewal, so I decided to move them somewhere else. Reddit had a nice thread about other registrars so I looked it over and picked 1&1.
I unlocked the domains, got the authorization codes and started the transfer. When I was signing up at 1&1 it was not obvious that they were going to park my domains on their name servers. Nor was it obvious where I could specify the name servers that I wanted to use. (I run a name server and have three secondarys around the country.)
So I approved the transfers and away we went. This morning I tried to load my web site and got a 1&1 parking page. Oh hell.
So I went and figured out how to change the DNS servers for my domains at 1&1, but the TTL is 48 hours, so it will probably be broken until late tomorrow. My web sites are down and email is busted. Again.
Final thoughts
So I still have one web site running Joomla that I need to bring up. Most of the others are up and running. I hope I can find good instructions for that.
Then I need to setup my backups again. This time I’ll include some config files I think. 🙂
I backup to my iomega NAS with rsync and it backs up to Crashplan, so I’m covered pretty well there.
I also sent a letter to 1&1 letting them know that I was unhappy with the way my domain transfer went. We’ll see if I get a response.
