Last night I sat down to watch an episode of The Practice that I had downloaded to my OpenFiler server in the basement.
I fired up the Xbox and launched Xbox Media Center. (I should probably upgrade that.) I tried to browse to the share on the OpenFiler server, but it told me it couldn’t connect.
That can’t be good.
So I headed down to the basement and turned on the monitor. No output. There’s a green light on the front of the box, so it’s on. I hit the reset button.
The server came up and booted, but the RAID 5 array with the data on it didn’t come online. Rut-row.
It appears that I have lost one of the disks in the array. According to the md man page, md won’t bring a degraded RAID array online at reboot unless you pass it a kernel argument:
If a RAID4, RAID5 or RAID6 array is degraded (missing at least one drive) when it is restarted after an unclean shutdown, it cannot recalculate parity, and so it is possible that data might be undetectably corrupted. The 2.4 md driver does not alert the operator to this condition. The 2.6 md driver will fail to start an array in this condition without manual intervention, though this behaviour can be over-ridden by a kernel parameter.
So, I have backups, right?
Nope.
I’ve been meaning to buy a 1TB USB hard drive to back up all my stuff (mp3s and photos mainly) but hadn’t gotten around to it yet. I bet you can guess what I’ll be buying on my way home.
It looks like I will be able to bring the RAID array up in degraded mode so I can copy the data off of it, so I think I’m okay there.
I went to Newegg today and bought three new 500GB hard drives. I’m just going to replace all the old drives that are currently in the server as they are all the same age and will probably start failing.
I’m also thinking about rebuilding the filer from scratch. The version of OpenFiler that I am running is a little old, and the last upgrade didn’t go so well so the OS is in a weird state. I might install FreeNAS instead and see if that’s a little easier. It doesn’t look like FreeNAS has any snapshot support, but it does have a Recycle Bin option on the CIFS share page so undelete might be possible (something I could have used last week when I deleted a bunch of stuff by accident.)
FreeNAS is also very small and can run off a USB drive. That could be entertaining to try. I’ll have to see if the motherboard I have will boot off a USB drive.
