Watching the master at work

There are a lot of Linux Engineers out there with more experience than me. I always read their posts when I get in trouble, and some times it helps me out. I was really close last night when one of the few guys I actually know who certifiably “Knows his sh*t waaay better than me” showed up.
My problem was I had half of a mirror set (Linux software RAID1) and the other half had gone south during a reboot. The half I had wouldn’t boot. Murphy’s law explicitly states that when I have part of a mirror set fail, the bootstrap will never be on the drive that survives. Murphy is one of the reasons I was suprised to see Dan L.
Given that I now have a reconstructed and booted system, I thought I would record the steps I took, so that if someone else encounters the same situation it may be of use to them.
The box all this was setup on is an entry level Intel server board, with integrated IDE controller and dual 80GB IDE drives. The RAID was setup as part of the installation process of RedHat 7.3, and I use GRUB as the bootloader.
To repair the RAID, I was using a RedHat install disk and booting to Linux Rescue. You can get a tiny Linux booted up to mount partitions and try to fix systems using this option. Debian has something like this too on their install CD.
I managed to partition the new drive exactly the same as the old drive. I should also mention that when the drive failed I initially tried some totally desparate switching of slave to master and cable swaping to get the other good drive to boot. No dice. Somehow at the end of this whole story when I finally booted and got everything running, it was OK that the drive on the second IDE channel had migrated to the primary IDE channel, and that the jumper setting had been changed. I really have to like an OS that can stand up to my hardware skills.
Once Dan became involved things started to come together. The first thing he noted was that none of the partitions were marked as bootable. Using fdisk, he made the partition which contained the kernel images (i.e. /boot) bootable. I am going to have to do more reading about this bootable partition thing; I have not used this in the past. On the other hand one of my most stable boxes has the swap partition on each drive marked as bootable. Making the /boot partition bootable makes a lot more sense to me. YMMV, this is a whole other topic.
In RH73 fdisk the option to mark a partition as bootable is “a”. I am told I should look into cfdisk, in fact the main page for fdisk says to choose cfdisk over fdisk.
Once the disk was setup, he mounted the root partition from the drive that was good, and performed a chroot to the temporarily mounted root partion. This enabled him to edit the raidtab to mark the new disk as “failed” so that once the operating system saw it the kernel wouldn’t accidentally reconstruct the wrong (i.e., blank) half of the mirror. That looks something like this for each md device:
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
chunk-size 64k
persistent-superblock 1
nr-spare-disks 0
device /dev/hda2
raid-disk 0
device /dev/hdd2
raid-disk 1
failed-disk 1
Dan also suggested we disconnect the new drive to be doubly sure there were no accidental reconstructions. Good plan.
Since I use GRUB, it is already installed. One of the very cool things about GRUB is that it has a tiny shell which allows you to setup the boot environment and also to install the bootloader. Once you’re in the shell you can tell grub to install a bootstrap using a command something like this where “hd0″ is the master drive on the primary IDE channel:
grub> install (hd0)And sure enough, the darn thing booted to grub and from there the system came up. The RAID barely complained about running in degraded mode because of the drive that we had disconnected and marked failed in the raidtab. It came up again when we reconnected the device too. Dan was hungry and I was tired, so we decided to call it a night after the box came up and the services were all running again.
Besides which, I couldn’t very well ask him for the help I needed to figure out how to reconstruct the arrays after everything else he had done. Fortunately the NEW documentation for NEW raidtools (2.4 kernel and above) is very good and it worked.
To reconstruct the array use the raidhotadd command:
raidhotadd /dev/md0 /dev/hdd2It doesn’t look like anything is happening, but you can check your /proc/mdstat to see whats going on. array reconsruction took a few minutes for small partitions and more than an hour for a 60GB partition. Reconstruction occurs in the background, so load on the server may effect this time. The drives can remain mounted and active while reconstruction occurs.
[root@cp root]# cat /proc/mdstat
Personalities : [raid1]
read_ahead 1024 sectors
md2 : active raid1 hdd9[0] hda9[1]
104320 blocks [2/2] [UU]
md6 : active raid1 hdd8[0] hda8[1]
66565184 blocks [2/2] [UU]
md3 : active raid1 hdd7[0] hda7[1]
3076352 blocks [2/2] [UU]
md1 : active raid1 hdd6[0] hda6[1]
513984 blocks [2/2] [UU]
md5 : active raid1 hdd5[0] hda5[1]
6144704 blocks [2/2] [UU]
md4 : active raid1 hdd3[0] hda3[1]
104320 blocks [2/2] [UU]
md0 : active raid1 hdd2[0] hda2[1]
722816 blocks [2/2] [UU]
unused devices:
I think since I manually marked the disk failed I am going to have to manually remove the failed entry from the raidtab, but I am going to deal with that when I am on-site.
I have been in this same situation three times in 5 years, and this is the first time I actually got reconstructed arrays working cleanly. I don’t really want to confess to the lengths I have gone to in order to fix this issue, but I guess I already have.

Comments are closed, but trackbacks and pingbacks are open.