LVM Blunders and Restoration Tricks

After upgrading to Fedora Core 5 recently, I was pleased to discover a new, easy-to-use GUI for the administration of LVM. This spiffy technology enables one to create 'logical' volumes from one or more physical disks -- that is, one can take two 100 GB drives and make a single virtual 200 GB drive from them, as far as the filesystem is concerned. I tried it out on a freshly-installed machine, and after success with it decided to migrate the data on my primary computer to LVM management as well.

So, here's what I ended up doing. In my primary machine, I had two hard disks which I used to store stuff; I'd rather just have one mega-volume instead. Since I've been experimenting with MythTV, having the ability to easily resize/repartition/reformat areas seems like fun as well. So, the goal was to combine two ext3 partitions, hdf5 and hde1, into one logical volume, a seemingly simple task!

To perform the conversion, I had to do the following:

  • Transfer all the data from hdf5 to hde1
  • Initialize hdf5 as a new LVM volume
  • Create a new logical volume, 'Data,' from the freshly-initialized partition as an ext3 area, in an existing LVM Volume Group 'VolGroup00' (the root filesystem on FC5)
  • Move the data from hde1 to the newly-minted LVM volume 'Data'
  • Initialize hde1 as a new LVM volume
  • Append the hde1 LVM volume to the logical volume 'Data'
  • Resize the filesystem on 'Data' to accomodate all the new space!

Not too bad, right? The provided system-config-lvm utility made the LVM wizardry a piece of cake, even adding a new entry to /etc/fstab so the freshly-minted logical volume would be accessible following a reboot of the machine:

/dev/VolGroup00/LogVol00 /                      ext3    defaults        1 1
LABEL=/boot             /boot                   ext3    defaults        1 2
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
sysfs                   /sys                    sysfs   defaults        0 0

/dev/hdf5		/mnt/data		ext3	defaults	1 1
/dev/hde1		/mnt/data2		ext3	defaults	1 1
/dev/VolGroup00/Data    /data    		ext3	defaults	1 2

But, that was not the case! Following a reboot, the init process threw up a big fit about the filesystems on /dev/hdf5 and /dev/hde1 being corrupt (their supernodes being bad) and dropped into the root-only filesystem maintainance mode.

Note: This is normally a Bad Thing. However, since I knew that I had just combined two ext3 filesystems into a mega-volume, which also used ext3, it seemed reasonable that there might be some confusion on the part of init. So, I then went on to do a Very Bad Thing, which happens to be the Correct Thing in the event of a regular, non-LVM partion:

#fsck

fsck then promptly found many, many errors in inode counts, etc; things that e2fsck is good at doing and fixing. Following another requisite reboot, I was then quite unsettled to see that I had induced a kernel panic and corrupted my LVM configuration quite well. D'oh!

Of course, I then realized what my fatal mistake was. The fstab entries for the old volumes were tripping up init. With the physical space for the prior ext3 filesystems being absorbed into a new one, and ext3 being placed on top of it, init was rightfully confused. As such, the fsck went and clobbered all the LVM-specific information stored in the filesystem-specific areas of the hard disk! If the old entries had been removed from /etc/fstab, there would have been no problem at all. (After I was able to fix the problem, this has become an outstanding bug report for the Fedora maintainers.)

The Fix

After several frantic Google searches I came across some helpful sites (below) that gave me critical insight as to repair the damage I'd done. The trick was to utilize the Fedora Core 5 Rescue CD. Although it was unable to detect my Linux partition, it did give me the ability to drop to a root shell. Added to the mix of handy utilities like fdisk, the Rescue CD also has the lvm pseudo-shell, which bundles together all the set of LVM command-line tools like pvcreate etc.

External Links

Bugzilla Filing for this problem
The Important Post Which Tipped Me Off to the Solution
Fedora Core 5 Rescue CD