Rudi Ahlers
2009-May-08 12:30 UTC
[CentOS] domU corrupt after server crash, help needed trying to recover domU LVM
Hi all, One of our Dell servers has failed badly, and one of the domU's has been corrupted in the process. It boots up to a point and then gives me a kernel panic: Loading dm-zero.ko module Loading dm-snapshot.ko module Scanning and configuring dmraid supported devices Scanning logical volumes Reading all physical volumes. This may take a while... No volume groups found Activating logical volumes Volume group "VolGroup00" not found Creating root device. Mounting root filesystem. mount: could not find filesystem '/dev/root' Setting up other filesystems. Setting up new root fs setuproot: moving /dev failed: No such file or directory no fstab.sys, mounting internal defaults setuproot: error mounting /proc: No such file or directory setuproot: error mounting /sys: No such file or directory Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys switchroot: mount failed: No such file or directory Kernel panic - not syncing: Attempted to kill init! It shows up as a Zombie: [root at xen ~]# xm list Name ID Mem(MiB) VCPUs State Time(s) Domain-0 0 1439 1 r----- 329.0 Zombie-hfserver2 15 1024 1 ----c- 0.5 hfdns02 10 519 2 r----- 1552.8 I can't mount either: root at xen ~]# mount /dev/data/hf hfdns02 hfserver2 [root at xen ~]# mount /dev/data/hfserver2 /mnt/cpanel/ mount: you must specify the filesystem type [root at xen ~]# mount -o loop /dev/data/hfserver2 /mnt/cpanel/ mount: you must specify the filesystem type Here's the output of the LVM partitions: [root at xen ~]# lvscan ACTIVE '/dev/data/cpanel002' [100.00 GB] inherit ACTIVE '/dev/data/windows2003_web' [30.00 GB] inherit ACTIVE '/dev/data/storage' [50.00 GB] inherit ACTIVE Original '/dev/data/hfserver2' [30.00 GB] inherit ACTIVE '/dev/data/hfdns02' [30.00 GB] inherit ACTIVE '/dev/data/pluto' [30.00 GB] inherit ACTIVE Snapshot '/dev/data/pluto_s' [30.00 GB] inherit ACTIVE '/dev/system/root' [39.06 GB] inherit ACTIVE '/dev/system/swap' [9.75 GB] inherit [root at xen ~]# vgscan Reading all physical volumes. This may take a while... Found volume group "data" using metadata type lvm2 Found volume group "system" using metadata type lvm2 [root at xen ~]# Does anyone know how to fix a LVM like this? -- Kind Regards Rudi Ahlers CEO, SoftDux Hosting Web: http://www.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos/attachments/20090508/1d743b38/attachment-0004.html>
Rudi Ahlers
2009-May-08 13:43 UTC
[CentOS] domU corrupt after server crash, help needed trying to recover domU LVM
On Fri, May 8, 2009 at 2:30 PM, Rudi Ahlers <rudiahlers at gmail.com> wrote:> Hi all, > > One of our Dell servers has failed badly, and one of the domU's has been > corrupted in the process. It boots up to a point and then gives me a kernel > panic: > > Loading dm-zero.ko module > Loading dm-snapshot.ko module > Scanning and configuring dmraid supported devices > Scanning logical volumes > Reading all physical volumes. This may take a while... > No volume groups found > Activating logical volumes > Volume group "VolGroup00" not found > Creating root device. > Mounting root filesystem. > mount: could not find filesystem '/dev/root' > Setting up other filesystems. > Setting up new root fs > setuproot: moving /dev failed: No such file or directory > no fstab.sys, mounting internal defaults > setuproot: error mounting /proc: No such file or directory > setuproot: error mounting /sys: No such file or directory > Switching to new root and running init. > unmounting old /dev > unmounting old /proc > unmounting old /sys > switchroot: mount failed: No such file or directory > Kernel panic - not syncing: Attempted to kill init! > > > It shows up as a Zombie: > > [root at xen ~]# xm list > Name ID Mem(MiB) VCPUs State Time(s) > Domain-0 0 1439 1 r----- 329.0 > Zombie-hfserver2 15 1024 1 ----c- 0.5 > hfdns02 10 519 2 r----- 1552.8 > > > I can't mount either: > > root at xen ~]# mount /dev/data/hf > hfdns02 hfserver2 > [root at xen ~]# mount /dev/data/hfserver2 /mnt/cpanel/ > mount: you must specify the filesystem type > [root at xen ~]# mount -o loop /dev/data/hfserver2 /mnt/cpanel/ > mount: you must specify the filesystem type > > Here's the output of the LVM partitions: > > [root at xen ~]# lvscan > ACTIVE '/dev/data/cpanel002' [100.00 GB] inherit > ACTIVE '/dev/data/windows2003_web' [30.00 GB] inherit > ACTIVE '/dev/data/storage' [50.00 GB] inherit > ACTIVE Original '/dev/data/hfserver2' [30.00 GB] inherit > ACTIVE '/dev/data/hfdns02' [30.00 GB] inherit > ACTIVE '/dev/data/pluto' [30.00 GB] inherit > ACTIVE Snapshot '/dev/data/pluto_s' [30.00 GB] inherit > ACTIVE '/dev/system/root' [39.06 GB] inherit > ACTIVE '/dev/system/swap' [9.75 GB] inherit > [root at xen ~]# vgscan > Reading all physical volumes. This may take a while... > Found volume group "data" using metadata type lvm2 > Found volume group "system" using metadata type lvm2 > [root at xen ~]# > > > Does anyone know how to fix a LVM like this? > > -- >Here's what I've done so far: [root at xen ~]# losetup /dev/loop4 /dev/data/hfserver2 # This mounts the LVM partition as imaginary physical to /dev/loop4 # data is the virtual group (VG) name [root at xen ~]# kpartx -va /dev/loop4 add map loop4p1 : 0 208782 linear /dev/loop4 63 add map loop4p2 : 0 62701695 linear /dev/loop4 208845 # This creates a device map with partitions in /dev/data/hfserver in /dev/mapper [root at xen ~]# vgscan Reading all physical volumes. This may take a while... Found volume group "VolGroup00" using metadata type lvm2 Found volume group "data" using metadata type lvm2 Found volume group "system" using metadata type lvm2 [root at xen ~]# lvscan inactive '/dev/VolGroup00/LogVol00' [27.94 GB] inherit inactive '/dev/VolGroup00/LogVol01' [1.94 GB] inherit ACTIVE '/dev/data/cpanel002' [100.00 GB] inherit ACTIVE '/dev/data/windows2003_web' [30.00 GB] inherit ACTIVE '/dev/data/storage' [50.00 GB] inherit ACTIVE Original '/dev/data/hfserver2' [30.00 GB] inherit ACTIVE '/dev/data/hfdns02' [30.00 GB] inherit ACTIVE '/dev/data/pluto' [30.00 GB] inherit ACTIVE Snapshot '/dev/data/pluto_s' [30.00 GB] inherit ACTIVE '/dev/system/root' [39.06 GB] inherit ACTIVE '/dev/system/swap' [9.75 GB] inherit [root at xen ~]# lvchange -ay VolGroup00 [root at xen ~]# lvscan ACTIVE '/dev/VolGroup00/LogVol00' [27.94 GB] inherit ACTIVE '/dev/VolGroup00/LogVol01' [1.94 GB] inherit ACTIVE '/dev/data/cpanel002' [100.00 GB] inherit ACTIVE '/dev/data/windows2003_web' [30.00 GB] inherit ACTIVE '/dev/data/storage' [50.00 GB] inherit ACTIVE Original '/dev/data/hfserver2' [30.00 GB] inherit ACTIVE '/dev/data/hfdns02' [30.00 GB] inherit ACTIVE '/dev/data/pluto' [30.00 GB] inherit ACTIVE Snapshot '/dev/data/pluto_s' [30.00 GB] inherit ACTIVE '/dev/system/root' [39.06 GB] inherit ACTIVE '/dev/system/swap' [9.75 GB] inherit [root at xen ~]# e2fsck /dev/VolGroup00/LogVol00 e2fsck 1.39 (29-May-2006) /dev/VolGroup00/LogVol00: clean, 631982/7325696 files, 4512772/7323648 blocks At first it found a whole lot of damages inodes which I repaired. Then, I reversed the process: [root at xen ~]# lvchange -an VolGroup00 [root at xen ~]# lvscan inactive '/dev/VolGroup00/LogVol00' [27.94 GB] inherit inactive '/dev/VolGroup00/LogVol01' [1.94 GB] inherit ACTIVE '/dev/data/cpanel002' [100.00 GB] inherit ACTIVE '/dev/data/windows2003_web' [30.00 GB] inherit ACTIVE '/dev/data/storage' [50.00 GB] inherit ACTIVE Original '/dev/data/hfserver2' [30.00 GB] inherit ACTIVE '/dev/data/hfdns02' [30.00 GB] inherit ACTIVE '/dev/data/pluto' [30.00 GB] inherit ACTIVE Snapshot '/dev/data/pluto_s' [30.00 GB] inherit ACTIVE '/dev/system/root' [39.06 GB] inherit ACTIVE '/dev/system/swap' [9.75 GB] inherit [root at xen ~]# vgchange -an VolGroup00 0 logical volume(s) in volume group "VolGroup00" now active [root at xen ~]# kpartx -d /dev/loop4 [root at xen ~]# losetup -d /dev/loop4 [root at xen ~]# [root at xen ~]# xm create -c /etc/xen/hfserver2 And then it dies: Reading all physical volumes. This may take a while... Found volume group "VolGroup00" using metadata type lvm2 Activating logical volumes 2 logical volume(s) in volume group "VolGroup00" now active Creating root device. Mounting root filesystem. kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. Setting up other filesystems. Setting up new root fs no fstab.sys, mounting internal defaults Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys exec of init (/sbin/init) failed!!!: No such file or directory Kernel panic - not syncing: Attempted to kill init! [root at xen ~]# -- Kind Regards Rudi Ahlers CEO, SoftDux Hosting Web: http://www.SoftDux.com Office: 087 805 9573 Cell: 082 554 7532 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos/attachments/20090508/2a0d505c/attachment-0004.html>