CentOS - Aug 2015 - a peculiar LVM failure on CentOS 6 run as a VMware 5.5 guest

Hello listmates,

I have encountered a rather peculiar situation.

We have a Centos 6 VM (64 bit) running on a VMware vSphere 5.5 server. It
was running just fine until one day I decided to reboot it and it just
would not boot up. Effectively, dracut failed to initialize the LVM, much
like under the scenario described here:

http://unix.stackexchange.com/questions/207593/how-to-make-lvms-at-available-boot-kernel-panic-dracut-cannot-find-logical-vo#

Following that I booted it off a Centos 6.6 install DVD in the rescue mode.
It recognized the LVM partition without a glitch, booted it and seemed
quite happy with it. having examined it, I discovered that, even though the
root partition on the original VM was an LV it lacked the lvm2 package. I
installed it, but that did not help. So at the moment I am stuck.

Hence if you can offer help on this one - please do, it will certainly be
appreciated.

Cheers,

Boris.

Hello again,

Not sure I have this down 100% but it appears that the main LVM config file
( /etc/lvm/lvm.conf ) has been changed on August 13, 2015 on a number of
machines we have - all of them configured for automatic yum updates. So I
presume that could be a change that came as part an update - potentially
making the machine no longer bootable.

I am going to investigate this angle.

Boris.


On Tue, Aug 25, 2015 at 3:59 PM, Boris Epstein <borepstein at gmail.com>
wrote:
> Hello listmates,
>
> I have encountered a rather peculiar situation.
>
> We have a Centos 6 VM (64 bit) running on a VMware vSphere 5.5 server. It
> was running just fine until one day I decided to reboot it and it just
> would not boot up. Effectively, dracut failed to initialize the LVM, much
> like under the scenario described here:
>
>
>
http://unix.stackexchange.com/questions/207593/how-to-make-lvms-at-available-boot-kernel-panic-dracut-cannot-find-logical-vo#
>
> Following that I booted it off a Centos 6.6 install DVD in the rescue
> mode. It recognized the LVM partition without a glitch, booted it and
> seemed quite happy with it. having examined it, I discovered that, even
> though the root partition on the original VM was an LV it lacked the lvm2
> package. I installed it, but that did not help. So at the moment I am
stuck.
>
> Hence if you can offer help on this one - please do, it will certainly be
> appreciated.
>
> Cheers,
>
> Boris.
>

Hello Boris,

On Tue, 2015-08-25 at 15:59 -0400, Boris Epstein wrote:
> We have a Centos 6 VM (64 bit) running on a VMware vSphere 5.5 server. It
> was running just fine until one day I decided to reboot it and it just
> would not boot up. Effectively, dracut failed to initialize the LVM, much
> like under the scenario described here:
> 
>
http://unix.stackexchange.com/questions/207593/how-to-make-lvms-at-available-boot-kernel-panic-dracut-cannot-find-logical-vo#
> 
Perhaps this is related?
https://rhn.redhat.com/errata/RHBA-2015-1615.html

Prior to this update, using the lvm utility when the persistent cache
file was
outdated caused devices that were stored in the persistent cache to
unintentionally bypass logical volume manager (LVM) filters set in the
LVM
configuration. As a consequence, Red Hat Enterprise Virtualization hosts
in some
cases failed to start with an outdated cache file. This update fixes
LVM's
internal cache handling so that the filters are applied properly, and
the
described problem no longer occurs. (BZ#1248032)

Try updating LVM to the latest version and see if it helps.

Regards,
Leonard.

-- 
mount -t life -o ro /dev/dna /genetic/research

Hello Leonard,

Thank you very much for your response.

While it most likely is related the problem description provided at the
link seems a bit vague, and tips on how to resolve the issue seem to be
even more so.

I have done some research and in the process stumbled upon this:

http://linoxide.com/linux-how-to/fixing-broken-initrd-image-linux/

The discussion there circled around using mkinitrd (as opposed to dracut
which in my case did not help).

So, while mounted off a Centos DVD ISO I chroot'ed into the root of my
installation on the disk and then ran the following:

mkinitrd /boot/initramfs-2.6.32-573.3.1.el6.x86_64.img 2.6.32-573.3.1

I saved the original content of
/boot/initramfs-2.6.32-573.3.1.el6.x86_64.img too. So now the problem is
resolved, and it is reproducible - it boots with the one I generated but
not with the original one which was the one that got there as a result of
an update.

I have not been able to see what the issue was with the original image.

Cheers,

Boris.


On Wed, Aug 26, 2015 at 9:47 AM, Leonard den Ottolander <
leonard at den.ottolander.nl> wrote:
> Hello Boris,
>
> On Tue, 2015-08-25 at 15:59 -0400, Boris Epstein wrote:
> > We have a Centos 6 VM (64 bit) running on a VMware vSphere 5.5 server.
It
> > was running just fine until one day I decided to reboot it and it just
> > would not boot up. Effectively, dracut failed to initialize the LVM,
much
> > like under the scenario described here:
> >
> >
>
http://unix.stackexchange.com/questions/207593/how-to-make-lvms-at-available-boot-kernel-panic-dracut-cannot-find-logical-vo#
> >
>
> Perhaps this is related?
> https://rhn.redhat.com/errata/RHBA-2015-1615.html
>
> Prior to this update, using the lvm utility when the persistent cache
> file was
> outdated caused devices that were stored in the persistent cache to
> unintentionally bypass logical volume manager (LVM) filters set in the
> LVM
> configuration. As a consequence, Red Hat Enterprise Virtualization hosts
> in some
> cases failed to start with an outdated cache file. This update fixes
> LVM's
> internal cache handling so that the filters are applied properly, and
> the
> described problem no longer occurs. (BZ#1248032)
>
> Try updating LVM to the latest version and see if it helps.
>
> Regards,
> Leonard.
>
> --
> mount -t life -o ro /dev/dna /genetic/research
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>