CentOS - Jul 2018 - Issues booting centos7 [dracut is failing to enable centos/root, centos/swap LVs]

Hello,

I'm having a strange problem booting a new centos7 installation. Below some
background on this. [I have attached the tech details at the bottom of this
message]

I started a new CentOS7 installation on a VM, so far all good, o/s boots
fine. Then I decided to increase VM disk size (initially was 10G) to 13G.
Powered off the VM, increased the vhd via the hypervisor, booted from
CentOS livecd, selected "recover my centos installation". Then I used
the
following sequence of commands to make the new vhd size "visible" to
the
o/s ...

- ran fdisk /dev/sda and deleted partition 2. (This is the LVM partition
where the o/s is stored. The first partition is /boot [xfs] partition *non
lvm*.)
- created a new partition with same starting sector as before deleting it,
but a different ending sector (to reflect to the increased space).
- set partition type to 8e (lvm), saved settings and exit.
- ran pvresize and lvresize to make new space visible to the o/s.All good,
I can see the space increase on 'lvdisplay centos/root'.
- ran 'mount /dev/mapper/centos-root /mnt/root' to temporarily mount the
o/s partition.
- ran 'xfs_growfs /mnt/root' to resize the o/s fs to the new size. It
was
successful and I could actually chroot to the o/s and verify new disk size.
- rebooted  and tried to boot from hdd this time. Grub menu shows up and
loads default kernel (3.10.0-862.9.1.el7.x86_64).
- after initial kernel boot up process, booting continues to initrd and
then it's where the problem starts...
- looks like dracut has issues locating/enabling /dev/mapper/centos-root
(lv) and as a result it cannot boot to the 'real' root fs (/).
- while in dracut shell, I execute the following command sequence:
      1. lvm lvchange -ay /dev/centos/root
      2. lvm lvchange -ay /dev/centos/swap
      3. ln -s /dev/mapper/centos-root /dev/root
      4. exit
...and the o/s boots fine...so looks like the pv,vg,lv is detected properly
while in initrd, but somehow dracut has difficulties enabling the root,swap
LVs ?

- While in o/s, I rebuild initrd by using: 'dracut -f -v -a lvm'
Note, I have to use '-a lvm' as for some reason, if don't, the lvm
utils
(lvm,lvm_scan) are not being included to the initrd, not sure why this
happens.
After rebooting, same thing happens, I have to manually boot the system via
dracut shell.

I'm a bit stuck at this point, any clues what I'm doing wrong in here ?

As I previously said, this is a new installation, so I could simply
reinstall the whole thing, but I would rather try to find out some answer
to why is this happening ... :-) [curiosity]

Some additional details:
=================I've attached /run/initramfs/rdsosreport.txt file
[generated while in
inird] here:
https://privatebin.net/?fdc4052c0c402884#gdB/QYR3IeR55SxUbjfrkZPQfJ7jMxiUxq5DWhTVbWc
o/s ver=CentOS Linux release 7.5.1804 (Core)

[root at localhost ~]# lsblk
NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda               8:0    0   13G  0 disk
??sda1            8:1    0  500M  0 part /boot
??sda2            8:2    0 12.5G  0 part
  ??centos-root 253:0    0 11.5G  0 lvm  /
  ??centos-swap 253:1    0    1G  0 lvm  [SWAP]

[root at localhost ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g'
/etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rd.shell
rhgb"
GRUB_DISABLE_RECOVERY="true"

Hi,
> Hello,
>
> I'm having a strange problem booting a new centos7 installation. Below
> some
> background on this. [I have attached the tech details at the bottom of
> this
> message]
>
> I started a new CentOS7 installation on a VM, so far all good, o/s boots
> fine. Then I decided to increase VM disk size (initially was 10G) to 13G.
> Powered off the VM, increased the vhd via the hypervisor, booted from
> CentOS livecd, selected "recover my centos installation". Then I
used the
> following sequence of commands to make the new vhd size "visible"
to the
> o/s ...
>
> - ran fdisk /dev/sda and deleted partition 2. (This is the LVM partition
> where the o/s is stored. The first partition is /boot [xfs] partition *non
> lvm*.)
> - created a new partition with same starting sector as before deleting it,
> but a different ending sector (to reflect to the increased space).
> - set partition type to 8e (lvm), saved settings and exit.
At this point I usually reboot and do the following steps while the system
is running. I don't say how you do it is wrong, it's just easier/faster
to
do things while the system is online if downtime matters.

I don't really know what causes your issues. The reason may be that I have
more and more problems understanding the whole boot process and all the
variants we used to have over more than two decades...

But, what happens if you let the kernel post install scripts do the work
or setting up the initrd things? If you reinstall the latest kernel, does
it change anything?

Regards,
Simon
> - ran pvresize and lvresize to make new space visible to the o/s.All good,
> I can see the space increase on 'lvdisplay centos/root'.
> - ran 'mount /dev/mapper/centos-root /mnt/root' to temporarily
mount the
> o/s partition.
> - ran 'xfs_growfs /mnt/root' to resize the o/s fs to the new size.
It was
> successful and I could actually chroot to the o/s and verify new disk
> size.
> - rebooted  and tried to boot from hdd this time. Grub menu shows up and
> loads default kernel (3.10.0-862.9.1.el7.x86_64).
> - after initial kernel boot up process, booting continues to initrd and
> then it's where the problem starts...
> - looks like dracut has issues locating/enabling /dev/mapper/centos-root
> (lv) and as a result it cannot boot to the 'real' root fs (/).
> - while in dracut shell, I execute the following command sequence:
>       1. lvm lvchange -ay /dev/centos/root
>       2. lvm lvchange -ay /dev/centos/swap
>       3. ln -s /dev/mapper/centos-root /dev/root
>       4. exit
> ...and the o/s boots fine...so looks like the pv,vg,lv is detected
> properly
> while in initrd, but somehow dracut has difficulties enabling the
> root,swap
> LVs ?
>
> - While in o/s, I rebuild initrd by using: 'dracut -f -v -a lvm'
> Note, I have to use '-a lvm' as for some reason, if don't, the
lvm utils
> (lvm,lvm_scan) are not being included to the initrd, not sure why this
> happens.
> After rebooting, same thing happens, I have to manually boot the system
> via
> dracut shell.
>
> I'm a bit stuck at this point, any clues what I'm doing wrong in
here ?
>
> As I previously said, this is a new installation, so I could simply
> reinstall the whole thing, but I would rather try to find out some answer
> to why is this happening ... :-) [curiosity]
>
> Some additional details:
> =================> I've attached /run/initramfs/rdsosreport.txt file
[generated while in
> inird] here:
>
https://privatebin.net/?fdc4052c0c402884#gdB/QYR3IeR55SxUbjfrkZPQfJ7jMxiUxq5DWhTVbWc>
> o/s ver=CentOS Linux release 7.5.1804 (Core)
>
> [root at localhost ~]# lsblk
> NAME            MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
> sda               8:0    0   13G  0 disk
> ??sda1            8:1    0  500M  0 part /boot
> ??sda2            8:2    0 12.5G  0 part
>   ??centos-root 253:0    0 11.5G  0 lvm  /
>   ??centos-swap 253:1    0    1G  0 lvm  [SWAP]
>
> [root at localhost ~]# cat /etc/default/grub
> GRUB_TIMEOUT=5
> GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g'
/etc/system-release)"
> GRUB_DEFAULT=saved
> GRUB_DISABLE_SUBMENU=true
> GRUB_TERMINAL_OUTPUT="console"
> GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap
rd.shell
> rhgb"
> GRUB_DISABLE_RECOVERY="true"
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>

>
> But, what happens if you let the kernel post install scripts do the work
> or setting up the initrd things?

Initially, I just rebooted from LiveCD and left Grub,kernel and dracut do
their job with the defaults, but unfortunately, boot process stops at
initrd (dracut) stage with the following error:

[  197.768159] localhost.localdomain dracut-initqueue[252]: Warning:
Could
> not boot.
> [  197.821409] localhost.localdomain systemd[1]: Received SIGRTMIN+20 from
> PID 254 (plymouthd).
> [  197.822328] localhost.localdomain dracut-initqueue[252]: Warning:
> /dev/centos/root does not exist
> [  197.823044] localhost.localdomain dracut-initqueue[252]: Warning:
> /dev/centos/swap does not exist
> [  197.846807] localhost.localdomain systemd[1]: Starting Dracut Emergency
> Shell...
> [  197.870983] localhost.localdomain systemd[1]: Received SIGRTMIN+21 from
> PID 254 (plymouthd).

Then, I'm dropped to dracut shell, where I boot the system by using the lvm
commands as described to my first post.
Once in O/S, I re-create initrd by using 'dracut -f -v -a lvm' command
and
reboot again the system.
But still I get the same error during boot ...

If you reinstall the latest kernel, does
> it change anything?
>
 This is actually the latest kernel (3.10.0-862.9.1.el7.x86_64) but I tried
to boot with the second (older) kernel
(vmlinuz-3.10.0-327.36.3.el7.x86_64), where I got exactly the same error
during boot.

Clearly, this is not a kernel issue, perhaps not even a dracut (initrd)
issue, since everything was working fine, before resizing the PV/LV. I
suspect that something went wrong during that step, but what? As I can
successfully boot the system manually via dracut, normally everything
should be ok? no?

What can prevent dracut from activating the 2 LVs (centos/root and
centos/swap) during initial boot phase ?

Thanks,
Yannis