CentOS virt - May 2017 - 4.9 kernel fails to boot because it didn't have the mpt3sas module

Howdy,

I hit a snag trying to install Xen4CentOS on a Supermicro based system
(X9DRD-7LN4F with the Broadcom/LSI 2308 chipset).  I spent a few hours on
this today, I'm posting this here in case it helps anyone else and saves
them the frustration I dealt with.

On this system I did a fresh install of CentOS 7, updated it, rebooted it,
then installed Xen.

The system was booting fine using the latest 3.10 kernel (3.10.0-514.16 at
the time of writing this).  But, when it tried to boot the kernel for Xen
(4.9.25-27) it failed to do so.

On first boot all it showed was a blinking cursor after grub attempted to
boot the 4.9 kernel, and it sat there indefinitely.

After some digging around I removed the "console=hvc0" setting
/etc/default/grub, ran grub2-mkconfig, and tried booting 4.9 again.  This
time it actually showed messages, leading up to this:


[  186.550326] dracut-initqueue[542]: Warning: dracut-iniqueue timeout -
starting timeout scripts
[  186.570380] dracut-initqueue[542]: Warning: dracut-iniqueue timeout -
starting timeout scripts
[  187.120156] dracut-initqueue[542]: Warning: dracut-iniqueue timeout -
starting timeout scripts
[  188.633714] dracut-initqueue[542]: Warning: dracut-iniqueue timeout -
starting timeout scripts
[  189.102305] dracut-initqueue[542]: Warning: dracut-iniqueue timeout -
starting timeout scripts
[...]
[  OK  ] Started dracut initqueue hook
[  OK  ] Reached target Remote File Systems (Pre).
[  OK  ] Reached target Remote File Systems.
[   ***] A start job is running for dev-disk...device (3min 35s / no limit)


I let this run for a while to see if anything changed (needed to turn my
attention elsewhere).  After a couple of hours it was still in the same
place.

This last line was confusing:


[   ***] A start job is running for dev-disk...device (3min 35s / no limit)


Had the console not truncated that line and instead just wrapped the full
message so I could see it I would have been a much happier person.

Turns out dracut was unable to mount the root file system.  So I went back
into the 3.10 kernel again to see if the mpt2sas or mpt3sas driver was in
its initramfs file... and it wasn't:


$ sudo lsinitrd -k 4.9.25-27.el7.x86_64 | grep mpt
-rw-r--r--   1 root     root            0 May 16 12:39 etc/fstab.empty
-rw-r--r--   1 root     root           22 Nov  5  2016
usr/lib/kbd/unimaps/empty.uni

For comparison:

$ sudo lsinitrd -k 3.10.0-514.16.1.el7.x86_64 | grep mpt
-rw-r--r--   1 root     root            0 May 16 04:37 etc/fstab.empty
-rw-r--r--   1 root     root           22 Nov  5  2016
usr/lib/kbd/unimaps/empty.uni
drwxr-xr-x   2 root     root            0 May 16 04:37
usr/lib/modules/3.10.0-514.16.1.el7.x86_64/kernel/drivers/scsi/mpt3sas
-rw-r--r--   1 root     root       379021 Apr 12 08:51
usr/lib/modules/3.10.0-514.16.1.el7.x86_64/kernel/drivers/scsi/mpt3sas/mpt2sas.ko


So I added it:


$ sudo dracut --force --add-drivers mpt3sas --kver=4.9.25-27.el7.x86_64
$ sudo lsinitrd -k 4.9.25-27.el7.x86_64 | grep mpt
Arguments: --force --add-drivers 'mpt3sas' --kver
'4.9.25-27.el7.x86_64'
-rw-r--r--   1 root     root            0 May 16 12:57 etc/fstab.empty
-rw-r--r--   1 root     root           22 Nov  5  2016
usr/lib/kbd/unimaps/empty.uni
drwxr-xr-x   2 root     root            0 May 16 12:57
usr/lib/modules/4.9.25-27.el7.x86_64/kernel/drivers/scsi/mpt3sas
-rwxr--r--   1 root     root       374152 May 16 12:57
usr/lib/modules/4.9.25-27.el7.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko


After this I was able to get the 4.9 kernel to boot and Xen is now working.

Jerry
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20170516/5f071271/attachment.html>

On 05/16/2017 09:04 PM, Jerry wrote:
<snip>
> 
> Turns out dracut was unable to mount the root file system.  So I went back
> into the 3.10 kernel again to see if the mpt2sas or mpt3sas driver was in
> its initramfs file... and it wasn't:
> 
> 
> $ sudo lsinitrd -k 4.9.25-27.el7.x86_64 | grep mpt
> -rw-r--r--   1 root     root            0 May 16 12:39 etc/fstab.empty
> -rw-r--r--   1 root     root           22 Nov  5  2016
> usr/lib/kbd/unimaps/empty.uni
> 
> For comparison:
> 
> $ sudo lsinitrd -k 3.10.0-514.16.1.el7.x86_64 | grep mpt
> -rw-r--r--   1 root     root            0 May 16 04:37 etc/fstab.empty
> -rw-r--r--   1 root     root           22 Nov  5  2016
> usr/lib/kbd/unimaps/empty.uni
> drwxr-xr-x   2 root     root            0 May 16 04:37
> usr/lib/modules/3.10.0-514.16.1.el7.x86_64/kernel/drivers/scsi/mpt3sas
> -rw-r--r--   1 root     root       379021 Apr 12 08:51
>
usr/lib/modules/3.10.0-514.16.1.el7.x86_64/kernel/drivers/scsi/mpt3sas/mpt2sas.ko
> 
> 
> So I added it:
> 
> 
> $ sudo dracut --force --add-drivers mpt3sas --kver=4.9.25-27.el7.x86_64
> $ sudo lsinitrd -k 4.9.25-27.el7.x86_64 | grep mpt
> Arguments: --force --add-drivers 'mpt3sas' --kver
'4.9.25-27.el7.x86_64'
> -rw-r--r--   1 root     root            0 May 16 12:57 etc/fstab.empty
> -rw-r--r--   1 root     root           22 Nov  5  2016
> usr/lib/kbd/unimaps/empty.uni
> drwxr-xr-x   2 root     root            0 May 16 12:57
> usr/lib/modules/4.9.25-27.el7.x86_64/kernel/drivers/scsi/mpt3sas
> -rwxr--r--   1 root     root       374152 May 16 12:57
> usr/lib/modules/4.9.25-27.el7.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko
> 
> 
> After this I was able to get the 4.9 kernel to boot and Xen is now working.
You should be able to make this change permanently from /etc/dracut.conf with
the add_drivers+= line, or build a generic initrd with hostonly="no" .

I don't know why it wasn't originally added.

--Sarah

On Wed, May 17, 2017 at 10:07 AM, Sarah Newman <srn at prgmr.com> wrote:
> On 05/16/2017 09:04 PM, Jerry wrote:
> <snip>
> >
> > Turns out dracut was unable to mount the root file system.  So I went
> back
> > into the 3.10 kernel again to see if the mpt2sas or mpt3sas driver was
in
> > its initramfs file... and it wasn't:
> >
> >
> > $ sudo lsinitrd -k 4.9.25-27.el7.x86_64 | grep mpt
> > -rw-r--r--   1 root     root            0 May 16 12:39 etc/fstab.empty
> > -rw-r--r--   1 root     root           22 Nov  5  2016
> > usr/lib/kbd/unimaps/empty.uni
> >
> > For comparison:
> >
> > $ sudo lsinitrd -k 3.10.0-514.16.1.el7.x86_64 | grep mpt
> > -rw-r--r--   1 root     root            0 May 16 04:37 etc/fstab.empty
> > -rw-r--r--   1 root     root           22 Nov  5  2016
> > usr/lib/kbd/unimaps/empty.uni
> > drwxr-xr-x   2 root     root            0 May 16 04:37
> > usr/lib/modules/3.10.0-514.16.1.el7.x86_64/kernel/drivers/scsi/mpt3sas
> > -rw-r--r--   1 root     root       379021 Apr 12 08:51
> > usr/lib/modules/3.10.0-514.16.1.el7.x86_64/kernel/drivers/
> scsi/mpt3sas/mpt2sas.ko
> >
> >
> > So I added it:
> >
> >
> > $ sudo dracut --force --add-drivers mpt3sas
--kver=4.9.25-27.el7.x86_64
> > $ sudo lsinitrd -k 4.9.25-27.el7.x86_64 | grep mpt
> > Arguments: --force --add-drivers 'mpt3sas' --kver
'4.9.25-27.el7.x86_64'
> > -rw-r--r--   1 root     root            0 May 16 12:57 etc/fstab.empty
> > -rw-r--r--   1 root     root           22 Nov  5  2016
> > usr/lib/kbd/unimaps/empty.uni
> > drwxr-xr-x   2 root     root            0 May 16 12:57
> > usr/lib/modules/4.9.25-27.el7.x86_64/kernel/drivers/scsi/mpt3sas
> > -rwxr--r--   1 root     root       374152 May 16 12:57
> > usr/lib/modules/4.9.25-27.el7.x86_64/kernel/drivers/scsi/
> mpt3sas/mpt3sas.ko
> >
> >
> > After this I was able to get the 4.9 kernel to boot and Xen is now
> working.
>
> You should be able to make this change permanently from /etc/dracut.conf
> with
> the add_drivers+= line, or build a generic initrd with
hostonly="no" .
>
Thank you for this.  Adding mpt2sas or mpt3sas to the drivers+ line
worked.  Running dracut generates a proper image now.

> I don't know why it wasn't originally added.
>
> --Sarah
>
For some reason each of the 5, 6 & 7 releases have issues with the mpt2sas
driver (there isn't enough swiotlb allocated by default for the driver to
work properly, and the units for that parameter changed between 5 and 6).
It's like this driver has bad luck or something.

Jerry
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20170517/0f97dfcd/attachment-0002.html>