CentOS virt - Dec 2015 - Missing module grub entry in xen-4.4.3-9 & boot issues

HI,

We've started to see several issues with the Xen releases. Going back to
basics I've used this guide
https://wiki.centos.org/HowTos/Xen/Xen4QuickStart

Once the install process is complete the grub.conf looks like this:

default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title CentOS (3.18.21-17.el6.x86_64)
        root (hd0,0)
        kernel /boot/xen.gz dom0_mem=1024M,max:1024M cpuinfo
com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all
        module /boot/vmlinuz-3.18.21-17.el6.x86_64 ro root=/dev/vda1
rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
title CentOS (2.6.32-573.12.1.el6.x86_64)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=/dev/vda1
rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
        initrd /boot/initramfs-2.6.32-573.12.1.el6.x86_64.img
title CentOS (2.6.32-358.el6.x86_64)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.32-358.el6.x86_64 ro
root=UUID=71b203ab-36ea-4aa8-9ba3-78e4109f0ca4 rd_NO_LUKS rd_NO_LVM
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto
KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
        initrd /boot/initramfs-2.6.32-358.el6.x86_64.img

As you can see the missing module line. If i manually run grub-bootxen.sh
nothing changes.

Manually adding the missing module entry:

module /boot/initramfs-3.18.21-17.el6.x86_64.img

Causes the following:

https://www.dropbox.com/s/ley5nj29ubwqogc/Screenshot%202015-12-20%2011.10.11.png?dl=0

Disabling APIC gives the following console output:

https://www.dropbox.com/s/a6zgioxl9xpg20y/Screenshot%202015-12-20%2011.15.29.png?dl=0

Now i know this is inside a KVM virtual machine, however we are starting to
see this issue on at least 7 standard dedicated servers.

[root at test ~]# rpm -qa | grep xen
xen-libs-4.4.3-9.el6.x86_64
xen-hypervisor-4.4.3-9.el6.x86_64
xen-4.4.3-9.el6.x86_64
centos-release-xen-7-11.el6.x86_64
xen-licenses-4.4.3-9.el6.x86_64
xen-runtime-4.4.3-9.el6.x86_64

[root at test ~]# cat /etc/redhat-release
CentOS release 6.7 (Final)
[root at test ~]#

[root at test ~]# cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 23
model name      : Intel(R) Core(TM)2 Quad CPU    Q9400  @ 2.66GHz
stepping        : 10
microcode       : 1
cpu MHz         : 2666.352
cache size      : 4096 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc
arch_perfmon unfair_spinlock pni ssse3 cx16 sse4_1 x2apic xsave hypervisor
lahf_lm
bogomips        : 5332.70

-- 
Regards,
Phill
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20151220/58c362cb/attachment-0002.html>

I've also had a few Xen systems grub entry issues. I'm not yet sure what
triggers the events, but I do think there *may* be a correlation to the
"/boot" part. I've found the Xen servers that have
"/boot" before any of
the the lines like "/boot/xen.gz" ended up missing complete grub
entries.
Not sure it's 100% true. Anyone else comment?

Thanks
PJ

On Sun, Dec 20, 2015 at 5:31 AM, Phill Bandelow <phill at onapp.com>
wrote:
> HI,
>
> We've started to see several issues with the Xen releases. Going back
to
> basics I've used this guide
> https://wiki.centos.org/HowTos/Xen/Xen4QuickStart
>
> Once the install process is complete the grub.conf looks like this:
>
> default=0
> timeout=5
> splashimage=(hd0,0)/boot/grub/splash.xpm.gz
> hiddenmenu
> title CentOS (3.18.21-17.el6.x86_64)
>         root (hd0,0)
>         kernel /boot/xen.gz dom0_mem=1024M,max:1024M cpuinfo
> com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all
>         module /boot/vmlinuz-3.18.21-17.el6.x86_64 ro root=/dev/vda1
> rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
> crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
> title CentOS (2.6.32-573.12.1.el6.x86_64)
>         root (hd0,0)
>         kernel /boot/vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=/dev/vda1
> rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
> crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
>         initrd /boot/initramfs-2.6.32-573.12.1.el6.x86_64.img
> title CentOS (2.6.32-358.el6.x86_64)
>         root (hd0,0)
>         kernel /boot/vmlinuz-2.6.32-358.el6.x86_64 ro
> root=UUID=71b203ab-36ea-4aa8-9ba3-78e4109f0ca4 rd_NO_LUKS rd_NO_LVM
> LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto
> KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
>         initrd /boot/initramfs-2.6.32-358.el6.x86_64.img
>
> As you can see the missing module line. If i manually run grub-bootxen.sh
> nothing changes.
>
> Manually adding the missing module entry:
>
> module /boot/initramfs-3.18.21-17.el6.x86_64.img
>
> Causes the following:
>
>
>
https://www.dropbox.com/s/ley5nj29ubwqogc/Screenshot%202015-12-20%2011.10.11.png?dl=0
>
> Disabling APIC gives the following console output:
>
>
>
https://www.dropbox.com/s/a6zgioxl9xpg20y/Screenshot%202015-12-20%2011.15.29.png?dl=0
>
> Now i know this is inside a KVM virtual machine, however we are starting
> to see this issue on at least 7 standard dedicated servers.
>
> [root at test ~]# rpm -qa | grep xen
> xen-libs-4.4.3-9.el6.x86_64
> xen-hypervisor-4.4.3-9.el6.x86_64
> xen-4.4.3-9.el6.x86_64
> centos-release-xen-7-11.el6.x86_64
> xen-licenses-4.4.3-9.el6.x86_64
> xen-runtime-4.4.3-9.el6.x86_64
>
> [root at test ~]# cat /etc/redhat-release
> CentOS release 6.7 (Final)
> [root at test ~]#
>
> [root at test ~]# cat /proc/cpuinfo
> processor       : 0
> vendor_id       : GenuineIntel
> cpu family      : 6
> model           : 23
> model name      : Intel(R) Core(TM)2 Quad CPU    Q9400  @ 2.66GHz
> stepping        : 10
> microcode       : 1
> cpu MHz         : 2666.352
> cache size      : 4096 KB
> physical id     : 0
> siblings        : 1
> core id         : 0
> cpu cores       : 1
> apicid          : 0
> initial apicid  : 0
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 13
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
> cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc
> arch_perfmon unfair_spinlock pni ssse3 cx16 sse4_1 x2apic xsave hypervisor
> lahf_lm
> bogomips        : 5332.70
>
> --
> Regards,
> Phill
>
> _______________________________________________
> CentOS-virt mailing list
> CentOS-virt at centos.org
> https://lists.centos.org/mailman/listinfo/centos-virt
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20151220/1b297409/attachment-0002.html>

On Sun, Dec 20, 2015 at 11:31 AM, Phill Bandelow <phill at onapp.com>
wrote:
> HI,
>
> We've started to see several issues with the Xen releases. Going back
to
> basics I've used this guide
> https://wiki.centos.org/HowTos/Xen/Xen4QuickStart
>
> Once the install process is complete the grub.conf looks like this:
>
> default=0
> timeout=5
> splashimage=(hd0,0)/boot/grub/splash.xpm.gz
> hiddenmenu
> title CentOS (3.18.21-17.el6.x86_64)
>         root (hd0,0)
>         kernel /boot/xen.gz dom0_mem=1024M,max:1024M cpuinfo
com1=115200,8n1
> console=com1,tty loglvl=all guest_loglvl=all
>         module /boot/vmlinuz-3.18.21-17.el6.x86_64 ro root=/dev/vda1
> rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
> crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
> title CentOS (2.6.32-573.12.1.el6.x86_64)
>         root (hd0,0)
>         kernel /boot/vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=/dev/vda1
> rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16
> crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
>         initrd /boot/initramfs-2.6.32-573.12.1.el6.x86_64.img
> title CentOS (2.6.32-358.el6.x86_64)
>         root (hd0,0)
>         kernel /boot/vmlinuz-2.6.32-358.el6.x86_64 ro
> root=UUID=71b203ab-36ea-4aa8-9ba3-78e4109f0ca4 rd_NO_LUKS rd_NO_LVM
> LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto
> KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
>         initrd /boot/initramfs-2.6.32-358.el6.x86_64.img
>
> As you can see the missing module line. If i manually run grub-bootxen.sh
> nothing changes.
Thanks for the detailed report.

The initrd thing is a bug in the upstream script that makes the grub
config; I haven't seen the behavior, so I haven't been able to track
down what the problem is, but other people have[1].

CentOS 7 uses grub2, which comes with its own grub-generation code
which is more reliable.

[1] marc.info/?i=<563670F4.9030701 at karlos.cz>
> Manually adding the missing module entry:
>
> module /boot/initramfs-3.18.21-17.el6.x86_64.img
>
> Causes the following:
>
>
https://www.dropbox.com/s/ley5nj29ubwqogc/Screenshot%202015-12-20%2011.10.11.png?dl=0
>
> Disabling APIC gives the following console output:
>
>
https://www.dropbox.com/s/a6zgioxl9xpg20y/Screenshot%202015-12-20%2011.15.29.png?dl=0
>
> Now i know this is inside a KVM virtual machine, however we are starting to
> see this issue on at least 7 standard dedicated servers.
You're starting to see that exact error message, you mean, about the
timer not connected to the IOAPIC?

In any case, it would be more helpful if you could manage to get a
serial console output from one of the physical boxes where you've got
a problem.  A lot of servers have built-in serial consoles you can
attach to over the network; otherwise, you can take a look at this
page:

http://wiki.xen.org/wiki/Xen_Serial_Console

Thanks,
 -George

On Sun, Dec 20, 2015 at 5:11 PM, PJ Welsh <pjwelsh at gmail.com>
wrote:
> I've also had a few Xen systems grub entry issues. I'm not yet sure
what
> triggers the events, but I do think there *may* be a correlation to the
> "/boot" part. I've found the Xen servers that have
"/boot" before any of the
> the lines like "/boot/xen.gz" ended up missing complete grub
entries. Not
> sure it's 100% true. Anyone else comment?
I can't quite figure out what you mean by having a "/boot" before
other lines.  Could you attach copies of "normal" vs "buggy"
grub.conf
files?

 -George