Xen devel - Mar 2013 - RHEL (CentOS, Oracle Liux) kernel 2.6.32-358 does not start at XCP

Hi, everyone,
Two days ago I have filed a bug to Red Hat  
https://bugzilla.redhat.com/show_bug.cgi?id=918239
The problem is, that new RHEL-6.4's kernel 2.6.32-358 does not start at HVM
on Xen Cloud Platform 1.6 (maybe, also affected XenServer 6.1)
I have tested kernels -358 from CentOS, Oracle and -356 from Red Hat both i686
and x86_64 at 2 different server hardware
The sympthoms: kernel does not boot at all, does not produce any log; previous
(all 2.6.32-279 and 2.6.32-220), and Oracle's kernel-uek work perfectly.

Andrew Jones from Red Hat advised me to ask, does anybody else have this issue?

Best regards,
-- 
Constantin Dunayev,
Kharkov, UA 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev
wrote:
>    Hi, everyone,
>    Two days ago I have filed a bug to Red Hat
>     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
>
That bug is marked as private, so it''s not visible to other people.
>    The problem is, that new RHEL-6.4''s kernel 2.6.32-358 does not
start at
>    HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer 6.1)
>    I have tested kernels -358 from CentOS, Oracle and -356 from Red Hat
both
>    i686 and x86_64 at 2 different server hardware
>    The sympthoms: kernel does not boot at all, does not produce any log;
>    previous (all 2.6.32-279 and 2.6.32-220), and Oracle''s
kernel-uek work
>    perfectly.
> 
>    Andrew Jones from Red Hat advised me to ask, does anybody else have this
>    issue?
> 
Did you try booting the domU kernel with "earlyprintk=xen
console=hvc0" cmdline options?

-- Pasi

On 07/03/13 14:13, Pasi Kärkkäinen wrote:
> On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev wrote:
>>    Hi, everyone,
>>    Two days ago I have filed a bug to Red Hat
>>     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
>>
> That bug is marked as private, so it''s not visible to other
people.
>
>>    The problem is, that new RHEL-6.4''s kernel 2.6.32-358 does
not start at
>>    HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer 6.1)
>>    I have tested kernels -358 from CentOS, Oracle and -356 from Red Hat
both
>>    i686 and x86_64 at 2 different server hardware
>>    The sympthoms: kernel does not boot at all, does not produce any
log;
>>    previous (all 2.6.32-279 and 2.6.32-220), and Oracle''s
kernel-uek work
>>    perfectly.
>>
>>    Andrew Jones from Red Hat advised me to ask, does anybody else have
this
>>    issue?
>>
> Did you try booting the domU kernel with "earlyprintk=xen
console=hvc0" cmdline options?
>
> -- Pasi
Or alternatively, start with pause-on-crash, and use xenctx with the
appropriate symbol table.

~Andrew (who also unable to view the bug ticket)
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

On Thu, Mar 07, 2013 at 02:39:24PM +0000, Andrew Cooper
wrote:
> On 07/03/13 14:13, Pasi Kärkkäinen wrote:
> > On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev wrote:
> >>    Hi, everyone,
> >>    Two days ago I have filed a bug to Red Hat
> >>     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
> >>
> > That bug is marked as private, so it''s not visible to other
people.
> >
> >>    The problem is, that new RHEL-6.4''s kernel 2.6.32-358
does not start at
> >>    HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer
6.1)
> >>    I have tested kernels -358 from CentOS, Oracle and -356 from
Red Hat both
> >>    i686 and x86_64 at 2 different server hardware
> >>    The sympthoms: kernel does not boot at all, does not produce
any log;
> >>    previous (all 2.6.32-279 and 2.6.32-220), and Oracle''s
kernel-uek work
> >>    perfectly.
> >>
> >>    Andrew Jones from Red Hat advised me to ask, does anybody else
have this
> >>    issue?
> >>
> > Did you try booting the domU kernel with "earlyprintk=xen
console=hvc0" cmdline options?
> >
> > -- Pasi
> 
> Or alternatively, start with pause-on-crash, and use xenctx with the
> appropriate symbol table.
> 
> ~Andrew (who also unable to view the bug ticket)
> 
Yep, instructions for using xenctx here:
http://wiki.xen.org/wiki/Xen_Common_Problems#My_Xen_PV_guest_kernel_crashes.2C_how_can_I_debug_it_and_get_a_stack.2Fcall_trace.3F

-- Pasi

----- Original Message -----
> On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev wrote:
> >    Hi, everyone,
> >    Two days ago I have filed a bug to Red Hat
> >     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
> >
> 
> That bug is marked as private, so it''s not visible to other
people.
> 
Strange. There''s no reason for the BZ to be private. It was opened
externally, and there''s nothing in the history indicating its
visibility
was changed. I also don''t see any way to make it any more public than
it already is (of course it doesn''t help that the bugzilla interface
keeps changing, so maybe there''s a new series of clicks that
aren''t
obvious to me...)
> >    The problem is, that new RHEL-6.4''s kernel 2.6.32-358 does
not
> >    start at
> >    HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer
> >    6.1)
> >    I have tested kernels -358 from CentOS, Oracle and -356 from Red
> >    Hat both
> >    i686 and x86_64 at 2 different server hardware
> >    The sympthoms: kernel does not boot at all, does not produce any
> >    log;
> >    previous (all 2.6.32-279 and 2.6.32-220), and Oracle''s
> >    kernel-uek work
> >    perfectly.
> > 
> >    Andrew Jones from Red Hat advised me to ask, does anybody else
> >    have this
> >    issue?
> > 
> 
> Did you try booting the domU kernel with "earlyprintk=xen
> console=hvc0" cmdline options?
> 
> -- Pasi
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

----- Original Message -----
> 
> 
> ----- Original Message -----
> > On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev wrote:
> > >    Hi, everyone,
> > >    Two days ago I have filed a bug to Red Hat
> > >     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
> > >
> > 
> > That bug is marked as private, so it''s not visible to other
people.
> > 
> 
> Strange. There''s no reason for the BZ to be private. It was opened
> externally, and there''s nothing in the history indicating its
> visibility
> was changed. I also don''t see any way to make it any more public
than
> it already is (of course it doesn''t help that the bugzilla
interface
> keeps changing, so maybe there''s a new series of clicks that
aren''t
> obvious to me...)
Ok, so there were a serious of clicks (which are obvious now in hind-
sight). I''ve opened the BZ to the public.
> 
> > >    The problem is, that new RHEL-6.4''s kernel 2.6.32-358
does not
> > >    start at
> > >    HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer
> > >    6.1)
> > >    I have tested kernels -358 from CentOS, Oracle and -356 from
> > >    Red
> > >    Hat both
> > >    i686 and x86_64 at 2 different server hardware
> > >    The sympthoms: kernel does not boot at all, does not produce
> > >    any
> > >    log;
> > >    previous (all 2.6.32-279 and 2.6.32-220), and
Oracle''s
> > >    kernel-uek work
> > >    perfectly.
> > > 
> > >    Andrew Jones from Red Hat advised me to ask, does anybody else
> > >    have this
> > >    issue?
> > > 
> > 
> > Did you try booting the domU kernel with "earlyprintk=xen
> > console=hvc0" cmdline options?
> > 
> > -- Pasi
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> > 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

On Fri, Mar 08, 2013 at 07:28:31AM -0500, Andrew Jones
wrote:
> 
> 
> ----- Original Message -----
> > 
> > 
> > ----- Original Message -----
> > > On Thu, Mar 07, 2013 at 02:26:10PM +0400, Constantin Dunayev
wrote:
> > > >    Hi, everyone,
> > > >    Two days ago I have filed a bug to Red Hat
> > > >     [1]https://bugzilla.redhat.com/show_bug.cgi?id=918239
> > > >
> > > 
> > > That bug is marked as private, so it''s not visible to
other people.
> > > 
> > 
> > Strange. There''s no reason for the BZ to be private. It was
opened
> > externally, and there''s nothing in the history indicating its
> > visibility
> > was changed. I also don''t see any way to make it any more
public than
> > it already is (of course it doesn''t help that the bugzilla
interface
> > keeps changing, so maybe there''s a new series of clicks that
aren''t
> > obvious to me...)
> 
> Ok, so there were a serious of clicks (which are obvious now in hind-
> sight). I''ve opened the BZ to the public.
> 
Thanks!

-- Pasi

On 07/03/13 10:26, Constantin Dunayev wrote:
> Hi, everyone,
> Two days ago I have filed a bug to Red Hat
>  https://bugzilla.redhat.com/show_bug.cgi?id=918239
> The problem is, that new RHEL-6.4''s kernel 2.6.32-358 does not
start
> at HVM on Xen Cloud Platform 1.6 (maybe, also affected XenServer 6.1)
> I have tested kernels -358 from CentOS, Oracle and -356 from Red Hat
> both i686 and x86_64 at 2 different server hardware
> The sympthoms: kernel does not boot at all, does not produce any log;
> previous (all 2.6.32-279 and 2.6.32-220), and Oracle''s kernel-uek
work
> perfectly.
>
> Andrew Jones from Red Hat advised me to ask, does anybody else have
> this issue?
>
> Best regards,
> -- 
> Constantin Dunayev,
> Kharkov, UA  
To follow up on this issue.  My findings are presented in full on the BZ
ticket, but in summary:

RHEL 6.4 appears to be somewhat viridian aware

The final message on the console before the VCPU spins is

"Switching to clocksource hyperv_clocksource"

and is presumably waiting for an interrupt from a clocksource we do not
implement.

Disabling viridian for the domain appears to work around the issue:

For XenServer,XCP use:

xe vm-param-set vm=$VM platform:viridian=false

For xl/xm, ensure viridian=false in the VM configuration file

~Andrew