Xen devel - May 2010 - can''t boot upstream kernel in PV guest with recent xen trees

After Xen summit, I tried to update a couple of my machines
to more recent bits.  Ever since then, I have been unable
to boot an upstream kernel in a PV guest!  I have tried
2.6.32, 2.6.33.1, and 2.6.34-rc5 kernels on tips of
xen-3.4-testing, xen-4.0-testing, and xen-unstable.
In each case, I am sure I am using a kernel config
file and the same process as a couple of months ago
when everything was working fine, probably on a pre-4.0
Xen.  I also found an old PV guest which I''m fairly sure
booted 2.6.30 fine and has sat unused for many months and
that failed as well.  However I am able to boot a
PV guest containing a pre-built RHEL6beta
2.6.32-19 kernel, so it is not ALL upstream kernels.

(Konrad pointed me offlist to a upstream-kernel-on-Xen
bug where the workaround was to de-config
CONFIG_DEBUG_RODATA... this made no difference.
And, yes, I''m following all the instructions in
http://wiki.xensource.com/xenwiki/2.6.18-to-2.6.31-and-higher) 

Every time, I get some kind of guest panic, not always
identical but often similar to below (and longer
version attached).  Every panic I''ve seen contains
xen_restore_fl_direct_end in the stacktrace, though
this could be a red herring.  And I''ve seen the
problem on two very different (though both Intel)
boxen.

Is it possible some bug fix was recently applied into
all three xen trees that creates or exposes a problem
in upstream kernels built with certain CONFIG_ options
booting in PV guests? (I''ve also attached the config
file used to build the 2.6.32 kernel that panic''ed.)

Or hopefully someone will find this familiar or
reproducible and can suggest an answer? I''m dead
in the water on one of my activities until I can
get an upstream kernel booting in a PV guest

Thanks for any help or suggestions!
Dan

<snip>
Freeing unused kernel memory: 468k freed
modprobe used greatest stack depth: 6120 bytes left
init used greatest stack depth: 5568 bytes left
Kernel panic - not syncing: Attempted to kill init!
Pid: 1, comm: init Not tainted 2.6.32 #2
Call Trace:
 [<ffffffff8103d87a>] ? panic+0x86/0x141
 [<ffffffff8100922a>] ? hypercall_page+0x22a/0x1001
 [<ffffffff8100e21f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8100dc15>] ? xen_force_evtchn_callback+0x9/0xa
 [<ffffffff8100e232>] ? check_events+0x12/0x20
 [<ffffffff8100e21f>] ? xen_restore_fl_direct_end+0x0/0x1
 [<ffffffff8100e1d9>] ? xen_irq_enable_direct_end+0x0/0x7
 [<ffffffff81046287>] ? exit_ptrace+0x8a/0xf3
 [<ffffffff81040301>] ? do_exit+0x72/0x68a
 [<ffffffff8104098c>] ? do_group_exit+0x73/0x9d
 [<ffffffff8135ed3c>] ? _spin_unlock_irqrestore+0xc/0xd
 [<ffffffff810409c8>] ? sys_exit_group+0x12/0x16
 [<ffffffff8135f2aa>] ? error_exit+0x2a/0x60
 [<ffffffff81010902>] ? system_call_fastpath+0x16/0x1b



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 12/05/2010 23:20, "Dan Magenheimer"
<dan.magenheimer@oracle.com> wrote:
> Is it possible some bug fix was recently applied into
> all three xen trees that creates or exposes a problem
> in upstream kernels built with certain CONFIG_ options
> booting in PV guests? (I''ve also attached the config
> file used to build the 2.6.32 kernel that panic''ed.)
> 
> Or hopefully someone will find this familiar or
> reproducible and can suggest an answer? I''m dead
> in the water on one of my activities until I can
> get an upstream kernel booting in a PV guest
Try 4.0.0 release. I''m sure people have been booting pv_ops kernels
with
that okay. I think this problem is somehow specific to your test machine.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> > Is it possible some bug fix was recently applied into
> > all three xen trees that creates or exposes a problem
> > in upstream kernels built with certain CONFIG_ options
> > booting in PV guests? (I''ve also attached the config
> > file used to build the 2.6.32 kernel that panic''ed.)
> >
> > Or hopefully someone will find this familiar or
> > reproducible and can suggest an answer? I''m dead
> > in the water on one of my activities until I can
> > get an upstream kernel booting in a PV guest
> 
> Try 4.0.0 release. I''m sure people have been booting pv_ops
kernels
> with
> that okay. I think this problem is somehow specific to your test
> machine.
Thanks for the reply!

After much teeth-gnashing and hair-pulling, I think I figured
out the problem -- though not yet a full solution.  When I
follow the exact build process and use the same config file
to boot an upstream kernel in a *64-bit* VM, it panics
every time (including on 4.0.0).  On a *32-bit* VM,
it boots fine every time.

I suspect that I have never (before last week) attempted
to build/boot an upstream kernel in a 64-bit VM.  I further
suspect that there is something in the config file
that I am using (possibly something annoyingly stupid)
that doesn''t cause a 64-bit *build* to fail but
does cause the 64-bit boot to fail.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel