Fedora xen - Mar 2007 - kernel OoPs after updating to 2.6.20-1.2933.fc6xen

Hi,

Today after updating kernel-xen to 2.6.20-1.2933.fc6xen on one of our host I
got kernel oops and machine rebooted, when it come back again i have no
other choice but to edit grub.conf and make the last working kernel ''
2.6.19-1.2911.6.5.fc6xen'' the default one which working fine.

DemU(s) working fine with with same kernel 2.6.20-1.2933.fc6xen, we use
loopback mounted fs for domU(s)

Here are the messages i found on screen before machine got rebooted (Note:
machine is remote to me thousands of miles away therefore i can''t check
its
boot screen :-S )

Now I am reluctant to update  kernel-xen on our other remote hosts, never
got into such issue on FC6 while updated kernel-xen. In help/suggestion will
be greatly appreciated.

------

[root@xxxxx ~]#
Message from syslogd@xxxxx at Mon Mar 26 20:17:02 2007 ...
xxxxx kernel: Oops: 0003 [#1]

Message from syslogd@xxxxx at Mon Mar 26 20:17:02 2007 ...
xxxxx kernel: SMP

Message from syslogd@xxxxx at Mon Mar 26 20:17:02 2007 ...
xxxxx kernel: CPU:    0

Message from syslogd@xxxxx at Mon Mar 26 20:17:02 2007 ...
xxxxx kernel: EIP:    0061:[<c0548ceb>]    Not tainted VLI

Message from syslogd@xxxxx at Mon Mar 26 20:17:02 2007 ...
xxxxx kernel: EFLAGS: 00010017   (2.6.20-1.2933.fc6xen #1)

Message from syslogd@xxxxx at Mon Mar 26 20:17:02 2007 ...
xxxxx kernel: EIP is at evtchn_do_upcall+0x55/0x97

Message from syslogd@xxxxx at Mon Mar 26 20:17:02 2007 ...
xxxxx kernel: eax: 00000010   ebx: 00000000   ecx: eaa35fe4   edx: ffffffec

Message from syslogd@xxxxx at Mon Mar 26 20:17:02 2007 ...
xxxxx kernel: esi: 00000001   edi: f5416000   ebp: fffffffe   esp: eaa35fc4

Message from syslogd@xxxxx at Mon Mar 26 20:17:02 2007 ...
xxxxx kernel: ds: 007b   es: 007b   ss: 0069

Message from syslogd@xxxxx at Mon Mar 26 20:17:02 2007 ...
xxxxx kernel: Process find (pid: 4000, ti=eaa35000 task=ed7b12f0
task.ti=eaa35000)

Message from syslogd@xxxxx at Mon Mar 26 20:17:03 2007 ...
xxxxx kernel: Stack: 00000000 00000000 00000003 eaa35fac eaa35fe4 eaa35000
c0404ff2 eaa35fe4

Message from syslogd@xxxxx at Mon Mar 26 20:17:03 2007 ...
xxxxx kernel:        b7ff0402 00000073 00000212 bf93c920 0000007b 00000000
00000000

Message from syslogd@xxxxx at Mon Mar 26 20:17:03 2007 ...
xxxxx kernel: Call Trace:

Message from syslogd@xxxxx at Mon Mar 26 20:17:03 2007 ...
xxxxx kernel:  ======================
Message from syslogd@xxxxx at Mon Mar 26 20:17:03 2007 ...
xxxxx kernel:  [<c0404ff2>] hypervisor_callback+0x46/0x50

Message from syslogd@xxxxx at Mon Mar 26 20:17:03 2007 ...
xxxxx kernel: Code: bd fe ff ff ff 88 d9 89 d8 c1 e0 05 d3 c5 89 04 24 eb 29
0f bc c0 03 04 24 8b 14 85 80 f0 6f c0 83 fa ff 74 12 8b 4c 24 1c f7 d2
<89>
51 28 89 c8 e8 16 db eb ff eb 05 e8 22 2e 00 00 8b 44 24 04

Message from syslogd@xxxxx at Mon Mar 26 20:17:03 2007 ...
xxxxx kernel: EIP: [<c0548ceb>] evtchn_do_upcall+0x55/0x97 SS:ESP
0069:eaa35fc4
Read from remote host xxxxx.xxxx.com: Connection reset by peer
Connection to xxxxx.xxxx.com closed.

On Tue, Mar 27, 2007 at 02:04:04AM +0500, Asrai khn
wrote:
> Hi,
> 
> Today after updating kernel-xen to 2.6.20-1.2933.fc6xen on one of our host
I
> got kernel oops and machine rebooted, when it come back again i have no
> other choice but to edit grub.conf and make the last working kernel
''
> 2.6.19-1.2911.6.5.fc6xen'' the default one which working fine.
> 
> DemU(s) working fine with with same kernel 2.6.20-1.2933.fc6xen, we use
> loopback mounted fs for domU(s)
> 
> Here are the messages i found on screen before machine got rebooted (Note:
> machine is remote to me thousands of miles away therefore i can''t
check its
> boot screen :-S )
> 
> Now I am reluctant to update  kernel-xen on our other remote hosts, never
> got into such issue on FC6 while updated kernel-xen. In help/suggestion
will
> be greatly appreciated.
> 
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=234008

Is it the same one?

-- Pasi

On 3/27/07, Pasi Kärkkäinen <pasik@iki.fi> wrote:
>
>
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=234008
>
> Is it the same one?

Yes, any fix for it?

Askar.

Still no update for this kernel bug

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=234008

I am wondering if they release another kernel-xen without fixing it, and we
go for update it will remove our last working kernel :(

And this is not something acceptable, so do you people want to to put
kernel-xen in yum "exclude" ?

Thanks. Askar

In general, my experience is that xen in fedora is unstable. It usually 
works pretty well, but it has its rough edges, and this is hardly the 
first time a kernel update has broken xen. In the past, on the FC5 line, 
it''s taken anywhere from a few days to a month or so for the fedora
guys
to fix an "unacceptable" problem.

That said, it''s not like you''re paying anybody money for
Fedora.... I''m
sure xen is much more stable in RHEL5.

BTW, you''re aware you can have yum stop erasing old kernels, right?

On Tue, 3 Apr 2007, Asrai khn wrote:
> And this is not something acceptable, so do you people want to to put
> kernel-xen in yum "exclude" ?

Hi Ben,

On 4/3/07, Ben <bench@silentmedia.com> wrote:
>
> In general, my experience is that xen in fedora is unstable. It usually
> works pretty well, but it has its rough edges, and this is hardly the
> first time a kernel update has broken xen. In the past, on the FC5 line,
> it''s taken anywhere from a few days to a month or so for the
fedora guys
> to fix an "unacceptable" problem.
>
> That said, it''s not like you''re paying anybody money for
Fedora.... I''m
> sure xen is much more stable in RHEL5.
>
> BTW, you''re aware you can have yum stop erasing old kernels,
right?

Nope I duno how to stop yum removing old kernels :-S

Thanks. Askar

Then you''ll be interested in this conf file:
/etc/yum/pluginconf.d/installonlyn.conf

Note that while it''s easy to roll back to an earlier kernel (assuming
you
still have it installed) the same is not true for rolling back xen 
userspace code updates, which have been just as risky to upgrade as the 
kernel, in my experience. Just so you know. :)

On Tue, 3 Apr 2007, Asrai khn wrote:
> Hi Ben,
>
> On 4/3/07, Ben <bench@silentmedia.com> wrote:
>> 
>> In general, my experience is that xen in fedora is unstable. It usually
>> works pretty well, but it has its rough edges, and this is hardly the
>> first time a kernel update has broken xen. In the past, on the FC5
line,
>> it''s taken anywhere from a few days to a month or so for the
fedora guys
>> to fix an "unacceptable" problem.
>> 
>> That said, it''s not like you''re paying anybody money
for Fedora.... I''m
>> sure xen is much more stable in RHEL5.
>> 
>> BTW, you''re aware you can have yum stop erasing old kernels,
right?
>
>
> Nope I duno how to stop yum removing old kernels :-S
>
> Thanks. Askar
>

On Tue, 27 Mar 2007 01:13:51 +0300 Pasi wrote:
PK> On Tue, Mar 27, 2007 at 02:04:04AM +0500, Asrai khn wrote:
PK> > Today after updating kernel-xen to 2.6.20-1.2933.fc6xen on one of
our host I
PK> > got kernel oops and machine rebooted, when it come back again i have
no
PK> > other choice but to edit grub.conf and make the last working kernel
''
PK> > 2.6.19-1.2911.6.5.fc6xen'' the default one which working
fine.
PK> 
PK> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=234008
PK> 
PK> Is it the same one?

I''m seeing pretty much the same thing with FC5 2307 xen0 kernel. :-(

> On Tue, 27 Mar 2007 01:13:51 +0300 Pasi wrote:
> PK> On Tue, Mar 27, 2007 at 02:04:04AM +0500, Asrai khn wrote:
> PK> > Today after updating kernel-xen to 2.6.20-1.2933.fc6xen on one
of our host I
> PK> > got kernel oops and machine rebooted, when it come back again i
have no
> PK> > other choice but to edit grub.conf and make the last working
kernel ''
> PK> > 2.6.19-1.2911.6.5.fc6xen'' the default one which
working fine.
> PK>
> PK> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=234008
> PK>
> PK> Is it the same one?
>
> I''m seeing pretty much the same thing with FC5 2307 xen0 kernel.
:-(
>
Same thing here. It seems to fail if there''s any significant network
activity. I''ve
also dropped back to a previous kernel.  Don''t have the OOPS now but
I''ll post it
later. There was something about rwsem.c errors at the top of the OOPs.

-- 
* The Digital Hermit   http://www.digitalhermit.com
* Unix and Linux Solutions   kwan@digitalhermit.com

Anyone tried the latest kernel-xen 2.6.20-1.2944.fc6 , wondering it solved
the problem that we had with previous release.

Thanks. Askar

Asrai khn wrote:
> Anyone tried the latest kernel-xen 2.6.20-1.2944.fc6 , wondering it 
> solved the problem that we had with previous release.
> 
> Thanks. Askar
> 
...

this version fixed my bugs (one of this was an ops...)

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221854
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221864
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=233918

-- 
      ronald

On 4/14/07, Ronald Warsow <rwarsow@online.de>
wrote:
>
>
> this version fixed my bugs (one of this was an ops...)
>
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221854
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221864
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=233918


Hi Ronald,

Yep same issues here with the previous release, so its time to kick the
kernel-xen update process :)

Thanks. Askar

On 4/14/07, Ronald Warsow <rwarsow@online.de>
wrote:
>
> ..
>
> this version fixed my bugs (one of this was an ops...)
>
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221854
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221864
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=233918


But wait, we were facing

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=234008

Duno if its also fix this bug.

Thanks. Askar

Asrai khn wrote:
> 
> 
> On 4/14/07, *Ronald Warsow* <rwarsow@online.de 
> <mailto:rwarsow@online.de>> wrote:
> 
>     ..
> 
>     this version fixed my bugs (one of this was an ops...)
> 
>     https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221854
>     https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=221864
>     https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=233918
> 
> 
> 
> But wait, we were facing
> 
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=234008
> 
> Duno if its also fix this bug.
> 
then give it a try, report and if fixed close the bug

-- 
      ronald

> Anyone tried the latest kernel-xen 2.6.20-1.2944.fc6 , wondering it solved
> the problem that we had with previous release.
It is still broken in kernel-xen 2.6.20-1.2944.fc6.
The last stable release appears to be:  2.6.19-1.2911.6.5.fc6xen

On 4/15/07, s@senator.org <s@senator.org> wrote:
>
>
> It is still broken in kernel-xen 2.6.20-1.2944.fc6.
> The last stable release appears to be:  2.6.19-1.2911.6.5.fc6xen

Yes you are 100% correct 2nd buggy kernel-xen from fedora folks. Here I fill
a bug, however they will look into it and accept it as bug. :-S

http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=236474

Machine was randomly rebooting, after sometime once gain fall back to last
working kernel ''2.6.19-1.2911.6.5.fc6xen''

Thanks. Askar

On 4/15/07, Asrai khn <asraikhn@gmail.com> wrote:
>
>
>
> On 4/15/07, s@senator.org <s@senator.org> wrote:
> >
> >
> > It is still broken in kernel-xen 2.6.20-1.2944.fc6.
> > The last stable release appears to be:  2.6.19-1.2911.6.5.fc6xen
>
>
> Yes you are 100% correct 2nd buggy kernel-xen from fedora folks. Here I
> fill a bug, however they will look into it and accept it as bug. :-S
>
> http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=236474
>
> Machine was randomly rebooting, after sometime once gain fall back to last
> working kernel '' 2.6.19-1.2911.6.5.fc6xen''
>


So what choice we have left, last kernel that worked for us and still
working is 2.6.19-1.2911.6.5.fc6xen the last two kernel-xen updates just
broke working installations.

Do we put kernel-xen in our yum.conf ''exclude'' or role our own
kernel. I
love to do it via yum however look like in end we will left with no working
kernel-xen if we keep updating kernel-xen from fc6 repo.

I wonders if anyone "accepting" our host reboots/crashing or its just
unacceptable bugs?

Thanks.
Askar Ali

Okay here we have another update for kernel-xen '' kernel-xen.i686[
2.6.20-1.2948.fc6'' wondering anyone tried it?

Does it fix the reboot/crashing problem ?

Thanks. Askar