Xen devel - Nov 2009 - xen-unstable: Xen BUG at sched_credit.c:925

I''m seeing this BUG/crash when shutting down a domain.
I''ve only seen it when running a domain with tmem, though
I''m not clear on what the relationship would be.

I haven''t tested tmem with xen-unstable for a couple of
months now and I see five patches to sched_credit.c
since Aug 25... should I just try backing those out,
or any other ideas on what might be going on?  The
code around the BUG is not very intuitive to me
and it looks like that chunk of code hasn''t been
changed for years.

Thanks,
Dan

(XEN)    0000000000000002 ffff82c48025f080 ffff82c4802e7f00 ffff82c48011da20
(XEN)    ffff82c48025fa00 ffff82c4802e7f28 ffff82c48025fa00 ffff83005c2aa000
(XEN)    ffff82c4802e7f20 ffff82c48014a4c3 0000000000000000 ffff83007d2d0000
(XEN)    ffff82c4802e7db8 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 00000000c05f0ffc 00000000deadbeef 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    00000000deadbeef 0000000000000000 0000000000000001 0000000000000000
(XEN)    0000010000000000 00000000c04013a7 0000000000000061 0000000000000246
(XEN)    00000000c06a0fc0 0000000000000069 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffff83007d2d0000
(XEN) Xen call trace:
(XEN)    [<ffff82c4801175e6>] csched_acct+0x25a/0x3e7
(XEN)    [<ffff82c48011f9fa>] execute_timer+0x2e/0x4c
(XEN)    [<ffff82c48011fbd2>] timer_softirq_action+0x1ba/0x328
(XEN)    [<ffff82c48011da20>] do_softirq+0x6a/0x77
(XEN)    [<ffff82c48014a4c3>] idle_loop+0x7a/0x7c
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Xen BUG at sched_credit.c:925
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 13/11/2009 17:04, "Dan Magenheimer"
<dan.magenheimer@oracle.com> wrote:
> I''m seeing this BUG/crash when shutting down a domain.
> I''ve only seen it when running a domain with tmem, though
> I''m not clear on what the relationship would be.
> 
> I haven''t tested tmem with xen-unstable for a couple of
> months now and I see five patches to sched_credit.c
> since Aug 25... should I just try backing those out,
> or any other ideas on what might be going on?  The
> code around the BUG is not very intuitive to me
> and it looks like that chunk of code hasn''t been
> changed for years.
Worth a try, but I don''t think any of those changesets are likely to
cause
this bug either. It''s not a BUG() I''ve seen or had reported
triggering
before.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

(Note this is clean xen-unstable with no changes.)
Well after compiling with debug=y, the following BUG
is triggered during a different activity.

(XEN) Xen call trace:
(XEN)    [<ffff82c480117a73>] csched_schedule+0xa8/0x557
(XEN)    [<ffff82c48011c647>] schedule+0xd6/0x46a
(XEN)    [<ffff82c48011da20>] do_softirq+0x6a/0x77
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 1:
(XEN) Xen BUG at sched_credit.c:326
(XEN) ****************************************

Looks like some memory corruption or something... maybe
a free''d pointer that is used as if never freed.

While this could be a tmem bug, tmem is so filled with
asserts when debug=y is enabled that it is less likely
(though certainly still possible).  Very likely, tmem
is churning otherwise unused memory a lot more than
normal, surfacing a (non-tmem) latent bent pointer bug.

This of course will be a real bear to track down,
especially in code with which I am completely unfamiliar.
Any ideas?

Thanks,
Dan
> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Friday, November 13, 2009 10:16 AM
> To: Dan Magenheimer; Xen-Devel (E-mail)
> Subject: Re: [Xen-devel] xen-unstable: Xen BUG at sched_credit.c:925
> 
> 
> On 13/11/2009 17:04, "Dan Magenheimer" 
> <dan.magenheimer@oracle.com> wrote:
> 
> > I''m seeing this BUG/crash when shutting down a domain.
> > I''ve only seen it when running a domain with tmem, though
> > I''m not clear on what the relationship would be.
> > 
> > I haven''t tested tmem with xen-unstable for a couple of
> > months now and I see five patches to sched_credit.c
> > since Aug 25... should I just try backing those out,
> > or any other ideas on what might be going on?  The
> > code around the BUG is not very intuitive to me
> > and it looks like that chunk of code hasn''t been
> > changed for years.
> 
> Worth a try, but I don''t think any of those changesets are 
> likely to cause
> this bug either. It''s not a BUG() I''ve seen or had
reported triggering
> before.
> 
>  -- Keir
> 
> 
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 13/11/2009 18:29, "Dan Magenheimer"
<dan.magenheimer@oracle.com> wrote:
> Looks like some memory corruption or something... maybe
> a free''d pointer that is used as if never freed.
> 
> While this could be a tmem bug, tmem is so filled with
> asserts when debug=y is enabled that it is less likely
> (though certainly still possible).  Very likely, tmem
> is churning otherwise unused memory a lot more than
> normal, surfacing a (non-tmem) latent bent pointer bug.
>
> This of course will be a real bear to track down,
> especially in code with which I am completely unfamiliar.
> Any ideas?
Not really. Generic memory corruption bug needing good old-fashioned
detective work.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Building Xen via xen-3.4.1-5.fc12.src.rpm requires e2fsprogs-devel to be
installed on Fedoras, then everything works fine.
In  case of setup Xen 3.4.2 Dom0 with 2.6.31.5 pvops kernel on top of Ubuntu
9.10 Server i was unable load via pygrub F12 PV DomU with ext4 for /boot.
Am i missing something ?

Boris.






      

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Sun, Nov 15, 2009 at 12:05:56PM -0800, Boris Derzhavets
wrote:
>    Building Xen via xen-3.4.1-5.fc12.src.rpm requires e2fsprogs-devel to be
>    installed on Fedoras, then everything works fine.
>
Yes, having e2fsprogs-devel installed when building Xen gives pygrub
ext4 support.
>    In  case of setup Xen 3.4.2 Dom0 with 2.6.31.5 pvops kernel on top of
>    Ubuntu 9.10 Server i was unable load via pygrub F12 PV DomU with ext4
for
>    /boot.
>    Am i missing something ?
> 
Do you have e2fsprogs-devel (whatever the matching headers package is in
ubuntu) installed?

-- Pasi


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

I''ve tried :-  
    e2fslibs-dev - (ext2/ext3/ext4 file system libraries-headers and static
libraries)
Installed package and rebuilt&reinstalled Xen 3.4.2.
No luck.

Boris.

--- On Mon, 11/16/09, Pasi Kärkkäinen <pasik@iki.fi> wrote:

From: Pasi Kärkkäinen <pasik@iki.fi>
Subject: Re: [Xen-devel] Does pygrub in Xen 3.4.2 support loading PV DomUs with
/boot on ext4fs ?
To: "Boris Derzhavets" <bderzhavets@yahoo.com>
Cc: "Xen-Devel" <xen-devel@lists.xensource.com>, "Keir
Fraser" <keir.fraser@eu.citrix.com>
Date: Monday, November 16, 2009, 2:16 AM

On Sun, Nov 15, 2009 at 12:05:56PM -0800, Boris Derzhavets
wrote:
>    Building Xen via xen-3.4.1-5.fc12.src.rpm requires e2fsprogs-devel to be
>    installed on Fedoras, then everything works fine.
>
Yes, having e2fsprogs-devel installed when building Xen gives pygrub
ext4 support.
>    In  case of setup Xen 3.4.2 Dom0 with 2.6.31.5 pvops kernel on top of
>    Ubuntu 9.10 Server i was unable load via pygrub F12 PV DomU with ext4
for
>    /boot.
>    Am i missing something ?
> 
Do you have e2fsprogs-devel (whatever the matching headers package is in
ubuntu) installed?

-- Pasi




      

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel