Xen devel - Apr 2006 - ksoftirq/0 eating all cpu time

Hi,

I am having serious trouble with the current xen unstable hg tree, and the
current linux hg tree.

My machine hangs at the moment. top shows ksoftirq/0 eating all cpu cycles in
dom0

What can I do, to get closer to the problem?

Many thanks in advance,

Johnny

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> I am having serious trouble with the current xen unstable hg 
> tree, and the current linux hg tree.
> 
> My machine hangs at the moment. top shows ksoftirq/0 eating 
> all cpu cycles in dom0
> 
> What can I do, to get closer to the problem?
What changeset number? 

How do you trigger the machine to get into this state? Can you isolate
whether its network or block device related? Does /proc/interrupts
indicate any kind of storm?

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Am Mittwoch, 12. April 2006 14:04 schrieb Ian Pratt:
> > I am having serious trouble with the current xen unstable hg 
> > tree, and the current linux hg tree.
> > 
> > My machine hangs at the moment. top shows ksoftirq/0 eating 
> > all cpu cycles in dom0
> > 
> > What can I do, to get closer to the problem?
> 
> What changeset number? 
kernel 2.6.16 20060410 smp
and xen unstable hg tree from 20060410

xm info says changeset: unavailable

Well, I found out that this might me a ipxripd problem. The IPX routing daemon
started by accident, and flooded my network.

But the other Problem I have with this machine is, that when I run a make -j3 in
dom0, the domUs are very slow, even if there is only one domU.

Any ideas?

Johnny
> 
> How do you trigger the machine to get into this state? Can you isolate
> whether its network or block device related? Does /proc/interrupts
> indicate any kind of storm?
> 
> Ian
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

> But the other Problem I have with this machine is, that when 
> I run a make -j3 in dom0, the domUs are very slow, even if 
> there is only one domU.
The default scheduling parameters are set assuming that the only thing
running in dom0 is the control tool stack and the physical device
drivers.
 
If you''re actually logged in and running stuff in dom0 then the
defaults
are terrible as they basically give dom0 much higher priority than
everything else.

Ian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Am Mittwoch, 12. April 2006 17:58 schrieben Sie:
> > But the other Problem I have with this machine is, that when 
> > I run a make -j3 in dom0, the domUs are very slow, even if 
> > there is only one domU.
> 
> The default scheduling parameters are set assuming that the only thing
> running in dom0 is the control tool stack and the physical device
> drivers.
>  
> If you''re actually logged in and running stuff in dom0 then the
defaults
> are terrible as they basically give dom0 much higher priority than
> everything else.
How can I change this?

Johnny
> 
> Ian
> 
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

On Wed, 2006-04-12 at 16:58 +0100, Ian Pratt wrote:
> The default scheduling parameters are set assuming that the only thing
> running in dom0 is the control tool stack and the physical device
> drivers.
>  
> If you''re actually logged in and running stuff in dom0 then the
defaults
> are terrible as they basically give dom0 much higher priority than
> everything else.
We may want to reconsider those defaults, as they can result in some
pathological cases even for the case where the dom0 is purely a service
OS.  There were results on the list a week or so ago from somebody
complaining about poor network performance: networking from a remote
host to a domU was dropping 99% or so of packets.  The problem was that
they were sending so much data that the dom0 was saturated, and the domU
never got a chance to see the incoming data; changing the scheduler
parameters to give the dom0 less priority actually vastly improved
performance.

This is a known problem for networking even on stock Linux without Xen,
as under extreme load you could in theory end up spending all your time
in network interrupts and starve the applications trying to consume the
data.  The NAPI networking code in Linux tries to deal with this by
enabling a polling mode precisely to avoid getting so bogged down. 
It''s
a similar case where you end up having to deprioritise your critical
handler in order to let the queues drain.

--Stephen



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

Am Mittwoch, 12. April 2006 18:22 schrieb Stephen C.
Tweedie:
> Hi,
> 
> On Wed, 2006-04-12 at 16:58 +0100, Ian Pratt wrote:
> 
> > The default scheduling parameters are set assuming that the only thing
> > running in dom0 is the control tool stack and the physical device
> > drivers.
> >  
> > If you''re actually logged in and running stuff in dom0 then
the defaults
> > are terrible as they basically give dom0 much higher priority than
> > everything else.
> 
> We may want to reconsider those defaults, as they can result in some
> pathological cases even for the case where the dom0 is purely a service
> OS.  There were results on the list a week or so ago from somebody
> complaining about poor network performance: networking from a remote
> host to a domU was dropping 99% or so of packets.  The problem was that
> they were sending so much data that the dom0 was saturated, and the domU
> never got a chance to see the incoming data; changing the scheduler
> parameters to give the dom0 less priority actually vastly improved
> performance.
What would be a better scheduler setting?
Could you point me to some docs?

I found only little documentation about xen and schedulers.

Many thanks in advance,

Johnny
> 
> This is a known problem for networking even on stock Linux without Xen,
> as under extreme load you could in theory end up spending all your time
> in network interrupts and starve the applications trying to consume the
> data.  The NAPI networking code in Linux tries to deal with this by
> enabling a polling mode precisely to avoid getting so bogged down. 
It''s
> a similar case where you end up having to deprioritise your critical
> handler in order to let the queues drain.
> 
> --Stephen
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hi,

On Thu, 2006-04-13 at 12:11 +0200, Hans-Christian Armingeon wrote:
> What would be a better scheduler setting?
> Could you point me to some docs?
Sure, there was a big thread on this recently on the xen-users list: see
the archives at

http://lists.xensource.com/archives/html/xen-users/2006-04/msg00143.html

for a pointer to possible scheduler settings.

Cheers,
 Stephen



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel