thr3ads.net - Xen users - [Xen-users] dom0 freezes under high IO load

If this information is useful, please help other people find it:
Share via:

Tom Mornini

2007-Mar-03 03:24 UTC

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Hello Daniel.

We''ve had similar problems, but have received very little feedback  
from our machines.

Our setup is also not entirely similar to yours. :-)

Could you try something out?

Set a cron job to run every 1 minute:

   cat /proc/slabinfo >> /root/slabinfo.txt

When we do this, our problem gets *much* worse. I''d love to know if  
these are similar problems.

-- 
-- Tom Mornini, CTO
-- Engine Yard, Ruby on Rails Hosting
-- Reliability, Ease of Use, Scalability
-- (866) 518-YARD (9273)

On Mar 2, 2007, at 7:27 PM, Daniel Mealha Cabrita wrote:
>
> hi there,
>
>
> 	Does anyone have have suggestions on how to proceed in this case?
>
>
> 	I''ve experiencing dom0 (xen 3.0.3, xen-3.0.4 and 3.0.4-testing)  
> lockups under
> heavy disk load (testing under dom0 directly).
> The hardware is a HP ML150 G2 with a HP 4ch SATA fakeraid (OEM  
> Adaptec 1420SA,
> sata_mv driver).
> 	The machine does not respond to network, keyboard not anything  
> noticeable
> when it happens.
>
> 	I''ve tried passing a number of parameters to kernel but no  
> success. The ones
> below even make things worse, causing a CPU0 soft lockup during boot:
> kernel = (hd0,0)/xen-3.0.4 dom0_mem=384M acpi=off noapic nolapic
>
> 	The disks runs in Linux kernel RAID5. High load to/from an  
> individual disk
> (the max an individual SATA HD can handle) does not cause any problem.
>
> 	Also, I''ve noticed that just after booting the machine does not  
> respond to
> pings nor anything from the network. If I locally ping from that to  
> another
> host, the network starts working. OR, if I wait enough time (several
> minutes), the machine''s network starts to respond normally.
>
> 	I''ve got no soft lockups logged in /var/log/messages. Nor
anything
> strange
> enough to call my attention.
>
> 	The problem does not happen with a non-Xen kernel.
> 	The machine firmware (BIOS if you like) is updated to the latest  
> version.
> 	Disabling all the non-essential hardware (USB, serial/parallel  
> ports, IDE
> ports, powersaving etc) makes no difference.
>
> 	When not under high disk load the machine seems stable, with  
> several domU VMs
> running happily under it.
>
> -- 
>  Daniel Mealha Cabrita
>  Divisao de Suporte Tecnico
>  AINFO / Reitoria / UTFPR
>  http://www.utfpr.edu.br
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel Mealha Cabrita

2007-Mar-03 03:27 UTC

head link

[Xen-users] dom0 freezes under high IO load - HP ML150 G2

hi there,


	Does anyone have have suggestions on how to proceed in this case?


	I''ve experiencing dom0 (xen 3.0.3, xen-3.0.4 and 3.0.4-testing)
lockups under
heavy disk load (testing under dom0 directly).
The hardware is a HP ML150 G2 with a HP 4ch SATA fakeraid (OEM Adaptec 1420SA, 
sata_mv driver).
	The machine does not respond to network, keyboard not anything noticeable 
when it happens.

	I''ve tried passing a number of parameters to kernel but no success.
The ones
below even make things worse, causing a CPU0 soft lockup during boot:
kernel = (hd0,0)/xen-3.0.4 dom0_mem=384M acpi=off noapic nolapic

	The disks runs in Linux kernel RAID5. High load to/from an individual disk 
(the max an individual SATA HD can handle) does not cause any problem.

	Also, I''ve noticed that just after booting the machine does not
respond to
pings nor anything from the network. If I locally ping from that to another 
host, the network starts working. OR, if I wait enough time (several 
minutes), the machine''s network starts to respond normally.

	I''ve got no soft lockups logged in /var/log/messages. Nor anything
strange
enough to call my attention.

	The problem does not happen with a non-Xen kernel.
	The machine firmware (BIOS if you like) is updated to the latest version.
	Disabling all the non-essential hardware (USB, serial/parallel ports, IDE 
ports, powersaving etc) makes no difference.

	When not under high disk load the machine seems stable, with several domU VMs 
running happily under it.

-- 
 Daniel Mealha Cabrita
 Divisao de Suporte Tecnico
 AINFO / Reitoria / UTFPR
 http://www.utfpr.edu.br

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

TMC

2007-Mar-03 05:58 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

I am not sure if testing like this in Dom0 is a good thing.  Dom0 is
"special" and should not run any precesses that hit disk hard. thats
the job for DomU

Regards
TMC

On 03/03/07, Tom Mornini <tmornini@engineyard.com>
wrote:> Hello Daniel.
>
> We''ve had similar problems, but have received very little feedback
> from our machines.
>
> Our setup is also not entirely similar to yours. :-)
>
> Could you try something out?
>
> Set a cron job to run every 1 minute:
>
>    cat /proc/slabinfo >> /root/slabinfo.txt
>
> When we do this, our problem gets *much* worse. I''d love to know
if
> these are similar problems.
>
> --
> -- Tom Mornini, CTO
> -- Engine Yard, Ruby on Rails Hosting
> -- Reliability, Ease of Use, Scalability
> -- (866) 518-YARD (9273)
>
> On Mar 2, 2007, at 7:27 PM, Daniel Mealha Cabrita wrote:
>
> >
> > hi there,
> >
> >
> >       Does anyone have have suggestions on how to proceed in this
case?
> >
> >
> >       I''ve experiencing dom0 (xen 3.0.3, xen-3.0.4 and
3.0.4-testing)
> > lockups under
> > heavy disk load (testing under dom0 directly).
> > The hardware is a HP ML150 G2 with a HP 4ch SATA fakeraid (OEM
> > Adaptec 1420SA,
> > sata_mv driver).
> >       The machine does not respond to network, keyboard not anything
> > noticeable
> > when it happens.
> >
> >       I''ve tried passing a number of parameters to kernel but
no
> > success. The ones
> > below even make things worse, causing a CPU0 soft lockup during boot:
> > kernel = (hd0,0)/xen-3.0.4 dom0_mem=384M acpi=off noapic nolapic
> >
> >       The disks runs in Linux kernel RAID5. High load to/from an
> > individual disk
> > (the max an individual SATA HD can handle) does not cause any problem.
> >
> >       Also, I''ve noticed that just after booting the machine
does not
> > respond to
> > pings nor anything from the network. If I locally ping from that to
> > another
> > host, the network starts working. OR, if I wait enough time (several
> > minutes), the machine''s network starts to respond normally.
> >
> >       I''ve got no soft lockups logged in /var/log/messages.
Nor anything
> > strange
> > enough to call my attention.
> >
> >       The problem does not happen with a non-Xen kernel.
> >       The machine firmware (BIOS if you like) is updated to the latest
> > version.
> >       Disabling all the non-essential hardware (USB, serial/parallel
> > ports, IDE
> > ports, powersaving etc) makes no difference.
> >
> >       When not under high disk load the machine seems stable, with
> > several domU VMs
> > running happily under it.
> >
> > --
> >  Daniel Mealha Cabrita
> >  Divisao de Suporte Tecnico
> >  AINFO / Reitoria / UTFPR
> >  http://www.utfpr.edu.br
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xensource.com
> > http://lists.xensource.com/xen-users
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

-- 
GPG key fingerprint: 3883 B308 8256 2246 D3ED  A1FF 3A1D 0EAD 41C4 C2F0
GPG public key availabe on pgp.mit .edu keyserver

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tom Mornini

2007-Mar-03 06:12 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

You don''t believe that *testing* is a good thing?

I''m pretty sure the Xen documentation points out that Dom0 is not  
particularly special, except that it is privileged to manipulate DomUs.

I''d love to hear other''s opinions on this topic. Should Dom0
be
entirely free of disk I/O?

Now, I should make it clear, I''m a big supporter of Dom0 doing  
essentially nothing. Yet, I''m also a supporter of not having  
difficulty sleeping at night, afraid that Dom0 might write a block or  
two to its own disks...

I have a job that runs every five minutes to grab CPU utilization,  
and writes that to disk. That job doesn''t cause destabilization. My  
problem seems to be related to kernel SLAB corruption, which is why I  
mentioned this as something to test, and made it clear that in my  
case, it made the machine unstable.

-- 
-- Tom Mornini, CTO
-- Engine Yard, Ruby on Rails Hosting
-- Reliability, Ease of Use, Scalability
-- (866) 518-YARD (9273)

On Mar 2, 2007, at 9:58 PM, TMC wrote:
> I am not sure if testing like this in Dom0 is a good thing.  Dom0 is
> "special" and should not run any precesses that hit disk hard.
thats
> the job for DomU
>
> Regards
> TMC
>
> On 03/03/07, Tom Mornini <tmornini@engineyard.com> wrote:
>> Hello Daniel.
>>
>> We''ve had similar problems, but have received very little
feedback
>> from our machines.
>>
>> Our setup is also not entirely similar to yours. :-)
>>
>> Could you try something out?
>>
>> Set a cron job to run every 1 minute:
>>
>>    cat /proc/slabinfo >> /root/slabinfo.txt
>>
>> When we do this, our problem gets *much* worse. I''d love to
know if
>> these are similar problems.
>>
>> --
>> -- Tom Mornini, CTO
>> -- Engine Yard, Ruby on Rails Hosting
>> -- Reliability, Ease of Use, Scalability
>> -- (866) 518-YARD (9273)
>>
>> On Mar 2, 2007, at 7:27 PM, Daniel Mealha Cabrita wrote:
>>
>> >
>> > hi there,
>> >
>> >
>> >       Does anyone have have suggestions on how to proceed in  
>> this case?
>> >
>> >
>> >       I''ve experiencing dom0 (xen 3.0.3, xen-3.0.4 and
3.0.4-
>> testing)
>> > lockups under
>> > heavy disk load (testing under dom0 directly).
>> > The hardware is a HP ML150 G2 with a HP 4ch SATA fakeraid (OEM
>> > Adaptec 1420SA,
>> > sata_mv driver).
>> >       The machine does not respond to network, keyboard not  
>> anything
>> > noticeable
>> > when it happens.
>> >
>> >       I''ve tried passing a number of parameters to kernel
but no
>> > success. The ones
>> > below even make things worse, causing a CPU0 soft lockup during  
>> boot:
>> > kernel = (hd0,0)/xen-3.0.4 dom0_mem=384M acpi=off noapic nolapic
>> >
>> >       The disks runs in Linux kernel RAID5. High load to/from an
>> > individual disk
>> > (the max an individual SATA HD can handle) does not cause any  
>> problem.
>> >
>> >       Also, I''ve noticed that just after booting the
machine
>> does not
>> > respond to
>> > pings nor anything from the network. If I locally ping from that
to
>> > another
>> > host, the network starts working. OR, if I wait enough time  
>> (several
>> > minutes), the machine''s network starts to respond
normally.
>> >
>> >       I''ve got no soft lockups logged in
/var/log/messages. Nor
>> anything
>> > strange
>> > enough to call my attention.
>> >
>> >       The problem does not happen with a non-Xen kernel.
>> >       The machine firmware (BIOS if you like) is updated to the  
>> latest
>> > version.
>> >       Disabling all the non-essential hardware (USB, serial/ 
>> parallel
>> > ports, IDE
>> > ports, powersaving etc) makes no difference.
>> >
>> >       When not under high disk load the machine seems stable, with
>> > several domU VMs
>> > running happily under it.
>> >
>> > --
>> >  Daniel Mealha Cabrita
>> >  Divisao de Suporte Tecnico
>> >  AINFO / Reitoria / UTFPR
>> >  http://www.utfpr.edu.br
>> >
>> > _______________________________________________
>> > Xen-users mailing list
>> > Xen-users@lists.xensource.com
>> > http://lists.xensource.com/xen-users
>>
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>>
>
>
> -- 
> GPG key fingerprint: 3883 B308 8256 2246 D3ED  A1FF 3A1D 0EAD 41C4  
> C2F0
> GPG public key availabe on pgp.mit .edu keyserver

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tim Post

2007-Mar-03 07:58 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

On Fri, 2007-03-02 at 22:12 -0800, Tom Mornini wrote:> You don''t believe that *testing* is a good thing?
> 
> I''m pretty sure the Xen documentation points out that Dom0 is not
> particularly special, except that it is privileged to manipulate DomUs.
> 
> I''d love to hear other''s opinions on this topic. Should
Dom0 be
> entirely free of disk I/O?
That''s impossible.
> Now, I should make it clear, I''m a big supporter of Dom0 doing  
> essentially nothing. Yet, I''m also a supporter of not having  
> difficulty sleeping at night, afraid that Dom0 might write a block or  
> two to its own disks...
That is a very silly extreme. When tweaking any system practical Linux
experience comes into play. This starts from selecting the right file
systems for the job at hand setup on networks that work well.

I''ve seen people using calls like :

cat /var/myfile | tr '':'' ''#'' | sed -e
''s/group:/q-group/d'' | grep
"#hello" | awk ''{ print $2,$3 }'' | tr
''#'' '':'' (silly extreme)

That run within loops of scripts and fork a thousand times. Then they
wonder why loads shoot so high when the scripts run and scan syslog. 

A big part of what Xen is bringing back to Linux is forcing integrators
and Administrators to once again be more conscious and aware of how
their operating systems and computers actually work.

> I have a job that runs every five minutes to grab CPU utilization,  
> and writes that to disk.
> That job doesn''t cause destabilization.
No. Its important to write scripts a little differently on dom-0 to be
sure you fork the least amount possible and you don''t get carried away
with over malloc()''ing utilities like sed awk and grep.

Keep perl and PHP down to a minimum and if you use PHP in your shell
scripts, make sure you built a small copy of PHP just for doing that.

Be sensible but don''t make a new phobia out of it.

There are *lots* of things that really should go on dom-0 just because
its the easiest place to put them. Bandwidth loggers, snort, traffic
shaping scripts, stuff scanning syslog every few minutes to see what Xen
has been up to, watching for brute force attacks on the public ports,
pinging iscsi or AoE targets and exports, etc. Its also a handy place to
stick rrdtool/mrtg style tools, or even centralize snmp data for all the
guests. Lots of good sane strategic uses for dom-0, its not nearly the
''black hole'' people make it to be.

There should also be room for some application that handles repetitive
tasks for you, such as setting up and managing the guests.

We made ''grawk'' to help do some of the scraping and sawing
more
efficiently : http://dev1.netkinetics.net/grawk/grawk.c on dom-0.
> My  
> problem seems to be related to kernel SLAB corruption, which is why I  
> mentioned this as something to test, and made it clear that in my  
> case, it made the machine unstable.
Time for a memtest and to make sure no dust bunnies have made their home
on your DDR depending on what exactly was in slabs. But sheesh, don''t
let a project list or some wiki make you afraid to touch your
computer :)

Best,
--Tim

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel Mealha Cabrita

2007-Mar-03 08:04 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

hello Tom,


	I''ve made a test remotely using the guinea-pig HP (I''m at
home now).
I ran the following during  ~5 minutes.

#!/bin/sh
#
while [ 1 ]; do
        cat /proc/slabinfo > /dev/null
done


Nothing happened.
I stopped the script.

A few minutes later I mounted the raid volume.
It might be a coincidence, anyways just a few seconds after that (I barely 
cd''ed to the mounted directory) the machine simply locked up. It never 
happened that fast before, much less before starting the stress test.


Are there any other symptoms you''ve noticed in your case?


Em Sáb 03 Mar 2007 00:24, você escreveu:> Hello Daniel.
>
> We''ve had similar problems, but have received very little feedback
> from our machines.
>
> Our setup is also not entirely similar to yours. :-)
>
> Could you try something out?
>
> Set a cron job to run every 1 minute:
>
>    cat /proc/slabinfo >> /root/slabinfo.txt
>
> When we do this, our problem gets *much* worse. I''d love to know
if
> these are similar problems.
-- 
 Daniel Mealha Cabrita
 Divisao de Suporte Tecnico
 AINFO / Reitoria / UTFPR
 http://www.utfpr.edu.br

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel Mealha Cabrita

2007-Mar-03 08:27 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Em Sáb 03 Mar 2007 03:12, você escreveu:> You don''t believe that *testing* is a good thing?
>
> I''m pretty sure the Xen documentation points out that Dom0 is not
> particularly special, except that it is privileged to manipulate DomUs.
>
> I''d love to hear other''s opinions on this topic. Should
Dom0 be
> entirely free of disk I/O?
I think that the every sysadmin should be able to use dom0 as seem fit.
I don''t think it''s a good idea running things there, but that
kernel is
expected to be fully functional. Security is another matter.

But I''ve already had the need for dom0 heavy disk IO while transferring
files
in order to fix a collapsed domU and dom0 crashed, taking 6 VMs with him.
> I have a job that runs every five minutes to grab CPU utilization,
> and writes that to disk. That job doesn''t cause destabilization.
My
> problem seems to be related to kernel SLAB corruption, which is why I
> mentioned this as something to test, and made it clear that in my
> case, it made the machine unstable.
Now that you mentioned that possibility, it could explain a Xen-related 
problem I''m experiencing with another hardware.

It''s heavy IO-related too.
In this case is a Dell Poweredge 1800.

Instead of just freezing as the HPs, at some point the machine reaches the 
point it has no memory and the kernel starts killing processes (typically 
SSHd and iozone). It seems to be back to normal after that (until you start 
another stress test).
At first I thought it was an iozone bug (it allocates lots of memory anyway), 
but the problem does not happen using non-Xen kernels.


-- 
 Daniel Mealha Cabrita
 Divisao de Suporte Tecnico
 AINFO / Reitoria / UTFPR
 http://www.utfpr.edu.br

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel Mealha Cabrita

2007-Mar-03 08:57 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Em Sáb 03 Mar 2007 04:58, Tim Post escreveu:> On Fri, 2007-03-02 at 22:12 -0800, Tom Mornini wrote:
> > You don''t believe that *testing* is a good thing?
> >
> > I''m pretty sure the Xen documentation points out that Dom0 is
not
> > particularly special, except that it is privileged to manipulate
DomUs.
> >
> > I''d love to hear other''s opinions on this topic.
Should Dom0 be
> > entirely free of disk I/O?
>
> That''s impossible.
I wouldn''t say that.
What prevents a domU from having a root in a block device exported by,
let''s
say, AoE? From dom0''s perspective it would be just network traffic.

Now that I''ve mentioned, I''ve had heavy
''disk'' dom0 problems using AoE aswell.


-- 
 Daniel Mealha Cabrita
 Divisao de Suporte Tecnico
 AINFO / Reitoria / UTFPR
 http://www.utfpr.edu.br

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

TMC

2007-Mar-03 10:24 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

I don''t actually think testing in Dom0 is a good thing unless you have
some very wierd situations.

As far as I understand Xen Dom0, and i might be wrong, it behaves just
like a plain Linux kernel does, in that it will hog CPU, had direct
access to hardware (DomU normally do not), and there is a whole set of
other issues associated with being the "hardware driver holder" for
the hypervisor...

I think a more telling tets would be to run indetical setup with a
plain Linux kenrel and see what that does. maybe youre running into a
driver/FS bug ?


regards
Tomasz

On 03/03/07, Tom Mornini <tmornini@engineyard.com>
wrote:> You don''t believe that *testing* is a good thing?
>
> I''m pretty sure the Xen documentation points out that Dom0 is not
> particularly special, except that it is privileged to manipulate DomUs.
>
> I''d love to hear other''s opinions on this topic. Should
Dom0 be
> entirely free of disk I/O?
>
> Now, I should make it clear, I''m a big supporter of Dom0 doing
> essentially nothing. Yet, I''m also a supporter of not having
> difficulty sleeping at night, afraid that Dom0 might write a block or
> two to its own disks...
>
> I have a job that runs every five minutes to grab CPU utilization,
> and writes that to disk. That job doesn''t cause destabilization.
My
> problem seems to be related to kernel SLAB corruption, which is why I
> mentioned this as something to test, and made it clear that in my
> case, it made the machine unstable.
>
> --
> -- Tom Mornini, CTO
> -- Engine Yard, Ruby on Rails Hosting
> -- Reliability, Ease of Use, Scalability
> -- (866) 518-YARD (9273)
>
> On Mar 2, 2007, at 9:58 PM, TMC wrote:
>
> > I am not sure if testing like this in Dom0 is a good thing.  Dom0 is
> > "special" and should not run any precesses that hit disk
hard. thats
> > the job for DomU
> >
> > Regards
> > TMC
> >
> > On 03/03/07, Tom Mornini <tmornini@engineyard.com> wrote:
> >> Hello Daniel.
> >>
> >> We''ve had similar problems, but have received very little
feedback
> >> from our machines.
> >>
> >> Our setup is also not entirely similar to yours. :-)
> >>
> >> Could you try something out?
> >>
> >> Set a cron job to run every 1 minute:
> >>
> >>    cat /proc/slabinfo >> /root/slabinfo.txt
> >>
> >> When we do this, our problem gets *much* worse. I''d love
to know if
> >> these are similar problems.
> >>
> >> --
> >> -- Tom Mornini, CTO
> >> -- Engine Yard, Ruby on Rails Hosting
> >> -- Reliability, Ease of Use, Scalability
> >> -- (866) 518-YARD (9273)
> >>
> >> On Mar 2, 2007, at 7:27 PM, Daniel Mealha Cabrita wrote:
> >>
> >> >
> >> > hi there,
> >> >
> >> >
> >> >       Does anyone have have suggestions on how to proceed in
> >> this case?
> >> >
> >> >
> >> >       I''ve experiencing dom0 (xen 3.0.3, xen-3.0.4
and 3.0.4-
> >> testing)
> >> > lockups under
> >> > heavy disk load (testing under dom0 directly).
> >> > The hardware is a HP ML150 G2 with a HP 4ch SATA fakeraid
(OEM
> >> > Adaptec 1420SA,
> >> > sata_mv driver).
> >> >       The machine does not respond to network, keyboard not
> >> anything
> >> > noticeable
> >> > when it happens.
> >> >
> >> >       I''ve tried passing a number of parameters to
kernel but no
> >> > success. The ones
> >> > below even make things worse, causing a CPU0 soft lockup
during
> >> boot:
> >> > kernel = (hd0,0)/xen-3.0.4 dom0_mem=384M acpi=off noapic
nolapic
> >> >
> >> >       The disks runs in Linux kernel RAID5. High load to/from
an
> >> > individual disk
> >> > (the max an individual SATA HD can handle) does not cause any
> >> problem.
> >> >
> >> >       Also, I''ve noticed that just after booting the
machine
> >> does not
> >> > respond to
> >> > pings nor anything from the network. If I locally ping from
that to
> >> > another
> >> > host, the network starts working. OR, if I wait enough time
> >> (several
> >> > minutes), the machine''s network starts to respond
normally.
> >> >
> >> >       I''ve got no soft lockups logged in
/var/log/messages. Nor
> >> anything
> >> > strange
> >> > enough to call my attention.
> >> >
> >> >       The problem does not happen with a non-Xen kernel.
> >> >       The machine firmware (BIOS if you like) is updated to
the
> >> latest
> >> > version.
> >> >       Disabling all the non-essential hardware (USB, serial/
> >> parallel
> >> > ports, IDE
> >> > ports, powersaving etc) makes no difference.
> >> >
> >> >       When not under high disk load the machine seems stable,
with
> >> > several domU VMs
> >> > running happily under it.
> >> >
> >> > --
> >> >  Daniel Mealha Cabrita
> >> >  Divisao de Suporte Tecnico
> >> >  AINFO / Reitoria / UTFPR
> >> >  http://www.utfpr.edu.br
> >> >
> >> > _______________________________________________
> >> > Xen-users mailing list
> >> > Xen-users@lists.xensource.com
> >> > http://lists.xensource.com/xen-users
> >>
> >>
> >> _______________________________________________
> >> Xen-users mailing list
> >> Xen-users@lists.xensource.com
> >> http://lists.xensource.com/xen-users
> >>
> >
> >
> > --
> > GPG key fingerprint: 3883 B308 8256 2246 D3ED  A1FF 3A1D 0EAD 41C4
> > C2F0
> > GPG public key availabe on pgp.mit .edu keyserver
>
>

-- 
GPG key fingerprint: 3883 B308 8256 2246 D3ED  A1FF 3A1D 0EAD 41C4 C2F0
GPG public key availabe on pgp.mit .edu keyserver

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tim Post

2007-Mar-03 12:57 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

On Sat, 2007-03-03 at 05:57 -0300, Daniel Mealha Cabrita
wrote:> Em Sáb 03 Mar 2007 04:58, Tim Post escreveu:
> > On Fri, 2007-03-02 at 22:12 -0800, Tom Mornini wrote:
> > > You don''t believe that *testing* is a good thing?
> > >
> > > I''m pretty sure the Xen documentation points out that
Dom0 is not
> > > particularly special, except that it is privileged to manipulate
DomUs.
> > >
> > > I''d love to hear other''s opinions on this
topic. Should Dom0 be
> > > entirely free of disk I/O?
> >
> > That''s impossible.
> 
> I wouldn''t say that.
Does dom-0 not require a base install of Linux? Does it not use
logrotate slocate brute force detection syslog sawing and other i/o
consuming needed but wasteful junk that any other server would?
> What prevents a domU from having a root in a block device exported by,
let''s
> say, AoE? From dom0''s perspective it would be just network
traffic.
That''s 100% correct. But that''s in instances where one sets up
a
scenario (such as AoE) where userspace in dom-0 is needed to make
dom-u''s happen. The trade off to that is use a routed protocol like
iscsi and follow the ever true credence that such drivers belong in
virtual disposable machines where crashes can be recovered and modules
can easily be destroyed. 

If you follow that, then you''re side swiped by the need for offload
engines and bonded gig paths defeating part of the purpose for not going
optical. Damned if you do or don''t no matter what when contemplating
network storage. So, I found its best to just buy yourself a few
options.
> Now that I''ve mentioned, I''ve had heavy
''disk'' dom0 problems using AoE aswell.
I''ve been opting for 2x 10G optical ethernet devices in most
"serious"
xen farms so I can be really flexible. One becomes a routed IP network,
the other one is just used for AoE. One good 12 - 24 port switch is all
I need. Its not cheap, but no longer so expensive that its totally
prohibitive. 

Using that + 15K SAS drives has really solved 99.9% of my problems with
really demanding guests. But this isn''t exactly ''off the
shelf'' either.

I think people just somehow lost sight of how Linux uses memory, and to
avoid needing to learn dom-0 was just named ''taboo''. 


Just treat it like any other vital system that has very little ram.
Vital systems having very little ram is not a familiar concept to many
newcomers to Linux since the days of 386 (and gasp) 286 / 8088 / 80886
users.

Best,
--Tim


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tim Post

2007-Mar-03 12:59 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Tom,

It may be helpful if the list knew what type of filesystem was backing
"/root" (ext3 resier jfs, etc?) as well as possibly see the contents
of
slabinfo.

I don''t think this is a Xen issue, but I''d like to see your
inodes
anyway.

Thanks
--Tim

On Fri, 2007-03-02 at 19:24 -0800, Tom Mornini wrote:> Hello Daniel.
> 
> We''ve had similar problems, but have received very little feedback
> from our machines.
> 
> Our setup is also not entirely similar to yours. :-)
> 
> Could you try something out?
> 
> Set a cron job to run every 1 minute:
> 
>    cat /proc/slabinfo >> /root/slabinfo.txt
> 
> When we do this, our problem gets *much* worse. I''d love to know
if
> these are similar problems.
> 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nico Kadel-Garcia

2007-Mar-03 14:05 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Tim Post wrote:> On Sat, 2007-03-03 at 05:57 -0300, Daniel Mealha Cabrita wrote:
>> Em Sáb 03 Mar 2007 04:58, Tim Post escreveu:
>>> On Fri, 2007-03-02 at 22:12 -0800, Tom Mornini wrote:
>>>> You don''t believe that *testing* is a good thing?
>>>>
>>>> I''m pretty sure the Xen documentation points out that
Dom0 is not
>>>> particularly special, except that it is privileged to
manipulate
>>>> DomUs.
>>>>
>>>> I''d love to hear other''s opinions on this
topic. Should Dom0 be
>>>> entirely free of disk I/O?
>>>
>>> That''s impossible.
>>
>> I wouldn''t say that.
>
> Does dom-0 not require a base install of Linux? Does it not use
> logrotate slocate brute force detection syslog sawing and other i/o
> consuming needed but wasteful junk that any other server would?
Welcome to LiveCD''s. With a grotesque amount RAM, you could potentially
do
entirely RAM based operating system images.
> Just treat it like any other vital system that has very little ram.
> Vital systems having very little ram is not a familiar concept to many
> newcomers to Linux since the days of 386 (and gasp) 286 / 8088 / 80886
> users.
Yeah, I''ve noticed that. Unfortunately, a lot of management tools these
days
are also quite.... generous with their requirements for successful 
operation. The virt-manager is one of those. 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tim Post

2007-Mar-03 15:04 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

On Sat, 2007-03-03 at 14:05 +0000, Nico Kadel-Garcia
wrote:> Tim Post wrote:
> > Just treat it like any other vital system that has very little ram.
> > Vital systems having very little ram is not a familiar concept to many
> > newcomers to Linux since the days of 386 (and gasp) 286 / 8088 / 80886
> > users.
> 
> Yeah, I''ve noticed that. Unfortunately, a lot of management tools
these days
> are also quite.... generous with their requirements for successful 
> operation. The virt-manager is one of those. 
Really depending on the structure of the management tool, they really
have no other choice. 

If you want a framework and interconnect for xen that steers clear of
blocking calls and remains (itself) transparent, you''re going to have
to
give a few extra pages to it so it has enough room for whatever results.

What''s critical is that if you have a reasonable expectation of
what''s
coming there''s really no reason to take much more, especially if you
know you have limited memory to begin with.

But everyone seems to have forgotten the days of 4 MB being a lot.

Anyway bordering on going off topic. I just find it ironic and humorous
that state of the art Virtualization technology is making problems that
went away magically when big ram came out re-present themselves with a
vengeance.

Best,
--Tim

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tom Mornini

2007-Mar-03 21:01 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

On Mar 2, 2007, at 11:58 PM, Tim Post wrote:
>> My
>> problem seems to be related to kernel SLAB corruption, which is why I
>> mentioned this as something to test, and made it clear that in my
>> case, it made the machine unstable.
>
> Time for a memtest and to make sure no dust bunnies have made their  
> home
> on your DDR depending on what exactly was in slabs. But sheesh,
don''t
> let a project list or some wiki make you afraid to touch your
> computer :)
I''ve run memtest, and this happens on more than one system. And, it now
appears that it happens on other systems as well, since doing this
provoked new crashing behavior for another lister as well.

-- 
-- Tom Mornini, CTO
-- Engine Yard, Ruby on Rails Hosting
-- Reliability, Ease of Use, Scalability
-- (866) 518-YARD (9273)


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tom Mornini

2007-Mar-03 21:15 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

On Mar 3, 2007, at 4:57 AM, Tim Post wrote:
>> Now that I''ve mentioned, I''ve had heavy
''disk'' dom0 problems using
>> AoE aswell.
>
> I''ve been opting for 2x 10G optical ethernet devices in most
"serious"
> xen farms so I can be really flexible. One becomes a routed IP  
> network,
> the other one is just used for AoE. One good 12 - 24 port switch is  
> all
> I need. Its not cheap, but no longer so expensive that its totally
> prohibitive.
We''ve considered this as well, but for now 4x GigE is working well, 2  
for
IP, 3 for AoE. Disk, or vis-a-vis network, congestion isn''t the
problem,
machines occasionally hanging is the problem.

We believe, largely from the test described previously, that we''re  
seeing
Dom0 SLAB corruption, and we''ve tried just about everything to identify
it, but alas, cannot track this down.

We''ve compiled kernels, Dom0 and DomU with full debugging, etc. but  
no luck.
> Using that + 15K SAS drives has really solved 99.9% of my problems  
> with
> really demanding guests. But this isn''t exactly ''off the
shelf''
> either.
>
> I think people just somehow lost sight of how Linux uses memory,  
> and to
> avoid needing to learn dom-0 was just named ''taboo''.
>
> Just treat it like any other vital system that has very little ram.
> Vital systems having very little ram is not a familiar concept to many
> newcomers to Linux since the days of 386 (and gasp) 286 / 8088 / 80886
> users.
Agreed. We have plenty of RAM to cat out /proc/slabinfo via cron,  
particularly
so if it helps us figure out what is going on. :-)

-- 
-- Tom Mornini, CTO
-- Engine Yard, Ruby on Rails Hosting
-- Reliability, Ease of Use, Scalability
-- (866) 518-YARD (9273)


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tom Mornini

2007-Mar-03 21:24 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

On Mar 3, 2007, at 7:04 AM, Tim Post wrote:
> But everyone seems to have forgotten the days of 4 MB being a lot.
>
> Anyway bordering on going off topic. I just find it ironic and  
> humorous
> that state of the art Virtualization technology is making problems  
> that
> went away magically when big ram came out re-present themselves with a
> vengeance.
The problem I''m seeing is *not* memory starvation, it''s memory
corruption.
We have almost nothing going on in Dom0... The only reason I put the  
cron
job there is the Dom0 kernel is crashing, so it needed to be tested  
there...

We go way back with Unix, and I''ve told people the same thing, that  
Xen is
going to teach people about system performance tuning again. :-)

-- 
-- Tom Mornini, CTO
-- Engine Yard, Ruby on Rails Hosting
-- Reliability, Ease of Use, Scalability
-- (866) 518-YARD (9273)


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

TMC

2007-Mar-03 22:20 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

On 04/03/07, Tom Mornini <tmornini@engineyard.com>
wrote:> On Mar 3, 2007, at 7:04 AM, Tim Post wrote:
>
> > But everyone seems to have forgotten the days of 4 MB being a lot.
> >
> > Anyway bordering on going off topic. I just find it ironic and
> > humorous
> > that state of the art Virtualization technology is making problems
> > that
> > went away magically when big ram came out re-present themselves with a
> > vengeance.
>
> The problem I''m seeing is *not* memory starvation, it''s
memory
> corruption.
are you seeing the same issues with the equivalent vanilla 2,6.xx
linux kernel without the Xen patches?
> We have almost nothing going on in Dom0... The only reason I put the
> cron
> job there is the Dom0 kernel is crashing, so it needed to be tested
> there...
>
> We go way back with Unix, and I''ve told people the same thing,
that
> Xen is
> going to teach people about system performance tuning again. :-)
>
> --
> -- Tom Mornini, CTO
> -- Engine Yard, Ruby on Rails Hosting
> -- Reliability, Ease of Use, Scalability
> -- (866) 518-YARD (9273)
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>

-- 
GPG key fingerprint: 3883 B308 8256 2246 D3ED  A1FF 3A1D 0EAD 41C4 C2F0
GPG public key availabe on pgp.mit .edu keyserver

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

TMC

2007-Mar-04 04:34 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

I seer.

what version of kernel are you using? have you actually check the
changelog for the newer kernel versions to see if there any related
issues being fixed in the device drivers, memory handling, block
devices ?

Tomasz



On 04/03/07, Tom Mornini <tmornini@engineyard.com>
wrote:> No, I''ve never seen this kind of behavior before.
>
> I cannot test it outside of Xen, because our entire system and
it''s
> reason for existence is based upon Xen.
>
> --
> -- Tom Mornini, CTO
> -- Engine Yard, Ruby on Rails Hosting
> -- Reliability, Ease of Use, Scalability
> -- (866) 518-YARD (9273)
>
>
> On Mar 3, 2007, at 2:20 PM, TMC wrote:
>
> > On 04/03/07, Tom Mornini <tmornini@engineyard.com> wrote:
> >> On Mar 3, 2007, at 7:04 AM, Tim Post wrote:
> >>
> >> > But everyone seems to have forgotten the days of 4 MB being a
lot.
> >> >
> >> > Anyway bordering on going off topic. I just find it ironic
and
> >> > humorous
> >> > that state of the art Virtualization technology is making
problems
> >> > that
> >> > went away magically when big ram came out re-present
themselves
> >> with a
> >> > vengeance.
> >>
> >> The problem I''m seeing is *not* memory starvation,
it''s memory
> >> corruption.
> >
> > are you seeing the same issues with the equivalent vanilla 2,6.xx
> > linux kernel without the Xen patches?
> >
> >> We have almost nothing going on in Dom0... The only reason I put
the
> >> cron
> >> job there is the Dom0 kernel is crashing, so it needed to be
tested
> >> there...
> >>
> >> We go way back with Unix, and I''ve told people the same
thing, that
> >> Xen is
> >> going to teach people about system performance tuning again. :-)
> >>
> >> --
> >> -- Tom Mornini, CTO
> >> -- Engine Yard, Ruby on Rails Hosting
> >> -- Reliability, Ease of Use, Scalability
> >> -- (866) 518-YARD (9273)
> >>
> >>
> >> _______________________________________________
> >> Xen-users mailing list
> >> Xen-users@lists.xensource.com
> >> http://lists.xensource.com/xen-users
> >>
> >
> >
> > --
> > GPG key fingerprint: 3883 B308 8256 2246 D3ED  A1FF 3A1D 0EAD 41C4
> > C2F0
> > GPG public key availabe on pgp.mit .edu keyserver
>
>

-- 
GPG key fingerprint: 3883 B308 8256 2246 D3ED  A1FF 3A1D 0EAD 41C4 C2F0
GPG public key availabe on pgp.mit .edu keyserver

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tom Mornini

2007-Mar-04 05:19 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

On Mar 3, 2007, at 8:34 PM, TMC wrote:
> what version of kernel are you using? have you actually check the
> changelog for the newer kernel versions to see if there any related
> issues being fixed in the device drivers, memory handling, block
> devices ?
2.6.16.33, which is the standard for Xen current (3.04)

I''ve just looked at changelogs for 2.6.16.34 and 2.6.16.43 and there
are a few potentially relevant changes. I''ve forwarded them on to
our kernel guy, and we''ll see if he can build Xen against current
2.6.16 kernel.

-- 
-- Tom Mornini, CTO
-- Engine Yard, Ruby on Rails Hosting
-- Reliability, Ease of Use, Scalability
-- (866) 518-YARD (9273)


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel Mealha Cabrita

2007-Mar-05 02:40 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Could you please let us know what happened, after he tries compiling the newer 
kernel with Xen?

I''ve already tried 2.6.16.38 (xen 3.0.4-testing) but nothing changed
here and
the problem persists.

Em Dom 04 Mar 2007 02:19, Tom Mornini escreveu:> On Mar 3, 2007, at 8:34 PM, TMC wrote:
> > what version of kernel are you using? have you actually check the
> > changelog for the newer kernel versions to see if there any related
> > issues being fixed in the device drivers, memory handling, block
> > devices ?
>
> 2.6.16.33, which is the standard for Xen current (3.04)
>
> I''ve just looked at changelogs for 2.6.16.34 and 2.6.16.43 and
there
> are a few potentially relevant changes. I''ve forwarded them on to
> our kernel guy, and we''ll see if he can build Xen against current
> 2.6.16 kernel.
-- 
 Daniel Mealha Cabrita
 Divisao de Suporte Tecnico
 AINFO / Reitoria / UTFPR
 http://www.utfpr.edu.br

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ulrich Windl

2007-Mar-05 07:15 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

On 2 Mar 2007 at 19:24, Tom Mornini wrote:
> Hello Daniel.
> 
> We''ve had similar problems, but have received very little feedback
> from our machines.
> 
> Our setup is also not entirely similar to yours. :-)
> 
> Could you try something out?
> 
> Set a cron job to run every 1 minute:
> 
>    cat /proc/slabinfo >> /root/slabinfo.txt
BTW: do you know slabtop?:

 Active / Total Objects (% used)    : 155588 / 295868 (52.6%)
 Active / Total Slabs (% used)      : 28469 / 28498 (99.9%)
 Active / Total Caches (% used)     : 89 / 128 (69.5%)
 Active / Total Size (% used)       : 56779.98K / 105070.07K (54.0%)
 Minimum / Average / Maximum Object : 0.02K / 0.35K / 128.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
 94170  23806  25%    0.25K   6278       15     25112K dentry_cache
 46880  30443  64%    0.09K   1172       40      4688K buffer_head
 44556  26860  60%    0.88K  11139        4     44556K ext3_inode_cache
 17175   6633  38%    0.75K   3435        5     13740K reiser_inode_cache
 15323   7591  49%    0.52K   2189        7      8756K radix_tree_node
 12992  12940  99%    0.03K    116      112       464K dm_io
 12960  12885  99%    0.02K     90      144       360K dm_tio
  9632   1706  17%    0.03K     86      112       344K size-32
  8445   6939  82%    0.25K    563       15      2252K size-256
  6728   3705  55%    0.06K    116       58       464K size-64
  4530   3598  79%    0.12K    151       30       604K size-128
  4158   3619  87%    0.17K    189       22       756K vm_area_struct
  2676   2676 100%    0.62K    446        6      1784K inode_cache
  1605   1326  82%    0.25K    107       15       428K filp
  1584    819  51%    0.02K     11      144        44K anon_vma
  1520   1489  97%    0.38K    152       10       608K ip_dst_cache

Ulrich


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Ulrich Windl

2007-Mar-05 07:17 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

On 3 Mar 2007 at 0:27, Daniel Mealha Cabrita wrote:
> 
> hi there,
> 
> 
> 	Does anyone have have suggestions on how to proceed in this case?
> 
> 
> 	I''ve experiencing dom0 (xen 3.0.3, xen-3.0.4 and 3.0.4-testing)
lockups under
> heavy disk load (testing under dom0 directly).
> The hardware is a HP ML150 G2 with a HP 4ch SATA fakeraid (OEM Adaptec
1420SA,
> sata_mv driver).
Which distribution do you use? I ran "iozone" under XEN and I had no
problems.

[...]
Ulrich


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tom Mornini

2007-Mar-05 14:51 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED (so it seems)

Can you try this while issuing:


   cat /proc/slabinfo >> slabinfo.txt


simultaneously?

--  
-- Tom Mornini, CTO
-- Engine Yard, Ruby on Rails Hosting
-- Reliability, Ease of Use, Scalability
-- (866) 518-YARD (9273)


On Mar 5, 2007, at 10:00 AM, Daniel Mealha Cabrita wrote:
>
> Apparently Xen-unstable (as of today) has this issue fixed.
>
> I left iozone running for several minutes and no hangups  
> (previously it
> wouldn''t past the random disk access tests, at best).
>
> No idea whether a kernel or a xen-specific bug. The kernel version  
> of unstable
> version is currently 2.6.18.
>
>
> -- 
>  Daniel Mealha Cabrita
>  Divisao de Suporte Tecnico
>  AINFO / Reitoria / UTFPR
>  http://www.utfpr.edu.br
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Itamar Reis Peixoto

2007-Mar-05 14:53 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED(so it seems)

Can you explain for me how to identify a problem using the slabinfo ?


--------------------

Itamar Reis Peixoto

e-mail: itamar@ispbrasil.com.br
msn: itamarjp@starmedia.com
skype: itamarjp
icq: 81053601
+55 34 3238 3845
+55 11 4063 5033


----- Original Message ----- 
From: "Tom Mornini" <tmornini@engineyard.com>
To: "Daniel Mealha Cabrita" <dancab@utfpr.edu.br>
Cc: <xen-users@lists.xensource.com>
Sent: Monday, March 05, 2007 12:51 PM
Subject: Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - 
FIXED(so it seems)

> Can you try this while issuing:
>
>
>   cat /proc/slabinfo >> slabinfo.txt
>
>
> simultaneously?
>
> --  -- Tom Mornini, CTO
> -- Engine Yard, Ruby on Rails Hosting
> -- Reliability, Ease of Use, Scalability
> -- (866) 518-YARD (9273)
>
>
> On Mar 5, 2007, at 10:00 AM, Daniel Mealha Cabrita wrote:
>
>>
>> Apparently Xen-unstable (as of today) has this issue fixed.
>>
>> I left iozone running for several minutes and no hangups  (previously
it
>> wouldn''t past the random disk access tests, at best).
>>
>> No idea whether a kernel or a xen-specific bug. The kernel version  of 
>> unstable
>> version is currently 2.6.18.
>>
>>
>> -- 
>>  Daniel Mealha Cabrita
>>  Divisao de Suporte Tecnico
>>  AINFO / Reitoria / UTFPR
>>  http://www.utfpr.edu.br
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
> 

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel Mealha Cabrita

2007-Mar-05 15:00 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED (so it seems)

Apparently Xen-unstable (as of today) has this issue fixed.

I left iozone running for several minutes and no hangups (previously it 
wouldn''t past the random disk access tests, at best).

No idea whether a kernel or a xen-specific bug. The kernel version of unstable 
version is currently 2.6.18.


-- 
 Daniel Mealha Cabrita
 Divisao de Suporte Tecnico
 AINFO / Reitoria / UTFPR
 http://www.utfpr.edu.br

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tom Mornini

2007-Mar-05 15:24 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED(so it seems)

I  noticed that if I set a cron job:

   cat /proc/slabinfo >> slabinfo.txt

to run every 5 minutes, my machines became far less stable.

Mr. Cabrita may have experienced the same issue.

--  
-- Tom Mornini, CTO
-- Engine Yard, Ruby on Rails Hosting
-- Reliability, Ease of Use, Scalability
-- (866) 518-YARD (9273)


On Mar 5, 2007, at 9:53 AM, Itamar Reis Peixoto wrote:
> Can you explain for me how to identify a problem using the slabinfo ?
>
>
> --------------------
>
> Itamar Reis Peixoto
>
> e-mail: itamar@ispbrasil.com.br
> msn: itamarjp@starmedia.com
> skype: itamarjp
> icq: 81053601
> +55 34 3238 3845
> +55 11 4063 5033
>
>
> ----- Original Message ----- From: "Tom Mornini"  
> <tmornini@engineyard.com>
> To: "Daniel Mealha Cabrita" <dancab@utfpr.edu.br>
> Cc: <xen-users@lists.xensource.com>
> Sent: Monday, March 05, 2007 12:51 PM
> Subject: Re: [Xen-users] dom0 freezes under high IO load - HP ML150  
> G2 - FIXED(so it seems)
>
>
>> Can you try this while issuing:
>>
>>
>>   cat /proc/slabinfo >> slabinfo.txt
>>
>>
>> simultaneously?
>>
>> --  -- Tom Mornini, CTO
>> -- Engine Yard, Ruby on Rails Hosting
>> -- Reliability, Ease of Use, Scalability
>> -- (866) 518-YARD (9273)
>>
>>
>> On Mar 5, 2007, at 10:00 AM, Daniel Mealha Cabrita wrote:
>>
>>>
>>> Apparently Xen-unstable (as of today) has this issue fixed.
>>>
>>> I left iozone running for several minutes and no hangups   
>>> (previously it
>>> wouldn''t past the random disk access tests, at best).
>>>
>>> No idea whether a kernel or a xen-specific bug. The kernel  
>>> version  of unstable
>>> version is currently 2.6.18.
>>>
>>>
>>> -- 
>>>  Daniel Mealha Cabrita
>>>  Divisao de Suporte Tecnico
>>>  AINFO / Reitoria / UTFPR
>>>  http://www.utfpr.edu.br
>>>
>>> _______________________________________________
>>> Xen-users mailing list
>>> Xen-users@lists.xensource.com
>>> http://lists.xensource.com/xen-users
>>
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Tom Mornini

2007-Mar-05 15:25 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED (so it seems)

Hmmm. So what is the status of the Xen-unstable branch?

Is there any way to read changeset info?

-- 
-- Tom Mornini, CTO
-- Engine Yard, Ruby on Rails Hosting
-- Reliability, Ease of Use, Scalability
-- (866) 518-YARD (9273)


On Mar 5, 2007, at 10:35 AM, Daniel Mealha Cabrita wrote:
>
> I can leave that running each second for > 20 min and no crashes.
>
> Em Seg 05 Mar 2007 11:51, Tom Mornini escreveu:
>> Can you try this while issuing:
>>
>>
>>    cat /proc/slabinfo >> slabinfo.txt
>>
>>
>> simultaneously?
>>
>> --
>> -- Tom Mornini, CTO
>> -- Engine Yard, Ruby on Rails Hosting
>> -- Reliability, Ease of Use, Scalability
>> -- (866) 518-YARD (9273)
>>
>> On Mar 5, 2007, at 10:00 AM, Daniel Mealha Cabrita wrote:
>>> Apparently Xen-unstable (as of today) has this issue fixed.
>>>
>>> I left iozone running for several minutes and no hangups
>>> (previously it
>>> wouldn''t past the random disk access tests, at best).
>>>
>>> No idea whether a kernel or a xen-specific bug. The kernel version
>>> of unstable
>>> version is currently 2.6.18.
>>>
>>>
>>> --
>>>  Daniel Mealha Cabrita
>>>  Divisao de Suporte Tecnico
>>>  AINFO / Reitoria / UTFPR
>>>  http://www.utfpr.edu.br
>>>
>>> _______________________________________________
>>> Xen-users mailing list
>>> Xen-users@lists.xensource.com
>>> http://lists.xensource.com/xen-users
>>
>> _______________________________________________
>> Xen-users mailing list
>> Xen-users@lists.xensource.com
>> http://lists.xensource.com/xen-users
>
> -- 
>  Daniel Mealha Cabrita
>  Divisao de Suporte Tecnico
>  AINFO / Reitoria / UTFPR
>  http://www.utfpr.edu.br

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Daniel Mealha Cabrita

2007-Mar-05 15:35 UTC

head link

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED (so it seems)

I can leave that running each second for > 20 min and no crashes.

Em Seg 05 Mar 2007 11:51, Tom Mornini escreveu:> Can you try this while issuing:
>
>
>    cat /proc/slabinfo >> slabinfo.txt
>
>
> simultaneously?
>
> --
> -- Tom Mornini, CTO
> -- Engine Yard, Ruby on Rails Hosting
> -- Reliability, Ease of Use, Scalability
> -- (866) 518-YARD (9273)
>
> On Mar 5, 2007, at 10:00 AM, Daniel Mealha Cabrita wrote:
> > Apparently Xen-unstable (as of today) has this issue fixed.
> >
> > I left iozone running for several minutes and no hangups
> > (previously it
> > wouldn''t past the random disk access tests, at best).
> >
> > No idea whether a kernel or a xen-specific bug. The kernel version
> > of unstable
> > version is currently 2.6.18.
> >
> >
> > --
> >  Daniel Mealha Cabrita
> >  Divisao de Suporte Tecnico
> >  AINFO / Reitoria / UTFPR
> >  http://www.utfpr.edu.br
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xensource.com
> > http://lists.xensource.com/xen-users
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
-- 
 Daniel Mealha Cabrita
 Divisao de Suporte Tecnico
 AINFO / Reitoria / UTFPR
 http://www.utfpr.edu.br

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Xen users - Mar 2007 - dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

[Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED (so it seems)

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED(so it seems)

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED (so it seems)

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED(so it seems)

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED (so it seems)

Re: [Xen-users] dom0 freezes under high IO load - HP ML150 G2 - FIXED (so it seems)