Xen users - Jun 2008 - development machine with xen on gentoo

Hi all.

We just put a relative large server in operation (for our measures ;-) 
), a 4x Dual Core / Intel with 16GB Ram and a 3ware 9550SX SATA-RAID 
with 4 drives. Operating system for the Dom0 is an up-to-date gentoo linux.

Everything runs really fine, there are 4 Domu''s running, 1x gentoo and 
2x debian and 1x Windows Server 2003. DomU''s are running blazing fast
in
normal work, no problems at all.

Problems appear on the Dom0 when doing a lot of traffic on the 
harddrives, for example when copying a large image-file or unpacking 
archived images. When doing that the performance on all Dom''s goes 
rapidly down. But _only_ when doing this on the Dom0. Traffic on a DomU 
doesn''t result in this behaviour. For example, copying an image with
1GB
took about an hour. IO waitstates are really high on the virtual 
machines in this situation.

Can someone give me a hint what this could be? Or maybe a hint about 
which tools can help me to find the source of the problem. I only have 
experience with smaller Xen-installations, and never experienced 
problems this way.

Any help would be appreciated.

Bye, Stephan



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Tue, Jun 3, 2008 at 2:45 PM, Stephan Wentz <stephan@wentz.it>
wrote:
> Problems appear on the Dom0 when doing a lot of traffic on the harddrives,
what happens if you use ionice to give Dom0 traffic just a little less
priority than serving DomUs?

-- 
Javier

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Javier Guerra wrote:
> On Tue, Jun 3, 2008 at 2:45 PM, Stephan Wentz <stephan@wentz.it>
wrote:
>> Problems appear on the Dom0 when doing a lot of traffic on the
harddrives,
> 
> what happens if you use ionice to give Dom0 traffic just a little less
> priority than serving DomUs?
> 
No that makes it even worse, iowaits stay the same and the transaction 
is even slower...

I digged a little deeper and did some tests, I found out the following. 
I stopped all DomU''s and rebootet to have a clean system with only the 
Dom0 running. Then I started to scp a 20 GB image from another machine.
Speed was ok for the first 8 GB, about 50mb/s. But then suddenly the 
throughput went slowly down, to about 2-3mb/s.
What was kinda weird, the memory usage of the Dom0 increased in the 
_same amount_ of the transferred data! When 9GB where tansferred free 
showed 9,5GB used memory, after 10GB transfer it showed 10,5GB used 
memory. That rather confused me - what can this be? Is the server 
holding the copied data in memory because it can''t write it fast
enough?
Looking at iostat -x I saw that %utilization was always at 100% for the 
target device.
Now what''s funny, when I killed scp on the remote machine the 
%utilization wasn''t going down, throughput was at constant 5mb/s, for 
about 5 minutes, it seemed like he was still writing the received data 
from memory to disk.

Btw, xen version is 3.2.1.

I think that the RAID controller has to do with it (3ware 9550SX), but I 
have no clue on how to get more details to find the bottleneck.

Anyone got an idea?

Bye,
Stephan



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stephan Wentz wrote:
> I think that the RAID controller has to do with it (3ware 9550SX), but 
> I have no clue on how to get more details to find the bottleneck.
I think you are correct. I have had some performance issues with 3ware 
controllers in the past.

What file system, have you done any of the 3ware tweaks? Your best bet 
is to start tracking down performance issues on the controller first off.

Do you have a BBU? if not you should get one in my tests a BBU on 3ware 
will increate write performance by up to 10%. What is storesave set to 
on the raid controller?


-- 
Nick Anderson <nick@anders0n.net>
http://www.cmdln.org
http://www.anders0n.net



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nick Anderson wrote:
> Stephan Wentz wrote:
>> I think that the RAID controller has to do with it (3ware 9550SX), but 
>> I have no clue on how to get more details to find the bottleneck.
> I think you are correct. I have had some performance issues with 3ware 
> controllers in the past.
> 
> What file system, have you done any of the 3ware tweaks? Your best bet 
> is to start tracking down performance issues on the controller first off.
> 
> Do you have a BBU? if not you should get one in my tests a BBU on 3ware 
> will increate write performance by up to 10%. What is storesave set to 
> on the raid controller?
I did filesystem tests with ext3 and xfs, xfs provided better performance.

Haven''t tried any 3ware tweaks - didn''t even know that they
exists ;-)
Can you provide a link or something where I can get information?

Thanks for the BBU-hint - we will order one tomorrow.

Stephan



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Nick Anderson wrote:
> Stephan Wentz wrote:
>> I think that the RAID controller has to do with it (3ware 9550SX), but 
>> I have no clue on how to get more details to find the bottleneck.
> I think you are correct. I have had some performance issues with 3ware 
> controllers in the past.
> 
> What file system, have you done any of the 3ware tweaks? Your best bet 
> is to start tracking down performance issues on the controller first off.
> 
> Do you have a BBU? if not you should get one in my tests a BBU on 3ware 
> will increate write performance by up to 10%. What is storesave set to 
> on the raid controller?
Oops, forgot to mention - will check the storesave parameter tomorrow. 
What setting is recommended?

And, additionally, would you recommend setting cache on or off? On only 
with a BBU?

Stephan



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Stephan Wentz wrote:
> Oops, forgot to mention - will check the storesave parameter tomorrow. 
> What setting is recommended?
> 
> And, additionally, would you recommend setting cache on or off? On only 
> with a BBU?
Storesave to perform and writecache on but both only with BBU.

What kind of read write performance did you get with your tests?

The 3ware knowledge base has some tips. For read performance 
specifically you will want to adjust the read ahead value. And I have 
found that if your VM is located on the raid set you will actually want 
to set the read ahead value inside of the vm as well.

-- 
Nick Anderson <nick@anders0n.net>
http://www.cmdln.org
http://www.anders0n.net


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users