n8&abby
2008-Apr-20 19:42 UTC
[Xen-users] creating domU''s consumes 100% of system resources
Hello, I have noticed that when I use xen-create-image to generate an domU, the whole server (dom0 and domU''s) basically hangs until it is finished. This happens primarily during the creation of the ext3 filesystem on an LVM partition. This creation of the file system can take up to 4 or 5 minutes. At which point, any other domU''s are basically paused... tcp connections time out.. etc. I tried ''xm vcpu-set 0 1'' to take the dom0 down to one virtual cpu.. but this didn''t seem to have any affect. I don''t mind if the generation of new domU''s takes 30 minutes, as long the performance of my other domU''s are not affected greatly. Any hints on how to fix this problem? Thanks, -=nathan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
n8&abby
2008-Apr-21 19:34 UTC
[Xen-users] creating domU''s consumes 100% of system resources
Hello, I have noticed that when I use xen-create-image to generate an domU, the whole server (dom0 and domU''s) basically hangs until it is finished. This happens primarily during the creation of the ext3 filesystem on an LVM partition. This creation of the file system can take up to 4 or 5 minutes. At which point, any other domU''s are basically paused... tcp connections time out.. etc. I tried ''xm vcpu-set 0 1'' to take the dom0 down to one virtual cpu.. but this didn''t seem to have any affect. I don''t mind if the generation of new domU''s takes 30 minutes, as long the performance of my other domU''s are not affected greatly. Any hints on how to fix this problem? Thanks, -=nathan _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
William Pitcock
2008-Apr-21 19:55 UTC
Re: [Xen-users] creating domU''s consumes 100% of system resources
On Mon, 2008-04-21 at 14:34 -0500, n8&abby wrote:> Hello, > > I have noticed that when I use xen-create-image to generate an domU, > the whole server (dom0 and domU''s) basically hangs until it is > finished. This happens primarily during the creation of the ext3 > filesystem on an LVM partition. > > This creation of the file system can take up to 4 or 5 minutes. At > which point, any other domU''s are basically paused... tcp connections > time out.. etc. > > I tried ''xm vcpu-set 0 1'' to take the dom0 down to one virtual cpu.. > but this didn''t seem to have any affect. > > I don''t mind if the generation of new domU''s takes 30 minutes, as long > the performance of my other domU''s are not affected greatly. Any > hints on how to fix this problem? >How are your disks set up? RAID 0, 1, 5, 10, or 50? What speed disks? Any number of things could be causing this, but I would check your storage situation first, as a failing disk could be very problematic. William _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
n8&abby
2008-Apr-21 22:55 UTC
Re: [Xen-users] creating domU''s consumes 100% of system resources
Thanks for your response William,
I have 8x 750GB 7200rpm SAMSUNG HD753LJ SATA Drives using a 3ware 9650SE
hardware raid controller. These 8 disks are configured using raid 5 and are
collected together into a single "unit" of about 4.77TB. The card
then
presents this array to look like 2 disks. Dom0 (Debian Etch amd64) is
installed on the 3GB /dev/sda and is limited to 1024MB ram. The other
15Gigs of memory on the server are for the domU's. The server has 1x quad
core amd64 2Ghz processor.
# fdisk -l /dev/sda
Disk /dev/sda: 3221 MB, 3221224960 bytes
255 heads, 63 sectors/track, 391 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 329 391 506047+ 5 Extended
/dev/sda2 * 1 328 2634628+ 83 Linux
/dev/sda5 330 391 498015 82 Linux swap / Solaris
Partition table entries are not in disk order
Data and domU's are installed on an LVM volume on /dev/sdb
# fdisk -l /dev/sdb
Disk /dev/sdb: 5246.6 GB, 5246699962880 bytes
255 heads, 63 sectors/track, 637874 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table
The 3ware raid card comes with a great web based interface and all drives
check out great.
Here are results from bonnie while only one domU was running. This domU was
serving a couple bittorents at about 250kB/s before running the bonnie
test. While running this test, the domU machine became almost completely
unresponsive and was only able to serve at a speed of about 10kB/s.
# bonnie++ -s 2048 -d /mnt/domU/tmp -u root
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
dom0 2G 5065 13 5095 1 3668 0 32225 74 205758 3
389.5 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++
+++
Here is the same test without any running domU's.
# bonnie++ -s 2048 -d /mnt/seed/tmp -u root
Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec
%CP
sbh2 2G 5320 13 5365 1 3931 0 34210 70 247347 4
513.2 0
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec
%CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++
+++
I would appreciate ideas anyone has about this.
Thanks,
-=nathan
How are your disks set up? RAID 0, 1, 5, 10, or 50? What speed
disks?>
> Any number of things could be causing this, but I would check your
> storage situation first, as a failing disk could be very problematic.
>
> William
>
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users
jim burns
2008-Apr-21 23:34 UTC
Re: [Xen-users] creating domU''s consumes 100% of system resources
On Sunday April 20 2008 03:42:47 pm n8&abby wrote:> I tried ''xm vcpu-set 0 1'' to take the dom0 down to one virtual cpu.. but > this didn''t seem to have any affect.You are setting the number of vcpus, but they are still allowed to raom between al available pcpus. Try xm vcpu-pin. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
n8&abby
2008-Apr-22 03:22 UTC
Re: [Xen-users] creating domU''s consumes 100% of system resources
Great, that really helped. And I think I also found the main problem.. Write caching was off on my raid controller from the last time it was rebuilt.. Of course turning this on made all the difference. Thanks, -=nathan On Mon, Apr 21, 2008 at 6:34 PM, jim burns <jim_burn@bellsouth.net> wrote:> On Sunday April 20 2008 03:42:47 pm n8&abby wrote: > > I tried ''xm vcpu-set 0 1'' to take the dom0 down to one virtual cpu.. but > > this didn''t seem to have any affect. > > You are setting the number of vcpus, but they are still allowed to raom > between al available pcpus. Try xm vcpu-pin. > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Nico Kadel-Garcia
2008-Apr-22 07:49 UTC
Re: [Xen-users] creating domU''s consumes 100% of system resources
n8&abby wrote:> Hello, > > I have noticed that when I use xen-create-image to generate an domU, > the whole server (dom0 and domU''s) basically hangs until it is > finished. This happens primarily during the creation of the ext3 > filesystem on an LVM partition.> This creation of the file system can take up to 4 or 5 minutes. At > which point, any other domU''s are basically paused... tcp connections > time out.. etc. > > I tried ''xm vcpu-set 0 1'' to take the dom0 down to one virtual cpu.. > but this didn''t seem to have any affect. > > I don''t mind if the generation of new domU''s takes 30 minutes, as long > the performance of my other domU''s are not affected greatly. Any > hints on how to fix this problem?Pre-create the file systems and partitions on the disk images. Write a kickstart image, or manage the creation tool, to assume the disk partitions already exist. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users