Hi, in a previously well behaving xen setup we are suddenly experiencing massive loss of disk performance. Setup: two identical machines (dell, 2 quadcore cpus each, 8 GB RAM each), attached to *one* SAS storagebox (Raid5 SAS disks), CentOS 5, Xen 3.04 running three domUs each. Storage partitions are exposed as block-devices to the guests. Filesystem is ext3. Since a domU running a rather heavily loaded mailserver (many small files, many reads/writes) has been added, io performance has deteriorated significantly. In dom0 about 100 MB/s can be achieved, in domU only about 50 MB/s. This looks even worse when there are massive concurrent / parallel io jobs, then throughput goes down to 20 MB/s. During all this the cpus are mostly idle... Any ideas? I know some performance degradation is to be expected but shurely not on this scale? Thanks, Andrej _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Are the disk I/O numbers quoted below aggregate for all machines, or just for the mailserver. -- -- Tom Mornini, CTO -- Engine Yard, Ruby on Rails Hosting -- Support, Scalability, Reliability -- (866) 518-YARD (9273) On Jun 20, 2007, at 10:49 PM, Andrej Radonic wrote:> Hi, > > in a previously well behaving xen setup we are suddenly > experiencing massive loss of disk performance. > > Setup: two identical machines (dell, 2 quadcore cpus each, 8 GB RAM > each), attached to *one* SAS storagebox (Raid5 SAS disks), CentOS > 5, Xen 3.04 running three domUs each. > Storage partitions are exposed as block-devices to the guests. > Filesystem is ext3. > > Since a domU running a rather heavily loaded mailserver (many small > files, many reads/writes) has been added, io performance has > deteriorated significantly. > > In dom0 about 100 MB/s can be achieved, in domU only about 50 MB/s. > This looks even worse when there are massive concurrent / parallel > io jobs, then throughput goes down to 20 MB/s. > > During all this the cpus are mostly idle... > > Any ideas? I know some performance degradation is to be expected > but shurely not on this scale? > Thanks, > Andrej > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> On Jun 20, 2007, at 10:49 PM, Andrej Radonic wrote: > >> Hi, >> >> in a previously well behaving xen setup we are suddenly experiencing >> massive loss of disk performance. >> >> Setup: two identical machines (dell, 2 quadcore cpus each, 8 GB RAM each), >> attached to *one* SAS storagebox (Raid5 SAS disks), CentOS 5, Xen 3.04 >> running three domUs each. >> Storage partitions are exposed as block-devices to the guests. Filesystem >> is ext3.raid5 ... so you''ve got the mailserver running on multiple hard drives?>> Since a domU running a rather heavily loaded mailserver (many small files, >> many reads/writes) has been added, io performance has deteriorated >> significantly. >> >> In dom0 about 100 MB/s can be achieved, in domU only about 50 MB/s. This >> looks even worse when there are massive concurrent / parallel io jobs, then >> throughput goes down to 20 MB/s.That is to be expected. If the drives are busy seeking all over the place, you are _not_ going to get high speed transfers off of them. I don''t see anything xen specific about this. Although it is more likely to show up when you put multiple tasks on a single set of hardware (and Xen encourages that).>> During all this the cpus are mostly idle... >> >> Any ideas? I know some performance degradation is to be expected but >> shurely not on this scale?Try it on a single drive/cpu machine... fire up something like bonnie to get it to do multiple parallel I/Os while timing a big dd run or something in the background. Maxing out your disk drives is the easiest way to drive performance into the ground :(. You can only get 50-100 MB/s transfers off a drive if the data is being read sequentially off the drive with an absolute minimum of head seeks. As soon as something else starts moving the drive head(s) around, your sustained transfer rate is going to fall dramatically... it comes down to how many seeks per second your drives can do. Building up a big RAID array improves the system''s ability to provide high throughput but is still limited by the abilities of the drives it is built from.... Mail servers, like news servers, "eat" drives.... If space allows it, you may be better off buying more smaller drives than fewer larger ones. With more drives you have more spindles and the ability to do more seeks per second (and higher combined throughput too). I try to isolate the drives for my domUs... often dedicating one spindle to one domU, and using drbd for raid1. It would be nicer if this wasn''t needed (since it is the opposite of virtualization), but AFAIK the one resource that is hard to schedule/limit/allocate/prioritize is disk access/bandwidth. -Tom _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tom,> Are the disk I/O numbers quoted below aggregate for all machines, or > just for the mailserver.so far we measured separately in dom0 and in the mail-domU. The 20 MB/s happen as soon as there is concurrent io from two or more domUs. Andrej> -- -- Tom Mornini, CTO -- Engine Yard, Ruby on Rails Hosting -- Support, Scalability, Reliability -- (866) 518-YARD (9273) On Jun 20, 2007, at 10:49 PM, Andrej Radonic wrote: >> > Hi, >> > >> > in a previously well behaving xen setup we are suddenly >> > experiencing massive loss of disk performance. >> > >> > Setup: two identical machines (dell, 2 quadcore cpus each, 8 GB RAM >> > each), attached to *one* SAS storagebox (Raid5 SAS disks), CentOS >> > 5, Xen 3.04 running three domUs each. >> > Storage partitions are exposed as block-devices to the guests. >> > Filesystem is ext3. >> > >> > Since a domU running a rather heavily loaded mailserver (many small >> > files, many reads/writes) has been added, io performance has >> > deteriorated significantly. >> > >> > In dom0 about 100 MB/s can be achieved, in domU only about 50 MB/s. >> > This looks even worse when there are massive concurrent / parallel >> > io jobs, then throughput goes down to 20 MB/s. >> > >> > During all this the cpus are mostly idle... >> > >> > Any ideas? I know some performance degradation is to be expected >> > but shurely not on this scale? >> > Thanks, >> > Andrej >> > >> > >> > _______________________________________________ >> > Xen-users mailing list >> > Xen-users@lists.xensource.com >> > http://lists.xensource.com/xen-usersinterSales AG Internet Commerce Andrej Radonic Vorstand _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Andrej Radonic > Sent: 22 June 2007 08:57 > To: xen-users@lists.xensource.com > Subject: Re: [Xen-users] Poor disk io performance in domUs > > Tom, > > > Are the disk I/O numbers quoted below aggregate for all > machines, or > > just for the mailserver. > > so far we measured separately in dom0 and in the mail-domU. > The 20 MB/s happen as soon as there is concurrent io from two > or more domUs.That would be a consequence of the two domains causing more head-movement on the drive(s) than a single domain, so you get more overhead. So if you get 50MB/s in a single domain, you don''t get 25MB/s in two parallel domains - you get a bit less. That''s just what I''d expect in this situation. [This is because the "disk" for each virtual machine is in a different part of the disk, so each time the first domain acesses the disk, it needs a (big) move of the read/write head, and then another big move when the second domain accesses it''s part of the disk]. -- Mats> > Andrej > > > -- -- Tom Mornini, CTO -- Engine Yard, Ruby on Rails > Hosting -- Support, Scalability, Reliability -- (866) > 518-YARD (9273) On Jun 20, 2007, at 10:49 PM, Andrej Radonic wrote: > >> > Hi, > >> > > >> > in a previously well behaving xen setup we are suddenly > >> > experiencing massive loss of disk performance. > >> > > >> > Setup: two identical machines (dell, 2 quadcore cpus > each, 8 GB RAM > >> > each), attached to *one* SAS storagebox (Raid5 SAS > disks), CentOS > >> > 5, Xen 3.04 running three domUs each. > >> > Storage partitions are exposed as block-devices to the guests. > >> > Filesystem is ext3. > >> > > >> > Since a domU running a rather heavily loaded mailserver > (many small > >> > files, many reads/writes) has been added, io performance has > >> > deteriorated significantly. > >> > > >> > In dom0 about 100 MB/s can be achieved, in domU only > about 50 MB/s. > >> > This looks even worse when there are massive concurrent > / parallel > >> > io jobs, then throughput goes down to 20 MB/s. > >> > > >> > During all this the cpus are mostly idle... > >> > > >> > Any ideas? I know some performance degradation is to be > expected > >> > but shurely not on this scale? > >> > Thanks, > >> > Andrej > >> > > >> > > >> > _______________________________________________ > >> > Xen-users mailing list > >> > Xen-users@lists.xensource.com > >> > http://lists.xensource.com/xen-users > > > > interSales AG Internet Commerce > Andrej Radonic > Vorstand > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Hi Mats,>> so far we measured separately in dom0 and in the mail-domU. >> > The 20 MB/s happen as soon as there is concurrent io from two >> > or more domUs. > > That would be a consequence of the two domains causing more > head-movement on the drive(s) than a single domain, so you get more > overhead. So if you get 50MB/s in a single domain, you don''t get 25MB/s > in two parallel domains - you get a bit less. That''s just what I''d > expect in this situation. [This is because the "disk" for each virtual > machine is in a different part of the disk, so each time the first > domain acesses the disk, it needs a (big) move of the read/write head, > and then another big move when the second domain accesses it''s part of > the disk].OK, I definitely agree, but I am still not sure about the extent of throughput degradation. I remeasured the setup. This is what I get using dd to write a large file to the disk: dd simultaneously in both dom0 = 170 MB/s dd simultaneously in two domU = 34 MB/s dd in a single dom0 = 120 MB/s Would you really say that one-third of io performance is what is to be expected? Thanks for your cooperation. Andrej interSales AG, Cologne/Germany _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Andrej Radonic > Sent: 22 June 2007 10:41 > To: xen-users@lists.xensource.com > Subject: RE: [Xen-users] Poor disk io performance in domUs > > Hi Mats, > > >> so far we measured separately in dom0 and in the mail-domU. > >> > The 20 MB/s happen as soon as there is concurrent io from two > >> > or more domUs. > > > > That would be a consequence of the two domains causing more > > head-movement on the drive(s) than a single domain, so you get more > > overhead. So if you get 50MB/s in a single domain, you > don''t get 25MB/s > > in two parallel domains - you get a bit less. That''s just what I''d > > expect in this situation. [This is because the "disk" for > each virtual > > machine is in a different part of the disk, so each time the first > > domain acesses the disk, it needs a (big) move of the > read/write head, > > and then another big move when the second domain accesses > it''s part of > > the disk]. > > OK, I definitely agree, but I am still not sure about the extent of > throughput degradation. > > I remeasured the setup. This is what I get using dd to write a large > file to the disk: > > dd simultaneously in both dom0 = 170 MB/sI take it you mean "two parallel ''dd'' commands at the same time"? That would still write to the same portion of disk (unless you specifically choose different partitions?) If it''s the same partition, then, althouth there is some head movement involved, there will be less head movement than two domains that start 10GB apart on a disk. Also, the filesystem driver in the Dom0 can re-arrange the disk accesses to make fewer movements.> dd simultaneously in two domU = 34 MB/sI take it this means two different DomU doing "dd"? Is that 34 MB/s "total" (i.e. 17MB/s per domain) or per domain (68 MB/s total)?> dd in a single dom0 = 120 MB/sSo this is "better" than half of 170 MB/s, agreed? So even in a single domain, running parallel sessions reduce the performance. Note also that there is overhead in transferring from Dom0 to DomU and the other way around. Even if this is fairly small, it''s not possible to ignore this.> > Would you really say that one-third of io performance is what > is to be > expected?It''s difficult to say - I''m just trying to give you some explanation to what you''re seeing. -- Mats> > Thanks for your cooperation. > Andrej > interSales AG, Cologne/Germany > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Lets say we have 4 domU heavy utilized. What about providing physical disk to every domU to avoid sharing single disk among all guests? Will it help to have didecated disk per domU? Peter> 2007/6/22, Petersson, Mats <Mats.Petersson@amd.com>: > > > > > > > -----Original Message----- > > > From: xen-users-bounces@lists.xensource.com > > > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > > > Andrej Radonic > > > Sent: 22 June 2007 10:41 > > > To: xen-users@lists.xensource.com > > > Subject: RE: [Xen-users] Poor disk io performance in domUs > > > > > > Hi Mats, > > > > > > >> so far we measured separately in dom0 and in the mail-domU. > > > >> > The 20 MB/s happen as soon as there is concurrent io from two > > > >> > or more domUs. > > > > > > > > That would be a consequence of the two domains causing more > > > > head-movement on the drive(s) than a single domain, so you get more > > > > overhead. So if you get 50MB/s in a single domain, you > > > don''t get 25MB/s > > > > in two parallel domains - you get a bit less. That''s just what I''d > > > > expect in this situation. [This is because the "disk" for > > > each virtual > > > > machine is in a different part of the disk, so each time the first > > > > domain acesses the disk, it needs a (big) move of the > > > read/write head, > > > > and then another big move when the second domain accesses > > > it''s part of > > > > the disk]. > > > > > > OK, I definitely agree, but I am still not sure about the extent of > > > throughput degradation. > > > > > > I remeasured the setup. This is what I get using dd to write a large > > > file to the disk: > > > > > > dd simultaneously in both dom0 = 170 MB/s > > I take it you mean "two parallel ''dd'' commands at the same time"? That > > would still write to the same portion of disk (unless you specifically > > choose different partitions?) > > > > If it''s the same partition, then, althouth there is some head movement > > involved, there will be less head movement than two domains that start > > 10GB apart on a disk. Also, the filesystem driver in the Dom0 can > > re-arrange the disk accesses to make fewer movements. > > > > > dd simultaneously in two domU = 34 MB/s > > I take it this means two different DomU doing "dd"? > > Is that 34 MB/s "total" (i.e. 17MB/s per domain) or per domain (68 MB/s > > total)? > > > > > dd in a single dom0 = 120 MB/s > > So this is "better" than half of 170 MB/s, agreed? So even in a single > > domain, running parallel sessions reduce the performance. > > > > Note also that there is overhead in transferring from Dom0 to DomU and > > the other way around. Even if this is fairly small, it''s not possible to > > ignore this. > > > > > > > > Would you really say that one-third of io performance is what > > > is to be > > > expected? > > > > It''s difficult to say - I''m just trying to give you some explanation to > > what you''re seeing. > > > > -- > > Mats > > > > > > Thanks for your cooperation. > > > Andrej > > > interSales AG, Cologne/Germany > > > > > > _______________________________________________ > > > Xen-users mailing list > > > Xen-users@lists.xensource.com > > > http://lists.xensource.com/xen-users > > > > > > > > > > > > > > > > > _______________________________________________ > > Xen-users mailing list > > Xen-users@lists.xensource.com > > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Peter Braun wrote:> Lets say we have 4 domU heavy utilized. > > What about providing physical disk to every domU to avoid sharing > single disk among all guests? > > Will it help to have didecated disk per domU? > > > PeterActual partitions rather than using image files on Dom0 is *always* a performance benefit. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
On Sat, 23 Jun 2007, Nico Kadel-Garcia wrote:> Peter Braun wrote: >> Lets say we have 4 domU heavy utilized. >> >> What about providing physical disk to every domU to avoid sharing >> single disk among all guests? >> >> Will it help to have didecated disk per domU?Having the domUs on separate disks will greatly help isolate them (e.g. reduce the effect they have on each other). The only bottleneck they would share is getting the dom0 to process their I/O requests (e.g. CPU time) ... but having one domU doing a pile of scattered reads should not have much affect on another domU trying to read in a large contiguous file (once you put them on separate drives).>> >> >> Peter > Actual partitions rather than using image files on Dom0 is *always* a > performance benefit.how is that relevent to the topic at hand? -Tom _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
> -----Original Message----- > From: xen-users-bounces@lists.xensource.com > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > Peter Braun > Sent: 23 June 2007 11:52 > To: Xen users mailing list > Subject: Re: [Xen-users] Poor disk io performance in domUs > > Lets say we have 4 domU heavy utilized. > > What about providing physical disk to every domU to avoid sharing > single disk among all guests? > > Will it help to have didecated disk per domU?Yes, that will help to some extent. You still have to consider that Dom0 is handling all the disk IO. -- Mats> > > Peter > > > > 2007/6/22, Petersson, Mats <Mats.Petersson@amd.com>: > > > > > > > > > > -----Original Message----- > > > > From: xen-users-bounces@lists.xensource.com > > > > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of > > > > Andrej Radonic > > > > Sent: 22 June 2007 10:41 > > > > To: xen-users@lists.xensource.com > > > > Subject: RE: [Xen-users] Poor disk io performance in domUs > > > > > > > > Hi Mats, > > > > > > > > >> so far we measured separately in dom0 and in the mail-domU. > > > > >> > The 20 MB/s happen as soon as there is concurrent > io from two > > > > >> > or more domUs. > > > > > > > > > > That would be a consequence of the two domains causing more > > > > > head-movement on the drive(s) than a single domain, > so you get more > > > > > overhead. So if you get 50MB/s in a single domain, you > > > > don''t get 25MB/s > > > > > in two parallel domains - you get a bit less. That''s > just what I''d > > > > > expect in this situation. [This is because the "disk" for > > > > each virtual > > > > > machine is in a different part of the disk, so each > time the first > > > > > domain acesses the disk, it needs a (big) move of the > > > > read/write head, > > > > > and then another big move when the second domain accesses > > > > it''s part of > > > > > the disk]. > > > > > > > > OK, I definitely agree, but I am still not sure about > the extent of > > > > throughput degradation. > > > > > > > > I remeasured the setup. This is what I get using dd to > write a large > > > > file to the disk: > > > > > > > > dd simultaneously in both dom0 = 170 MB/s > > > I take it you mean "two parallel ''dd'' commands at the > same time"? That > > > would still write to the same portion of disk (unless you > specifically > > > choose different partitions?) > > > > > > If it''s the same partition, then, althouth there is some > head movement > > > involved, there will be less head movement than two > domains that start > > > 10GB apart on a disk. Also, the filesystem driver in the Dom0 can > > > re-arrange the disk accesses to make fewer movements. > > > > > > > dd simultaneously in two domU = 34 MB/s > > > I take it this means two different DomU doing "dd"? > > > Is that 34 MB/s "total" (i.e. 17MB/s per domain) or per > domain (68 MB/s > > > total)? > > > > > > > dd in a single dom0 = 120 MB/s > > > So this is "better" than half of 170 MB/s, agreed? So > even in a single > > > domain, running parallel sessions reduce the performance. > > > > > > Note also that there is overhead in transferring from > Dom0 to DomU and > > > the other way around. Even if this is fairly small, it''s > not possible to > > > ignore this. > > > > > > > > > > > Would you really say that one-third of io performance is what > > > > is to be > > > > expected? > > > > > > It''s difficult to say - I''m just trying to give you some > explanation to > > > what you''re seeing. > > > > > > -- > > > Mats > > > > > > > > Thanks for your cooperation. > > > > Andrej > > > > interSales AG, Cologne/Germany > > > > > > > > _______________________________________________ > > > > Xen-users mailing list > > > > Xen-users@lists.xensource.com > > > > http://lists.xensource.com/xen-users > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > Xen-users mailing list > > > Xen-users@lists.xensource.com > > > http://lists.xensource.com/xen-users > > > > > > > _______________________________________________ > Xen-users mailing list > Xen-users@lists.xensource.com > http://lists.xensource.com/xen-users > > >_______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users