Xen users - Jun 2007 - Subject: RE: Poor disk io performance in domUs

Mats,
>> dd simultaneously in both dom0 = 170 MB/s
> I take it you mean "two parallel ''dd'' commands at
the same time"? That
> would still write to the same portion of disk (unless you specifically
> choose different partitions?)
it''s different partitions - one dedicated partition for each domU. The 
partition are created as "virtual" block devices with the dell storage
box manager.
>> dd simultaneously in two domU = 34 MB/s
> I take it this means two different DomU doing "dd"? 
> Is that 34 MB/s "total" (i.e. 17MB/s per domain) or per domain
(68 MB/s
> total)? 
sorry, good you asked: it''s the total, i.e. 17MB/s per domain! I guess 
you are getting the picture now as to my feelings... ;-)

Kind regards,
Andrej

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 6/22/07, Andrej Radonic <rado@intersales.de>
wrote:
> Mats,
>
> >> dd simultaneously in both dom0 = 170 MB/s
> > I take it you mean "two parallel ''dd'' commands
at the same time"? That
> > would still write to the same portion of disk (unless you specifically
> > choose different partitions?)
>
> it''s different partitions - one dedicated partition for each domU.
The
> partition are created as "virtual" block devices with the dell
storage
> box manager.
>
> >> dd simultaneously in two domU = 34 MB/s
> > I take it this means two different DomU doing "dd"?
> > Is that 34 MB/s "total" (i.e. 17MB/s per domain) or per
domain (68 MB/s
> > total)?
>
> sorry, good you asked: it''s the total, i.e. 17MB/s per domain! I
guess
> you are getting the picture now as to my feelings... ;-)
>
Yeah I''ve experienced some interesting things with very good I/O
performance and xen not handling it very well with the domU''s. Since
there''s a little kernel process running on the dom0 for each virtual
block device exported to domU''s, which does translation mostly,
I''ve
found that the more domU''s you bring up all doing I/O the dom0
processes tend to do just as much work as all the dd operations of all
the domU''s combined. So if you have 6 domU''s all doing about
15% using
dd''s your dom0 is going to be pushing 100% of its cpu usage and going
to be doing a crap load of work and the I/O performance in the domU''s
will be failing. So it does pay to make sure your dom0 can handle
translating everything (note this should go away with the IOMMU
support, I would hope).

Also I''d check what the dell virtual block manager can do, try
creating virtual block devices then try dd''ing to them in parallel in
the dom0 it might only be that the dell virtual block device manager
can handle 60Mb/s total to any of the block devices it creates.

I''ve got experience with the HP virtual block device thingy and you
can actually specify to only use two of the disks to raid0/1 them
depending on what you want to do. Then export the raided disks to the
kernel. Since we have 6 drives that gives at most 3 block devices
getting to disk somehow to test if the pipe between the disk and OS
can''t handle the data. I would suggest doing something similar.

Thanks,
- David Brown

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com 
> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of 
> David Brown
> Sent: 22 June 2007 16:16
> To: Andrej Radonic
> Cc: xen-users@lists.xensource.com
> Subject: Re: Subject: RE: [Xen-users] Poor disk io 
> performance in domUs
> 
> On 6/22/07, Andrej Radonic <rado@intersales.de> wrote:
> > Mats,
> >
> > >> dd simultaneously in both dom0 = 170 MB/s
> > > I take it you mean "two parallel ''dd''
commands at the
> same time"? That
> > > would still write to the same portion of disk (unless you 
> specifically
> > > choose different partitions?)
> >
> > it''s different partitions - one dedicated partition for 
> each domU. The
> > partition are created as "virtual" block devices with the 
> dell storage
> > box manager.
> >
> > >> dd simultaneously in two domU = 34 MB/s
> > > I take it this means two different DomU doing "dd"?
> > > Is that 34 MB/s "total" (i.e. 17MB/s per domain) or per
> domain (68 MB/s
> > > total)?
> >
> > sorry, good you asked: it''s the total, i.e. 17MB/s per 
> domain! I guess
> > you are getting the picture now as to my feelings... ;-)
> >
> 
> Yeah I''ve experienced some interesting things with very good I/O
> performance and xen not handling it very well with the domU''s.
Since
> there''s a little kernel process running on the dom0 for each
virtual
> block device exported to domU''s, which does translation mostly,
I''ve
> found that the more domU''s you bring up all doing I/O the dom0
> processes tend to do just as much work as all the dd operations of all
> the domU''s combined. So if you have 6 domU''s all doing
about 15% using
> dd''s your dom0 is going to be pushing 100% of its cpu usage and
going
> to be doing a crap load of work and the I/O performance in the
domU''s
> will be failing. So it does pay to make sure your dom0 can handle
> translating everything (note this should go away with the IOMMU
> support, I would hope).
Actually, if you expect IOMMU to solve the problem, you can do the same
(with slightly less security, admittedly) in Para-virtual domains today
- since IOMMU can only translate and protect on a per-device level, so
you need to have one device per domain. 

So if you have a disk-controller with disk for each domain, you could do
that today. Same with network controllers [there are even some network
controllers which are "multihead", meaning that they present
themselves
as multiple individual devices, even though it all goes onto a single
network connection].

The other point that immediately comes to mind here is that the Dom0
should definitely have it''s own CPU if you''re doing a lot of
disk/network IO through it. 

--
Mats 



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> Actually, if you expect IOMMU to solve the problem, you can do the same
> (with slightly less security, admittedly) in Para-virtual domains today
> - since IOMMU can only translate and protect on a per-device level, so
> you need to have one device per domain.
Well yeah the same thing can be accomplished today but there''s that
little kernel process that has to do the mapping in software and that
takes away cpu time from doing other work.

Ideally, you would bring up a box with say 6+ domains on it plus a
dom0 and no matter what you did with the domains it wouldn''t adversely
affect the other domains, even dom0. So this theoretical box would
probably have 7 network devices (with IOMMU) mapped to the various
domains in hardware so no software has to do the mapping to the
different address spaces. Similarly with disk devices, there would
have to be 7 scsi devices so that each could be mapped the the various
domains. Then there would be only one box and it could really look
like 7 different machines and have the same performance as it would
normally.
> So if you have a disk-controller with disk for each domain, you could do
> that today. Same with network controllers [there are even some network
> controllers which are "multihead", meaning that they present
themselves
> as multiple individual devices, even though it all goes onto a single
> network connection].
>
> The other point that immediately comes to mind here is that the Dom0
> should definitely have it''s own CPU if you''re doing a lot
of
> disk/network IO through it.
Well, I should have said hardware IOMMU, but you''re right, its on a
device based level which doesn''t help I/O so much since that''s
not
based on disks. Really what we need is a scsi based IOMMU controler of
some kind that can do the translation on the device itself per disk,
now that''d be cool :)

- David Brown

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: David Brown [mailto:dmlb2000@gmail.com] 
> Sent: 22 June 2007 17:00
> To: Petersson, Mats
> Cc: Andrej Radonic; xen-users@lists.xensource.com
> Subject: Re: Subject: RE: [Xen-users] Poor disk io 
> performance in domUs
> 
> > Actually, if you expect IOMMU to solve the problem, you can 
> do the same
> > (with slightly less security, admittedly) in Para-virtual 
> domains today
> > - since IOMMU can only translate and protect on a 
> per-device level, so
> > you need to have one device per domain.
> 
> Well yeah the same thing can be accomplished today but there''s
that
> little kernel process that has to do the mapping in software and that
> takes away cpu time from doing other work.
Assuming we''re talking about Para-virtual domains, the guest already
knows the physical address, so there''s no need to do any extra work
here. [this assumes the packet of data doesn''t have to be copied into a
contiguous buffer, that''s a differrent matter - but it''s quite
likely
that it doesn''t actually have to be - particularly not for small
packets
of data such as network traffic]. 
> 
> Ideally, you would bring up a box with say 6+ domains on it plus a
> dom0 and no matter what you did with the domains it wouldn''t
adversely
> affect the other domains, even dom0. So this theoretical box would
> probably have 7 network devices (with IOMMU) mapped to the various
> domains in hardware so no software has to do the mapping to the
> different address spaces. Similarly with disk devices, there would
> have to be 7 scsi devices so that each could be mapped the the various
> domains. Then there would be only one box and it could really look
> like 7 different machines and have the same performance as it would
> normally.
IOMMU on Para-virtual domains isn''t going to give you any advantage
other than proper protection. For HVM domains, it''s the key that
unlocks
guest-access to hardware (and of course giving protection).
> 
> > So if you have a disk-controller with disk for each domain, 
> you could do
> > that today. Same with network controllers [there are even 
> some network
> > controllers which are "multihead", meaning that they 
> present themselves
> > as multiple individual devices, even though it all goes 
> onto a single
> > network connection].
> >
> > The other point that immediately comes to mind here is that the Dom0
> > should definitely have it''s own CPU if you''re doing
a lot of
> > disk/network IO through it.
> 
> Well, I should have said hardware IOMMU, but you''re right, its on
a
> device based level which doesn''t help I/O so much since
that''s not
> based on disks. Really what we need is a scsi based IOMMU controler of
> some kind that can do the translation on the device itself per disk,
> now that''d be cool :)
It''s not beyond the realms of possibility that future disk controllers
will have "multi-head" capabilities too - so that you could have a
single SCSI-card in the machine and 4 disks, but to the software, it
would appear like 4 SCSI controllers with one disk each. 

Note this is PURE speculation - I have no idea if anyone is working on
something like that. I only know of network cards like that. But froma
conceptual standpoint, there''s no real problem doing that. 

--
Mats
> 
> - David Brown
> 
> 
> 


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users