Xen users - May 2006 - poor harddisk performance HVM domain (Windows 2003 guest)

I am now running into another problem, the networking one has been solved 
(thanks Dave). As I posted to my previous thread, the poor networking 
performance is most probably caused by the harddisk. Running some 
benchmarks gives a max of 10MB/s read throughput, while write activities 
don''t go beyond 5MB/s. The guest''s write cache on the disk is
enabled.

I tried both image and physical disk partition, no difference. The CPU and 
memory benchmarks on the other hand, are close to native. So the bottleneck 
is definitely the (virtual) harddisk.

Dave (and ofcourse others), what is your experience, do you have better 
read/write results?

- Joost

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi,

I''d like to see some confirmation on this one, since I''m
experimenting with
this for days and being unable to get acceptable transfer speeds. I thought 
such poor performance should not happen with VT? 

It even gets worse when installing and using Ubuntu HVM, can''t enable
DMA
for the QEMU harddisk, resulting in a very slow +-3.5MB/s read. Isn''t
there
any way to resolve this?

 - Joost

On Thursday 4 May 2006 16:25, Joost van den Broek wrote:
> I am now running into another problem, the networking one has been solved
> (thanks Dave). As I posted to my previous thread, the poor networking
> performance is most probably caused by the harddisk. Running some
> benchmarks gives a max of 10MB/s read throughput, while write activities
> don''t go beyond 5MB/s. The guest''s write cache on the
disk is enabled.
>
> I tried both image and physical disk partition, no difference. The CPU
> and memory benchmarks on the other hand, are close to native. So the
> bottleneck is definitely the (virtual) harddisk.
>
> Dave (and ofcourse others), what is your experience, do you have better
> read/write results?
>
> - Joost
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sunday 07 May 2006 8:37 am, Joost van den Broek
wrote:
> It even gets worse when installing and using Ubuntu HVM, can''t
enable DMA
> for the QEMU harddisk, resulting in a very slow +-3.5MB/s read.
Isn''t there
> any way to resolve this?
qemu''s HD emulation is very primitive; no DMA, no LBA48.  but it
shouldn''t be
such slow, since the real device (in dom0) does have DMA.  of course, the 
best solution is still in the future, when a HVM domain would accept special 
paravirtualized drivers for HD and network.  those will be needed to get 
better performance for windows.

PD, the newest version of qemu does have both DMA and LBA48, but i think 
nobody has bothered porting it to Xen

-- 
Javier


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Sunday 07 May 2006 6:11 pm, Andy Clayton wrote:
> If there is interest in a patch with LBA48 stuff ported over I can post
> it.  I am not sure about DMA, but LBA48 falls in easily and seems to
> work fine.
i would vote for this.  this is one reason (not the only, unfortunately) why 
i''m still deploying VMWare. hoping to move all to Xen someday.  having
LBA48
would be one step forward for this.

(still, it would be even better to have paravirtualized HD drivers for 
windows)

-- 
Javier


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Monday 8 May 2006 04:33, Javier Guerra wrote:
> On Sunday 07 May 2006 6:11 pm, Andy Clayton wrote:
> > If there is interest in a patch with LBA48 stuff ported over I can
post
> > it.  I am not sure about DMA, but LBA48 falls in easily and seems to
> > work fine.
>
> i would vote for this.  this is one reason (not the only, unfortunately)
> why i''m still deploying VMWare. hoping to move all to Xen someday.
> having LBA48 would be one step forward for this.
>
> (still, it would be even better to have paravirtualized HD drivers for
> windows)
Indeed, this would be very nice. So until there are paravirtualized drivers, 
the performance won''t get as good as VMware? I''m currently
installing a new
server and yet not decided to go with Xen or Vservers + VMware Server 
(beta). Last one seems to have very good performance, but I''d rather
use
Xen instead.

Though, the Windows (2003 Web Server) will not be very busy (for now) so it 
might be worth using it for the time being, until stuff gets optimized. My 
only concern is if the current HVM image will work with upcoming features, 
such as paravirtualized drivers..

- Joost

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> -----Original Message-----
> From: xen-users-bounces@lists.xensource.com 
> [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of 
> Joost van den Broek
> Sent: 07 May 2006 14:38
> To: xen-users@lists.xensource.com
> Subject: Re: [Xen-users] poor harddisk performance HVM domain 
> (Windows 2003 guest)
> 
> Hi,
> 
> I''d like to see some confirmation on this one, since I''m 
> experimenting with this for days and being unable to get 
> acceptable transfer speeds. I thought such poor performance 
> should not happen with VT? 
> 
> It even gets worse when installing and using Ubuntu HVM, 
> can''t enable DMA for the QEMU harddisk, resulting in a very 
> slow +-3.5MB/s read. Isn''t there any way to resolve this?
> 
>  - Joost
> 
As discussed before, the QEMU is pretty primitive, but further on that,
the HVM solution (or any other virtualized solution aside from giivng
the virtual machine direct access to the real hardware, which in 90% of
cases with Hard-disks isn''t an option - you''d need as many HD
controllers as DomU''s (not just disks, but actual physical PCI
controller devices) - and that is assuming the device operates in
non-DMA mode, or otherwise we''d also need to have a IOMMU allowing the
guest OS''s "physical" address into "real physical"
address [which we
fake to the GUEST os so that it believes memory starts at 0 and goes to,
say, 256MB, when it REALLY starts at 256MB and goes to 512MB, for
example]. So a DMA operation where the OS says "send bytes at address
123456 to hard-disk", would have to be translated to "send bytes as
address 256MB + 123456 to hard-disk". There''s three ways to solve
this:
1. IOMMU - this works without any modification of the guest-OS. 
2. Para-virtualized OS - the base-OS is modified so that when it''s
asking for a physical address, it knows that there are two kinds of
physical address: Guest physical and "real" physical (machine
physical).

3. Para-virtualized driver - although the main OS isn''t modified, the
guest''s driver for the hard-disk would have some sort of modifications
to allow it to understand the actual double-layering of physical
addresses. 

Option one is the best choice of these - but still relies on each guest
having it''s own hardware device (controller + disk) [or a multi-port
device that allows several guests to access it and the controller has a
separate "device" for each guest - I''ve heard that there are
network
controllers that allow this at the moment]. 

Option two is fine for Linux (that''s how a Linux DomU built for Xen
works), but OS''s where the source code isn''t easily
accessible, this
doesn''t quite work...

Option three is the only remaining viable option. 

Some further background: Whilst the actual access to the hard-disk is
done pretty well by Dom0 - at least it SHOULD be, or it needs fixing in
Dom0 - the overhead of running the disk access through the two layers of
Hypervisor (HVM code) and QEMU does add some noticable delays - bear in
mind that each sector being accessed in IDE will require writes to the
following registers:
Sector count
Sector number
Cylinder LSB
Cylinder MSB
Drive/Head
Command
[These CAN be combined, but many drivers write at least most of it
individually]
For a read, the driver will wait for an interrupt, for a write, it will
perform further IO reads to detect the drive being ready to accept the
data (QEMU is usually ready after the first poll) followed by a 512-byte
OUTS operation. 

Each one of these accesses cause an intercept, which means that the
guest is halted, the hypervisor invoked and the cause of the intercept
is checked and action taken. In the case of IO operations, this means
sending a message to QEMU, a context switch in the hypervisor to Dom0
(where QEMU runs), qemu interpreting the message and performing the
operation (storing the information about what we want to access on the
hard-disk until we get to the Command bit). 

When the command is complete, it will then perform the actual
disk-read/write (in the form of a fread/fwrite from the image file or
physical disk). Dom0 may well "sleep" (allow another context to run)
whilst it''s waiting for the actual data to arrive from/to the disk.
[And
here, DMA may well be used]

Once the data transfer is complete, an interrupt is issued to the guest
and it''s restarted. The guest then reads the status register (another
intercept) to see if it worked out OK, and then, if it was a read, a
512-byte INS operation. 

There''s a lot of passing information around, and there''s a lot
of
state-saving/restoring when context switching. Just a simple intercept
with no "real work" in the Hypervisor will take, probably, a couple of
thousand cycles to perform - and we have somewhere between 4 and 8 of
those for each disk-operation. We also have full-context switches
between DomU and Dom0 to run QEMU. 

A para-virtualized driver could do the same work in perhaps one set of
context switching and no more than two intercepts - so the task would be
done with much less overhead. 

--
Mats


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

<wishful thinking>

Would a AoE driver for windows do the job too in combination with a AoE
server xen machine?

</wishful thinking>


----- Original Message -----
From: "Joost van den Broek" <joost@seat-ibiza.nl>
To: <xen-users@lists.xensource.com>
Sent: Sunday, May 07, 2006 3:37 PM
Subject: Re: [Xen-users] poor harddisk performance HVM domain (Windows
2003guest)

> Hi,
>
> I''d like to see some confirmation on this one, since I''m
experimenting
with
> this for days and being unable to get acceptable transfer speeds. I
thought
> such poor performance should not happen with VT?
>
> It even gets worse when installing and using Ubuntu HVM, can''t
enable DMA
> for the QEMU harddisk, resulting in a very slow +-3.5MB/s read.
Isn''t
there
> any way to resolve this?
>
>  - Joost
>
> On Thursday 4 May 2006 16:25, Joost van den Broek wrote:
> > I am now running into another problem, the networking one has been
solved
> > (thanks Dave). As I posted to my previous thread, the poor networking
> > performance is most probably caused by the harddisk. Running some
> > benchmarks gives a max of 10MB/s read throughput, while write
activities
> > don''t go beyond 5MB/s. The guest''s write cache on
the disk is enabled.
> >
> > I tried both image and physical disk partition, no difference. The CPU
> > and memory benchmarks on the other hand, are close to native. So the
> > bottleneck is definitely the (virtual) harddisk.
> >
> > Dave (and ofcourse others), what is your experience, do you have
better
> > read/write results?
> >
> > - Joost
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xensource.com
> > http://lists.xensource.com/xen-users
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Monday 08 May 2006 2:54 am, Joost van den Broek
wrote:
> Indeed, this would be very nice. So until there are paravirtualized
> drivers, the performance won''t get as good as VMware? I''m
currently
> installing a new server and yet not decided to go with Xen or Vservers +
> VMware Server (beta). Last one seems to have very good performance, but
I''d
> rather use Xen instead.
i haven''t compared performance, as i still don''t have any
VT-capable hardware
(and won''t for several months more).  in fact, that''s my
biggest obstacle to
using all Xen.  still, your performance numbers are too low, i don''t
think
that''s all you should get.
> Though, the Windows (2003 Web Server) will not be very busy (for now) so it
> might be worth using it for the time being, until stuff gets optimized. My
> only concern is if the current HVM image will work with upcoming features,
> such as paravirtualized drivers..
i guess it should;  the usage scenario for paravirtualized drivers (when and 
if they''re implemented) is to first install windows on a fully
virtualized
hardware and after that installing an ''enhancement pack'' with
drivers for HD
and network that can detect the Xen hypervisor underneath, and optimize for 
that.

-- 
Javier


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Javier Guerra wrote:
> On Sunday 07 May 2006 8:37 am, Joost van den Broek wrote:
>   
>> It even gets worse when installing and using Ubuntu HVM, can''t
enable DMA
>> for the QEMU harddisk, resulting in a very slow +-3.5MB/s read.
Isn''t there
>> any way to resolve this?
>>     
>
> qemu''s HD emulation is very primitive; no DMA, no LBA48.  but it
shouldn''t be
> such slow, since the real device (in dom0) does have DMA.  of course, the 
> best solution is still in the future, when a HVM domain would accept
special
> paravirtualized drivers for HD and network.  those will be needed to get 
> better performance for windows.
>
> PD, the newest version of qemu does have both DMA and LBA48, but i think 
> nobody has bothered porting it to Xen
>
>   
> ------------------------------------------------------------------------
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
Gah! I missed the reply to all button.  Some have already replied, but 
anyways...

If there is interest in a patch with LBA48 stuff ported over I can post 
it.  I am not sure about DMA, but LBA48 falls in easily and seems to 
work fine.

Andy Clayton

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Monday 8 May 2006 22:13, Andy Clayton wrote:
> Gah! I missed the reply to all button.  Some have already replied, but
> anyways...
>
> If there is interest in a patch with LBA48 stuff ported over I can post
> it.  I am not sure about DMA, but LBA48 falls in easily and seems to
> work fine.
>
> Andy Clayton
>
Ofcourse such features are nice, but I''m really looking for some
performance
increasing solutions. Guess LBA48 has nothing to do with that. But maybe 
DMA will help? If that''s possible, it would be very nice.

- Joost

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users