xen discuss - Jul 2008 - Solaris xVM dom0 vs. Solaris base compared with Linux dom0 vs. Linux base

Hi All,

We are doing some performance testing of virtualization
on Solaris Nevada build 87.

For the xVM case, we noticed a higher than expected
overhead in Dom0. In an effort to make sense of it, we
compared the results with a Linux system vs. its Dom0.

We ran an apache compile on each system.

Here are the compile time results:

Solaris base                1m24.478sSolaris xVM Dom0     2m32.772s

Linux base                  1m32.632s
Linux Dom0                1m54.301s

As you can see there is much more overhead for the
xVM Dom0 case relative to the Linux Dom0 case. Note
that we are not comparing Solaris to Linux in a general
sense, we are only comparing the overhead of the base
compared to its Dom0.

We have the complete output for the average of 3 trials that
we ran on each, and the breakdown from the output of the time
command included in a spreadsheet, which you can find here:

http://spreadsheets.google.com/pub?key=pYKv0vD13oyMfkORpXSLDMQ

In particular for the raw data we see that the overhead from
the compile on the xVM Dom0 comes from the sys field (or time
spent in system mode)

Our understanding is that the Solaris port of Xen is working,
but is still under heavy development. Is it the case that the
Solaris Dom0, at least in build 87, is not optimized yet? And
that the results we are seeing are in fact correct, but that work
ongoing will show better Solaris Dom0 results in the future?

We carefully compared the differences between Solaris base
and xVM Dom0

Here are the process lists, with the xVM difference highlighted
in yellow:

http://docs.google.com/View?docid=dc5sf6db_122wmdfpfct


All of Linux, Linux Dom0, Solaris base, and xVM are 64bit
and SMP capable.

The uname and isainfo commands follow.

solaris-base# uname -a
SunOS solaris 5.11 snv_87 i86pc i386 i86pc

solaris-base# isainfo
amd64 i386

xVM# uname -a
SunOS solaris 5.11 snv_87 i86pc i386 i86xpv

xVM# isainfo
amd64 i386


The system details are:

Dell OptiPlex 745 with a 2.4 GHz Intel Core 2  CPU 6600,
4 GB of RAM


Thanks for you time and any feedback.

Best Regards,
Todd

-- 
Todd Deshane
http://todddeshane.net
check out our book: http://runningxen.com

On Fri, Jul 11, 2008 at 01:16:13AM -0400, Todd Deshane wrote:
> We ran an apache compile on each system.
Won''t this perturb the results, because presumably different things get
compiled on the two platforms? Have you verified that the results are at
least fairly similar?
> Solaris base                1m24.478sSolaris xVM Dom0     2m32.772s
> 
> Linux base                  1m32.632s
> Linux Dom0                1m54.301s
> 
> As you can see there is much more overhead for the
> xVM Dom0 case relative to the Linux Dom0 case. Note
Are your disks in SATA or IDE mode? (I think eeprom bootpath will tell
you)
> Here are the process lists, with the xVM difference highlighted
> in yellow:
> 
> http://docs.google.com/View?docid=dc5sf6db_122wmdfpfct
This doesn''t seem accurate, that is the highlighted parts are not the
only differences.

regards
john

On Sat, Jul 12, 2008 at 02:29:47AM +0800, HU wenjin wrote:
> # eeprom bootpath
> bootpath=/pci@0,0/pci-ide@1f,2/ide@0/cmdk@0,0:a
> I wonder how that matters. Could you explain more to us? Thanks.
That''s IDE not SATA.

It makes a huge difference to performance since the Solaris IDE driver
has lots of inb()/outb()s which are trapped (you could probably dtrace
them?). If you look in /sys/block you can probably check that Linux is
using SATA...

regards
john

--- On Fri, 7/11/08, John Levon <john.levon@sun.com> wrote:
From: John Levon <john.levon@sun.com>
Subject: Re: [xen-discuss] Solaris xVM dom0 vs. Solaris base compared with Linux
dom0 vs. Linux base
To: "HU wenjin" <wenjin.hu@gmail.com>
Cc: "Todd Clayton" <Todd.Clayton@sun.com>, "Jeanna
Matthews" <jeanna.matthews@gmail.com>, "Mark Johnson"
<mark.johnson@sun.com>, xen-discuss@opensolaris.org
Date: Friday, July 11, 2008, 2:38 PM

On Sat, Jul 12, 2008 at 02:29:47AM +0800, HU wenjin wrote:
> # eeprom bootpath
> bootpath=/pci@0,0/pci-ide@1f,2/ide@0/cmdk@0,0:a
> I wonder how that matters. Could you explain more to us? Thanks.
That''s IDE not SATA.

It makes a huge difference to performance since the Solaris IDE driver
has lots of inb()/outb()s which are trapped (you could probably dtrace
them?). If you look in /sys/block you can probably check that Linux is
using SATA...
*****************************************************
If Nevada runs in IDE mode it means the South Bridge
(most probably ICH8R) is set in BIOS to IDE Emulation
AHCI support has been implemented on Nevada a while 
ago. Current Nevada builds do detect ICHR9R (ICH8R) 
BIOS settings and load Solaris AHCI driver.
How can Linux run with AHCI driver if South Bridge
is set up to IDE Emulation ? Even if it does so sata
features appear to be disabled on BIOS level anyway.
*****************************************************

regards
john
_______________________________________________
xen-discuss mailing list
xen-discuss@opensolaris.org

OK. Dell OptiPlex 745 with a 2.4 GHz Intel Core 2  CPU 6600 is based on
Intel Q965 ATX Motherboard :-
http://www.protech-ipc.com/ipc/news_news_content.asp?NID=80
South Bridge as expected is ICH8R
 
 
This message posted from opensolaris.org

On Sat, Jul 12, 2008 at 06:43:06AM -0500, Jeanna Matthews wrote:
> of Xen on Linux. Our tests on the same hardware platform showed only an X%
> performance overhead with a Solaris guest and a Linux dom0. However, this
> comparison is likely to vary substantially with hardware. In particular,
our
> tests used IDE drives and we suspect this may introduce more overhead in
the
> Solaris case than the Linux case because the Solaris IDE drivers are not as
> optimized."
> 
> Does this seem like an accurate/fair statement?
Possibly. It depends on what sort of overhead you get if you *do* use
SATA. Otherwise it''s a (slightly) educated guess from me, only.

I don''t have a reference, sorry.

regards
john

> It makes a huge difference to performance since the Solaris IDE driver
> has lots of inb()/outb()s which are trapped
Is there any work in progress to update/optimize the Solaris IDE driver for Xen?
I''ve got a server with an Intel 5000 series chipset that supports SATA
devices, but not in AHCI mode.  Performance isn''t pretty under Xen, and
is horrible with Xen+ZFS, despite applying recommended tuning to each.
 
 
This message posted from opensolaris.org

Is Intel providing SATA drivers for any OS''s ?
 
 
This message posted from opensolaris.org

On Mon, Jul 21, 2008 at 01:08:52PM -0700, Chris wrote:
> > It makes a huge difference to performance since the Solaris IDE driver
> > has lots of inb()/outb()s which are trapped
> 
> Is there any work in progress to update/optimize the Solaris IDE
> driver for Xen?  I''ve got a server with an Intel 5000 series
chipset
> that supports SATA devices, but not in AHCI mode.  Performance
isn''t
> pretty under Xen, and is horrible with Xen+ZFS, despite applying
> recommended tuning to each.
I don''t think that there are any plans to do this. The main challenge
is the testing associated with such a large change. Even if the
implementation too a couple of weeks, getting good test coverage would
be a nightmare.

Perhaps we could build an i86xpv variant of the driver that didn''t use
inb/outb. We''d have to manage any divergence carefully.

Chris wrote:
>> It makes a huge difference to performance since the Solaris IDE driver
>> has lots of inb()/outb()s which are trapped
> 
> Is there any work in progress to update/optimize the 
 > Solaris IDE driver for Xen?  I''ve got a server with
 > an Intel 5000 series chipset that supports SATA devices,
 > but not in AHCI mode.  Performance isn''t pretty under
 > Xen, and is horrible with Xen+ZFS, despite applying
 > recommended tuning to each.

On a related note, I found a couple of bugs last
night while chasing down a hang in the IDE driver.

I have fixes for both and they should improve IDE
performance some..

But if your trying to do anything moderate to
serious with Xen, and have more than ~ 3G of memory
in the system, you should not be using IDE.

You need a disk hba and NIC which can reach all of
physical memory.



MRJ

> But if your trying to do anything moderate to
> serious with Xen, and have more than ~ 3G of memory
> in the system, you should not be using IDE.
Well OK, but reality is that I''m working with integrated SATA
controllers that only operate in IDE mode.  Solaris performs well enough on this
hardware when running on bare metal, but chokes under Xen.  If the only answer
is to buy a new SATA controller, I can live with that.  However, before
springing for any new hardware I thought I''d ask if this known problem
was being worked.  Thanks for the response.
 
 
This message posted from opensolaris.org

> Well OK, but reality is that I''m working with
> integrated SATA controllers that only operate in IDE
> mode.  Solaris performs well enough on this hardware
> when running on bare metal, but chokes under Xen.  If
> the only answer is to buy a new SATA controller, I
> can live with that.  However, before springing for
> any new hardware I thought I''d ask if this known
> problem was being worked.  Thanks for the response.
What is logged to /var/adm/messages during (dom0) boot
when the ata driver is probing for HDDs?  Are the
HDDs used in some DMA mode, or is it perhaps using
PIO mode for data transfers?
 
 
This message posted from opensolaris.org