Xen devel - Dec 2010 - domU using linux-2.6.37-xen-next pvops kernel with CONFIG_PARAVIRT_SPINLOCKS disabled results in 150% performance improvement

For a couple of months now, we''ve been trying to track down the slow
I/O
performance in pvops domU. Our system has 16 Fibre Channel devices, all
PCI-passthrough to domU. We were previously using a 2.6.32 (Ubuntu version)
HVM kernel and were getting 511k IOPS. We switched to pvops with
Konrad''s
xen-pcifront-0.8.2 kernel and were disappointed to see the performance
degrade to 11k IOPS. After disabling some kernel debug options including
KMEMLEAK, the performance jumped to 186k IOPS but still well below what we
were getting with the HVM kernel. We tried disabling spinlock debugging in
the kernel but it actually resulted in a drop in performance to 70k IOPS.

Last week we switched to linux-2.6.37-xen-next and with the same kernel
debug options disabled, the I/O performance was slightly better at 211k
IOPS. We tried disabling spinlock debugging again and saw a similar drop in
performance to 58k IOPS. We searched around for any performance-related
posts regarding pvops and found two references to CONFIG_PARAVIRT_SPINLOCKS
(one from Jeremy and one from Konrad):
http://lists.xensource.com/archives/html/xen-devel/2009-05/msg00660.html
http://lists.xensource.com/archives/html/xen-devel/2010-11/msg01111.html

Both posts recommended (Konrad strongly) enabling PARAVIRT_SPINLOCKS when
running under Xen. Since it''s enabled by default, we decided to see
what
would happen if we disabled CONFIG_PARAVIRT_SPINLOCKS. With the spinlock
debugging enabled, we were getting 205k IOPS but with spinlock debugging
disabled, the performance leaped to 522k IOPS !!!

So here''s a summary of the kernels, config changes and performance (in
IOPS

                      2.6.32   pcifront   linux
                      Ubuntu   0.8.2      2.6.37-xen-next
                      HVM      pvops      pvops
Spinlock
debugging enabled,     511k     186k       205k
PARAVIRT_SPINLOCKS=n

Spinlock
debugging enabled,
PARAVIRT_SPINLOCKS=y

Spinlock
debugging disabled,
PARAVIRT_SPINLOCKS=n



                        Performance (in thousands IOPS)
                        kernel debug    spinlock disabled       spinlock
disabled
                        disabled        PARAVIRT_SPINLOCKS=y
 PARAVIRT_SPINLOCKS not set
2.6.32 Ubuntu HVM       511
pcifront-0.8.2          186             70
linux-2.6.37-xen-next   205             58                      522

system: HP Proliant DL380G7
Xen: 4.1-unstable (Thu Oct 14 12:46:29 2010 +0100 22241:3a5755249361)
dom0: 2.6.32.24-pvops-stable (xen/stable)
domU: linux-2.6.37-xen-next (2.6.37-rc5-pvops)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel