CentOS virt - Apr 2016 - CentOS7, Xen 4.6.1, kernel 3.18.25-19 strange performance problem

Hi all,

i just stumbled over a strange performance issue with my Xen setup.

I use centos-virt Xen since a long time on my workstation and usually 
never check performance. However, yesterday I booted into the 3.18.25-19 
Dom0 kernel *without* Xen and found the system noticably more 
responsive. That triggered me into running a simple kernel compile 
benchmark and compare the results:

Testsequence:
echo 3 > /proc/sys/vm/drop_caches
make distclean
cp ../.config . #copies an existing valid config for compile
make oldconfig
time make -j <threads> bzImage

Results:

xen 4.6.1 - kernel 3.18.25-19
16 threads:
real    1m41.983s
user    16m37.247s
sys 5m7.121s

32 threads:
real    1m29.240s
user    23m7.170s
sys 6m31.497s

kernel 3.18.25-19
16 threads:
real    1m12.909s
user    14m37.627s
sys 1m36.521s

32 threads:
real    1m0.530s
user    18m45.569s
sys 1m58.807s

Some notes:
- System is a HP z840 with 2*8core E5 v3 Xeon and 128gig Ram. Disk is an 
Intel 750 ssd (PCIe nvme)
- While normally in Xen, I limit Dom0 Memory via dom0_mem, but for the 
tests, Dom0 was allowed to use full memory
- While running the test, no guest was running except Dom0. System was 
booted to graphical desktop and was completely unloaded otherwise
- (boot-)logs showed no issues with ACPI or hardware-support or whatsoever.

I have not yet run any in-depth investigation or tried too many 
different settings, but I find the differences quite noteworthy. So, if 
anybody came across the same issues, has some hints or suggestions, feel 
free. I'll continue poking around and if there is interest, i'll post 
updates.

Regards,
Thomas

On Sun, Apr 10, 2016 at 11:20 AM, T.Weyergraf <T.Weyergraf at
virtfinity.de> wrote:
> Hi all,
>
> i just stumbled over a strange performance issue with my Xen setup.
>
> I use centos-virt Xen since a long time on my workstation and usually never
> check performance. However, yesterday I booted into the 3.18.25-19 Dom0
> kernel *without* Xen and found the system noticably more responsive. That
> triggered me into running a simple kernel compile benchmark and compare the
> results:
Thanks for the report.  64-bit PV is known to be slow for Xen for
workloads that involve a lot of system calls (which kernel build
certainly will).  You can read an explanation for the issue, and our
intended solution, in this blog post I wrote a few years ago:

https://blog.xenproject.org/2012/10/31/the-paravirtualization-spectrum-part-2-from-poles-to-a-spectrum/

Unfortunately the initial implementation of PVH turned out to have
some technical issues, and it's in the process of being revamped.  The
work is actively ongoing, but it's likely to be another year at least
before we're going to be able to default to a PVH dom0 kernel for
CentOS.

Still, over 3x for a kernel build is rather surprising -- I'll ask
around and see what other people's expectations / experiences are.
It's possible there's a regression in the CentOS build that hasn't
been noticed.  In particular, if you happen to find any other
combination of Xen / kernel which is significantly faster, do report
it so we can look into it.

 -George

On 04/11/2016 12:34 PM, George Dunlap wrote:
> On Sun, Apr 10, 2016 at 11:20 AM, T.Weyergraf <T.Weyergraf at
virtfinity.de> wrote:
>> [...my stuff deleted...]
> Thanks for the report.  64-bit PV is known to be slow for Xen for
> workloads that involve a lot of system calls (which kernel build
> certainly will).  You can read an explanation for the issue, and our
> intended solution, in this blog post I wrote a few years ago:
>
>
https://blog.xenproject.org/2012/10/31/the-paravirtualization-spectrum-part-2-from-poles-to-a-spectrum/
Right, thanks for the link. I am aware, that the PV-part of PVHVM is 
eating some performance. Usually, this is something in the 10-20% 
ballpark, give or take the workload.
But this is more a 50% hit in walltime and close to a 5-fold jump in 
system time. Also keep in mind, that this is an E5 v3 based machine, 
with APIC-V (and Xen detecting it). I was expecting:
- that Dom0 utilizes it, however I cannot confirm it's usage
- that this will at least help with irq processing
times.
>
> Unfortunately the initial implementation of PVH turned out to have
> some technical issues, and it's in the process of being revamped.  The
> work is actively ongoing, but it's likely to be another year at least
> before we're going to be able to default to a PVH dom0 kernel for
> CentOS.
I am aware of PVH (and it's potential merits). However, as it is not 
there yet, I have no reference as to what to expect in real life. The 
point I am making, is that the performance hit observed is quite hefty, 
IMHO. I have never observed something in that magnitude before.

Note, that I might be looking at a problem, entirely unrelated to
Xen.
>
> Still, over 3x for a kernel build is rather surprising -- I'll ask
> around and see what other people's expectations / experiences are.
> It's possible there's a regression in the CentOS build that
hasn't
> been noticed.  In particular, if you happen to find any other
> combination of Xen / kernel which is significantly faster, do report
> it so we can look into it.
Yes, that is precisely what I was hoping for - other people's 
expectations/experiences. Maybe even some repeats of my simple test and 
posted results. On that machine, I use Xen as a workstation setup, where 
Dom0 also serves as my regular work environment. On the infrastructure, 
I maintain at work, Dom0 performance is next to irrelevant.
I have tried some additional things. The whole thing started with a 
4.6.0-rc2 kernel tried as Dom0 kernel, as I wanted to try the new 
multiqueue blkback/blkfront patches. Unfortunately, that did not work 
due to my graphics not being initialized properly, which I assume is 
unrelated to Xen. I'll continue poking around.
I have tried some non-Xen bare-metal kernels, including said 4.6.0-rc2, 
which yielded results consistent with the 3.18.29 being used as 
bare-metal kernel. I even applied a BIOS-update, which did not change 
things either, just to make sure (according to my experience, the 
z-series Workstations are pretty good at delivering working BIOSes, 
which is one of the main reasons for me to buy them in the first place).

I'll continue investigating and post updates, if i find anything 
noteworthy. I'd be more than happy to hear, what kinds of results others 
see and i am open to suggestions for things to try.
>
>   -George
Regards,
Thomas