Hi,
I found an article about forking time for redis NoSQL database in
different systems:
http://redis.io/topics/latency
---------Quote------------------
Fork time in different systems
Modern hardware is pretty fast to copy the page table, but Xen is not.
The problem with Xen is not virtualization-specific, but Xen-specific.
For instance using VMware or Virutal Box does not result into slow fork
time. The following is a table that compares fork time for different
Redis instance size. Data is obtained performing a BGSAVE and looking at
the latest_fork_usec filed in the INFO command output.
* Linux beefy VM on VMware 6.0GB RSS forked in 77 milliseconds
(12.8 milliseconds per GB).
* Linux running on physical machine (Unknown HW) 6.1GB RSS forked
in 80 milliseconds (13.1 milliseconds per GB)
* Linux running on physical machine (Xeon @ 2.27Ghz) 6.9GB RSS
forked into 62 millisecodns (9 milliseconds per GB).
* Linux VM on 6sync (KVM) 360 MB RSS forked in 8.2 milliseconds
(23.3 millisecond per GB).
* Linux VM on EC2 (Xen) 6.1GB RSS forked in 1460 milliseconds
(239.3 milliseconds per GB).
* Linux VM on Linode (Xen) 0.9GBRSS forked into 382 millisecodns
(424 milliseconds per GB).
As you can see a VM running on Xen has a performance hit that is between
one order to two orders of magnitude. We believe this is a severe
problem with Xen and we hope it will be addressed ASAP.
----------End of quote-----------------
I made my own test with Xen 4.1 and Redis 2.4 with 7.04GB dataset. The
test was performed on Intel Core I5 2500 processor unit. Forking time
was about 1 sec or 151 ms/GB - it''s faster then tests over Amazon
EC2/Linode were mentioned in the article, but still much slower then
VMWare or physical machines. Has anyone running with this issue? Or may
be there is a way to tune Xen for less forking times?
Sergey Zhukov
Hey Sergey, interesting read!!!! 2012/6/16 Sergey Zhukov <svg@ngs.ru>:> Hi, > > I found an article about forking time for redis NoSQL database in > different systems: > > http://redis.io/topics/latency > ---------Quote------------------ > Fork time in different systems > Modern hardware is pretty fast to copy the page table, but Xen is not. > The problem with Xen is not virtualization-specific, but Xen-specific.Maybe thats one of the reasons why Xen constantly looks slow in "unixbench". I don''t know the reasons. Would you mind re-testing this with vcpu-pinning the VM? And how about asking on xen-devel? Florian
On Sat, 2012-06-16 at 23:37 +0200, Florian Heigl wrote:> Hey Sergey, > > interesting read!!!! > > 2012/6/16 Sergey Zhukov <svg@ngs.ru>: > > Hi, > > > > I found an article about forking time for redis NoSQL database in > > different systems: > > > > http://redis.io/topics/latency > > ---------Quote------------------ > > Fork time in different systems > > Modern hardware is pretty fast to copy the page table, but Xen is not. > > The problem with Xen is not virtualization-specific, but Xen-specific. > > Maybe thats one of the reasons why Xen constantly looks slow in "unixbench". > > I don''t know the reasons. Would you mind re-testing this with > vcpu-pinning the VM?I think I can re-test with other vcpu pinning, currently I''ve got one vcpu in DomU with the mapping: VCPU: 0 Pinning: 0,1,2,3 Which one pinning should be tested?> > And how about asking on xen-devel?Do you think the topic is more suitable for devel mail list? I''ve never posted there before, because I thought it''s more related to the patches, code submissions etc... Sergey Zhukov
On Sat, Jun 16, 2012 at 10:15 AM, Sergey Zhukov <svg@ngs.ru> wrote:> Hi, > > I found an article about forking time for redis NoSQL database in > different systems: > > http://redis.io/topics/latency > ---------Quote------------------ > Fork time in different systems > Modern hardware is pretty fast to copy the page table, but Xen is not. > The problem with Xen is not virtualization-specific, but Xen-specific. > For instance using VMware or Virutal Box does not result into slow fork > time. The following is a table that compares fork time for different > Redis instance size. Data is obtained performing a BGSAVE and looking at > the latest_fork_usec filed in the INFO command output. > > * Linux beefy VM on VMware 6.0GB RSS forked in 77 milliseconds > (12.8 milliseconds per GB). > * Linux running on physical machine (Unknown HW) 6.1GB RSS forked > in 80 milliseconds (13.1 milliseconds per GB) > * Linux running on physical machine (Xeon @ 2.27Ghz) 6.9GB RSS > forked into 62 millisecodns (9 milliseconds per GB). > * Linux VM on 6sync (KVM) 360 MB RSS forked in 8.2 milliseconds > (23.3 millisecond per GB). > * Linux VM on EC2 (Xen) 6.1GB RSS forked in 1460 milliseconds > (239.3 milliseconds per GB). > * Linux VM on Linode (Xen) 0.9GBRSS forked into 382 millisecodns > (424 milliseconds per GB). > > As you can see a VM running on Xen has a performance hit that is between > one order to two orders of magnitude. We believe this is a severe > problem with Xen and we hope it will be addressed ASAP. > ----------End of quote----------------- > > I made my own test with Xen 4.1 and Redis 2.4 with 7.04GB dataset. The > test was performed on Intel Core I5 2500 processor unit. Forking time > was about 1 sec or 151 ms/GB - it''s faster then tests over Amazon > EC2/Linode were mentioned in the article, but still much slower then > VMWare or physical machines. Has anyone running with this issue? Or may > be there is a way to tune Xen for less forking times?Which version of the kernel are you using for Xen? And is it a pvops kernel or a "classic xen" kernel? Also, have you tried running Linux in HVM mode? The original "classic Xen" kernel had a lot of performance tuning done to it. But the focus of pvops development for years has just been *getting the necessary support into the kernel*, so these kinds of microbenchmark performance things have suffered just because no one has been looking at it. We''ve got engineers looking more closely at this now, so it''s good to know what areas we need to look at. If you use HVM mode, you''ll most likely be using the same exact codepaths that KVM will be using (hardware pagetable virtualization), and so will likely get the same speed. -George
Sorry for being so late with the reply, I was on the road and just remembered the thread :) 2012/6/17 Sergey Zhukov <svg@ngs.ru>:> I think I can re-test with other vcpu pinning, currently I''ve got one > vcpu in DomU with the mapping: > VCPU: 0 > Pinning: 0,1,2,3 > > Which one pinning should be tested?Pin the vcpu to one phyiscal core, instead of allowing it to be floating between them. i.e. xm cpu-pin DomU 0 2 pins it to the second core. If your system has hyperthreading, make sure you''re not pinning it to the hyperthread, you can see those via xenpm topology-something (see the xenpm help output) In theory you could also make sure no other VM or dom0 is using that core, but that''s not relevant here i think. Xen will push any other dom''s from that busy core. If your results show that the vcpu migration is not the culprit then it would really be some dragons in the domU kernel. :) Florian