Johnn Tan
2007-Aug-13 21:16 UTC
[Xen-users] disk performance about half in domU? + question about XenSource
Based on some tests we ran, it seems the biggest performance hit you get from running within domU is from disk I/O. We did some mysql read and write tests, and for both, our performance is about half that compared to native. Has that been others'' experience? Is there any way to make this better? We are using physical partitions. In contrast, cpu/memory tests appear to be near native. Does buying commercial Xen (XenSource) help in anyway? Do they have optimized disk drivers? johnn _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Tony Hoyle
2007-Aug-13 21:47 UTC
Re: [Xen-users] disk performance about half in domU? + question about XenSource
Johnn Tan wrote:> Based on some tests we ran, it seems the biggest performance hit you get > from running within domU is from disk I/O. We did some mysql read and > write tests, and for both, our performance is about half that compared > to native. Has that been others'' experience? > > Is there any way to make this better? We are using physical partitions. > In contrast, cpu/memory tests appear to be near native. > > > Does buying commercial Xen (XenSource) help in anyway? Do they have > optimized disk drivers? >There''s also a fairly big hit on network I/O. I don''t believe you can get it any faster with the current architecture as it doesn''t support any kind of DMA.. it''s all software copying of data from one place to another. One thing to do is give the dom0 its own processor - makes quite a big difference. If course because there''s only one process in dom0 doing the whole I/O (qemu-dm) it''ll slow down with each new domU you start up. loopback partitions are faster because a lot gets cached in RAM, although because of that they''re not so great if you get a sudden power loss. Never had a kernel that supported blktap so I don''t know what kind of speed you can get out of that. Tony _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson
2007-Aug-14 01:37 UTC
Re: [Xen-users] disk performance about half in domU? + question about XenSource
> Based on some tests we ran, it seems the biggest performance > hit you get from running within domU is from disk I/O. We > did some mysql read and write tests, and for both, our > performance is about half that compared to native. Has that > been others'' experience? > > Is there any way to make this better? We are using physical > partitions. In contrast, cpu/memory tests appear to be near > native.Sounds a bit weird. How many CPUs in this box? My memories of the benchmarks suggest this should be better than you''re seeing, but maybe your workload is tickling some bad cases or something... I know this sounds really weird but when comparing to native performance you do need to test on the same area of the disk, have you done this? Portions of the disk nearer to the outside edge of the platter can have significantly higher transfer rates due to moving at a higher linear velocity.> Does buying commercial Xen (XenSource) help in anyway? Do > they have optimized disk drivers?XenEnterprise and friends add optimised disk drivers for Windows, but if you''re running paravirtualised Linux then you''ve already got optimised disk drivers, so the commercial product probably won''t help. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson
2007-Aug-14 01:44 UTC
Re: [Xen-users] disk performance about half in domU? + question about XenSource
> There''s also a fairly big hit on network I/O.Network IO is particularly hard to virtualise with good performance. The trick you suggested with a multi-processor system does help. Actually, even just dedicating a hyperthread to dom0 helps (if your system has HT).> I don''t believe you can get it any faster with the current architecture > as it doesn''t support any kind of DMA.. it''s all software copying of > data from one place to another.Disk data is DMAed to /from the correct location for paravirt devices. I''m not sure how it works for HVM emulated devices. Received network data is copied these days because it actually gives better average case performance than the previous copy avoidance scheme. Ideally the network card would know to DMA straight to the correct virtual machine without dom0 getting involved.> loopback partitions are faster because a lot gets cached in RAM, > although because of that they''re not so great if you get a sudden power > loss. Never had a kernel that supported blktap so I don''t know what > kind of speed you can get out of that.Loopback devices can be a bit hairy though, both because of the power outage issue you mentioned and also because they''re heavy RAM users. The caching improvements are probably better provided by reducing the size of dom0 and increasing the size of the domUs so that they can cache more stuff themselves. Blktap is supposed to be a nicer solution for file-backed devices. Still, using raw partitions should really be plenty fast enough, there''s not really a lower overhead solution than that. And performance problems with one solution are probably going to crop up with another, anyhow. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John A. Sullivan III
2007-Aug-14 14:02 UTC
Re: [Xen-users] disk performance about half in domU? + question about XenSource
On Tue, 2007-08-14 at 02:37 +0100, Mark Williamson wrote:> > Based on some tests we ran, it seems the biggest performance > > hit you get from running within domU is from disk I/O. We > > did some mysql read and write tests, and for both, our > > performance is about half that compared to native. Has that > > been others'' experience? > > > > Is there any way to make this better? We are using physical > > partitions. In contrast, cpu/memory tests appear to be near > > native. > > Sounds a bit weird. How many CPUs in this box? My memories of the benchmarks > suggest this should be better than you''re seeing, but maybe your workload is > tickling some bad cases or something... > > I know this sounds really weird but when comparing to native performance you > do need to test on the same area of the disk, have you done this? Portions > of the disk nearer to the outside edge of the platter can have significantly > higher transfer rates due to moving at a higher linear velocity. > > > Does buying commercial Xen (XenSource) help in anyway? Do > > they have optimized disk drivers? > > XenEnterprise and friends add optimised disk drivers for Windows, but if > you''re running paravirtualised Linux then you''ve already got optimised disk > drivers, so the commercial product probably won''t help. > > Cheers, > Mark > >I don''t know how relevant our experience is since we are still on Xen 2.0.7 and just starting to test 3.1, however, we have noticed the same thing. We assumed it was simply I/O bottleneck - we have a single RAID controller trying to service several virtual servers. We did dedicate a hyperthread to dom0 and changed the scheduler (I do not recall offhand to what) and those made significant improvements but we still cringe when we see how many CPU cycles are simply spent waiting for disk I/O. This is especially scary because, although it is a nasty mix of email, web and database servers along with intensive network I/O on two of the domUs which serve as VPN gateways, the actual usage is quite low - a handful of users coming across an Internet connection. I don''t know how these would fare under LAN load with a few hundred users. Not complaining - just sharing. Thanks for a great product - John -- John A. Sullivan III Open Source Development Corporation +1 207-985-7880 jsullivan@opensourcedevel.com Financially sustainable open source development http://www.opensourcedevel.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson
2007-Aug-14 14:21 UTC
Re: [Xen-users] disk performance about half in domU? + question about XenSource
> I don''t know how relevant our experience is since we are still on Xen > 2.0.7 and just starting to test 3.1, however, we have noticed the same > thing.Yay - it''s always somewhat satisfying to see how stable the 2.x series was for people and that they''re still using it. ISTR that 2.x (or maybe it was the early 3.x?) might have had some weird block performance regression that crept in. The fact you''re seeing it for 3.1 is a little weird though.> We assumed it was simply I/O bottleneck - we have a single RAID > controller trying to service several virtual servers. We did dedicate a > hyperthread to dom0 and changed the scheduler (I do not recall offhand > to what) and those made significant improvements but we still cringe > when we see how many CPU cycles are simply spent waiting for disk I/O. > This is especially scary because, although it is a nasty mix of email, > web and database servers along with intensive network I/O on two of the > domUs which serve as VPN gateways, the actual usage is quite low - a > handful of users coming across an Internet connection. I don''t know how > these would fare under LAN load with a few hundred users. > > Not complaining - just sharing. Thanks for a great product - JohnWell, it sounds slower than I''d expect things to be from what I remember of the benchmarks... Have you tried just doing a big dd (e.g. dd a big file into /dev/null using a decent block size) and seeing what raw bandwidth you can get? Could you check what scheduler this is? Credit is the most well supported at the moment. Is it an SMP or a UP box? Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
John A. Sullivan III
2007-Aug-14 14:58 UTC
Re: [Xen-users] disk performance about half in domU? + question about XenSource
On Tue, 2007-08-14 at 15:21 +0100, Mark Williamson wrote:> > I don''t know how relevant our experience is since we are still on Xen > > 2.0.7 and just starting to test 3.1, however, we have noticed the same > > thing. > > Yay - it''s always somewhat satisfying to see how stable the 2.x series was for > people and that they''re still using it. ISTR that 2.x (or maybe it was the > early 3.x?) might have had some weird block performance regression that crept > in. The fact you''re seeing it for 3.1 is a little weird though. > > > We assumed it was simply I/O bottleneck - we have a single RAID > > controller trying to service several virtual servers. We did dedicate a > > hyperthread to dom0 and changed the scheduler (I do not recall offhand > > to what) and those made significant improvements but we still cringe > > when we see how many CPU cycles are simply spent waiting for disk I/O. > > This is especially scary because, although it is a nasty mix of email, > > web and database servers along with intensive network I/O on two of the > > domUs which serve as VPN gateways, the actual usage is quite low - a > > handful of users coming across an Internet connection. I don''t know how > > these would fare under LAN load with a few hundred users. > > > > Not complaining - just sharing. Thanks for a great product - John > > Well, it sounds slower than I''d expect things to be from what I remember of > the benchmarks... Have you tried just doing a big dd (e.g. dd a big file > into /dev/null using a decent block size) and seeing what raw bandwidth you > can get? > > Could you check what scheduler this is? Credit is the most well supported at > the moment. Is it an SMP or a UP box? > > Cheers, > Mark >We haven''t tested 3.1 enough to know if this is still a problem. If it is important to gather this information for 2.0.7, I''ll do so and submit it. Please let me know so I don''t spend the time unnecessarily. If it will be helpful, I most gladly try the dd and figure out what scheduler we ultimately used. Thanks - John -- John A. Sullivan III Open Source Development Corporation +1 207-985-7880 jsullivan@opensourcedevel.com Financially sustainable open source development http://www.opensourcedevel.com _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Johnn Tan
2007-Aug-14 15:04 UTC
Re: [Xen-users] disk performance about half in domU? + question about XenSource
Mark Williamson wrote:> Sounds a bit weird. How many CPUs in this box? My memories of the benchmarks > suggest this should be better than you''re seeing, but maybe your workload is > tickling some bad cases or something...Hi Mark: We have a quad-core processor and 8GB RAM. I gave the domU all 4 VCPUs (didn''t know you could do that! -- you can''t do it with memory). I assigned 7.5GB RAM to the domU. This was using CentOS-5 x86_64 and mysql 5.1.20-beta.> I know this sounds really weird but when comparing to native performance you > do need to test on the same area of the disk, have you done this? Portions > of the disk nearer to the outside edge of the platter can have significantly > higher transfer rates due to moving at a higher linear velocity.No, we didn''t specifically test the same area of the disk. We have a hardware RAID-10 (PERC 5 on Dell PE 2950) using 128K chunks. But I think we did enough tests to see a pattern emerging. Granted, we also used sql-bench and the results for the domU were actually okay. But we believe our own sql tests more specifically focus on the disk I/O. But I also ran unixbench since that''s a bit more standardized, and it seems to indicate some of the same results. I don''t know how helpful it is, but I went ahead and enclosed those results below.> XenEnterprise and friends add optimised disk drivers for Windows, but if > you''re running paravirtualised Linux then you''ve already got optimised disk > drivers, so the commercial product probably won''t help.Ok thanks for clarifying. We haven''t yet tested Xen 3.1 -- do you know if that includes any I/O performance enhancements? johnn Here''s the unixbench results for CentOS-5 x86_64 on a physical machine (non-xen): ===================================================== BYTE UNIX Benchmarks (Version 4.1.0) System -- Linux sql1 2.6.18-8.1.8.el5 #1 SMP Tue Jul 10 06:39:17 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux Start Benchmark Run: Wed Aug 8 18:15:57 EDT 2007 1 interactive users. 18:15:57 up 9 min, 1 user, load average: 0.01, 0.04, 0.00 lrwxrwxrwx 1 root root 4 Aug 8 16:41 /bin/sh -> bash /bin/sh: symbolic link to `bash'' /dev/sda1 8123168 1189948 6513928 16% / Dhrystone 2 using register variables 8893810.7 lps (10.0 secs, 10 samples) Double-Precision Whetstone 1781.2 MWIPS (9.9 secs, 10 samples) System Call Overhead 473046.7 lps (10.0 secs, 10 samples) Pipe Throughput 492008.6 lps (10.0 secs, 10 samples) Pipe-based Context Switching 96405.6 lps (10.0 secs, 10 samples) Process Creation 8338.6 lps (30.0 secs, 3 samples) Execl Throughput 2372.1 lps (29.8 secs, 3 samples) File Read 1024 bufsize 2000 maxblocks 844117.0 KBps (30.0 secs, 3 samples) File Write 1024 bufsize 2000 maxblocks 478062.0 KBps (30.0 secs, 3 samples) File Copy 1024 bufsize 2000 maxblocks 283041.0 KBps (30.0 secs, 3 samples) File Read 256 bufsize 500 maxblocks 235142.0 KBps (30.0 secs, 3 samples) File Write 256 bufsize 500 maxblocks 126786.0 KBps (30.0 secs, 3 samples) File Copy 256 bufsize 500 maxblocks 80142.0 KBps (30.0 secs, 3 samples) File Read 4096 bufsize 8000 maxblocks 1755004.0 KBps (30.0 secs, 3 samples) File Write 4096 bufsize 8000 maxblocks 931432.0 KBps (30.0 secs, 3 samples) File Copy 4096 bufsize 8000 maxblocks 601062.0 KBps (30.0 secs, 3 samples) Shell Scripts (1 concurrent) 5410.3 lpm (60.0 secs, 3 samples) Shell Scripts (8 concurrent) 1721.7 lpm (60.0 secs, 3 samples) Shell Scripts (16 concurrent) 925.7 lpm (60.0 secs, 3 samples) Arithmetic Test (type = short) 1243892.9 lps (10.0 secs, 3 samples) Arithmetic Test (type = int) 1265520.4 lps (10.0 secs, 3 samples) Arithmetic Test (type = long) 326020.1 lps (10.0 secs, 3 samples) Arithmetic Test (type = float) 925118.2 lps (10.0 secs, 3 samples) Arithmetic Test (type = double) 512452.7 lps (10.0 secs, 3 samples) Arithoh 227641190.3 lps (10.0 secs, 3 samples) C Compiler Throughput 978.7 lpm (60.0 secs, 3 samples) Dc: sqrt(2) to 99 decimal places 84417.7 lpm (30.0 secs, 3 samples) Recursion Test--Tower of Hanoi 81500.4 lps (20.0 secs, 3 samples) INDEX VALUES TEST BASELINE RESULT INDEX Dhrystone 2 using register variables 116700.0 8893810.7 762.1 Double-Precision Whetstone 55.0 1781.2 323.9 Execl Throughput 43.0 2372.1 551.7 File Copy 1024 bufsize 2000 maxblocks 3960.0 283041.0 714.8 File Copy 256 bufsize 500 maxblocks 1655.0 80142.0 484.2 File Copy 4096 bufsize 8000 maxblocks 5800.0 601062.0 1036.3 Pipe Throughput 12440.0 492008.6 395.5 Process Creation 126.0 8338.6 661.8 Shell Scripts (8 concurrent) 6.0 1721.7 2869.5 System Call Overhead 15000.0 473046.7 315.4 ======== FINAL SCORE 640.2 ===================================================== And here''s the results for the exact same setup on a domU using same physical hardware: BYTE UNIX Benchmarks (Version 4.1.0) System -- Linux store-nyc377-vmsql02.limewire.com 2.6.18-8.1.8.el5xen #1 SMP Tue Jul 10 07:06:45 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux Start Benchmark Run: Thu Aug 9 14:34:36 EDT 2007 1 interactive users. 14:34:36 up 1 day, 20:41, 1 user, load average: 0.16, 0.04, 0.01 lrwxrwxrwx 1 root root 4 Jul 17 17:54 /bin/sh -> bash /bin/sh: symbolic link to `bash'' /dev/xvda1 6092360 3795932 1981960 66% / Dhrystone 2 using register variables 8931957.1 lps (10.0 secs, 10 samples) Double-Precision Whetstone 1788.3 MWIPS (9.9 secs, 10 samples) System Call Overhead 174628.7 lps (10.0 secs, 10 samples) Pipe Throughput 211195.6 lps (10.0 secs, 10 samples) Pipe-based Context Switching 54065.7 lps (10.0 secs, 10 samples) Process Creation 2336.0 lps (30.0 secs, 3 samples) Execl Throughput 948.3 lps (29.7 secs, 3 samples) File Read 1024 bufsize 2000 maxblocks 404623.0 KBps (30.0 secs, 3 samples) File Write 1024 bufsize 2000 maxblocks 254555.0 KBps (30.0 secs, 3 samples) File Copy 1024 bufsize 2000 maxblocks 145873.0 KBps (30.0 secs, 3 samples) File Read 256 bufsize 500 maxblocks 105386.0 KBps (30.0 secs, 3 samples) File Write 256 bufsize 500 maxblocks 65650.0 KBps (30.0 secs, 3 samples) File Copy 256 bufsize 500 maxblocks 39343.0 KBps (30.0 secs, 3 samples) File Read 4096 bufsize 8000 maxblocks 1103909.0 KBps (30.0 secs, 3 samples) File Write 4096 bufsize 8000 maxblocks 645148.0 KBps (30.0 secs, 3 samples) File Copy 4096 bufsize 8000 maxblocks 398281.0 KBps (30.0 secs, 3 samples) Shell Scripts (1 concurrent) 2693.7 lpm (60.0 secs, 3 samples) Shell Scripts (8 concurrent) 638.0 lpm (60.0 secs, 3 samples) Shell Scripts (16 concurrent) 335.3 lpm (60.0 secs, 3 samples) Arithmetic Test (type = short) 1244421.3 lps (10.0 secs, 3 samples) Arithmetic Test (type = int) 1267033.4 lps (10.0 secs, 3 samples) Arithmetic Test (type = long) 326565.1 lps (10.0 secs, 3 samples) Arithmetic Test (type = float) 926358.0 lps (10.0 secs, 3 samples) Arithmetic Test (type = double) 513037.0 lps (10.0 secs, 3 samples) Arithoh 227902041.2 lps (10.0 secs, 3 samples) C Compiler Throughput 761.3 lpm (60.0 secs, 3 samples) Dc: sqrt(2) to 99 decimal places 37142.6 lpm (30.0 secs, 3 samples) Recursion Test--Tower of Hanoi 81742.8 lps (20.0 secs, 3 samples) INDEX VALUES TEST BASELINE RESULT INDEX Dhrystone 2 using register variables 116700.0 8931957.1 765.4 Double-Precision Whetstone 55.0 1788.3 325.1 Execl Throughput 43.0 948.3 220.5 File Copy 1024 bufsize 2000 maxblocks 3960.0 145873.0 368.4 File Copy 256 bufsize 500 maxblocks 1655.0 39343.0 237.7 File Copy 4096 bufsize 8000 maxblocks 5800.0 398281.0 686.7 Pipe Throughput 12440.0 211195.6 169.8 Process Creation 126.0 2336.0 185.4 Shell Scripts (8 concurrent) 6.0 638.0 1063.3 System Call Overhead 15000.0 174628.7 116.4 ======== FINAL SCORE 324.3 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Mark Williamson
2007-Aug-14 15:06 UTC
Re: [Xen-users] disk performance about half in domU? + question about XenSource
> > We haven''t tested 3.1 enough to know if this is still a problem. If it > is important to gather this information for 2.0.7, I''ll do so and submit > it. Please let me know so I don''t spend the time unnecessarily. If it > will be helpful, I most gladly try the dd and figure out what scheduler > we ultimately used. Thanks - JohnTo be honest, for 2.x I don''t think it''ll be worth your time: it''s too out of date and development has moved on a longer way... I don''t think anybody is going to fix anything Xen 2 based. Sorry :-( I guess Xen isn''t a big enough project to have an "old stable" patching scheme as the Linux kernel does. But if you observe worrying performance behaviour on 3.x please do report it. Doubly so if it''s a regression wrt Xen 2. I''ll have my fingers crossed that your problems are resolved under Xen 3.1. By the way, remember that if you need to run legacy virtual machines e.g. 2.4 you could run them in HVM mode on Xen 3 (if your hardware supports it). I believe there are even some paravirtualised drivers around for 2.4 Linux that you could use to boost IO performance in HVM. Actually, you could run the entirety of Xen 2.x and its guests in HVM mode, but this probably wouldn''t do much for the IO performance (and there would quite likely be problems compiling optimised drivers for it!). Good luck with it anyhow. Hope you find that Xen 3 is to your liking; there are lots of cool features there. If you don''t have a requirement to run less-common OSes (e.g. NetBSD, Plan9 etc) I''d suggest that you try to use PAE-enabled 32-bit guests under Xen 3. This will enable you to migrate them more easily to a 64-bit Xen at a future date. Cheers, Mark -- Dave: Just a question. What use is a unicyle with no seat? And no pedals! Mark: To answer a question with a question: What use is a skateboard? Dave: Skateboards have wheels. Mark: My wheel has a wheel! _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users