Robert Taylor
2009-Nov-06 19:55 UTC
[Xen-users] XEN 3.1 on 2.6.18-164.el5xen kernel - Slow IO & Pauses
I''ve seen tons of posts about slow IO under Xen and others without slow IO under xen. Here''s what I''m experiencing. I''ve got a XEN system on an 2 dual core AMD Opteron 275 processors. I''m running DRBD for disk, but have also tested without DRBD. Throughput in Dom-0 seems to be 50-80MB/sec for a file of any size. Throughput in DomU is 20-30MB/sec for a file under 2GB, and 1-2MB/sec for a larger file. I''ve seen many other posts which are similar. I''ve tried diagnosing and I''m seeing the the IO seems to be throttled somewhere between the DomU and the Dom0. (Since they''re LVM partitions, I''m assuming Dom0 handles the LVM, because there is LVM xen module.) The strange thing I''m finding is that under the DomU, when the IO is being blocked, its blocking the entire kernel. In other words, even non-disk bound processes are pausing. For instance if you ssh to the box, even the network IO for SSH will pause for 10-20 seconds before it responds. Its very strange. I was on 3.0 and didn''t seem to see it as bad as when I moved to 3.1. xm info: release : 2.6.18-164.6.1.el5xen version : #1 SMP Tue Nov 3 17:53:47 EST 2009 machine : i686 nr_cpus : 4 nr_nodes : 1 sockets_per_node : 2 cores_per_socket : 2 threads_per_core : 1 cpu_mhz : 2210 hw_caps : 178bfbff:e3d3fbff:00000000:00000010:00000001:00000000:00000003 total_memory : 7422 free_memory : 0 node_to_cpu : node0:0-3 xen_major : 3 xen_minor : 1 xen_extra : .2-164.6.1.el5 xen_caps : xen-3.0-x86_32p xen_pagesize : 4096 platform_params : virt_start=0xf5800000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-46) cc_compile_by : mockbuild cc_compile_domain : centos.org cc_compile_date : Tue Nov 3 16:10:34 EST 2009 xend_config_format : 2 Any help would be appreciated. This is under CentOS 5.4, 8GB memory, 1TB SATA nv-sata controller. Tyan K8SRE S2891 motherboard. NVidia nForce 4Port SATAlink controller. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Nov-09 02:00 UTC
Re: [Xen-users] XEN 3.1 on 2.6.18-164.el5xen kernel - Slow IO & Pauses
On Sat, Nov 7, 2009 at 2:55 AM, Robert Taylor <rob@mpcs.com> wrote:> I''ve seen tons of posts about slow IO under Xen and others without slow IO > under xen. Here''s what I''m experiencing. I''ve got a XEN system on an 2 > dual core AMD Opteron 275 processors. > > I''m running DRBD for disk, but have also tested without DRBD. > Throughput in Dom-0 seems to be 50-80MB/sec for a file of any size. > Throughput in DomU is 20-30MB/sec for a file under 2GB, and 1-2MB/sec for > a larger file.How do you measure the throughput?> Any help would be appreciated. This is under CentOS 5.4, 8GB memory, 1TB > SATA nv-sata controller. Tyan K8SRE S2891 motherboard. NVidia nForce > 4Port SATAlink controller.I''m using a similar software setup (RHEL5), different hardware though. I''d suggest you start with doing "iostat -mx 3" on both dom0 and domU to see where the bottleneck is. My guess is that some process is using lots of I/O (possibly in small sizes) so that you''re practically IOPS bound, and the problem is not directly related to Xen. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Rob Taylor
2009-Nov-09 13:17 UTC
RE: [Xen-users] XEN 3.1 on 2.6.18-164.el5xen kernel - Slow IO & Pauses
Well, for the most part I''m using dd if=/dev/zero of=test bs=4k count=1000000 This writes a 4gig file to the disk. I''ve tried with 100M, 200M, 400M, 800M, 1G, 2G, 4G file sizes. 100M-800M the throughput is quite impressive. 1G the throughput is still 100+M/sec. 2G+ it goes down to 1-2M/sec on DomU. ---- I''ve done the iostat, and in most all cases the most of the usage is in the "await" state on the drive I''m testing. I''ll keep digging to see if I can find out more information. Thanks for the reply. -----Original Message----- From: Fajar A. Nugraha [mailto:fajar@fajar.net] Sent: Sunday, November 08, 2009 9:01 PM To: Robert Taylor Cc: xen-users@lists.xensource.com Subject: Re: [Xen-users] XEN 3.1 on 2.6.18-164.el5xen kernel - Slow IO & Pauses On Sat, Nov 7, 2009 at 2:55 AM, Robert Taylor <rob@mpcs.com> wrote:> I''ve seen tons of posts about slow IO under Xen and others without > slow IO under xen. Here''s what I''m experiencing. I''ve got a XEN > system on an 2 dual core AMD Opteron 275 processors. > > I''m running DRBD for disk, but have also tested without DRBD. > Throughput in Dom-0 seems to be 50-80MB/sec for a file of any size. > Throughput in DomU is 20-30MB/sec for a file under 2GB, and 1-2MB/sec > for a larger file.How do you measure the throughput?> Any help would be appreciated. This is under CentOS 5.4, 8GB memory, > 1TB SATA nv-sata controller. Tyan K8SRE S2891 motherboard. NVidia > nForce 4Port SATAlink controller.I''m using a similar software setup (RHEL5), different hardware though. I''d suggest you start with doing "iostat -mx 3" on both dom0 and domU to see where the bottleneck is. My guess is that some process is using lots of I/O (possibly in small sizes) so that you''re practically IOPS bound, and the problem is not directly related to Xen. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Fajar A. Nugraha
2009-Nov-10 03:41 UTC
Re: [Xen-users] XEN 3.1 on 2.6.18-164.el5xen kernel - Slow IO & Pauses
On Mon, Nov 9, 2009 at 8:17 PM, Rob Taylor <rob@mpcs.com> wrote:> Well, for the most part I''m using dd if=/dev/zero of=test bs=4k > count=1000000 > > This writes a 4gig file to the disk. I''ve tried with 100M, 200M, 400M, > 800M, 1G, 2G, 4G file sizes. > 100M-800M the throughput is quite impressive. > 1G the throughput is still 100+M/sec. > 2G+ it goes down to 1-2M/sec on DomU.Next question: what is the allocated memory for domU and dom0? I''m guessing around 1GB? To get meaningful results, you should use file size about 2x available memory. Otherwise it''d just be cached. -- Fajar _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users