Hello. We have a number of Xen 3.4.2. boxes which have constant iowaits at around 10% with spikes up to 100% when accessing data over NFS. We have been unable to nail down the issue. Any advice? System info: release : 2.6.18-194.3.1.el5xen version : #1 SMP Thu May 13 13:49:53 EDT 2010 machine : x86_64 nr_cpus : 16 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2261 hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 32758 free_memory : 3974 node_to_cpu : node0:0-15 node_to_memory : node0:3974 xen_major : 3 xen_minor : 4 xen_extra : .2 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-44) cc_compile_by : root cc_compile_domain : gitco.tld cc_compile_date : Wed Nov 11 21:16:28 CET 2009 xend_config_format : 4 And lsmod from a VM: Module Size Used by nfsd 284041 17 exportfs 38849 1 nfsd auth_rpcgss 81889 1 nfsd ipv6 424737 24 xfrm_nalgo 43333 1 ipv6 crypto_api 42945 1 xfrm_nalgo i2c_dev 43849 0 i2c_core 56129 1 i2c_dev nfs 289549 1 lockd 99185 3 nfsd,nfs fscache 52385 1 nfs nfs_acl 36673 2 nfsd,nfs sunrpc 197513 18 nfsd,auth_rpcgss,nfs,lockd,nfs_acl iptable_filter 36161 1 ip_tables 55329 1 iptable_filter x_tables 50377 1 ip_tables dm_mirror 54345 0 dm_multipath 55385 0 scsi_dh 41665 1 dm_multipath scsi_mod 197337 1 scsi_dh parport_pc 62313 0 lp 47121 0 parport 73293 2 parport_pc,lp pcspkr 36289 0 xennet 61129 0 [permanent] dm_raid45 98897 0 dm_message 36161 1 dm_raid45 dm_region_hash 46401 1 dm_raid45 dm_log 44993 3 dm_mirror,dm_raid45,dm_region_hash dm_mod 100753 4 dm_mirror,dm_multipath,dm_raid45,dm_log dm_mem_cache 39617 1 dm_raid45 ext3 168529 1 jbd 94257 1 ext3 uhci_hcd 57561 0 ohci_hcd 56053 0 ehci_hcd 65869 0 xenblk 50729 1 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Please inform the parameters you use in fstab when you mount your NFS share. Something like this: <server>:/<share> <mountpoint> nfs rsize=8192, wsize=8192, timeo=14, intr, tcp, bg, rw, bg, hard, nointr, rsize=32768, wsize=32768, tcp, actimeo=0, vers=3,timeo=600 Emerson -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Geoff Galitz Sent: Tuesday, September 14, 2010 11:04 AM To: xen-users@lists.xensource.com Subject: [Xen-users] IOwaits over NFS Hello. We have a number of Xen 3.4.2. boxes which have constant iowaits at around 10% with spikes up to 100% when accessing data over NFS. We have been unable to nail down the issue. Any advice? System info: release : 2.6.18-194.3.1.el5xen version : #1 SMP Thu May 13 13:49:53 EDT 2010 machine : x86_64 nr_cpus : 16 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2261 hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 32758 free_memory : 3974 node_to_cpu : node0:0-15 node_to_memory : node0:3974 xen_major : 3 xen_minor : 4 xen_extra : .2 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-44) cc_compile_by : root cc_compile_domain : gitco.tld cc_compile_date : Wed Nov 11 21:16:28 CET 2009 xend_config_format : 4 And lsmod from a VM: Module Size Used by nfsd 284041 17 exportfs 38849 1 nfsd auth_rpcgss 81889 1 nfsd ipv6 424737 24 xfrm_nalgo 43333 1 ipv6 crypto_api 42945 1 xfrm_nalgo i2c_dev 43849 0 i2c_core 56129 1 i2c_dev nfs 289549 1 lockd 99185 3 nfsd,nfs fscache 52385 1 nfs nfs_acl 36673 2 nfsd,nfs sunrpc 197513 18 nfsd,auth_rpcgss,nfs,lockd,nfs_acl iptable_filter 36161 1 ip_tables 55329 1 iptable_filter x_tables 50377 1 ip_tables dm_mirror 54345 0 dm_multipath 55385 0 scsi_dh 41665 1 dm_multipath scsi_mod 197337 1 scsi_dh parport_pc 62313 0 lp 47121 0 parport 73293 2 parport_pc,lp pcspkr 36289 0 xennet 61129 0 [permanent] dm_raid45 98897 0 dm_message 36161 1 dm_raid45 dm_region_hash 46401 1 dm_raid45 dm_log 44993 3 dm_mirror,dm_raid45,dm_region_hash dm_mod 100753 4 dm_mirror,dm_multipath,dm_raid45,dm_log dm_mem_cache 39617 1 dm_raid45 ext3 168529 1 jbd 94257 1 ext3 uhci_hcd 57561 0 ohci_hcd 56053 0 ehci_hcd 65869 0 xenblk 50729 1 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
We`ve used various NFS parameters in our testing to resolve this. Here are three different examples of what we have used on a single machine, with no significant improvement: 1) defaults,auto,tcp,bg,hard,intr,noatime 0 0 2) auto,noatime,rsize=32768,wsize=32768,bg,nfsvers=3,intr,tcp,retrans=0 0 0 3) noatime,bg,intr,nolock,tcp 0 0 Thanks. From: Ribeiro Emerson Gomes Sent: Tuesday, September 14, 2010 4:20 PM To: Geoff Galitz ; xen-users@lists.xensource.com Subject: RE: [Xen-users] IOwaits over NFS Please inform the parameters you use in fstab when you mount your NFS share. Something like this: <server>:/<share> <mountpoint> nfs rsize=8192, wsize=8192, timeo=14, intr, tcp, bg, rw, bg, hard, nointr, rsize=32768, wsize=32768, tcp, actimeo=0, vers=3,timeo=600 Emerson -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Geoff Galitz Sent: Tuesday, September 14, 2010 11:04 AM To: xen-users@lists.xensource.com Subject: [Xen-users] IOwaits over NFS Hello. We have a number of Xen 3.4.2. boxes which have constant iowaits at around 10% with spikes up to 100% when accessing data over NFS. We have been unable to nail down the issue. Any advice? System info: release : 2.6.18-194.3.1.el5xen version : #1 SMP Thu May 13 13:49:53 EDT 2010 machine : x86_64 nr_cpus : 16 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2261 hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 32758 free_memory : 3974 node_to_cpu : node0:0-15 node_to_memory : node0:3974 xen_major : 3 xen_minor : 4 xen_extra : .2 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-44) cc_compile_by : root cc_compile_domain : gitco.tld cc_compile_date : Wed Nov 11 21:16:28 CET 2009 xend_config_format : 4 And lsmod from a VM: Module Size Used by nfsd 284041 17 exportfs 38849 1 nfsd auth_rpcgss 81889 1 nfsd ipv6 424737 24 xfrm_nalgo 43333 1 ipv6 crypto_api 42945 1 xfrm_nalgo i2c_dev 43849 0 i2c_core 56129 1 i2c_dev nfs 289549 1 lockd 99185 3 nfsd,nfs fscache 52385 1 nfs nfs_acl 36673 2 nfsd,nfs sunrpc 197513 18 nfsd,auth_rpcgss,nfs,lockd,nfs_acl iptable_filter 36161 1 ip_tables 55329 1 iptable_filter x_tables 50377 1 ip_tables dm_mirror 54345 0 dm_multipath 55385 0 scsi_dh 41665 1 dm_multipath scsi_mod 197337 1 scsi_dh parport_pc 62313 0 lp 47121 0 parport 73293 2 parport_pc,lp pcspkr 36289 0 xennet 61129 0 [permanent] dm_raid45 98897 0 dm_message 36161 1 dm_raid45 dm_region_hash 46401 1 dm_raid45 dm_log 44993 3 dm_mirror,dm_raid45,dm_region_hash dm_mod 100753 4 dm_mirror,dm_multipath,dm_raid45,dm_log dm_mem_cache 39617 1 dm_raid45 ext3 168529 1 jbd 94257 1 ext3 uhci_hcd 57561 0 ohci_hcd 56053 0 ehci_hcd 65869 0 xenblk 50729 1 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
rsize and wsize are the most important ones to reduce your iowait. Those can dramatically change performance. The theory is that the best performance is reached when you match the geometry of the actual disks to the rsize/wsize nfs parameter. Use dd to emulate different block sizes and check the best performance for your environment. Emerson -----Original Message----- From: Geoff Galitz [mailto:geoff@galitz.org] Sent: Tuesday, September 14, 2010 11:33 AM To: Ribeiro Emerson Gomes; xen-users@lists.xensource.com Subject: Re: [Xen-users] IOwaits over NFS We`ve used various NFS parameters in our testing to resolve this. Here are three different examples of what we have used on a single machine, with no significant improvement: 1) defaults,auto,tcp,bg,hard,intr,noatime 0 0 2) auto,noatime,rsize=32768,wsize=32768,bg,nfsvers=3,intr,tcp,retrans=0 0 0 3) noatime,bg,intr,nolock,tcp 0 0 Thanks. From: Ribeiro Emerson Gomes <mailto:Emerson.Ribeiro@scania.com> Sent: Tuesday, September 14, 2010 4:20 PM To: Geoff Galitz <mailto:geoff@galitz.org> ; xen-users@lists.xensource.com Subject: RE: [Xen-users] IOwaits over NFS Please inform the parameters you use in fstab when you mount your NFS share. Something like this: <server>:/<share> <mountpoint> nfs rsize=8192, wsize=8192, timeo=14, intr, tcp, bg, rw, bg, hard, nointr, rsize=32768, wsize=32768, tcp, actimeo=0, vers=3,timeo=600 Emerson -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Geoff Galitz Sent: Tuesday, September 14, 2010 11:04 AM To: xen-users@lists.xensource.com Subject: [Xen-users] IOwaits over NFS Hello. We have a number of Xen 3.4.2. boxes which have constant iowaits at around 10% with spikes up to 100% when accessing data over NFS. We have been unable to nail down the issue. Any advice? System info: release : 2.6.18-194.3.1.el5xen version : #1 SMP Thu May 13 13:49:53 EDT 2010 machine : x86_64 nr_cpus : 16 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2261 hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 32758 free_memory : 3974 node_to_cpu : node0:0-15 node_to_memory : node0:3974 xen_major : 3 xen_minor : 4 xen_extra : .2 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-44) cc_compile_by : root cc_compile_domain : gitco.tld cc_compile_date : Wed Nov 11 21:16:28 CET 2009 xend_config_format : 4 And lsmod from a VM: Module Size Used by nfsd 284041 17 exportfs 38849 1 nfsd auth_rpcgss 81889 1 nfsd ipv6 424737 24 xfrm_nalgo 43333 1 ipv6 crypto_api 42945 1 xfrm_nalgo i2c_dev 43849 0 i2c_core 56129 1 i2c_dev nfs 289549 1 lockd 99185 3 nfsd,nfs fscache 52385 1 nfs nfs_acl 36673 2 nfsd,nfs sunrpc 197513 18 nfsd,auth_rpcgss,nfs,lockd,nfs_acl iptable_filter 36161 1 ip_tables 55329 1 iptable_filter x_tables 50377 1 ip_tables dm_mirror 54345 0 dm_multipath 55385 0 scsi_dh 41665 1 dm_multipath scsi_mod 197337 1 scsi_dh parport_pc 62313 0 lp 47121 0 parport 73293 2 parport_pc,lp pcspkr 36289 0 xennet 61129 0 [permanent] dm_raid45 98897 0 dm_message 36161 1 dm_raid45 dm_region_hash 46401 1 dm_raid45 dm_log 44993 3 dm_mirror,dm_raid45,dm_region_hash dm_mod 100753 4 dm_mirror,dm_multipath,dm_raid45,dm_log dm_mem_cache 39617 1 dm_raid45 ext3 168529 1 jbd 94257 1 ext3 uhci_hcd 57561 0 ohci_hcd 56053 0 ehci_hcd 65869 0 xenblk 50729 1 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
I neglected to mention that the problem is specifically for NFS mounts within the DomUs. Regardless, I''ll make a point of continuing our tuning using dd. Thanks. From: Ribeiro Emerson Gomes Sent: Tuesday, September 14, 2010 4:42 PM To: Geoff Galitz ; xen-users@lists.xensource.com Subject: RE: [Xen-users] IOwaits over NFS rsize and wsize are the most important ones to reduce your iowait. Those can dramatically change performance. The theory is that the best performance is reached when you match the geometry of the actual disks to the rsize/wsize nfs parameter. Use dd to emulate different block sizes and check the best performance for your environment. Emerson -----Original Message----- From: Geoff Galitz [mailto:geoff@galitz.org] Sent: Tuesday, September 14, 2010 11:33 AM To: Ribeiro Emerson Gomes; xen-users@lists.xensource.com Subject: Re: [Xen-users] IOwaits over NFS We`ve used various NFS parameters in our testing to resolve this. Here are three different examples of what we have used on a single machine, with no significant improvement: 1) defaults,auto,tcp,bg,hard,intr,noatime 0 0 2) auto,noatime,rsize=32768,wsize=32768,bg,nfsvers=3,intr,tcp,retrans=0 0 0 3) noatime,bg,intr,nolock,tcp 0 0 Thanks. From: Ribeiro Emerson Gomes Sent: Tuesday, September 14, 2010 4:20 PM To: Geoff Galitz ; xen-users@lists.xensource.com Subject: RE: [Xen-users] IOwaits over NFS Please inform the parameters you use in fstab when you mount your NFS share. Something like this: <server>:/<share> <mountpoint> nfs rsize=8192, wsize=8192, timeo=14, intr, tcp, bg, rw, bg, hard, nointr, rsize=32768, wsize=32768, tcp, actimeo=0, vers=3,timeo=600 Emerson -----Original Message----- From: xen-users-bounces@lists.xensource.com [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of Geoff Galitz Sent: Tuesday, September 14, 2010 11:04 AM To: xen-users@lists.xensource.com Subject: [Xen-users] IOwaits over NFS Hello. We have a number of Xen 3.4.2. boxes which have constant iowaits at around 10% with spikes up to 100% when accessing data over NFS. We have been unable to nail down the issue. Any advice? System info: release : 2.6.18-194.3.1.el5xen version : #1 SMP Thu May 13 13:49:53 EDT 2010 machine : x86_64 nr_cpus : 16 nr_nodes : 1 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2261 hw_caps : bfebfbff:28100800:00000000:00000340:009ce3bd:00000000:00000001:00000000 virt_caps : hvm total_memory : 32758 free_memory : 3974 node_to_cpu : node0:0-15 node_to_memory : node0:3974 xen_major : 3 xen_minor : 4 xen_extra : .2 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xffff800000000000 xen_changeset : unavailable cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-44) cc_compile_by : root cc_compile_domain : gitco.tld cc_compile_date : Wed Nov 11 21:16:28 CET 2009 xend_config_format : 4 And lsmod from a VM: Module Size Used by nfsd 284041 17 exportfs 38849 1 nfsd auth_rpcgss 81889 1 nfsd ipv6 424737 24 xfrm_nalgo 43333 1 ipv6 crypto_api 42945 1 xfrm_nalgo i2c_dev 43849 0 i2c_core 56129 1 i2c_dev nfs 289549 1 lockd 99185 3 nfsd,nfs fscache 52385 1 nfs nfs_acl 36673 2 nfsd,nfs sunrpc 197513 18 nfsd,auth_rpcgss,nfs,lockd,nfs_acl iptable_filter 36161 1 ip_tables 55329 1 iptable_filter x_tables 50377 1 ip_tables dm_mirror 54345 0 dm_multipath 55385 0 scsi_dh 41665 1 dm_multipath scsi_mod 197337 1 scsi_dh parport_pc 62313 0 lp 47121 0 parport 73293 2 parport_pc,lp pcspkr 36289 0 xennet 61129 0 [permanent] dm_raid45 98897 0 dm_message 36161 1 dm_raid45 dm_region_hash 46401 1 dm_raid45 dm_log 44993 3 dm_mirror,dm_raid45,dm_region_hash dm_mod 100753 4 dm_mirror,dm_multipath,dm_raid45,dm_log dm_mem_cache 39617 1 dm_raid45 ext3 168529 1 jbd 94257 1 ext3 uhci_hcd 57561 0 ohci_hcd 56053 0 ehci_hcd 65869 0 xenblk 50729 1 _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users
Geoff Galitz wrote:>I neglected to mention that the problem is specifically for NFS >mounts within the DomUs.Is it the same problems as using iSCSI mounts in DomUs ? There the underlying problems is, I believe, that the Dom0 networking is single threaded and introduces a bottleneck. I only use iSCSI on one of my VMs, but it has to be mounted in Dom0 as mounting it in DomU (as I''d have preferred) gives lousy performance. -- Simon Hobson Visit http://www.magpiesnestpublishing.co.uk/ for books by acclaimed author Gladys Hobson. Novels - poetry - short stories - ideal as Christmas stocking fillers. Some available as e-books. _______________________________________________ Xen-users mailing list Xen-users@lists.xensource.com http://lists.xensource.com/xen-users