xen discuss - Apr 2009 - bad nfs performance with opensolaris dom0 server and ubuntu intrepid domu client

Hi

I am trying to track down a problem with nfs performance between an opensolaris
nfs server in dom0 and an ubuntu pv domu client. I would be
gratefull for any tips on how to identify the problem.
.
I have /home from the opensolaris dom0 mounted as /home in the ubuntu domu. 
Read performance on the client is very slow:

robbie@ubuntu:~$ dd if=intrepid-desktop-i386.iso  of=/dev/null
1424672+0 records in
1424672+0 records out
729432064 bytes (729 MB) copied, 1389.73 s, 525 kB/s
robbie@ubuntu:~$ dd if=intrepid-desktop-i386.iso  bs=32k[K[K[K[K=32k
of=/dev/null
22260+1 records in
22260+1 records out
729432064 bytes (729 MB) copied, 1376.52 s, 530 kB/s
robbie@ubuntu:~$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:16:3e:52:b7:35  
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::216:3eff:fe52:b735/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3425312 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2270502 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:4775043508 (4.7 GB)  TX bytes:1016651840 (1.0 GB)
          Interrupt:9 

robbie@ubuntu:~$ dd if=intrepid-desktop-i386.iso  bs=32k of=/dev/null
22260+1 records in
22260+1 records out
729432064 bytes (729 MB) copied, 1395.64 s, 523 kB/s
robbie@ubuntu:~$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:16:3e:52:b7:35  
          inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
          inet6 addr: fe80::216:3eff:fe52:b735/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3937117 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2547334 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5534387560 (5.5 GB)  TX bytes:1036869559 (1.0 GB)
          Interrupt:9 

It is much faster if I use cifs:

dd if=intrepid-desktop-i386.iso  bs=32k of=/dev/null
22260+1 records in
22260+1 records out
729432064 bytes (729 MB) copied, 13.0277 s, 56.0 MB/s

Regards

Robbie

Robert Murray wrote:
> Hi
> 
> I am trying to track down a problem with nfs performance between an
opensolaris nfs server in dom0 and an ubuntu pv domu client. I would be
> gratefull for any tips on how to identify the problem.
> .
> I have /home from the opensolaris dom0 mounted as /home in the ubuntu domu.
Read performance on the client is very slow:

Did you try mounting with nfsv3 or nfsv2?  Is it using tcp or udp?



MRJ


> robbie@ubuntu:~$ dd if=intrepid-desktop-i386.iso  of=/dev/null
> 1424672+0 records in
> 1424672+0 records out
> 729432064 bytes (729 MB) copied, 1389.73 s, 525 kB/s
> robbie@ubuntu:~$ dd if=intrepid-desktop-i386.iso  bs=32k[K[K[K[K=32k
of=/dev/null
> 22260+1 records in
> 22260+1 records out
> 729432064 bytes (729 MB) copied, 1376.52 s, 530 kB/s
> robbie@ubuntu:~$ ifconfig eth0
> eth0      Link encap:Ethernet  HWaddr 00:16:3e:52:b7:35  
>           inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
>           inet6 addr: fe80::216:3eff:fe52:b735/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:3425312 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:2270502 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000 
>           RX bytes:4775043508 (4.7 GB)  TX bytes:1016651840 (1.0 GB)
>           Interrupt:9 
> 
> robbie@ubuntu:~$ dd if=intrepid-desktop-i386.iso  bs=32k of=/dev/null
> 22260+1 records in
> 22260+1 records out
> 729432064 bytes (729 MB) copied, 1395.64 s, 523 kB/s
> robbie@ubuntu:~$ ifconfig eth0
> eth0      Link encap:Ethernet  HWaddr 00:16:3e:52:b7:35  
>           inet addr:10.0.0.2  Bcast:10.0.0.255  Mask:255.255.255.0
>           inet6 addr: fe80::216:3eff:fe52:b735/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:3937117 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:2547334 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000 
>           RX bytes:5534387560 (5.5 GB)  TX bytes:1036869559 (1.0 GB)
>           Interrupt:9 
> 
> It is much faster if I use cifs:
> 
> dd if=intrepid-desktop-i386.iso  bs=32k of=/dev/null
> 22260+1 records in
> 22260+1 records out
> 729432064 bytes (729 MB) copied, 13.0277 s, 56.0 MB/s
> 
> Regards
> 
> Robbie
> 
> _______________________________________________
> xen-discuss mailing list
> xen-discuss@opensolaris.org

On Thu, Apr 16, 2009 at 03:18:57PM -0400, Mark Johnson
wrote:
>
>
>
> Did you try mounting with nfsv3 or nfsv2?  Is it using tcp or udp?
I did try nfs v3 with the same result, and it is tcp.

I think I have found the problem.  The default rsize and wsize are both 1048576.
I have set rsize and wsize to 32768 in /etc/fstab for /home, but
the filesystems under /home which are automatically mounted don''t
inherit these options.

I have no idea how to fix this, but as it is not xen related I will look
elsewhere.

Thanks for your help, I found this when looking at /proc/mounts to see if it was
udp or tcp:-)

Regards

Robbie

On Thu, Apr 16, 2009 at 08:06:14PM +0100, Robert Murray
wrote:
> Hi
> 
> I am trying to track down a problem with nfs performance between an
opensolaris nfs server in dom0 and an ubuntu pv domu client. I would be
> gratefull for any tips on how to identify the problem.
> .
> I have /home from the opensolaris dom0 mounted as /home in the ubuntu domu.
Read performance on the client is very slow:
> 

I am curious how you got that far.  I have an opensolaris and an ubuntu
VM - the automount works for the opensolaris, but not for the ubuntu.  I
can ls -l /net/mydom0/export, but not /etc/mydom0/export/home.  [And
both have mode dr-xr-xr-x].


-- 
/*********************************************************************\
**
** Joe Yao				jsdy@tux.org - Joseph S. D. Yao
**
\*********************************************************************/

On Fri, Apr 17, 2009 at 01:02:48PM -0400, Joseph S D Yao
wrote:
> I am curious how you got that far.  I have an opensolaris and an ubuntu
> VM - the automount works for the opensolaris, but not for the ubuntu.  I
> can ls -l /net/mydom0/export, but not /etc/mydom0/export/home.  [And
> both have mode dr-xr-xr-x].
I am not using an automounter, /home is listed in /etc/fstab in the Ubuntu
domain.  The filesystems under /home are mounted automatically by NFSv4
when they are accessed, but the rsize and wsize set on /home are not propagated.
The default of 1048576 seams to cause very bad performance, so I
have listed each filesystem in /etc/fstab with rsize=32768,wsize=32768.

Regards

Robbie

I have been noticing this problem as well. The Ubuntu domU ends up spinning on
iowait with average sysload getting as high as 12! Any idea? This seems to be
new behavior, I suspect an Ubuntu update caused it, but I don''t know
which one. Both read and write have been crippled with NFS. SMB/CIFS works
significantly better. I''m moving performance sensitive files that
don''t need to be shared from the dom0 to mounts in the VM with
"virsh attach-disk". That seems to be helping. I would like to fix the
problem though. I believe there was an Ubuntu kernel update recently. Perhaps I
should try the prior kernel.

Doing a "cp" to or from the NFS mount causes iowait to peg at 50% (2
CPUs in the VM) and causes NFS performance problems for other clients on the
LAN. CPU and RAM are fine both in the dom0 and domU. Even "ls" is
incredibly slow. The same operation with SMB/CIFS is significantly faster,
doesn''t leave iowait pegged high, and actually uses some CPU for the
"cp" and SMB processes, while not causing the problems for other
clients on the LAN. This is very strange as NFS used to be slightly more
performant.

I tried disabling ZIL with no change, not that I expected anything as zilstat
was showing rather low ZIL usage.

Ubuntu: 
Linux mythtv 2.6.31-20-server #58-Ubuntu SMP Fri Mar 12 05:40:05 UTC 2010 x86_64
GNU/Linux

OpenSolaris b129. I''ve been holding on this build as it fixed the MPT
errors I was having and the 2010.03 release was getting close. That''s
been delayed, but I haven''t noticed any trouble until now.
-- 
This message posted from opensolaris.org

I figured out the problem for my setup. I think it was connecting with NFS4. I
added nfsvers=3 to the options list for the automounter and all is well now. I
suspect the default changed with the updates to Ubuntu.
-- 
This message posted from opensolaris.org