Apologies for being terse, in a hurry here.
1) -o async doesn't work with NFS, don't use that.
2) how big are the text versus binary files?
3) how are you copying them over nfs?
I suspect, (could be wrong of course) that the ascii files
are a lot smaller than the binary files, so what's happening
is that for binary files, the client is issuing write-behind
async, however for ascii files its issuing the writes at
close time which will force the sync flag.
-Alfred
* Brent Jones <brent@servuhome.net> [090128 19:38]
wrote:> Hello FreeBSD users,
> I am running into some performance problems with NFSv3/v4 mounts.
> I have a Sun X4540 running OpenSolaris 2008.11 with ZFS exporting NFS
shares
> The NFS clients are a FreeBSD 6.3 32 bit, quad core xeon with 4GB ram
> and a FreeBSD 7.1 32bit with same hardware.
>
> The issue I am seeing, is that for certain file types, the FreeBSD NFS
> client will either issue an ASYNC write, or an FSYNC.
> However, NFSv3 and v4 both support "safe" ASYNC writes in the TCP
> versions of the protocol, so that should be the default.
> Issuing FSYNC's for every compete block transmitted adds substantial
> overhead and slows everything down.
>
> The two test files I have that can reproduce this data are a file
> created by 'dump' which is just binary data:
>
> $ file testbinery
> testbinery: data
>
> ASCII text file from a Maildir format:
>
> $ file ascittest
> ascittest: ASCII mail text
>
> My NFS mount command lines I have tried to get all data to ASYNC write:
>
> $ mount_nfs -3T -o async 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/
> $ mount_nfs -3T 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/
> $ mount_nfs -4TL 192.168.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/
>
> Here is an excerpt from a snoop from the binary data file:
>
> $ snoop rpc nfs
>
> obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3
> (read,lookup,modify,extend,delete,execute)
> pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend)
> obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 testbinery
> pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 OK FH=57D3
> obsmtp02.local -> pdxfilu01 NFS C ACCESS3 FH=57D3
> (read,lookup,modify,extend,delete,execute)
> pdxfilu01 -> obsmtp02.local NFS R ACCESS3 OK (read,modify,extend)
> obsmtp02.local -> pdxfilu01 NFS C SETATTR3 FH=57D3
> pdxfilu01 -> obsmtp02.local NFS R SETATTR3 OK
> obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 0 for 32768
(ASYNC)
> pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)
> obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 582647808 for
> 32768 (ASYNC)
> pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)
> obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 592871424 for
> 32768 (ASYNC)
> pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)
> obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=57D3 at 605421568 for
> 32768 (ASYNC)
> pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (ASYNC)
>
>
> And on and on.. it will acheive near full wire-speed, about 110MB/sec
> during the copy
>
>
> Here is the same snoop, only copying the ASCII mail file:
>
> $ snoop rpc nfs
>
> obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest
> pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory
> obsmtp02.local -> pdxfilu01 NFS C LOOKUP3 FH=BB85 ascittest
> pdxfilu01 -> obsmtp02.local NFS R LOOKUP3 No such file or directory
> obsmtp02.local -> pdxfilu01 NFS C CREATE3 FH=BB85 (UNCHECKED)
ascittest
> pdxfilu01 -> obsmtp02.local NFS R CREATE3 OK FH=69D3
> obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 0 for 32768
(FSYNC)
> pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC)
> obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 32768 for 32768
(FSYNC)
> pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC)
> obsmtp02.local -> pdxfilu01 NFS C WRITE3 FH=69D3 at 65536 for 32768
(FSYNC)
> pdxfilu01 -> obsmtp02.local NFS R WRITE3 OK 32768 (FSYNC)
>
>
> And so on. I've reproduced this with several files, and the only
> difference between tests is the file type.
> Is the FreeBSD NFS client requesting FSYNC or ASYNC depending on the
> file type/contents?
> If so, is there a tuneable setting to make all write ASYNC?
> Otherwise, FSYNC'ing for every block written over NFS will cause so
> many IOPS on the NFS server, that performance will degrade severely.
>
> Testing with an OpenSolaris 2008.11 client will issue ASYNC writes for
> any file type, if mounted with NFSv3 of NFSv4 (TCP).
>
> Any ideas?
>
> Thanks in advance!
>
> --
> Brent Jones
> brent@servuhome.net
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"
--
- Alfred Perlstein