Chad Leigh
2006-Sep-25 08:10 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
I have set up a Solaris 10 U2 06/06 system that has basic patches to the latest -19 kernel patch and latest zfs genesis etc as recommended. I have set up a basic pool (local) and a bunch of sub-pools (local/mail, local/mail/shire.net, local/mail/shire.net/o, local/jailextras/shire.net/irsfl, etc). I am exporting these with sharenfs=rw=@mysubnet,root=@mysubnet and then mounting a few of these pools on a FreeBSD system using nfsv3. The FreeBSD has about 4 of my 10 or so subpools mounted. 2 are email imap account tests, 1 is generic storage, and one is a FreeBSD jail root. FreeBSD mounts them with using TCP /sbin/mount_nfs -s -i -3 -T foo-i1:/local/mail/shire.net/o/obar /local/2/hobbiton/local/mail/shire.net/o/obar The systems are both directly connected to a gigabit switch using 1000btx-fdx and both have an MTU set at 9000. The Solaris side is an e1000g port (the system has 2 bge and 2 e1000g ports all configured) and the FreeBSD is a bge port. etc. I have heard that there are some ZFS/NFS sync performance problems etc that will be fixed in U3 or are fixed in OpenSolaris. I do not think my issue is related to that. I have also seen some of that with sometimes having pisspoor performance on writing. I have experienced the following issue several times since I started experimenting with this a few days ago. I periodically will get NFS server not responding errors on the FreeBSD machine for one of the mounted pools, and it will last 4-8 minutes or so and then come alive again and be fine for many hours. When this happens, access to the other mounted pools still works fine and logged directly in to the Solaris machine I am able to access the file systems (pools) just fine. Example error message: Sep 24 03:09:44 freebsdclient kernel: nfs server solzfs-i1:/local/jailextras/shire.net/irsfl: not responding Sep 24 03:10:15 freebsdclient kernel: nfs server solzfs-i1:/local/jailextras/shire.net/irsfl: not responding Sep 24 03:12:19 freebsdclient last message repeated 4 times Sep 24 03:14:54 freebsdclient last message repeated 5 times I would be interested in getting feedback on what might be the problem and also ways to track this down etc. Is this a know issue? Have others seen the nfs server sharing ZFS time out (but not for all pools)? Etc. Is there any functional difference with setting up the ZFS pools as legacy mounts and using a traditional share command to share them over nfs? I am mostly a Solaris noob and am happy to learn and can try anything people want me to test. Thanks in advance for any comments or help. thanks Chad This message posted from opensolaris.org
eric kustarz
2006-Sep-25 18:18 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
Chad Leigh wrote:>I have set up a Solaris 10 U2 06/06 system that has basic patches to the latest -19 kernel patch and latest zfs genesis etc as recommended. I have set up a basic pool (local) and a bunch of sub-pools (local/mail, local/mail/shire.net, local/mail/shire.net/o, local/jailextras/shire.net/irsfl, etc). I am exporting these with sharenfs=rw=@mysubnet,root=@mysubnet and then mounting a few of these pools on a FreeBSD system using nfsv3. > >The FreeBSD has about 4 of my 10 or so subpools mounted. 2 are email imap account tests, 1 is generic storage, and one is a FreeBSD jail root. FreeBSD mounts them with using TCP > >/sbin/mount_nfs -s -i -3 -T foo-i1:/local/mail/shire.net/o/obar /local/2/hobbiton/local/mail/shire.net/o/obar > >The systems are both directly connected to a gigabit switch using 1000btx-fdx and both have an MTU set at 9000. The Solaris side is an e1000g port (the system has 2 bge and 2 e1000g ports all configured) and the FreeBSD is a bge port. > >etc. > >I have heard that there are some ZFS/NFS sync performance problems etc that will be fixed in U3 or are fixed in OpenSolaris. I do not think my issue is related to that. I have also seen some of that with sometimes having pisspoor performance on writing. > >I have experienced the following issue several times since I started experimenting with this a few days ago. I periodically will get NFS server not responding errors on the FreeBSD machine for one of the mounted pools, and it will last 4-8 minutes or so and then come alive again and be fine for many hours. When this happens, access to the other mounted pools still works fine and logged directly in to the Solaris machine I am able to access the file systems (pools) just fine. > >Example error message: > >Sep 24 03:09:44 freebsdclient kernel: nfs server solzfs-i1:/local/jailextras/shire.net/irsfl: not responding >Sep 24 03:10:15 freebsdclient kernel: nfs server solzfs-i1:/local/jailextras/shire.net/irsfl: not responding >Sep 24 03:12:19 freebsdclient last message repeated 4 times >Sep 24 03:14:54 freebsdclient last message repeated 5 times > >I would be interested in getting feedback on what might be the problem and also ways to track this down etc. Is this a know issue? Have others seen the nfs server sharing ZFS time out (but not for all pools)? Etc. > >Could be lots of things - network partition, bad hardware, overloaded server, bad routers, etc. What''s the server''s load like (vmstat, prstat)? If you''re banging on the server too hard and using up the server''s resources then nfsd may not be able to respond to your client''s requests. You can also grab a snoop trace to see what packets are not being responded too? What are clients and local apps doing to the machine? What is your server hardware (# processors, memory) - is it underprovisioned for what you''re doing to it? How is the freeBSD NFS client code - robust? Are there any disk errors on the server (iostat -E, check /var/adm/messages, zpool iostat -x)? Is the network being flaky? eric>Is there any functional difference with setting up the ZFS pools as legacy mounts and using a traditional share command to share them over nfs? > >I am mostly a Solaris noob and am happy to learn and can try anything people want me to test. > >Thanks in advance for any comments or help. >thanks >Chad > > >This message posted from opensolaris.org >_______________________________________________ >zfs-discuss mailing list >zfs-discuss at opensolaris.org >http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > >
Chad Leigh -- Shire.Net LLC
2006-Sep-25 18:35 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
On Sep 25, 2006, at 12:18 PM, eric kustarz wrote:> Chad Leigh wrote: > >> I have set up a Solaris 10 U2 06/06 system that has basic patches >> to the latest -19 kernel patch and latest zfs genesis etc as >> recommended. I have set up a basic pool (local) and a bunch of >> sub-pools (local/mail, local/mail/shire.net, local/mail/shire.net/ >> o, local/jailextras/shire.net/irsfl, etc). I am exporting these >> with sharenfs=rw=@mysubnet,root=@mysubnet and then mounting a few >> of these pools on a FreeBSD system using nfsv3. >> >> The FreeBSD has about 4 of my 10 or so subpools mounted. 2 are >> email imap account tests, 1 is generic storage, and one is a >> FreeBSD jail root. FreeBSD mounts them with using TCP >> >> /sbin/mount_nfs -s -i -3 -T foo-i1:/local/mail/shire.net/o/obar / >> local/2/hobbiton/local/mail/shire.net/o/obar >> >> The systems are both directly connected to a gigabit switch using >> 1000btx-fdx and both have an MTU set at 9000. The Solaris side is >> an e1000g port (the system has 2 bge and 2 e1000g ports all >> configured) and the FreeBSD is a bge port. >> >> etc. >> >> I have heard that there are some ZFS/NFS sync performance problems >> etc that will be fixed in U3 or are fixed in OpenSolaris. I do >> not think my issue is related to that. I have also seen some of >> that with sometimes having pisspoor performance on writing. >> >> I have experienced the following issue several times since I >> started experimenting with this a few days ago. I periodically >> will get NFS server not responding errors on the FreeBSD machine >> for one of the mounted pools, and it will last 4-8 minutes or so >> and then come alive again and be fine for many hours. When this >> happens, access to the other mounted pools still works fine and >> logged directly in to the Solaris machine I am able to access the >> file systems (pools) just fine. >> >> Example error message: >> >> Sep 24 03:09:44 freebsdclient kernel: nfs server solzfs-i1:/local/ >> jailextras/shire.net/irsfl: not responding >> Sep 24 03:10:15 freebsdclient kernel: nfs server solzfs-i1:/local/ >> jailextras/shire.net/irsfl: not responding >> Sep 24 03:12:19 freebsdclient last message repeated 4 times >> Sep 24 03:14:54 freebsdclient last message repeated 5 times >> >> I would be interested in getting feedback on what might be the >> problem and also ways to track this down etc. Is this a know >> issue? Have others seen the nfs server sharing ZFS time out >> (but not for all pools)? Etc. >> > > Could be lots of things - network partition, bad hardware, > overloaded server, bad routers, etc. > > What''s the server''s load like (vmstat, prstat)? If you''re banging > on the server too hard and using up the server''s resources then > nfsd may not be able to respond to your client''s requests.The server is not doing anything except this ZFS / NFS serving and only 1 client is attached to it (the one with the problems). prstat shows a load of 0.00 continually and vmstat is typically like # vmstat kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s1 s2 -- -- in sy cs us sy id 0 0 0 10640580 691412 0 1 0 0 0 0 2 0 11 0 0 421 85 120 0 0 100 #> > You can also grab a snoop trace to see what packets are not being > responded too?If I can catch it happening. Most of the time I am not around and I just see it in the logs. Sometimes it happens when I do a "df -h" on the client for example.> > What are clients and local apps doing to the machine?Almost nothing. No local apps are running on the server. It is basically just doing ZFS and NFS. The client has 4 mounts from ZFS, all of them very low usage. 2 email accounts storage (imap maildir) are mounted for testing. Each receives 10-100 messages a day. 1 extra storage space is mounted and once a day rsync copies 2 files to it in the middle of the night -- one around 70mb and one 7mb. The other is being used as the root for a FreeBSD jail which is not being used for anything. Just proof of concept. No processes are running in the jail that are doing much of anything to the NFS mounted fiel system -- occasional log writes.> > What is your server hardware (# processors, memory) - is it > underprovisioned for what you''re doing to it?Tyan 2892 MB with a single dual core Opteron at 2.0 GHZ. 2GB memory. Single Areca 1130 raid card with 1gb RAM cache. Works very well with ZFS without the NFS component. (Has a 9 disk RAID 6 array on it). I have done lots of testing with this card and Solaris with and without ZFS and it has held up very well without any sort of IO issues. (Except the fact that it does not get a flush when the system powers down with init 5). The ZFS pools are currently on this single "disk" (to be augmented later this year when more funding comes through to buy more stuff) A dual port e1000g intel server card over PCIe is the Solaris side of the network.> > How is the freeBSD NFS client code - robust?I have not had issues with it over the last 10 years when mounting from other FreeBSD boxes. It seems to be robust.> > Are there any disk errors on the server (iostat -E, check /var/adm/ > messages, zpool iostat -x)?Nothing in /var/adm/messages . zpool iostat shows nothing iostat -E shows one illegal request error (I have had 3+ nfs episodes per day over the last 3 or 4 days) # iostat -E sd1 Soft Errors: 1 Hard Errors: 0 Transport Errors: 0 Vendor: Areca Product: ARC-1130-VOL#00 Revision: R001 Serial No: Size: 22.00GB <21999648256 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 1 Predictive Failure Analysis: 0 sd2 Soft Errors: 1 Hard Errors: 0 Transport Errors: 0 Vendor: Areca Product: ARC-1130-VOL#01 Revision: R001 Serial No: Size: 1898.00GB <1897998581248 bytes> Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 1 Predictive Failure Analysis: 0 #> > Is the network being flaky?Nothing else is running on it but when I do tests across it it seems to be running fine. I don''t get any errors. FreeBSD box is directly connected to the GB switch (which supports jumbo frames) and the Solaris box is directly connected. While an episode is happening, I have no problems at all accessing the other nfs shares on the FreeBSD box from the same ZFS pool. Nor do I have problems on the Solaris side with direct access to the ZFS pool being affected (ie, I can log in to the solaris box and cd to the space and do thing with no issues). Thanks Chad> > eric > >> Is there any functional difference with setting up the ZFS pools >> as legacy mounts and using a traditional share command to share >> them over nfs? >> >> I am mostly a Solaris noob and am happy to learn and can try >> anything people want me to test. >> >> Thanks in advance for any comments or help. >> thanks >> Chad >> This message posted from opensolaris.org >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> > > >--- Chad Leigh -- Shire.Net LLC Your Web App and Email hosting provider chad at shire.net -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2411 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060925/a6568219/attachment.bin>
Mike Kupfer
2006-Sep-25 19:15 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
>>>>> "Chad" == Chad Leigh <-- Shire.Net LLC" <chad at shire.net>> writes:Chad> On Sep 25, 2006, at 12:18 PM, eric kustarz wrote:>> You can also grab a snoop trace to see what packets are not being >> responded too?Chad> If I can catch it happening. Most of the time I am not around and Chad> I just see it in the logs. I''ve attached a hack script that runs snoop in the background and rotates the capture files. If you start it as (for example) bgsnoop <client> <server> it will save the last 6 hours of capture files between the two hosts. If you notice a problem in the logs, you can find the corresponding capture file and extract from it what you need. mike -------------- next part -------------- A non-text attachment was scrubbed... Name: bgsnoop Type: application/x-sh Size: 2246 bytes Desc: bgsnoop script URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060925/40f5150c/attachment.sh>
Chad Leigh -- Shire.Net LLC
2006-Sep-25 20:13 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
On Sep 25, 2006, at 1:15 PM, Mike Kupfer wrote:>>>>>> "Chad" == Chad Leigh <-- Shire.Net LLC" <chad at shire.net>> writes: > > Chad> On Sep 25, 2006, at 12:18 PM, eric kustarz wrote: > >>> You can also grab a snoop trace to see what packets are not being >>> responded too? > > Chad> If I can catch it happening. Most of the time I am not > around and > Chad> I just see it in the logs. > > I''ve attached a hack script that runs snoop in the background and > rotates the capture files. If you start it as (for example) > > bgsnoop <client> <server> > > it will save the last 6 hours of capture files between the two hosts. > If you notice a problem in the logs, you can find the corresponding > capture file and extract from it what you need.Hi Mike Thanks. I set this up like so ./bgsnoop.sh -d e1000g0 freebsd-internal since my nfs is not going out the "default" interface. Soon thereafter I caught the problem. In looking at the snoop.trace file I am not sure what to look for. There seems to be no packet headers or time stamps or anything -- just a lot of binary data. What am I looking for? Thanks Chad> > mike > <bgsnoop>--- Chad Leigh -- Shire.Net LLC Your Web App and Email hosting provider chad at shire.n -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2411 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060925/274739c8/attachment.bin>
Mike Kupfer
2006-Sep-25 20:49 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
>>>>> "Chad" == Chad Leigh <-- Shire.Net LLC" <chad at shire.net>> writes:Chad> There seems to be no packet headers or time stamps or anything -- Chad> just a lot of binary data. What am I looking for? Use "snoop -i <capture_file>" to decode the capture file. cheers, mike
Chad Leigh -- Shire.Net LLC
2006-Sep-25 21:28 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
On Sep 25, 2006, at 2:49 PM, Mike Kupfer wrote:>>>>>> "Chad" == Chad Leigh <-- Shire.Net LLC" <chad at shire.net>> writes: > > Chad> There seems to be no packet headers or time stamps or > anything -- > Chad> just a lot of binary data. What am I looking for? > > Use "snoop -i <capture_file>" to decode the capture file.OK, a little snoop help is required. I ran bgsnoop as follows: # ./bgsnoop.sh -t a -r -d e1000g0 According to the snoop man page -t [ r | a | d ] Time-stamp presentation. Time-stamps are accurate to within 4 microseconds. The default is for times to be presented in d (delta) format (the time since receiving the previous packet). Option a (abso- lute) gives wall-clock time. Option r (relative) gives time relative to the first packet displayed. This can be used with the -p option to display time relative to any selected packet. so -t a should show wall clock time But my feed looks like the following and I don''t see any "wall clock" time stamps. I need to be able to get some sort of wall-time stamp on this so that I can know where to look in my snoop dump for offending issues... 1 0.00000 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=50E5 (read,lookup,modify,extend,delete,execute) 2 0.00045 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=339B (read,lookup,modify,extend,delete,execute) 3 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=339B 1159219290.M400972P15189_courierlock.freebsd.shire.net 4 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=339B 1159219290.M400972P15189_courierlock.freebsd.shire.net 5 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C CREATE3 FH=339B (UNCHECKED) 1159219290.M400972P15189_courierlock.freebsd.shire.net 6 0.00045 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=878C (read,lookup,modify,extend,delete,execute) 7 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=50E5 tmp 8 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=339B 1159219290.M400972P15189_courierlock.freebsd.shire.net 9 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=878C (read,lookup,modify,extend,delete,execute) 10 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=878C (read,lookup,modify,extend,delete,execute) 11 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C WRITE3 FH=878C at 0 for 24 (ASYNC) 12 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=878C (read,lookup,modify,extend,delete,execute) 13 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=339B courier.lock 14 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C COMMIT3 FH=878C at 0 for 24 15 0.00032 freebsd-internal.shire.net -> bagend-i1 NFS C LINK3 FH=878C to FH=339B courier.lock 16 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=339B 1159219290.M400972P15189_courierlock.freebsd.shire.net 17 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C REMOVE3 FH=339B 1159219290.M400972P15189_courierlock.freebsd.shire.net 18 0.00032 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=339B (read,lookup,modify,extend,delete,execute) 19 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C FSSTAT3 FH=50E5 20 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C READDIR3 FH=339B Cookie=0 for 8192 21 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=339B courier.lock 22 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=339B 1159219290.M405999P15189_imapuid_164.freebsd.shire.net 23 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=339B 1159219290.M405999P15189_imapuid_164.freebsd.shire.net 24 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C CREATE3 FH=339B (UNCHECKED) 1159219290.M405999P15189_imapuid_164.freebsd.shire.net 25 0.00032 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=868C (read,lookup,modify,extend,delete,execute) 26 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=339B 1159219290.M405999P15189_imapuid_164.freebsd.shire.net 27 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=EE81 (read,lookup,modify,extend,delete,execute) 28 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=EE81 (read,lookup,modify,extend,delete,execute) 29 0.05840 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=868C (read,lookup,modify,extend,delete,execute) 30 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=868C (read,lookup,modify,extend,delete,execute) 31 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C SETATTR3 FH=868C 32 0.04253 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=87E5 (read,lookup,modify,extend,delete,execute) 33 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=87E5 (read,lookup,modify,extend,delete,execute) 34 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C FSSTAT3 FH=50E5 35 0.09871 freebsd-internal.shire.net -> bagend-i1 TCP D=2049 S=828 Ack=1388080234 Seq=414843977 Len=0 Win=65535 Options=<nop,nop,tstamp 633351866 31062869> 36 0.09588 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=3C9B (read,lookup,modify,extend,delete,execute) 37 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=3C9B (read,lookup,modify,extend,delete,execute) 38 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C FSSTAT3 FH=50E5 39 0.00432 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=339B 1159219290.M405999P15189_imapuid_164.freebsd.shire.net 40 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C REMOVE3 FH=339B 1159219290.M405999P15189_imapuid_164.freebsd.shire.net 41 0.00544 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=A6E5 (read,lookup,modify,extend,delete,execute) 42 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=A6E5 :list 43 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C SETATTR3 FH=A6E5 44 0.00078 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=50E5 courierimapkeywords 45 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=A6E5 (read,lookup,modify,extend,delete,execute) 46 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=A6E5 (read,lookup,modify,extend,delete,execute) 47 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C FSSTAT3 FH=50E5 48 0.02208 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=A6E5 .3767952.1130327968.H497313P29943.freebsd.shire.net 49 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=D4BD (read,lookup,modify,extend,delete,execute) 50 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=D4BD (read,lookup,modify,extend,delete,execute) 51 0.00032 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=A6E5 .3806200.1141843328.H928068P3390.freebsd.shire.net 52 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=DABD (read,lookup,modify,extend,delete,execute) 53 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=DABD (read,lookup,modify,extend,delete,execute) 54 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=A6E5 .3830089.1149025540.H610574P94084.freebsd.shire.net 55 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=CEBD (read,lookup,modify,extend,delete,execute) 56 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=CEBD (read,lookup,modify,extend,delete,execute) 57 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=A6E5 .3863990.1159190932.H248287P24183.freebsd.shire.net 58 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=6C9F (read,lookup,modify,extend,delete,execute) 59 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=6C9F (read,lookup,modify,extend,delete,execute) 60 0.00026 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=A6E5 .3864038.1159209259.H328539P97232.freebsd.shire.net 61 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=5E8F (read,lookup,modify,extend,delete,execute) 62 0.00013 freebsd-internal.shire.net -> bagend-i1 NFS C ACCESS3 FH=5E8F (read,lookup,modify,extend,delete,execute) 63 0.00111 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=50E5 tmp 64 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C LOOKUP3 FH=339B courier.lock 65 0.00019 freebsd-internal.shire.net -> bagend-i1 NFS C REMOVE3 FH=339B courier.lock 66 0.09890 freebsd-internal.shire.net -> bagend-i1 TCP D=2049 S=828 Ack=1388085198 Seq=414847545 Len=0 Win=65535 Options=<nop,nop,tstamp 633351971 31062892> freebsd> > cheers, > mike--- Chad Leigh -- Shire.Net LLC Your Web App and Email hosting provider chad at shire.net -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2411 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060925/613ec592/attachment.bin>
Mike Kupfer
2006-Sep-25 21:54 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
>>>>> "Chad" == Chad Leigh <-- Shire.Net LLC" <chad at shire.net>> writes:Chad> so -t a should show wall clock time The capture file always records absolute time. So you (just) need to use "-t a" when you decode the capture file. Sorry for not making the clear earlier. mike
Chad Leigh -- Shire.Net LLC
2006-Sep-25 22:23 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
On Sep 25, 2006, at 3:54 PM, Mike Kupfer wrote:>>>>>> "Chad" == Chad Leigh <-- Shire.Net LLC" <chad at shire.net>> writes: > > Chad> so -t a should show wall clock time > > The capture file always records absolute time. So you (just) need to > use "-t a" when you decode the capture file. > > Sorry for not making the clear earlier.OK, thanks. Sorry for being such a noob with "snoop". I guess it is kind of obvious now that you would put that on the snoop that reads the file and outputs the human readable one and not the one that saves things away... This appears to be the only stuff having to do with the hanging server (lots of other stuff that is with other zfs pools that are served over nfs) 68 15:29:27.53298 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC 72 15:29:28.54294 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC (retransmit) 73 15:29:29.54312 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC (retransmit) 74 15:29:31.54356 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC (retransmit) 75 15:29:35.54443 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC (retransmit) 76 15:29:43.54610 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC (retransmit) 5890 15:29:59.55835 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC 5993 15:30:31.56506 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC (retransmit) 6124 15:31:35.58971 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC (retransmit) 6346 15:32:44.23048 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC 6347 15:32:44.23585 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC (retransmit) 6755 15:34:40.56138 freebsd-internal.shire.net -> solaris-zfs-i1 NFS C FSSTAT3 FH=84EC comes alive again again right about 6347 15:32:22.23585 based on matching log entries and this snoop snoop does not show me the reply packets going back. What do I need to do to go both ways? Thanks Chad> > mike--- Chad Leigh -- Shire.Net LLC Your Web App and Email hosting provider chad at shire.net -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2411 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060925/f56c404f/attachment.bin>
Chad Leigh -- Shire.Net LLC
2006-Sep-26 06:22 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
Chad Leigh -- Shire.Net LLC wrote:> I have set up a Solaris 10 U2 06/06 system that has basic patches > to the latest -19 kernel patch and latest zfs genesis etc as > recommended. I have set up a basic pool (local) and a bunch of sub- > pools (local/mail, local/mail/shire.net, local/mail/shire.net/o, > local/jailextras/shire.net/irsfl, etc). I am exporting these with > sharenfs=rw=@mysubnet,root=@mysubnet and then mounting a few of > these pools on a FreeBSD system using nfsv3. > > The FreeBSD has about 4 of my 10 or so subpools mounted. 2 are > email imap account tests, 1 is generic storage, and one is a > FreeBSD jail root. FreeBSD mounts them with using TCP > > /sbin/mount_nfs -s -i -3 -T foo-i1:/local/mail/shire.net/o/obar / > local/2/hobbiton/local/mail/shire.net/o/obar > > The systems are both directly connected to a gigabit switch using > 1000btx-fdx and both have an MTU set at 9000. The Solaris side is > an e1000g port (the system has 2 bge and 2 e1000g ports all > configured) and the FreeBSD is a bge port. > > etc. > > I have heard that there are some ZFS/NFS sync performance problems > etc that will be fixed in U3 or are fixed in OpenSolaris. I do not > think my issue is related to that. I have also seen some of that > with sometimes having pisspoor performance on writing. > > I have experienced the following issue several times since I > started experimenting with this a few days ago. I periodically > will get NFS server not responding errors on the FreeBSD machine > for one of the mounted pools, and it will last 4-8 minutes or so > and then come alive again and be fine for many hours. When this > happens, access to the other mounted pools still works fine and > logged directly in to the Solaris machine I am able to access the > file systems (pools) just fine. > > Example error message: > > Sep 24 03:09:44 freebsdclient kernel: nfs server solzfs-i1:/local/ > jailextras/shire.net/irsfl: not responding > Sep 24 03:10:15 freebsdclient kernel: nfs server solzfs-i1:/local/ > jailextras/shire.net/irsfl: not responding > Sep 24 03:12:19 freebsdclient last message repeated 4 times > Sep 24 03:14:54 freebsdclient last message repeated 5 times > > I would be interested in getting feedback on what might be the > problem and also ways to track this down etc. Is this a know > issue? Have others seen the nfs server sharing ZFS time out (but > not for all pools)? Etc.Offline someone suggested I put some snoop files up so people can look at them The following is a snoop file created on the Solaris ZFS server. The incident happens at the very end. http://www.shire.net/snoop-solzfs.txt.gz The following is a tcpdump on the BSD nfs client. The system clocks are pretty much in sync so it should be easy to find the location in this file to match up. I am not sure about the various checksum issues tcpdump shows but I think they are a tcpdump artifact as they are throughout the file and not associated with the issue I am having http://www.shire.net/bsd-tcpdump.txt.gz The system log messages that show the issue are: (help you find the location in the snoop file) (this was a short lived one or I caught it in the middle of an episode when I did a "df -h" on the client. Sep 25 21:11:59 bywater kernel: nfs server bagend-i1:/local/ jailextras/shire.net/irsfl: not responding Sep 25 21:12:02 bywater kernel: nfs server bagend-i1:/local/ jailextras/shire.net/irsfl: is alive again --- Chad Leigh -- Shire.Net LLC Your Web App and Email hosting provider chad at shire.net -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2411 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060926/ac11eda2/attachment.bin>
Mike Kupfer
2006-Sep-26 18:24 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
>>>>> "Chad" == Chad Leigh <-- Shire.Net LLC" <chad at shire.net>> writes:Chad> snoop does not show me the reply packets going back. What do I Chad> need to do to go both ways? It''s possible that performance issues are causing snoop to miss the replies. If your server has multiple network interfaces, it''s more likely that the server is routing the replies back on a different interface. We''ve run into that problem many times with the NFS server that has my home directory on it. If that is what''s going on, you need to fire up multiple instances of snoop, one per interface. mike
Chad Leigh -- Shire.Net LLC
2006-Sep-26 18:26 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
On Sep 26, 2006, at 12:24 PM, Mike Kupfer wrote:>>>>>> "Chad" == Chad Leigh <-- Shire.Net LLC" <chad at shire.net>> writes: > > Chad> snoop does not show me the reply packets going back. What do I > Chad> need to do to go both ways? > > It''s possible that performance issues are causing snoop to miss the > replies. > > If your server has multiple network interfaces, it''s more likely that > the server is routing the replies back on a different interface. > We''ve > run into that problem many times with the NFS server that has my home > directory on it. If that is what''s going on, you need to fire up > multiple instances of snoop, one per interface.OK, I will try that. I did run tcpdump on the BSD client as well so the responses should show up there as well as it only has the 1 interface on that net while the Solaris box has 3. Thanks Chad> > mike--- Chad Leigh -- Shire.Net LLC Your Web App and Email hosting provider chad at shire.net -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2411 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060926/a52f7b44/attachment.bin>
Chad Leigh -- Shire.Net LLC
2006-Sep-28 07:54 UTC
[zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
On Sep 26, 2006, at 12:26 PM, Chad Leigh -- Shire.Net LLC wrote:> > On Sep 26, 2006, at 12:24 PM, Mike Kupfer wrote: > >>>>>>> "Chad" == Chad Leigh <-- Shire.Net LLC" <chad at shire.net>> >>>>>>> writes: >> >> Chad> snoop does not show me the reply packets going back. What do I >> Chad> need to do to go both ways? >> >> It''s possible that performance issues are causing snoop to miss the >> replies. >> >> If your server has multiple network interfaces, it''s more likely that >> the server is routing the replies back on a different interface. >> We''ve >> run into that problem many times with the NFS server that has my home >> directory on it. If that is what''s going on, you need to fire up >> multiple instances of snoop, one per interface. > > OK, I will try that. I did run tcpdump on the BSD client as well > so the responses should show up there as well as it only has the 1 > interface on that net while the Solaris box has 3.That got me thinking. Since I had 3 "dedicated" ports to use for nfs, I changed it so each is on its own network (192.168.2 .3 . 4) so there is no port switcheroo on incoming and outgoing port. I also upgraded the FreeBSD to catch any bge updates and patches (there were some I think but I am not sure they had anything to do with my issue). Anyway, after doing both of these my issue seems to have gone away... I am still testing / watching but I have not seen or experienced the issue in a day. I am not sure which one "fixed" my problem but it seems to have gone away. Thanks Chad> > Thanks > Chad > >> >> mike > > --- > Chad Leigh -- Shire.Net LLC > Your Web App and Email hosting provider > chad at shire.net > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss--- Chad Leigh -- Shire.Net LLC Your Web App and Email hosting provider chad at shire.net -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2411 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20060928/3537113c/attachment.bin>
Chad Leigh -- Shire.Net LLC
2006-Oct-06 08:07 UTC
It''s Back (but worse and different) Re: [zfs-discuss] problem ZFS / NFS from FreeBSD nfsv3 client -- periodic NFS server not resp
On Sep 28, 2006, at 1:54 AM, Chad Leigh -- Shire.Net LLC wrote:> > On Sep 26, 2006, at 12:26 PM, Chad Leigh -- Shire.Net LLC wrote: > >> >> On Sep 26, 2006, at 12:24 PM, Mike Kupfer wrote: >> >>>>>>>> "Chad" == Chad Leigh <-- Shire.Net LLC" <chad at shire.net>> >>>>>>>> writes: >>> >>> Chad> snoop does not show me the reply packets going back. What >>> do I >>> Chad> need to do to go both ways? >>> >>> It''s possible that performance issues are causing snoop to miss the >>> replies. >>> >>> If your server has multiple network interfaces, it''s more likely >>> that >>> the server is routing the replies back on a different interface. >>> We''ve >>> run into that problem many times with the NFS server that has my >>> home >>> directory on it. If that is what''s going on, you need to fire up >>> multiple instances of snoop, one per interface. >> >> OK, I will try that. I did run tcpdump on the BSD client as well >> so the responses should show up there as well as it only has the 1 >> interface on that net while the Solaris box has 3. > > That got me thinking. Since I had 3 "dedicated" ports to use for > nfs, I changed it so each is on its own network (192.168.2 .3 . > 4) so there is no port switcheroo on incoming and outgoing port. I > also upgraded the FreeBSD to catch any bge updates and patches > (there were some I think but I am not sure they had anything to do > with my issue). Anyway, after doing both of these my issue seems > to have gone away... I am still testing / watching but I have not > seen or experienced the issue in a day. I am not sure which one > "fixed" my problem but it seems to have gone away. >Ok, the problem started again today or yesterday (wed or tuesday) and seems to be "worse" than ever. Some symptoms are different. Before it was just the one specific zfs pool (say local/mail/foo.com/ c/chad ) that was not responding. And, if I remember correctly, I could have a shell on the Solaris side and still do things on the local zfs side -- only the nfs server is not responding. Now, the whole zfs system seems to freeze up and all nfs served pools are unavailable at the same time and I cannot access them independently on the solaris side in a shell (in fact, I cannot even log in during the episodes since my user home directory in in a zfs pool -- /local/ home/chad in the local/home/chad pool). Previously they woujld last 3-10 min. Now they seem to be lasting 15-20 minutes or so. I do not think this is the whole-pool-sync issue with nfs as it shouldn''t take 15-20 minutes to do a sync of the pool (about 1.6-2.0GB across all zfs based pools/FS [about 10, mostly empty or almost empty] on the machine with almost all of that in one pool -- an email store for one account that is about 1.6GB) and it doesn''t happen on every file write. No patches have been applied since I last reported it was OK about a week ago. Chad --- Chad Leigh -- Shire.Net LLC Your Web App and Email hosting provider chad at shire.net -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2411 bytes Desc: not available URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20061006/1b40c2a7/attachment.bin>