On Mon, Dec 22, 2014 at 10:04:48AM -0500, John Baldwin wrote:> On Sunday, December 21, 2014 5:27:46 am Richard Perini wrote: > > > > We're struggling with an NFS negative name caching issue that results in > > a file created by an NFS client 'A' being invisible on client 'B' for up > > to client A's negnametimeo value. In our scenario, a process on client > > A creates a file, and passes a message to another process which may > > run on client B. The second process expects the file created by A to > > be available. > > Which NFS server are you using? If it is a FreeBSD NFS server, try changing > vfs.timestamp_precision to 2 (or 3) and seeing if that reduces the amount of > time you have to wait until the directory's ac timeout.Yes, we are running FreeBSD on the server machines. Unfortunately, our process really can't tolerate a delay of any length - either the file is present or its not.> Another possible the fix is to be careful to not open the file until you know > it exists if you still want to keep the reduced LOOKUP RPC load from caching > negative lookups.We have coded around the most common failure points with retry logic, but this is a hack, and there are some third party libraries involved that are not practical to fix in this manner.> > We're running a mix of 9-stable and 10-stable machines, and the problem is > > common to both. > > > > The obvious fix is to set the nfs mount option 'negnametimeo' to 0, but > > unfortunately we also have 'amd' in the picture (which we also need in our > > environment). Amd doesn't understand negnametimeo and ignores it, leaving > > it set to the system default of 60 seconds (as shown by nfsstat -m). > > Have you tried autofs for 10-stable? Is it able to pass this option to NFS > if you use it? If that works, I would prefer that to be the long term > solution for this. I'm not a huge fan of adding kernel options to override > each NFS default mount option if we can help it.I just ran up autofs and automountd on 10-stable, set the negnametimeo option in auto_master and it works a treat. However it will be quite some time before we're able to shift off 9 which leaves us with the kernel option as the easiest path. I'd point out that the nfs client code in /usr/src/sys/fs/nfsclient/nfsmount.h is already coded to allow override: ifndef NFS_DEFAULT_NEGNAMETIMEO #define NFS_DEFAULT_NEGNAMETIMEO 60 #endif so all that is required is the entry in the "options" file. Naturally we can add that ourselves (the beauty of open source :-) but it would be the only change to the native FreeBSD code for us, so of course we'd prefer to see it in the tree. Regards, and compliments of the season. --R
On Tuesday, December 23, 2014 10:25:27 AM Richard Perini wrote:> I just ran up autofs and automountd on 10-stable, set the negnametimeo > option in auto_master and it works a treat. However it will be quite > some time before we're able to shift off 9 which leaves us with the > kernel option as the easiest path. > > I'd point out that the nfs client code in > /usr/src/sys/fs/nfsclient/nfsmount.h is already coded to allow override: > > ifndef NFS_DEFAULT_NEGNAMETIMEO > #define NFS_DEFAULT_NEGNAMETIMEO 60 > #endif > > so all that is required is the entry in the "options" file. Naturally > we can add that ourselves (the beauty of open source :-) but it would > be the only change to the native FreeBSD code for us, so of course > we'd prefer to see it in the tree. > > Regards, and compliments of the season.I think for the future (10.x and later) the autofs solution is preferred, so the option would be unused if added to HEAD. (I also think setting it via autofs is preferable as it is better documented and is more fine-grained since it is per-mount.) Given that, I would only add the option on 9.x. However, I think there's another route you can use to set this option using a stock source tree. First, you could set COPTFLAGS in /etc/src.conf and add '-DNFS_DEFAULT_NEGNAMETIMEO=0' to it (e.g. set COPTFLAGS to "-O2 -pipe -D<blah"). Second, you could add the compiler directive to the 'DEBUG' line in your kernel config since that is added to CFLAGS, so if you have this in your config file: makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols you can add: makeoptions DEBUG+=-DNFS_DEFAULT_NEGNAMETIMEO=0 # Disable -ve name caching -- John Baldwin
On Sun, Dec 28, 2014 at 11:27:29AM -0500, John Baldwin wrote:> On Tuesday, December 23, 2014 10:25:27 AM Richard Perini wrote: > > I just ran up autofs and automountd on 10-stable, set the negnametimeo > > option in auto_master and it works a treat. However it will be quite > > some time before we're able to shift off 9 which leaves us with the > > kernel option as the easiest path. > > > > I'd point out that the nfs client code in > > /usr/src/sys/fs/nfsclient/nfsmount.h is already coded to allow override: > > > > ifndef NFS_DEFAULT_NEGNAMETIMEO > > #define NFS_DEFAULT_NEGNAMETIMEO 60 > > #endif > > > > so all that is required is the entry in the "options" file. Naturally > > we can add that ourselves (the beauty of open source :-) but it would > > be the only change to the native FreeBSD code for us, so of course > > we'd prefer to see it in the tree. > > > > Regards, and compliments of the season. > > I think for the future (10.x and later) the autofs solution is preferred, > so the option would be unused if added to HEAD. (I also think setting it > via autofs is preferable as it is better documented and is more fine-grained > since it is per-mount.) Given that, I would only add the option on 9.x. > However, I think there's another route you can use to set this option using a > stock source tree. First, you could set COPTFLAGS in /etc/src.conf and add > '-DNFS_DEFAULT_NEGNAMETIMEO=0' to it (e.g. set COPTFLAGS to > "-O2 -pipe -D<blah"). Second, you could add the compiler directive to > the 'DEBUG' line in your kernel config since that is added to CFLAGS, so if > you have this in your config file: > > makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols > > you can add: > > makeoptions DEBUG+=-DNFS_DEFAULT_NEGNAMETIMEO=0 # Disable -ve name cachingThanks John, The makeoptions magic is the route we'll take. It was the solution I was looking for when I encountered the options file. --R