CentOS - May 2019 - Slow performance with NFSv4.1 on CentOS 7.5 ?

We have a number of identical NFS clients mounting a server using 
NFSv4.1 - server and clients are all running CentOS 7.5 (kernel 
3.10.0-862.14.4.el7.x86_64)

However, on some clients, the NFS performance 'degrades' with time ...

Running a simple test - a python script that just imports a module 
(python and its modules are installed on the NFS share) can be an order 
of magnitude or more slower on some clients. i.e. very little data is 
transferred, it is the rate of stat'ing and opening files on the NFS 
server that is 'slow'

Running a tcpdump on a 'slow' client shows that the NFS traffic 
generated on the 'slow' client is again an order of magnitude or more 
when compared with that generated by a 'fast' client

The majority of the extra NFS traffic in the slow case, appears to be a 
large number of NFS 'TEST_STATEID' calls the client makes - which are 
not there in the tcpdump on the fast client

The issue can be 'fixed' in the short term by rebooting the affected 
client - and after a reboot, running the same tcpdump shows no 
TEST_STATEID calls - however after a while (several days), the 
performance might degrade again

I've found a number of reports of excessive TEST_STATEID calls - but 
most seem to relate to NFSv4 client hangs - which is not happening here 
- things are working, but much slower than they should be ...

Has anyone come across this issue - and have any fixes/workarounds?

Thanks

James Pearson

James Pearson wrote:
> We have a number of identical NFS clients mounting a server using
> NFSv4.1 - server and clients are all running CentOS 7.5 (kernel
> 3.10.0-862.14.4.el7.x86_64)
>
> However, on some clients, the NFS performance 'degrades' with time
...
>
> Running a simple test - a python script that just imports a module
> (python and its modules are installed on the NFS share) can be an order
> of magnitude or more slower on some clients. i.e. very little data is
> transferred, it is the rate of stat'ing and opening files on the NFS
> server that is 'slow'
>
> Running a tcpdump on a 'slow' client shows that the NFS traffic
> generated on the 'slow' client is again an order of magnitude or
more when
> compared with that generated by a 'fast' client
>
> The majority of the extra NFS traffic in the slow case, appears to be a
> large number of NFS 'TEST_STATEID' calls the client makes - which
are not
> there in the tcpdump on the fast client
>
> The issue can be 'fixed' in the short term by rebooting the
affected
> client - and after a reboot, running the same tcpdump shows no TEST_STATEID
> calls - however after a while (several days), the performance might
> degrade again
>
> I've found a number of reports of excessive TEST_STATEID calls - but
> most seem to relate to NFSv4 client hangs - which is not happening here -
> things are working, but much slower than they should be ...
>
> Has anyone come across this issue - and have any fixes/workarounds?
You might find some useful info in
<http://www.admin-magazine.com/HPC/Articles/Useful-NFS-Options-for-Tuning-and-Management>

On the other hand, we've had some issues, and on at least a couple
servers, I've forced it to use 4.0 instead of 4.1, based on some
information I found online, and it helped.

      mark

On Tue, May 07, 2019 at 03:00:54PM +0000, James Pearson
wrote:
> We have a number of identical NFS clients mounting a server using 
> NFSv4.1 - server and clients are all running CentOS 7.5 (kernel 
> 3.10.0-862.14.4.el7.x86_64)
Have you updated to the supported CentOS7 kernel and NFS utilities?
There have been dozens of updates to the NFS code in the kernel since
the kernel you're running.

-- 
Jonathan Billings <billings at negate.org>

James Pearson wrote:
> 
> We have a number of identical NFS clients mounting a server using
> NFSv4.1 - server and clients are all running CentOS 7.5 (kernel
> 3.10.0-862.14.4.el7.x86_64)
> 
> However, on some clients, the NFS performance 'degrades' with time
...
> 
> Running a simple test - a python script that just imports a module
> (python and its modules are installed on the NFS share) can be an order
> of magnitude or more slower on some clients. i.e. very little data is
> transferred, it is the rate of stat'ing and opening files on the NFS
> server that is 'slow'
> 
> Running a tcpdump on a 'slow' client shows that the NFS traffic
> generated on the 'slow' client is again an order of magnitude or
more
> when compared with that generated by a 'fast' client
> 
> The majority of the extra NFS traffic in the slow case, appears to be a
> large number of NFS 'TEST_STATEID' calls the client makes - which
are
> not there in the tcpdump on the fast client
> 
> The issue can be 'fixed' in the short term by rebooting the
affected
> client - and after a reboot, running the same tcpdump shows no
> TEST_STATEID calls - however after a while (several days), the
> performance might degrade again
> 
> I've found a number of reports of excessive TEST_STATEID calls - but
> most seem to relate to NFSv4 client hangs - which is not happening here
> - things are working, but much slower than they should be ...
> 
> Has anyone come across this issue - and have any fixes/workarounds?
After a bit of further poking about, it looks like we are hitting the 
issue described in:

  https://bugzilla.redhat.com/show_bug.cgi?id=1552203

The fix should be in 7.7, but the linked:

  https://access.redhat.com/solutions/3915571

(login required)

has a few suggested workarounds which includes mounting with NFSv4.0 or 
disabling 'NFSv4 delegations' on the server via:

  sysctl -w fs.leases-enable=0

I've used the above sysctl which appears to have fixed the issue - 
although I had to restart NFS on the server to notice any change with 
the affected clients - so I'm not sure if the sysctl change or the NFS 
restart 'fixed' the issue ...

James Pearson