dtrace discuss - Jun 2009 - NFS file locking issues on an X4500 running OpenSolaris 2008.11 w/ZFS

I have an X4500 running OpenSolaris 2008.11 (snv_101b) operating as an NFS v3/v4
server to about 80 clients.

Over the last few months, we''ve been seeing an issue where file locks
start to take a very long time (60-90 seconds), and the number of running lockd
threads rises up into the 50-100 range from its normal range of 3-5.

During these storms, the system seems otherwise healthy. File serving
performance is speedy,  the load is below 2, we have lots of free RAM and swap,
and we''re not pushing more IOs that normal.

Sun support wants us to panic the box during one of these storms and collect a
crash dump, but that has little chance of happening on our live production
server, and I can''t reproduce this on any other machines.

Has anyone seen anything similar to this, or have any tips on how to further
debug this?  Maybe a dtrace script to see where it''s spending so much
time?
-- 
This message posted from opensolaris.org