nfs lockd errors after NetApp software upgrade.

Rick Macklem
2019-12-22T17:01:24Z

Daniel Braniss wrote:
>> On 21 Dec 2019, at 19:32, Rick Macklem <rmacklem at uoguelph.ca>
wrote:
>>
>> Daniel Braniss wrote:
>>>> On 20 Dec 2019, at 19:19, Rick Macklem >>><rmacklem
at uoguelph.ca<mailto:rmacklem at uoguelph.ca>> wrote:
>>>>
>>>> Adam McDougall wrote:
>>>>> Try changing bool_t do_tcp = FALSE; to TRUE in
>>>>> /usr/src/sys/nlm/nlm_prot_impl.c, recompile the kernel and
try again. I
>>>>> think this makes it match Linux client behavior. I suspect
I ran into
>>>>> the same issue as you. I do think I used nolockd is a
workaround
>>>>> temporarily. I can provide some more details if it works.
>>>> If this fixes the problem, please let me know.
>>>>
>>>> I'm not sure I'd want to change the default, since it
might break things for
>>>> others, but I can definitely make it a tunable, so that people
don't need to
>>>> recompile a kernel to deal with it.
>>>>
>>>>
>>> great! I was just about to see how it can be done(tunable) but need
to check if it can >be done
>>> at any time, or just at boot time.
>> I haven't looked at the code, but I suspect changing it on the fly
could cause problems,
>> so I am inclined to make it a tunable (boot time only).
my feelings too.
>>
>>> thanks.
>>> btw, currently, from several hours of analysing the traffic, it
seems that nlm is UDP.
>> I assume that means you haven't tried flipping it to TCP yet.
>I will soon, but I have my doubts, the problem is caused my multiple events,
i.e, it >happened once while
>I was doing svn checkout, but i have done it several times since, and no
issues. So it >must be
>an aggregation of factors. Other hosts are reporting locks times too.
Well, I've noted the flawed protocol. Here's an example (from my limited
understanding of these protocols, where there has never been a published spec) :
- The NLM supports a "blocking lock request" that goes something like
this...
   - client requests lock and is willing to wait for it
   - if server has a conflicting lock on the file, it replies "I'll
acquire the lock for
      you when I can and let you know".
     --> When the conflicting lock is released, the server acquires the lock
and does
            a callback (server->client RPC) to tell the client it now has the
lock.
You don't have to think about this for long to realize that any network
unreliability
or partitioning could result in trouble.
The kernel RPC layer may do some retries of the RPCs (this is controlled by the
parameters set for the RPC), but at some point the protocol asks the NSM
(rpc.statd) if the machine is "up" and then uses the NSM's answer
to deal with it.
(The NSM basically pokes other systems and notes they are "up" if they
get
 replies to these pokes. It uses IP broadcast at some point.)

Now, maybe switching to TCP will make the RPCs reliable enough that it will
work, or maybe it won't? (It certainly sounds like the Netapp upgrade is
causing
some kind of network issue, and the NLM doesn't tolerate that well.)

rick

danny

>
> Please let us know how it goes, rick
>
> danny
>
>
> rick
>
> On 12/19/19 9:21 AM, Daniel Braniss wrote:
>
>
> On 19 Dec 2019, at 16:09, Rick Macklem <rmacklem at
uoguelph.ca<mailto:rmacklem at uoguelph.ca>> wrote:
>
> Daniel Braniss wrote:
> [stuff snipped]
> all mounts are nfsv3/tcp
> This doesn't affect what the NLM code (rpc.lockd) uses. I honestly
don't know when
> the NLM uses tcp vs udp. I think rpc.statd still uses IP broadcast at
times.
> can the replay cache have any influence here? I tend to remember way back
issues
> with it,
>
> To me, it looks like a network configuration issue.
> that was/is my gut feelings too, but, as far as we can tell, nothing has
changed in the network infrastructure,
> the problems appeared after the NetAPP?s software was updated, it was
working fine till then.
>
> the problems are also happening on freebsd 12.1
>
> You could capture packets (maybe when a client first starts rpc.statd and
rpc.lockd)
> and then look at them in wireshark. I'd disable statup of rpc.lockd and
rpc.statd
> at boot for a test client and then run something like:
> # tcpdump -s 0 -s out.pcap host <netapp-host>
> - and then start rpc.statd and rpc.lockd
> Then I'd look at out.pcap in wireshark (much better at decoding this
stuff than
> tcpdump). I'd look for things like different reply IP addresses from
the Netapp,
> which might confuse this tired old NLM protocol Sun devised in the
mid-1980s.
>
> it?s going to be an interesting week end :-(
>
> the error is also appearing on freebsd-11.2-stable, I?m now checking if
it?s also
> happening on 12.1
> btw, the NetApp version is 9.3P17
> Yes. I wasn't the author of the NSM and NLM code (long ago I refused to
even
> try to implement it, because I knew the protocol was badly broken) and I
avoid
> fiddling with. As such, it won't have change much since around
FreeBSD7.
> and we haven?t had any issues with it for years, so you must have done
something good
>
> cheers,
>     danny
>
>
> rick
>
> cheers,
>      danny
>
> rick
>
> Cheers
>
> Richard
> (NetApp admin)
>
> On Wed, 18 Dec 2019 at 15:46, Daniel Braniss <danny at
cs.huji.ac.il<mailto:danny at cs.huji.ac.il><mailto:danny at
cs.huji.ac.il>> wrote:
>
>
> On 18 Dec 2019, at 16:55, Rick Macklem <rmacklem at
uoguelph.ca<mailto:rmacklem at uoguelph.ca><mailto:rmacklem at
uoguelph.ca>> wrote:
>
> Daniel Braniss wrote:
>
> Hi,
> The server with the problems is running FreeBSD 11.1 stable, it was working
fine for >several months,
> but after a software upgrade of our NetAPP server it?s reporting many lockd
errors >and becomes catatonic,
> ...
> Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not
responding
> Dec 18 13:11:45 moo-09 last message repeated 7 times
> Dec 18 13:12:55 moo-09 last message repeated 8 times
> Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is alive
again
> Dec 18 13:13:10 moo-09 last message repeated 8 times
> Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen
queue >overflow: 194 already in queue awaiting acceptance (1 occurrences)
> Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen
queue >overflow: 193 already in queue awaiting acceptance (3957 occurrences)
> Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen
queue >overflow: 193 already in queue awaiting acceptance ?
> Seems like their software upgrade didn't improve handling of NLM RPCs?
> Appears to be handling RPCs slowly and/or intermittently. Note that no one
> tests it with IPv6, so at least make sure you are still using IPv4 for the
mounts and
> try and make sure IP broadcast works between client and Netapp. I think the
NLM
> and NSM (rpc.statd) still use IP broadcast sometimes.
>
> we are ipv4 - we have our own class c :-)
> Maybe the network guys can suggest more w.r.t. why, but as I've stated
before,
> the NLM is a fundamentally broken protocol which was never published by
Sun,
> so I suggest you avoid using it if at all possible.
> well, at the moment the ball is on NetAPP court, and switching to NFSv4 at
the moment is out of the question, it?s
> a production server used by several thousand students.
>
>
> - If the locks don't need to be seen by other clients, you can just use
the "nolockd"
> mount option.
> or
> - If locks need to be seen by other clients, try NFSv4 mounts. Netapp
filers
> should support NFSv4.1, which is a much better protocol that NFSv4.0.
>
> Good luck with it, rick
> thanks
>     danny
>
> ?
> any ideas?
>
> thanks,
>    danny
>
> _______________________________________________
> freebsd-stable at freebsd.org<mailto:freebsd-stable at
freebsd.org><mailto:freebsd-stable at freebsd.org> mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org<mailto:freebsd-stable-unsubscribe at freebsd.org>"
>
> _______________________________________________
> freebsd-stable at freebsd.org<mailto:freebsd-stable at
freebsd.org><mailto:freebsd-stable at freebsd.org> mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org<mailto:freebsd-stable-unsubscribe at freebsd.org>"
>
>
> _______________________________________________
> freebsd-stable at freebsd.org<mailto:freebsd-stable at freebsd.org>
mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"
>
>
> _______________________________________________
> freebsd-stable at freebsd.org<mailto:freebsd-stable at freebsd.org>
mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org<mailto:freebsd-stable-unsubscribe at freebsd.org>"
> _______________________________________________
> freebsd-stable at freebsd.org<mailto:freebsd-stable at freebsd.org>
mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org<mailto:freebsd-stable-unsubscribe at freebsd.org>"
>

_______________________________________________
freebsd-stable at freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscribe at
freebsd.org"