freebsd stable - Feb 2021 - Suspected mbuf leak with Nginx + sendfile + TLS in 12.2-STABLE

On 2021-02-05 09:11, GomoR wrote:
>> The first step I would do if possible would be to bisect between the 
>> last
>> known working version and the version that is known to be broken to
>> determine which commit introduced the problem.  One thing that could 
>> help
>> here is to see if you can reproduce the problem using a 12.2 kernel on 
>> a
>> 12.1 world + ports.  If you can, then you can limit your bisecting to 
>> just
>> building new kernels which will make that process quicker.
We have reinstalled from scratch our system with FreeBSD 12.1-RELEASE. 
We then
have installed just enough of our software stack to reproduce the issue.

No problem with a stock 12.1-RELEASE kernel, but problem arise after 
installkernel
with the latest 12.2-STABLE. We then turned off all our customizations, 
including
some specific sysctl.conf values. The bug didn't triggered.

After dissecting our sysctl values, the faulty one has been identified:

kern.ipc.maxsockbuf=157286400

This value is 75 times the default value (2097152). Restoring the 
default value
fixes the issue. After some tests, the bug is triggered starting 
somewhere to
64 times the default value (134217728).

There was no issue with this setting in 12.1-RELEASE, but there is in 
12.2-RELEASE.

Do you have some insights onto why it causes that mbuf problems? In the 
meantime,
we have our solution, but we are willing to help identify if that's a 
kernel bug
or just a real bad idea to set maxsockbuf to such a high value.

Best regards,

On Fri, Feb 05, 2021 at 11:54:07AM +0100, GomoR wrote:
> On 2021-02-05 09:11, GomoR wrote:
> >> The first step I would do if possible would be to bisect between
the
> >> last
> >> known working version and the version that is known to be broken
to
> >> determine which commit introduced the problem.  One thing that
could
> >> help
> >> here is to see if you can reproduce the problem using a 12.2
kernel on
> >> a
> >> 12.1 world + ports.  If you can, then you can limit your bisecting
to
> >> just
> >> building new kernels which will make that process quicker.
> 
> We have reinstalled from scratch our system with FreeBSD 12.1-RELEASE. 
> We then
> have installed just enough of our software stack to reproduce the issue.
> 
> No problem with a stock 12.1-RELEASE kernel, but problem arise after 
> installkernel
> with the latest 12.2-STABLE. We then turned off all our customizations, 
> including
> some specific sysctl.conf values. The bug didn't triggered.
> 
> After dissecting our sysctl values, the faulty one has been identified:
> 
> kern.ipc.maxsockbuf=157286400
> 
> This value is 75 times the default value (2097152). Restoring the 
> default value
> fixes the issue. After some tests, the bug is triggered starting 
> somewhere to
> 64 times the default value (134217728).
> 
> There was no issue with this setting in 12.1-RELEASE, but there is in 
> 12.2-RELEASE.
> 
> Do you have some insights onto why it causes that mbuf problems? In the 
> meantime,
> we have our solution, but we are willing to help identify if that's a 
> kernel bug
> or just a real bad idea to set maxsockbuf to such a high value.
==> Each time a user downloads a file, mbuf & mbuf_clusters are raising
to
> reach the maximum limit in a matter of seconds. Those values are
> asserted by 'netstat -m' as follows:
>
> Normal situation:
>
> mbuf:                   256, 26031105,   16767,    5974,428087938,   0,   0
> mbuf_cluster:          2048, 8135232,   18408,    2704,101644203,   0,   0
>
> Warning situtation:
>
> mbuf:                   256, 26031105, 2981516,  151205,1109483561,   0,  
0
> mbuf_cluster:          2048, 8135232, 2983155,    4201,319714617,   0,   0
==
Can you clarified what is problem?
I.e. under load system used more resources and this is not bug.
Do you see more resources usage compared to load?
Or resources don't freed after drop load?