freebsd stable - Feb 2021 - Suspected mbuf leak with Nginx + sendfile + TLS in 12.2-STABLE

On 2/4/21 8:08 AM, GomoR wrote:
> Dear FreeBSD community,
> 
> we are encountering a DoS condition on our production machines.
> Our use case is an Nginx reverse proxy serving large files via HTTPS.
> This problem arose when switching kernel and userland from 12.1-RELEASE
> to 12.2-RELEASE. Ports were not upgraded (at first).
> 
> Each time a user downloads a file, mbuf & mbuf_clusters are raising to
> reach the maximum limit in a matter of seconds. Those values are
> asserted by 'netstat -m' as follows:
> 
> Normal situation:
> 
> mbuf:                   256, 26031105,   16767,    5974,428087938,   0,
>    0
> mbuf_cluster:          2048, 8135232,   18408,    2704,101644203,   0,
> 0
> 
> Warning situtation:
> 
> mbuf:                   256, 26031105, 2981516,  151205,1109483561,   0,
>     0
> mbuf_cluster:          2048, 8135232, 2983155,    4201,319714617,   0,
> 0
> 
> We have seen a patch related to sendfile + KTLS + mbuf at the below link
> and we updated to -STABLE to apply:
None of the sendfile or KTLS changes from Netflix are in 12, they are only
in 13 and later.
> Don't transmit mbufs that aren't yet ready on TOE sockets.
> This includes mbufs waiting for data from sendfile() I/O requests, or
> mbufs awaiting encryption for KTLS.
>
https://github.com/freebsd/freebsd-src/commit/14c77f30b201bf76119d59678e72051c093333c2
This patch only applies to Chelsio T5/T6 NICs when using TOE (TCP offload)
and doesn't affect freeing mbufs, it just fixes a race when the NIC could
potentially send random garbage if it sends the mbuf before the scheduled
disk I/O to populate it with data from disk has completed.
> NIC is:
> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver>
> 
> What can we do to help you find the root cause?
The first step I would do if possible would be to bisect between the last
known working version and the version that is known to be broken to
determine which commit introduced the problem.  One thing that could help
here is to see if you can reproduce the problem using a 12.2 kernel on a
12.1 world + ports.  If you can, then you can limit your bisecting to just
building new kernels which will make that process quicker.

You might also see if using a different NIC shows the same problem.  If
not, then it might point to a regression in the NIC driver (or perhaps in
iflib as ix uses iflib I believe).

-- 
John Baldwin

On 2021-02-04 19:33, John Baldwin wrote:
> None of the sendfile or KTLS changes from Netflix are in 12, they are 
> only
> in 13 and later.
I thought about that possibility, thank you for the clarification.
>> Don't transmit mbufs that aren't yet ready on TOE sockets.
>> This includes mbufs waiting for data from sendfile() I/O requests, or
>> mbufs awaiting encryption for KTLS.
>>
https://github.com/freebsd/freebsd-src/commit/14c77f30b201bf76119d59678e72051c093333c2
> 
> This patch only applies to Chelsio T5/T6 NICs when using TOE (TCP 
> offload)
> and doesn't affect freeing mbufs, it just fixes a race when the NIC 
> could
> potentially send random garbage if it sends the mbuf before the 
> scheduled
> disk I/O to populate it with data from disk has completed.
Understood.
>> NIC is:
>> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver>
>> 
>> What can we do to help you find the root cause?
> 
> The first step I would do if possible would be to bisect between the 
> last
> known working version and the version that is known to be broken to
> determine which commit introduced the problem.  One thing that could 
> help
> here is to see if you can reproduce the problem using a 12.2 kernel on 
> a
> 12.1 world + ports.  If you can, then you can limit your bisecting to 
> just
> building new kernels which will make that process quicker.
Thank you for the tip, I'll try that path and let you know.
> You might also see if using a different NIC shows the same problem.  If
> not, then it might point to a regression in the NIC driver (or perhaps 
> in
> iflib as ix uses iflib I believe).
Unfortunately, not a possibility here.

I did some other tests and found where the problem arise. In fact, we 
use
proxy_pass directive within Nginx and the network flow goes through one 
public
interface (ix0) and proxy_pass through a second (ix1) towards a remote 
machine.

Changing the Nginx configuration to only go through ix0 does not cause 
the
issue. So that's something about with passing packets between 2 NICs.

I'll keep you posted.

Regards,