samba - Sep 2021 - Possible memory leak in

On Tue, Sep 14, 2021 at 08:16:49PM +0200, Ralph Boehme via samba
wrote:
>
>Am 12.09.21 um 21:48 schrieb Matt Oursbourn via samba:
>>What I have noticed from looking at the Processes tab in System
>>Monitor on the server: The server is on a 10GB network.  When I start
>>a chia plot (~106Gb file) transfer from a client computer that is
>>also connected to the 10Gb network the transfer starts off at
>>~1000Mb/s until the samba process is larger
>than
>>32.5GiB. The largest I have seen is 39GiB.   The transfer rate then
>>drops down to the hdd write speed of ~170Mb/s.  That samba process
>>never
>gets any
>>smaller than 32.5GiB.  Even after the transfer is complete.
>>
>>If a client computer that is connected with a 1Gb nic starts a
>>transfer, another samba process is spawned. That transfer stays
>>around 130Mb/s
>to 150
>>Mb/s and the samba process grows to ~9GiB. After that transfer is
>>complete the process drops down to ~5MiB.
>>
>>After a day of plotting and transferring ~100 plot files to the
>>server
>from
>>those two computers there are still only the two samba processes
>>running. Plotting has been stopped for several hours  and one is
>>32.5Gib, the other is 4.9Mib.
>
>the pool-usage of the large process you'd sent me privately contains:
>
>$ grep pthreadpool_tevent_job_state pool-usage\ smbd_32GiB.txt | wc -l
>6468
>
>which means there are 6468 SMB2 WRITE requests pending at the server.
>
>This is likely triggered because
>
>- the client is sending data much faster then the server is writing data
>to disk
>
>- Samba returns SMB2 Async Interim Response for every received WRITE request
>
>- These grant SMB2 credits to the client
>
>- As a result the client is not throttled by a crediting window
>
>Iirc we've seen this before some time ago and iirc a Windows server
>behaves either doesn't send interim async responses or stops sending
>them at some point.
>
>To me this looks like a design flaw in the crediting logic, but we can't
>change that so we have to adjust our write processing code to handle this.
>
>Until that fix materializes (which may take some tim) you could disable
>aio as a workaround in smb.conf by setting:
>
>aio read size = 0
>aio write size = 0
Hmmm. Can't we create some back-pressure on the crediting algorithm
by forcing writes to go synchronous after the queue async io events
reaches a (parameterized) size ?

We already count fsp->num_aio_requests, so we have a per-fd
count. We could add a per-process count ? Or maybe start
forcing reads/writes to go synchronous once the time between
request/response goes over a certain amount ?

Am 14.09.21 um 20:42 schrieb Jeremy Allison:
> On Tue, Sep 14, 2021 at 08:16:49PM +0200, Ralph Boehme via samba wrote:
>>
>> Am 12.09.21 um 21:48 schrieb Matt Oursbourn via samba:
>>> What I have noticed from looking at the Processes tab in System
>>> Monitor on the server: The server is on a 10GB network.? When I
start
>>> a chia plot (~106Gb file) transfer from a client computer that is
>>> also connected to the 10Gb network the transfer starts off at
>>> ~1000Mb/s until the samba process is larger
>> than
>>> 32.5GiB. The largest I have seen is 39GiB.?? The transfer rate then
>>> drops down to the hdd write speed of ~170Mb/s.? That samba process
>>> never
>> gets any
>>> smaller than 32.5GiB.? Even after the transfer is complete.
>>>
>>> If a client computer that is connected with a 1Gb nic starts a
>>> transfer, another samba process is spawned. That transfer stays
>>> around 130Mb/s
>> to 150
>>> Mb/s and the samba process grows to ~9GiB. After that transfer is
>>> complete the process drops down to ~5MiB.
>>>
>>> After a day of plotting and transferring ~100 plot files to the
>>> server
>> from
>>> those two computers there are still only the two samba processes
>>> running. Plotting has been stopped for several hours? and one is
>>> 32.5Gib, the other is 4.9Mib.
>>
>> the pool-usage of the large process you'd sent me privately
contains:
>>
>> $ grep pthreadpool_tevent_job_state pool-usage\ smbd_32GiB.txt | wc -l
>> 6468
>>
>> which means there are 6468 SMB2 WRITE requests pending at the server.
>>
>> This is likely triggered because
>>
>> - the client is sending data much faster then the server is writing
data
>> to disk
>>
>> - Samba returns SMB2 Async Interim Response for every received WRITE 
>> request
>>
>> - These grant SMB2 credits to the client
>>
>> - As a result the client is not throttled by a crediting window
>>
>> Iirc we've seen this before some time ago and iirc a Windows server
>> behaves either doesn't send interim async responses or stops
sending
>> them at some point.
>>
>> To me this looks like a design flaw in the crediting logic, but we
can't
>> change that so we have to adjust our write processing code to handle 
>> this.
>>
>> Until that fix materializes (which may take some tim) you could disable
>> aio as a workaround in smb.conf by setting:
>>
>> aio read size = 0
>> aio write size = 0
> 
> Hmmm. Can't we create some back-pressure on the crediting algorithm
> by forcing writes to go synchronous after the queue async io events
> reaches a (parameterized) size ?
> 
> We already count fsp->num_aio_requests, so we have a per-fd
> count. We could add a per-process count ? Or maybe start
> forcing reads/writes to go synchronous once the time between
> request/response goes over a certain amount ?
first we have to check how Windows handles this. Iirc they don't send 
interim async responses. Problem solved. But my memory may not serve me 
right.

-slow

-- 
Ralph Boehme, Samba Team                 https://samba.org/
SerNet Samba Team Lead      https://sernet.de/en/team-samba

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 840 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20210914/c4c0b9c7/OpenPGP_signature.sig>