On Tue, Sep 14, 2021 at 08:16:49PM +0200, Ralph Boehme via samba wrote:> >Am 12.09.21 um 21:48 schrieb Matt Oursbourn via samba: >>What I have noticed from looking at the Processes tab in System >>Monitor on the server: The server is on a 10GB network. When I start >>a chia plot (~106Gb file) transfer from a client computer that is >>also connected to the 10Gb network the transfer starts off at >>~1000Mb/s until the samba process is larger >than >>32.5GiB. The largest I have seen is 39GiB. The transfer rate then >>drops down to the hdd write speed of ~170Mb/s. That samba process >>never >gets any >>smaller than 32.5GiB. Even after the transfer is complete. >> >>If a client computer that is connected with a 1Gb nic starts a >>transfer, another samba process is spawned. That transfer stays >>around 130Mb/s >to 150 >>Mb/s and the samba process grows to ~9GiB. After that transfer is >>complete the process drops down to ~5MiB. >> >>After a day of plotting and transferring ~100 plot files to the >>server >from >>those two computers there are still only the two samba processes >>running. Plotting has been stopped for several hours and one is >>32.5Gib, the other is 4.9Mib. > >the pool-usage of the large process you'd sent me privately contains: > >$ grep pthreadpool_tevent_job_state pool-usage\ smbd_32GiB.txt | wc -l >6468 > >which means there are 6468 SMB2 WRITE requests pending at the server. > >This is likely triggered because > >- the client is sending data much faster then the server is writing data >to disk > >- Samba returns SMB2 Async Interim Response for every received WRITE request > >- These grant SMB2 credits to the client > >- As a result the client is not throttled by a crediting window > >Iirc we've seen this before some time ago and iirc a Windows server >behaves either doesn't send interim async responses or stops sending >them at some point. > >To me this looks like a design flaw in the crediting logic, but we can't >change that so we have to adjust our write processing code to handle this. > >Until that fix materializes (which may take some tim) you could disable >aio as a workaround in smb.conf by setting: > >aio read size = 0 >aio write size = 0Hmmm. Can't we create some back-pressure on the crediting algorithm by forcing writes to go synchronous after the queue async io events reaches a (parameterized) size ? We already count fsp->num_aio_requests, so we have a per-fd count. We could add a per-process count ? Or maybe start forcing reads/writes to go synchronous once the time between request/response goes over a certain amount ?
Am 14.09.21 um 20:42 schrieb Jeremy Allison:> On Tue, Sep 14, 2021 at 08:16:49PM +0200, Ralph Boehme via samba wrote: >> >> Am 12.09.21 um 21:48 schrieb Matt Oursbourn via samba: >>> What I have noticed from looking at the Processes tab in System >>> Monitor on the server: The server is on a 10GB network.? When I start >>> a chia plot (~106Gb file) transfer from a client computer that is >>> also connected to the 10Gb network the transfer starts off at >>> ~1000Mb/s until the samba process is larger >> than >>> 32.5GiB. The largest I have seen is 39GiB.?? The transfer rate then >>> drops down to the hdd write speed of ~170Mb/s.? That samba process >>> never >> gets any >>> smaller than 32.5GiB.? Even after the transfer is complete. >>> >>> If a client computer that is connected with a 1Gb nic starts a >>> transfer, another samba process is spawned. That transfer stays >>> around 130Mb/s >> to 150 >>> Mb/s and the samba process grows to ~9GiB. After that transfer is >>> complete the process drops down to ~5MiB. >>> >>> After a day of plotting and transferring ~100 plot files to the >>> server >> from >>> those two computers there are still only the two samba processes >>> running. Plotting has been stopped for several hours? and one is >>> 32.5Gib, the other is 4.9Mib. >> >> the pool-usage of the large process you'd sent me privately contains: >> >> $ grep pthreadpool_tevent_job_state pool-usage\ smbd_32GiB.txt | wc -l >> 6468 >> >> which means there are 6468 SMB2 WRITE requests pending at the server. >> >> This is likely triggered because >> >> - the client is sending data much faster then the server is writing data >> to disk >> >> - Samba returns SMB2 Async Interim Response for every received WRITE >> request >> >> - These grant SMB2 credits to the client >> >> - As a result the client is not throttled by a crediting window >> >> Iirc we've seen this before some time ago and iirc a Windows server >> behaves either doesn't send interim async responses or stops sending >> them at some point. >> >> To me this looks like a design flaw in the crediting logic, but we can't >> change that so we have to adjust our write processing code to handle >> this. >> >> Until that fix materializes (which may take some tim) you could disable >> aio as a workaround in smb.conf by setting: >> >> aio read size = 0 >> aio write size = 0 > > Hmmm. Can't we create some back-pressure on the crediting algorithm > by forcing writes to go synchronous after the queue async io events > reaches a (parameterized) size ? > > We already count fsp->num_aio_requests, so we have a per-fd > count. We could add a per-process count ? Or maybe start > forcing reads/writes to go synchronous once the time between > request/response goes over a certain amount ?first we have to check how Windows handles this. Iirc they don't send interim async responses. Problem solved. But my memory may not serve me right. -slow -- Ralph Boehme, Samba Team samba.org SerNet Samba Team Lead sernet.de/en/team-samba -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: <lists.samba.org/pipermail/samba/attachments/20210914/c4c0b9c7/OpenPGP_signature.sig>