samba - May 2020 - Performance Degradation when copying >1500 files to mac

Hello all,

I'm running into a significant performance issue where copying a couple
thousand <3kb files from my stretch SMB server to a mac client. Dragging +
dropping the files from the gui results in a long period of time in which
the client is "preparing to copy" and then actually transferring,
during
this time transfer starts around 300MB/s, then after around 5-10 seconds
drops to <50MB/s and eventually recovers after a few minutes. The more
files being copied, the more pronounced the drop off is and the slower the
bandwidth. If two clients are doing the copy at the same time, they'll both
slow eachother down significantly, even if it's done manually via cp on the
command line.

The server is running debian stretch, SMB 4.5, backed by ZFS 0.7.13. The
mac os client is running Darwin 18.7.0 (have also tested and confirmed
behavior on Mojave 10.14.6) and writing to SSD, and I confirmed that
comparable FS operations locally do not incur the performance penalty, only
from SMB to disk.

iperf gives the connection speed its full line rate of 10gbps, and disk
read performance of the smb server is >10gbps.

I've tested the same operations from linux <-> linux via the smb
share, but
no performance drop off was seen. Further, copying large files back and
forth also saw no problem.

I'm wondering if anyone might have any knowledge as to particular smb
client configuration settings on the mac side that may help alleviate this
issue, or if this is a known problem with current mac smb clients?

Thanks for your time

Best regards,
Christian

-- 
 <https://opendrives.com/wp-content/uploads/2020/04/OD-Anywhere.pdf>

On Thu, May 28, 2020 at 04:02:53PM -0700, Christian Kuntz via samba
wrote:
> Hello all,
> 
> I'm running into a significant performance issue where copying a couple
> thousand <3kb files from my stretch SMB server to a mac client. Dragging
+
> dropping the files from the gui results in a long period of time in which
> the client is "preparing to copy" and then actually transferring,
during
> this time transfer starts around 300MB/s, then after around 5-10 seconds
> drops to <50MB/s and eventually recovers after a few minutes. The more
> files being copied, the more pronounced the drop off is and the slower the
> bandwidth. If two clients are doing the copy at the same time, they'll
both
> slow eachother down significantly, even if it's done manually via cp on
the
> command line.
> 
> The server is running debian stretch, SMB 4.5, backed by ZFS 0.7.13. The
> mac os client is running Darwin 18.7.0 (have also tested and confirmed
> behavior on Mojave 10.14.6) and writing to SSD, and I confirmed that
> comparable FS operations locally do not incur the performance penalty, only
> from SMB to disk.
> 
> iperf gives the connection speed its full line rate of 10gbps, and disk
> read performance of the smb server is >10gbps.
> 
> I've tested the same operations from linux <-> linux via the smb
share, but
> no performance drop off was seen. Further, copying large files back and
> forth also saw no problem.
> 
> I'm wondering if anyone might have any knowledge as to particular smb
> client configuration settings on the mac side that may help alleviate this
> issue, or if this is a known problem with current mac smb clients?
Are you using vfs_fruit on the Mac's ? Samba 4.5 is a little old,
I don't think it uses pthread aio by default.

If you're willing to get a little more adventurous then the latest
Samba does async io by default and also has io_uring available
(which just got a fix with latest kernel issues).