samba - Sep 2017 - poor performance when copying files with windows client

Hello everyone,

in my setup I have two samba file server with clustered samba 4.6.7 and
glusterfs 3.10. The server are connected via a 10 Gb network (one for
clients and an extra network for gluster). When I am copying files from
different server ( centos 7 1611, samba 4.4.4, connected with 10Gb) to
the file servers with scp, the maximum speed is 70 MB/s (for small files
this is decreasing). However if I create a samba share on the
fileservers and a samba share on the other server and use a windows 7
client to copy the same files, the speed is decreased to less than 10
MB/s. While capturing the the network traffic, I noticed several access
denied messages in response to a file request message. But in the end
the file is still written and the permissions (windows acl) are set
correctly. Does anybody know if this is an intended behaviour and how
one could increase the performance when copying files with a windows
client from network shares? If you need more informations please let me
know.

Best regards,

Nicolas

Am 04.09.2017 um 20:25 schrieb Nicolas Zuber via samba:
> in my setup I have two samba file server with clustered samba 4.6.7 and
> glusterfs 3.10. The server are connected via a 10 Gb network (one for
> clients and an extra network for gluster). When I am copying files from
> different server ( centos 7 1611, samba 4.4.4, connected with 10Gb) to
> the file servers with scp, the maximum speed is 70 MB/s (for small files
> this is decreasing)
for 10 GBE 70 MB/sec are a joke to start with, any average 1 GBE network 
can do that and given that samba has a lot of roundtrips when your 
network has general problems with SMB they are always much worser (just 
look at the horrible performance over OpenVPN)

[harry at rh:/downloads]$ scp harry at remote-machine:/www/bench.bin ./
bench.bin

100%  256MB  94.2MB/s   00:02

Hello Nicolas,

in case your testing scenario includes a large number of small files, 
and directories with a huge number of files, I would also think that it 
is possible that you are hitting the exact same issue that I have uncovered:

https://lists.samba.org/archive/samba/2017-August/210509.html
https://lists.samba.org/archive/samba/2017-August/210337.html
https://lists.samba.org/archive/samba/2017-July/209749.html
https://lists.samba.org/archive/samba/2017-July/209750.html
https://lists.samba.org/archive/samba/2017-July/209751.html

Base line is that there are two big root causes that make SMB2 behave 
much worse than SMB1 with a huge number of small files even in an 
unencrypted scenario:

(a) as you seem to have uncovered as well, the Microsoft SMB2 client 
uses very inefficient SMB2 Find commands (always with Pattern "*")
(b) if the SMB2 server is Samba (as opposed to running on Windows), the 
fact that Samba does not yet implement "directory handle leases" is 
especially detrimental to performance due to the fact that issue (a) in 
the Windows client requires re-reading the whole directory structure for 
each and every inefficient SMB2 Find request sent by the Windows client

I am about to post a detailed testing scenario and Wireshark traces on 
samba-technical (during the next couple of days) in order to properly 
document this and make the Samba team and their colleagues at Microsoft 
aware of the issue(s).

So what does your testing scenario look like when it comes to number of 
files per directory and file sizes? Also, what maximum speed do you get 
from a Linux "mount.cifs vers=3" client as opposed to a Windows client
when accessing the Samba server? And finally, will your scenario run 
faster when you use the same Windows client, but store your files on a 
Windows 10 SMB2 server share (only for testing purposes)?

Hope this helps & best regards
Andreas


Am 04.09.2017 um 20:25 schrieb Nicolas Zuber via samba:
> Hello everyone,
>
> in my setup I have two samba file server with clustered samba 4.6.7 and
> glusterfs 3.10. The server are connected via a 10 Gb network (one for
> clients and an extra network for gluster). When I am copying files from
> different server ( centos 7 1611, samba 4.4.4, connected with 10Gb) to
> the file servers with scp, the maximum speed is 70 MB/s (for small files
> this is decreasing). However if I create a samba share on the
> fileservers and a samba share on the other server and use a windows 7
> client to copy the same files, the speed is decreased to less than 10
> MB/s. While capturing the the network traffic, I noticed several access
> denied messages in response to a file request message. But in the end
> the file is still written and the permissions (windows acl) are set
> correctly. Does anybody know if this is an intended behaviour and how
> one could increase the performance when copying files with a windows
> client from network shares? If you need more informations please let me
> know.
>
> Best regards,
>
> Nicolas
>
>
>

Am 05.09.2017 um 10:17 schrieb Reindl Harald via samba:
>
>
> Am 04.09.2017 um 20:25 schrieb Nicolas Zuber via samba:
>> in my setup I have two samba file server with clustered samba 4.6.7 and
>> glusterfs 3.10. The server are connected via a 10 Gb network (one for
>> clients and an extra network for gluster). When I am copying files from
>> different server ( centos 7 1611, samba 4.4.4, connected with 10Gb) to
>> the file servers with scp, the maximum speed is 70 MB/s (for small
files
>> this is decreasing)
>
> for 10 GBE 70 MB/sec are a joke to start with, any average 1 GBE
> network can do that and given that samba has a lot of roundtrips when
> your network has general problems with SMB they are always much worser
> (just look at the horrible performance over OpenVPN)
>
> [harry at rh:/downloads]$ scp harry at remote-machine:/www/bench.bin ./
> bench.bin
>
> 100%  256MB  94.2MB/s   00:02
>
The speed of 70MB/s was limited by the test file size. In the meantime a
run some more test to find the limiting factors. Coping a 10 GB file
with scp results in 365MB/s, with a windows client 110 MB/s are
possible. I have also tested the network speed with iperf which results
in 9.3 Gb/s for TCP. Hence, I think the network is fine. The problem
arises when copying larger amounts of small files. For the following
tests I used a folder containing 2000 files with 1kB of size each.
Because gluster seems also to slow down work with small files I also
tested shares without gluster. The results are the following:

scp:
       no gluster -> no gluster : <1s
       no gluster -> gluster :      <5s
       gluster      -> no gluster : <2s
       gluster      -> gluster :      <5s

samba share using a win 7 client to copy from share to share:
      no gluster -> no gluster : <23s
       no gluster -> gluster :      <4min
       gluster      -> no gluster : <40s
       gluster      -> gluster :      <4min

Using a windows 10 client to copy the date from share to share did not
improve the performance (testing protocol verison smb2 vs smb3). For now
it seems that using samba shares is causing most of the slowdown.

I will try also to use a linux with mount.cifs as suggested by Andreas.
Maybe someone has additional ideas for testing to exclude possible
bottlenecks.

Regards,
Nicolas