samba - Jun 2017 - Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Hello Samba experts,

I have just successfully replaced the old default Thecus Samba version 
3.5.16 on my Thecus NAS (32-bit Intel Atom, 3 GB RAM, i686 Thecus kernel 
2.6.33) by a current Samba 4.6.5 build that I have cross-compiled myself 
from scratch.

Note that so far, I am using the unchanged (i.e. exactly identical) old 
3.5.16 smb.conf file for 4.6.5.

Everything seems to work fine in terms of functionality. From Windows 
PowerShell, I have been also able to verify with "Get-SmbConnection" 
that I am now using SMB3 ("dialect 3.1.1"), and I have noticed decent 
performance gains for copying large files as expected:

  * single large file copy, Win10 NTFS client to Thecus NAS ("write"):
    before: Samba 3.5.16/SMB 1.5: ~18 MB per second
    after: Samba 4.6.5/SMB 3.1.1: ~23 MB per second

  * single large file copy, Thecus NAS to Win10 NTFS client ("read"):
    before: Samba 3.5.16/SMB 1.5: ~60 MB per second
    after: Samba 4.6.5/SMB 3.1.1: ~85 MB per second


But unfortunately, I have also run into a severe performance degradation 
for copying a huge number of small files between a Win10 client and the 
NAS in both directions. My test scenario here is copying a single 
directory containing 5400 files of between less than 1kB and max 8kB in 
size in both directions (Win10 -> Thecus and the other way round).

To the best of my knowledge, performance-related parameters in smb.conf 
had already been tuned for 3.5.16 with good results - IIRC, it was 
especially crucial for write performance to switch from

case sensitive = auto

to

case sensitive = true
default case = lower
preserve case = yes
short preserve case = yes

in order to stop smbd from repeatedly looping through those ~5000 trying 
to check whether any new file name was unique in a case insensitive way...

But with the exact same smb.conf as attached (global section), I now 
observe

  * many small files copy, Win10 NTFS client to Thecus NAS ("write"):
    before: Samba 3.5.16/SMB 1.5: ~120 kB per second
    after: Samba 4.6.5/SMB 3.1.1: ~4 (!!!) kB per second (this is
    completely inacceptable: on average, only 1-2 files per second!!!)

  * many small files copy, Thecus NAS to Win10 NTFS client ("read"):
    before: Samba 3.5.16/SMB 1.5: ~400 kB per second
    after: Samba 4.6.5/SMB 3.1.1: ~130 kB per second (even though still
    acceptable, slower by a factor of 3!!!)

Note that also, in both the "write" and the "read" scenario,
CPU load
for the smbd process on the NAS is much higher for the 4.6.5 than the 
3.5.16 scenario - and in the "write" scenario, CPU load even increases
over time (i.e. with the number of files that are already copied) and 
continuously uses up to about 80% of one of my four (hyper-threaded) 
Atom cores (1.8 GHz)...

Unfortunately, this performance issue makes Samba 4.6.5 pretty much a 
pain to use for any "development" scenarios with huge numbers of small
files - so for now, I had to revert back my NAS to 3.5.16 to be usable - 
even though I definitely want to upgrade asap in order to remove the 
risk to be affected by the "SambaCry" issue (CVE-2017-7494)... :-(


My questions now are:

  * Am I still doing something wrong in terms of configuration, i.e.
    using inappropriate settings in smb.conf?
  * Am I perhaps hitting an already known performance issue, and if so,
    do you have any plans/timeline for fixing it?
  * And if this issue is indeed new to you, then how can I help the
    Samba team in tracking down the root cause of this and hopefully
    fixing the issue, i.e. can I enable my build to support code
    profiling to see where all that CPU power and time is lost?

Many thanks in advance for looking into this and advising how to 
proceed! :-)

Best regards
Andreas


-------------- next part --------------
[global]
server string = %h
max open files = 100000
deadtime = 15
dead time = 15
hide unreadable = yes
load printers = no
log file = /var/log/samba.%m
max log size = 50
strict locking = no
lock directory = /var/samba
encrypt passwords = yes
case sensitive = true
default case = lower
preserve case = yes
short preserve case = yes
passdb backend = tdbsam
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE
aio read size = 1
aio write size = 1
write cache size = 2097152
read raw = yes
write raw = yes
min receivefile size = 0
use sendfile = yes
large readwrite = yes
max xmit = 32768
getwd cache = true
map untrusted to domain = yes
os level = 1
local master = yes
unix extensions = yes
domain master = no
preferred master = no
dns proxy = no
dos charset = cp1252
unix charset = utf8
client ldap sasl wrapping = seal
allow trusted domains = yes
idmap uid = 20000-60000000
idmap gid = 20000-60000000
winbind separator = +
winbind nested groups = yes
winbind enum users = yes
winbind enum groups = yes
create mask = 0644
winbind use default domain = yes
map acl inherit = yes
nt acl support = yes
#map system = yes
bind interfaces only = yes
interfaces = lo,bond*
guest account = nobody
map to guest = Bad User
guest only = yes
follow symlinks = no
block size = 262144
dfree cache time = 5
large readwrite = yes
getwd cache = yes
oplocks = yes
kernel oplocks = yes
veto files = /folder.db/.AppleDouble/.AppleDB/.bin/.AppleDesktop/Network Trash
Folder/:2eDS_Store/.DS_Store/TheFindByContentFolder/TheVolumeSettingsFolder/Temporary
Items/.AppleDBcnid.lock/.VolumeIcon.icns/.Temporary
Items/.Parent/.HSicon/._*/:*/
veto oplock files = /J0*.WMF/*_.GIF/J0*.JPG/*_.WMF/
workgroup = WORKGROUP
password server = *
security = user
auth methods = guest sam_ignoredomain
realm idmap backend = rid:WORKGROUP=20000-60000000
wins server = 192.168.2.110
client ntlmv2 auth = no
server signing = disabled
delete veto files = yes

On Tue, Jun 13, 2017 at 12:09:41PM +0200, awl1 via samba
wrote:
> Hello Samba experts,
> 
> I have just successfully replaced the old default Thecus Samba
> version 3.5.16 on my Thecus NAS (32-bit Intel Atom, 3 GB RAM, i686
> Thecus kernel 2.6.33) by a current Samba 4.6.5 build that I have
> cross-compiled myself from scratch.
> 
> Note that so far, I am using the unchanged (i.e. exactly identical)
> old 3.5.16 smb.conf file for 4.6.5.
> 
> Everything seems to work fine in terms of functionality. From
> Windows PowerShell, I have been also able to verify with
> "Get-SmbConnection" that I am now using SMB3 ("dialect
3.1.1"), and
> I have noticed decent performance gains for copying large files as
> expected:
> 
>  * single large file copy, Win10 NTFS client to Thecus NAS
("write"):
>    before: Samba 3.5.16/SMB 1.5: ~18 MB per second
>    after: Samba 4.6.5/SMB 3.1.1: ~23 MB per second
> 
>  * single large file copy, Thecus NAS to Win10 NTFS client
("read"):
>    before: Samba 3.5.16/SMB 1.5: ~60 MB per second
>    after: Samba 4.6.5/SMB 3.1.1: ~85 MB per second
> 
> 
> But unfortunately, I have also run into a severe performance
> degradation for copying a huge number of small files between a Win10
> client and the NAS in both directions. My test scenario here is
> copying a single directory containing 5400 files of between less
> than 1kB and max 8kB in size in both directions (Win10 -> Thecus and
> the other way round).
> 
> To the best of my knowledge, performance-related parameters in
> smb.conf had already been tuned for 3.5.16 with good results - IIRC,
> it was especially crucial for write performance to switch from
Can you get comparitive wireshark traces for the two cases ?

That would help discover what the bottleneck is.

Hello Jeremy,

thanks a million for your help and interest in tracking this down! :-)

Am 13.06.2017 um 18:36 schrieb Jeremy Allison:
> Can you get comparitive wireshark traces for the two cases ?
>
> That would help discover what the bottleneck is.
I am not at all a network guy, but I hope that - maybe with a little 
more help from your part once I have tried to do so in practice - I 
should be able to do so...

Follow-up questions:

Which machine do you want me to run wireshark on? Can this be the 
Windows machine, or will I need to cross-compile a wireshark version to 
run on my NAS first (which might take some days)?

Based on the description here:

     https://wiki.samba.org/index.php/Capture_Packets

I assume that you want me to record ports 139 and 445?

In the first step, I will try and make four records, right:

  * the "write" scenario, copying from Win10 to the NAS with 3.5.16
and
    4.6.5
  * the "read" scenario, copying from the NAS to Win10 with 3.5.16 and
4.6.5

How many files should I copy during each recording?

I'm a little bit worried whether a network trace will indeed pont to the 
root cause, as I fear from looking at smbd CPU usage on the NAS that 
there will be some CPU-bound busy activity (i.e. not just I/O wait) at 
the heart of the issue, but we will hopefully see this during the process.

And finally: Where do you want me to upload the recorded traces to?

Many thanks so far & best regards
Andreas