samba - Jun 2017 - Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf

Hello Jeremy,

thanks a million for your help and interest in tracking this down! :-)

Am 13.06.2017 um 18:36 schrieb Jeremy Allison:
> Can you get comparitive wireshark traces for the two cases ?
>
> That would help discover what the bottleneck is.
I am not at all a network guy, but I hope that - maybe with a little 
more help from your part once I have tried to do so in practice - I 
should be able to do so...

Follow-up questions:

Which machine do you want me to run wireshark on? Can this be the 
Windows machine, or will I need to cross-compile a wireshark version to 
run on my NAS first (which might take some days)?

Based on the description here:

     https://wiki.samba.org/index.php/Capture_Packets

I assume that you want me to record ports 139 and 445?

In the first step, I will try and make four records, right:

  * the "write" scenario, copying from Win10 to the NAS with 3.5.16
and
    4.6.5
  * the "read" scenario, copying from the NAS to Win10 with 3.5.16 and
4.6.5

How many files should I copy during each recording?

I'm a little bit worried whether a network trace will indeed pont to the 
root cause, as I fear from looking at smbd CPU usage on the NAS that 
there will be some CPU-bound busy activity (i.e. not just I/O wait) at 
the heart of the issue, but we will hopefully see this during the process.

And finally: Where do you want me to upload the recorded traces to?

Many thanks so far & best regards
Andreas

Hello again, Jeremy,

first of all, I am terribly sorry for my late reply. I tried to send my 
posting many times, but my mail has always been silently discarded by 
the Samba mail servers due to my main mail provider (GMX - a very large 
German mail provider with millions of customers) having been blacklisted 
by SORBS.

For the time being, SORBS is still unwilling to delist them for unknown 
reasons (which I consider a clear malpractice by SORBS, as GMX has 
sophisticated spam/abuse management in place), so I had to switch to 
another mail provider just in order to be able to post again on the 
Samba list... :-(


Am 13.06.2017 um 19:00 schrieb awl1:
> Am 13.06.2017 um 18:36 schrieb Jeremy Allison:
>> Can you get comparitive wireshark traces for the two cases ?
>>
>> That would help discover what the bottleneck is.
> I am not at all a network guy, but I hope that - maybe with a little 
> more help from your part once I have tried to do so in practice - I 
> should be able to do so...
OK, so it looks like I have been able to successfully produce Wireshark 
capture files for the four scenarios... :-)

As I am almost certain that these packet captures will contain at least 
some sensitive information from my environment - such as e.g. user, 
share and machine names, IP addresses (possibly in the old SMB dialect 
1.5 even the clear-text password for the share?) - I will only send the 
link to the captures ZIP file stored in my cloud space to you via 
private mail. So please keep the packet dumps confidential, and only 
share them with other Samba developers after getting my explicit consent!

The ZIP file contains four Wireshark captures for the two scenarios 
(write to and read from share) and the two Samba/SMB versions 
(4.6.5/SMB2/dialect 3.1.1 and 3.5.16/SMB/dialect 1.5) in "pcapng"
format:

  * smb311_write - Win10 client writing to Samba 4.6.5 using SMB2
    protocol (dialect 3.1.1), copying ~ 1000 files from local hard disk
    onto the share, documenting the issue with very slow throughput of
    below 10 kB/sec (especially in the range of file 300-600, most
    interestingly throughput improved again after some time)
  * smb15_write - Win10 client writing to Samba 3.5.16 using SMB
    protocol (dialect 1.5), copying ~ 1000 files from local hard disk
    onto the share, with much better throughput than in smb311_write

  * smb311_read - Win10 client reading from Samba 4.6.5 using SMB2
    protocol (dialect 3.1.1), copying ~ 2000 files from the share to
    local hard disk, with acceptable throughput, but consistently slower
    than in smb15_read
  * smb15_read - Win10 client reading from Samba 3.5.16 using SMB
    protocol (dialect 1.5), copying ~ 2000 files from the share to local
    hard disk, with consistently better throughput than in smb311_read

Fingers crossed that you will be able to determine why 4.6.5 is slower 
in both scenarios, and especially so much slower when writing to the 
share (smb311_write) and one more time, thanks a million for digging 
into these packet dumps...

Best regards
Andreas

Hello again, Jeremy and other Samba experts,

I'm sorry to be such a pain in your neck(s), but I still need your help 
in looking for help trying to find out why SMB2/3.1.1 in Samba 4.6.5 
performs so much worse than SMB/1.5 in Samba 3.6.15 in scenarios with a 
huge number of small files.

As requested by Jeremy, I have done wireshark "pcapng" captures of the
four scenarios as described in my original post below:

  * smb311_write - Win 10 client storing ~ 1000 small files onto Samba
    4.6.5/Thecus NAS
  * smb15_write - Win 10 client storing ~ 1000 small files on Samba
    3.6.15/Thecus NAS
  * smb311_read - Win 10 client reading ~ 2000 small files from Samba
    4.6.5/Thecus NAS
  * smb15_read - Win 10 client reading ~ 2000 small files from Samba
    3.6.15/Thecus NAS

These recordings do indeed contain confidential data from my machine, 
which is why I have so far only sent a download link to Jeremy via 
private mail.

In case others from the Samba team would also like to look into the 
wireshark capture traces, please get back to me directly and request 
access: I will then also send you a download link/password to the 
capture files ZIP via private mail.

I would really appreciate to be able to switch this old Thecus N4200PRO 
NAS away from Thecus' outdated 3.6.15 version (prone to
"SambaCry") to a
self-compiled, but secure 4.6.x version asap.

Many thanks one more time for your kind help with this & best regards
Andreas


-------- Weitergeleitete Nachricht --------
Betreff: 	Re: [Samba] Huge number of small files performance regression 
from 3.5.16 to 4.6.5 with identical smb.conf
Datum: 	Tue, 20 Jun 2017 14:30:22 +0200
Von: 	awl1 <awl1 at mnet-online.de>
An: 	Jeremy Allison <jra at samba.org>, samba at lists.samba.org



Hello again, Jeremy,

first of all, I am terribly sorry for my late reply. I tried to send my
posting many times, but my mail has always been silently discarded by
the Samba mail servers due to my main mail provider (GMX - a very large
German mail provider with millions of customers) having been blacklisted
by SORBS.

For the time being, SORBS is still unwilling to delist them for unknown
reasons (which I consider a clear malpractice by SORBS, as GMX has
sophisticated spam/abuse management in place), so I had to switch to
another mail provider just in order to be able to post again on the
Samba list... :-(


Am 13.06.2017 um 19:00 schrieb awl1:
> Am 13.06.2017 um 18:36 schrieb Jeremy Allison:
>> Can you get comparitive wireshark traces for the two cases ?
>>
>> That would help discover what the bottleneck is.
> I am not at all a network guy, but I hope that - maybe with a little 
> more help from your part once I have tried to do so in practice - I 
> should be able to do so...
OK, so it looks like I have been able to successfully produce Wireshark
capture files for the four scenarios... :-)

As I am almost certain that these packet captures will contain at least
some sensitive information from my environment - such as e.g. user,
share and machine names, IP addresses (possibly in the old SMB dialect
1.5 even the clear-text password for the share?) - I will only send the
link to the captures ZIP file stored in my cloud space to you via
private mail. So please keep the packet dumps confidential, and only
share them with other Samba developers after getting my explicit consent!

The ZIP file contains four Wireshark captures for the two scenarios
(write to and read from share) and the two Samba/SMB versions
(4.6.5/SMB2/dialect 3.1.1 and 3.5.16/SMB/dialect 1.5) in "pcapng"
format:

  * smb311_write - Win10 client writing to Samba 4.6.5 using SMB2
    protocol (dialect 3.1.1), copying ~ 1000 files from local hard disk
    onto the share, documenting the issue with very slow throughput of
    below 10 kB/sec (especially in the range of file 300-600, most
    interestingly throughput improved again after some time)
  * smb15_write - Win10 client writing to Samba 3.5.16 using SMB
    protocol (dialect 1.5), copying ~ 1000 files from local hard disk
    onto the share, with much better throughput than in smb311_write

  * smb311_read - Win10 client reading from Samba 4.6.5 using SMB2
    protocol (dialect 3.1.1), copying ~ 2000 files from the share to
    local hard disk, with acceptable throughput, but consistently slower
    than in smb15_read
  * smb15_read - Win10 client reading from Samba 3.5.16 using SMB
    protocol (dialect 1.5), copying ~ 2000 files from the share to local
    hard disk, with consistently better throughput than in smb311_read

Fingers crossed that you will be able to determine why 4.6.5 is slower
in both scenarios, and especially so much slower when writing to the
share (smb311_write) and one more time, thanks a million for digging
into these packet dumps...

Best regards
Andreas