awl1
2017-Jun-29 09:03 UTC
[Samba] Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf
Hello again, Jeremy and other Samba experts,
I'm sorry to be such a pain in your neck(s), but I still need your help
in looking for help trying to find out why SMB2/3.1.1 in Samba 4.6.5
performs so much worse than SMB/1.5 in Samba 3.6.15 in scenarios with a
huge number of small files.
As requested by Jeremy, I have done wireshark "pcapng" captures of the
four scenarios as described in my original post below:
* smb311_write - Win 10 client storing ~ 1000 small files onto Samba
4.6.5/Thecus NAS
* smb15_write - Win 10 client storing ~ 1000 small files on Samba
3.6.15/Thecus NAS
* smb311_read - Win 10 client reading ~ 2000 small files from Samba
4.6.5/Thecus NAS
* smb15_read - Win 10 client reading ~ 2000 small files from Samba
3.6.15/Thecus NAS
These recordings do indeed contain confidential data from my machine,
which is why I have so far only sent a download link to Jeremy via
private mail.
In case others from the Samba team would also like to look into the
wireshark capture traces, please get back to me directly and request
access: I will then also send you a download link/password to the
capture files ZIP via private mail.
I would really appreciate to be able to switch this old Thecus N4200PRO
NAS away from Thecus' outdated 3.6.15 version (prone to
"SambaCry") to a
self-compiled, but secure 4.6.x version asap.
Many thanks one more time for your kind help with this & best regards
Andreas
-------- Weitergeleitete Nachricht --------
Betreff: Re: [Samba] Huge number of small files performance regression
from 3.5.16 to 4.6.5 with identical smb.conf
Datum: Tue, 20 Jun 2017 14:30:22 +0200
Von: awl1 <awl1 at mnet-online.de>
An: Jeremy Allison <jra at samba.org>, samba at lists.samba.org
Hello again, Jeremy,
first of all, I am terribly sorry for my late reply. I tried to send my
posting many times, but my mail has always been silently discarded by
the Samba mail servers due to my main mail provider (GMX - a very large
German mail provider with millions of customers) having been blacklisted
by SORBS.
For the time being, SORBS is still unwilling to delist them for unknown
reasons (which I consider a clear malpractice by SORBS, as GMX has
sophisticated spam/abuse management in place), so I had to switch to
another mail provider just in order to be able to post again on the
Samba list... :-(
Am 13.06.2017 um 19:00 schrieb awl1:> Am 13.06.2017 um 18:36 schrieb Jeremy Allison:
>> Can you get comparitive wireshark traces for the two cases ?
>>
>> That would help discover what the bottleneck is.
> I am not at all a network guy, but I hope that - maybe with a little
> more help from your part once I have tried to do so in practice - I
> should be able to do so...
OK, so it looks like I have been able to successfully produce Wireshark
capture files for the four scenarios... :-)
As I am almost certain that these packet captures will contain at least
some sensitive information from my environment - such as e.g. user,
share and machine names, IP addresses (possibly in the old SMB dialect
1.5 even the clear-text password for the share?) - I will only send the
link to the captures ZIP file stored in my cloud space to you via
private mail. So please keep the packet dumps confidential, and only
share them with other Samba developers after getting my explicit consent!
The ZIP file contains four Wireshark captures for the two scenarios
(write to and read from share) and the two Samba/SMB versions
(4.6.5/SMB2/dialect 3.1.1 and 3.5.16/SMB/dialect 1.5) in "pcapng"
format:
* smb311_write - Win10 client writing to Samba 4.6.5 using SMB2
protocol (dialect 3.1.1), copying ~ 1000 files from local hard disk
onto the share, documenting the issue with very slow throughput of
below 10 kB/sec (especially in the range of file 300-600, most
interestingly throughput improved again after some time)
* smb15_write - Win10 client writing to Samba 3.5.16 using SMB
protocol (dialect 1.5), copying ~ 1000 files from local hard disk
onto the share, with much better throughput than in smb311_write
* smb311_read - Win10 client reading from Samba 4.6.5 using SMB2
protocol (dialect 3.1.1), copying ~ 2000 files from the share to
local hard disk, with acceptable throughput, but consistently slower
than in smb15_read
* smb15_read - Win10 client reading from Samba 3.5.16 using SMB
protocol (dialect 1.5), copying ~ 2000 files from the share to local
hard disk, with consistently better throughput than in smb311_read
Fingers crossed that you will be able to determine why 4.6.5 is slower
in both scenarios, and especially so much slower when writing to the
share (smb311_write) and one more time, thanks a million for digging
into these packet dumps...
Best regards
Andreas
Andrew Walker
2017-Jun-29 20:14 UTC
[Samba] Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf
Andreas, A few thoughts regarding your system 1) If it's a home system and you're specifically concerned about mitigating CVE 2017-7494, (a) verify that your share isn't mounted 'noexec' - if it's mounted this way then you're safe (b) if not (a), then add the [global] parameter "nt pipe support = no". This will break functionality that relies on support for named pipes, but downloading / uploading files should still work normally. 2) If you need to use Samba 4.6.5, try starting with a minimal smb.conf with logging turned up. Then review your samba logs. Note that setting log level to "10' will probably be more verbose than you want. Choose an appropriate level. Here's one from on of my testing machines. : [global] guest account = awalker map to guest = Bad User log level = 10 [Donkey Vol] path = "/mnt/Donkey/Vol1" writeable = yes vfs objects = zfs_space guest ok = yes guest only = yes 3) The last firmware update looks like it was from 2014. You're probably vulnerable to a lot more than just that single Samba CVE. If this is in a business environment, perhaps look into migrating to a new appliance / server that's not EOL. If it's a home environment, and you like to tinker with things look for guides on installing stock Debian on the Thecus (it looks the Thecus has x86 hardware and an IDE DOM), and then adding Louis's Samba repo / installing the package. I did this with an old WD MyCloud about a year or so ago, and was much happier with the system afterwards. It's hackish, but can be a fun side-project. Andrew
awl1
2017-Jun-30 22:22 UTC
[Samba] Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf
Hello Andrew, many thanks for your thoughts and clarifications - here's my take on them: Regarding 1), I had indeed started to look into simply adding "nt pipe support = no" to the original 3.5.6 Samba smb.conf first. But in order to do so, I would also need to create my own Samba module, replacing the Thecus built-in: Without deactivating the built-in Samba, it is not possible to add custom smb.conf entries (anything not settable through the admin GUI), and the admin GUI backs out/overwrites any custom changes to the default smb.conf from an SSH command line every time I execute any non-readonly operation from the admin GUI... :-( So I ended up with not simply providing my own smb.conf, but also compiling my own Samba version, and I have indeed successfully cross-compiled Samba 4.6.5 including all dependencies without any issues: Besides the performance regression for huge numbers of small files, my self-compiled 4.6.5 works completely fine and as expected. Would you indeed think that the performance issue (SMB2 3.1.1 notably slower than SMB 1.5 when copying a huge number of small files) will document itself in debug level log entries? As these log entries will also need to be written to the exact same NAS HDDs, they would also inevitably come with a throughput/performance overhead/bottleneck for this debug logging...!? (With Thecus' default log level settings, there are no error messages at all in any of the Samba logs when I execute the file copy tests...) And regarding 3): Is there any particular reason why you seem to expect Louis' Debian modules for 4.6.5 to perform better than my self-compiled version? Could this indeed be due to the fact that my Samba 4.6.5 on the NAS is still running on Thecus kernel 2.6.33 (I'd rather question this, as I have compiled Samba itself as well as all its required dependencies in up-to-date versions with gcc-5.2 from the latest crosstools-ng that works on the NAS's kernel)? This is my private NAS located in my Home Office LAN, and it also runs a custom (and more or less up-to-date) OpenSSH and Dovecot service compiled by myself. The NAS is mainly used as a backup target and for development purposes, so the huge number of small files scenario unfortunately is not that rare... Base line: From my guts feeling, most probably comparing the Wireshark packet captures as proposed by Jeremy would be a good way to move forward and find out why the number of packets in the Samba 4.6.5/SMB2 3.1.1 scenario seems to be higher than using Samba 3.6.15/SMB 1.5, but I will clearly need some help by you, SMB protocol/Wireshark experts, to know what to look for in those packet captures... ;-) So I'd still prefer to stay with my self-compiled 4.6.x version and am looking forward for some help from the experts in how to read/analyze the Wireshark captures, trying to find out why SMB2/3.1.1 in Samba 4.6.5 seems to be hitting a performance regression... Thanks anyway & best regards Andreas Am 29.06.2017 um 22:14 schrieb Andrew Walker via samba:> Andreas, > > A few thoughts regarding your system > 1) If it's a home system and you're specifically concerned about mitigating > CVE 2017-7494, (a) verify that your share isn't mounted 'noexec' - if it's > mounted this way then you're safe (b) if not (a), then add the [global] > parameter "nt pipe support = no". This will break functionality that relies > on support for named pipes, but downloading / uploading files should still > work normally. > > 2) If you need to use Samba 4.6.5, try starting with a minimal smb.conf > with logging turned up. Then review your samba logs. Note that setting log > level to "10' will probably be more verbose than you want. Choose an > appropriate level. Here's one from on of my testing machines. : > > [global] > guest account = awalker > map to guest = Bad User > log level = 10 > > [Donkey Vol] > path = "/mnt/Donkey/Vol1" > writeable = yes > vfs objects = zfs_space > guest ok = yes > guest only = yes > > 3) The last firmware update looks like it was from 2014. You're probably > vulnerable to a lot more than just that single Samba CVE. If this is in a > business environment, perhaps look into migrating to a new appliance / > server that's not EOL. > > If it's a home environment, and you like to tinker with things look for > guides on installing stock Debian on the Thecus (it looks the Thecus has > x86 hardware and an IDE DOM), and then adding Louis's Samba repo / > installing the package. I did this with an old WD MyCloud about a year or > so ago, and was much happier with the system afterwards. It's hackish, but > can be a fun side-project. > > Andrew
awl1
2017-Jul-03 10:08 UTC
[Samba] Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf
Hello one more time, Jeremy & fellow Samba experts/developers,
over the weekend, I have done some more reading about wireshark
tooling/statistics - the answer to
https://ask.wireshark.org/questions/58970/analysing-performance-issues-with-storage-smb2
was very helpful - and am now able to provide very clear and simple
proof of the performance regression that I am seeing between SMB/1.5 in
Samba 3.5.16 and SMB2/3.1.1 in Samba 4.6.5, using Wireshark's
"Statistics -> Service Response Times -> SMB(2)" tool.
I really hope that someone from the development team is now interested
in taking over and starts looking into this with me. Please rest assured
that I will be happy to do everything I can to support the analysis and
testing process. Please get back to me, and I will be sending you the
access information and password for the respective wireshark PCAPNG
traces ZIP file.
My test scenario: Win 10 client, copying files from/to share with Total
Commander, Thecus N4200pro NAS (Linux kernel 2.6.33) with 3 GB RAM and
either Thecus original Samba 3.5.16 or self-compiled (using gcc-5.2)
Samba 4.6.5, exact same smb.conf for both versions, definitely no other
load on /access to the NAS during my testing.
A) "Write" Scenario: Win 10 client copying/storing ~ 1000 small files
onto a Samba share on Thecus NAS
A1) smb15_write: Samba share running Thecus Samba 3.5.16 (writing 1024
small files to share)
================================================================================================SMB
Service Response Time Statistics - smb15_write:
Index Procedure Calls Min SRT (s) Max SRT (s)
Avg SRT (s) Sum SRT (s)
-------------------------------------------------------------------------------------------------
SMB Commands
Close 4 2791 0.000210 0.020178
0.000831 2.320021
Negotiate Protocol 114 1 0.001986 0.001986
0.001986 0.001986
NT Create AndX 162 3066 0.000650 0.019631
0.003122 9.570728
Session Setup AndX 115 2 0.001048 0.005596
0.003322 0.006644
Tree Connect AndX 117 1 0.012328 0.012328
0.012328 0.012328
Write AndX 47 1024 0.000473 0.350664
0.001528 1.564348
Transaction2 Sub-Commands
FIND_FIRST2 1 2042 0.001599 0.017518
0.003191 6.515678
QUERY_FILE_INFO 7 3071 0.000260 0.007440
0.000421 1.292491
QUERY_FS_INFO 3 1024 0.000259 0.017036
0.000363 0.371787
QUERY_PATH_INFO 5 2 0.000437 0.001040
0.000739 0.001477
SET_FILE_INFO 8 2037 0.001218 0.008207
0.001812 3.691841
NT Transaction Sub-Commands
-------------------------------------------------------------------------------------------------
Grand Total Sum of SRT (s) 25.349329
A2) smb31_write: Samba share running self-compiled Samba 4.6.5 (writing
1015 small files to share)
=======================================================================================SMB2
Service Response Time Statistics - smb31_write:
Index Procedure Calls Min SRT (s) Max SRT (s) Avg SRT
(s) Sum SRT (s)
----------------------------------------------------------------------------------------
SMB2
Close 6 3059 0.000496 0.590329
0.004469 13.670456
Create 5 3061 0.001683 0.023491
0.005686 17.405265
Find 14 1607 0.001383 0.746684
0.193413 310.814294
GetInfo 16 46 0.000424 0.009404
0.001441 0.066298
Ioctl 11 1 0.000510 0.000510
0.000510 0.000510
Negotiate Protocol 0 1 0.043381 0.043381
0.043381 0.043381
Session Setup 1 2 0.001535 0.037150
0.019343 0.038685
SetInfo 17 2033 0.000757 0.007015
0.001111 2.259172
Tree Connect 3 2 0.001591 0.031307
0.016449 0.032898
Tree Disconnect 4 1 0.004142 0.004142
0.004142 0.004142
Write 9 1015 0.000590 0.358855
0.001798 1.825072
----------------------------------------------------------------------------------------
Grand Total Sum of SRT
(s) 346.160173
As you can easily see, the big difference here is in the "Find"
Operation:
A2a) While in Samba 4.6.5, 1607 calls to SMB2 "Find" take 310.81
seconds, in Samba 3.5.16 2042 calls to SMB "FIND_FIRST2" take only
6.51
seconds.
The obvious consequence is that the overall grand total sum of service
response time for copying ~ 1000 small files goes up from 25 to a really
annoying 346 seconds.
B) "Read" Scenario: Win 10 client copying/reading ~ 2000 small files
from a Samba share on Thecus NAS
B1) smb15_read: Samba share running Thecus Samba 3.5.16 (reading 2040
small files from share)
================================================================================================SMB
Service Response Time Statistics - smb15_read:
Index Procedure Calls Min SRT (s) Max SRT (s)
Avg SRT (s) Sum SRT (s)
-------------------------------------------------------------------------------------------------
SMB Commands
Close 4 1630 0.000234 0.051417
0.001525 2.486313
Negotiate Protocol 114 1 0.001941 0.001941
0.001941 0.001941
NT Create AndX 162 2040 0.009347 0.170227
0.012715 25.938098
Read AndX 46 2185 0.000267 0.008728
0.000842 1.838953
Session Setup AndX 115 2 0.001074 0.005605
0.003340 0.006679
Tree Connect AndX 117 1 0.012529 0.012529
0.012529 0.012529
Transaction2 Sub-Commands
FIND_FIRST2 1 10 0.014211 0.817830
0.218505 2.185050
FIND_NEXT2 2 117 0.167130 0.868053
0.547014 64.000643
QUERY_FILE_INFO 7 10200 0.000267 0.003532
0.000363 3.706283
QUERY_FS_INFO 3 2048 0.000254 0.017356
0.000334 0.683642
QUERY_PATH_INFO 5 54 0.000424 0.055262
0.007493 0.404625
NT Transaction Sub-Commands
-------------------------------------------------------------------------------------------------
Grand Total Sum of SRT (s) 101.264756
B2) smb31_read: Samba share running self-compiled Samba 4.6.5 (reading
2033 small files from share)
=======================================================================================SMB2
Service Response Time Statistics - smb31_read:
Index Procedure Calls Min SRT (s) Max SRT (s) Avg SRT
(s) Sum SRT (s)
----------------------------------------------------------------------------------------
SMB2
Close 6 1754 0.000443 0.188892
0.006119 10.732004
Create 5 2038 0.001203 2.296105
0.026880 54.782150
Find 14 52 0.000435 2.540043
1.351859 70.296650
GetInfo 16 6091 0.000656 0.066154
0.001229 7.485929
Negotiate Protocol 0 1 0.032793 0.032793
0.032793 0.032793
Read 8 2033 0.000300 0.071186
0.000742 1.507737
Session Setup 1 2 0.001522 0.036670
0.019096 0.038192
Tree Connect 3 1 0.030543 0.030543
0.030543 0.030543
----------------------------------------------------------------------------------------
Grand Total Sum of SRT
(s) 144.905998
In this case, the difference in the overall grand total sum of response
times is "only" 101 seconds to 145 seconds, which is much less
drastic,
but still pretty much noticeable.
Unfortunately, it seems less clear here what the main culprit is - the
difference rather seems to come from three different SMB(2) calls:
B2a) While we have 10+117 = 127 calls to
"FIND_FIRST2"/"FIND_NEXT2" in
SMB that take ~ 66 seconds, only 52 calls to SMB2 "Find" take ~ 70
seconds.
B2b) SMB2 "Close" takes ~ 10 seconds for 1754 calls, while SMB
"Close"
takes only ~ 2.5 seconds for 1630 calls.
B2c) SMB2 "Create" takes ~ 55 seconds for 2038 calls, while SMB
"NT
Create AndX" takes only ~ 26 seconds for 2040 calls.
I truly hope we will be able to improve general Samba 4.6 / SMB2
performance for the "huge number of small files" scenario as a result
of
this exercise...
Many thanks in advance for your kind help & best regards
Andreas
Am 29.06.2017 um 11:03 schrieb awl1:> Hello again, Jeremy and other Samba experts,
>
> I'm sorry to be such a pain in your neck(s), but I still need your
> help in looking for help trying to find out why SMB2/3.1.1 in Samba
> 4.6.5 performs so much worse than SMB/1.5 in Samba 3.5.16 in scenarios
> with a huge number of small files.
>
> As requested by Jeremy, I have done wireshark "pcapng" captures
of the
> four scenarios as described in my original post below:
>
> * smb311_write - Win 10 client storing ~ 1000 small files onto Samba
> 4.6.5/Thecus NAS
> * smb15_write - Win 10 client storing ~ 1000 small files on Samba
> 3.5.16/Thecus NAS
> * smb311_read - Win 10 client reading ~ 2000 small files from Samba
> 4.6.5/Thecus NAS
> * smb15_read - Win 10 client reading ~ 2000 small files from Samba
> 3.5.16/Thecus NAS
>
> These recordings do indeed contain confidential data from my machine,
> which is why I have so far only sent a download link to Jeremy via
> private mail.
>
> In case others from the Samba team would also like to look into the
> wireshark capture traces, please get back to me directly and request
> access: I will then also send you a download link/password to the
> capture files ZIP via private mail.
>
> I would really appreciate to be able to switch this old Thecus
> N4200PRO NAS away from Thecus' outdated 3.5.16 version (prone to
> "SambaCry") to a self-compiled, but secure 4.6.x version asap.
>
> Many thanks one more time for your kind help with this & best regards
> Andreas
>
> Am 13.06.2017 um 18:36 schrieb Jeremy Allison:
>>> Can you get comparitive wireshark traces for the two cases ?
>>>
>>> That would help discover what the bottleneck is.
awl1
2017-Aug-04 14:49 UTC
[Samba] Could you please comment on my findings? (was: Huge number of small files performance regression from 3.5.16 to 4.6.5)
Hello again, Jeremy (and hello, fellow Samba experts - maybe someone of you can comment on this, too!?), please accept my apologies in case you are on vacation - otherwise in case you are still terribly busy, please at least reply very shortly stating when you will finally have some time to look into this issue and my findings... Sorry to still be such a pain in your neck(s), but another two weeks passed, and I still did not receive any feedback on my findings (and questions re making Microsoft aware of their issue in case my analysis is correct). Looking forward to your assessment and proposal on how to move forward... ;-) Thanks a million, best regards & have a great weekend Andreas Am 21.07.2017 um 15:56 schrieb awl1:> Hello Jeremy, > > Am 14.07.2017 um 23:33 schrieb Jeremy Allison: >> Look into it some more (I'm a bit busy with other stuff right now). > when will you be available again to look into my latest findings (see > previous posts) and provide your feedback? > > If you can confirm that it is Microsoft to blame, what would you > propose to raise this issue with them? Wouldn't this ideally be done > by somebody from the Samba team rather than me? > > Many thanks & best regards > Andreas
awl1
2017-Aug-18 10:51 UTC
[Samba] Friendly Reminder: Would you please comment on my findings?
Hello one more time, unfortunately, one more time, another two weeks have passed without any reaction from neither Jeremy in person nor the Samba team/community. I am quite a bit astonished, disappointed and negatively surprised indeed: Did I do anything wrong when raising my issue with you? Everybody else asking questions or raising issues on this list seems to receive great advice, while I no longer get any reactions at all... Is my issue (a huge performance regression when copying a large number of small files both from and to a share with SMB2/3 as compared to SMB1 using a Windows 10 client) too technical/complicated (can hardly believe that), too hard to understand, or simply (for whatever else reasons) otherwise not welcome!? Note that - as requested by Jeremy on July 14th: "Look into it some more (I'm a bit busy with other stuff right now)." - I think have successfully tracked down the performance regression to at least one major root cause, which is a client-side issue in SMB2/SMB3 on the Windows platform - see my several posts on this list dated July 15th: https://lists.samba.org/archive/samba/2017-July/209749.html https://lists.samba.org/archive/samba/2017-July/209750.html https://lists.samba.org/archive/samba/2017-July/209751.html So basically at this point this analysis seems mostly done and I am "only" waiting for Jeremy and/or the Samba team to look into my assessment and provide your feedback: a) Do you think that my analysis as laid out in this full thread, particularly the posts referenced above, is valid? b) In case you think it is, how can we proceed in terms of making Microsoft aware of their performance regression? As I feel that it would be much more likely to be successful, I would like the Samba team to raise this issue with your Microsoft contacts, rather than me as a private end customer (there's no company backing me with regards to this) complaining about this (which will only have a very low chance of success)... Eagerly looking forward to some feedback and support from the Samba community... Many thanks for your understanding and support and best regards Andreas Am 04.08.2017 um 16:49 schrieb awl1:> Hello again, Jeremy (and hello, fellow Samba experts - maybe someone > of you can comment on this, too!?), > > please accept my apologies in case you are on vacation - otherwise in > case you are still terribly busy, please at least reply very shortly > stating when you will finally have some time to look into this issue > and my findings... > > Sorry to still be such a pain in your neck(s), but another two weeks > passed, and I still did not receive any feedback on my findings (and > questions re making Microsoft aware of their issue in case my analysis > is correct). > > Looking forward to your assessment and proposal on how to move > forward... ;-) > > Thanks a million, best regards & have a great weekend > Andreas > > > Am 21.07.2017 um 15:56 schrieb awl1: >> Hello Jeremy, >> >> Am 14.07.2017 um 23:33 schrieb Jeremy Allison: >>> Look into it some more (I'm a bit busy with other stuff right now). >> when will you be available again to look into my latest findings (see >> previous posts) and provide your feedback? >> >> If you can confirm that it is Microsoft to blame, what would you >> propose to raise this issue with them? Wouldn't this ideally be done >> by somebody from the Samba team rather than me? >> >> Many thanks & best regards >> Andreas >
Possibly Parallel Threads
- Friendly Reminder: Would you please comment on my findings?
- Friendly Reminder: Would you please comment on my findings?
- Friendly Reminder: Would you please comment on my findings?
- Friendly Reminder: Would you please comment on my findings?
- Friendly Reminder: Huge number of small files performance regression from 3.5.16 to 4.6.5 with identical smb.conf