Hi Volker, Thanks for the response. The clients are Win 7, Win 10 and various flavours of Linux, with Samba client versions ranging from 3.6 to 4.5.3. There is no perceptible difference between any of them. Network latency is never above 0.2s and mostly sits at around 0.1s. If that's a problem it is so only for Samba. I did as you suggested with the smb options but that just made it worse. Using the same test environment and file set, the final speed dropped from average of around 3.5MB/s to an average of about 3.0MB/s over a series if test runs. Again, the OS, and therefore the client version, made no difference. I guess I won't be using those settings in production. regards, John On 09/01/17 18:00, Volker Lendecke wrote:> On Mon, Jan 09, 2017 at 09:39:31AM +1100, John Gardeniers via samba wrote: >> We are experiencing painfully slow performance when transferring many small >> files via Samba. We are currently running Samba 4.5.1, although based on my >> searches the version doesn't seem to make any difference and this is a long >> standing problem. > SMB performance critically depends on the client. What client are you > using? > >> Based on my searches I've been playing around with a number of settings. The >> testbed right now has the following options added to the basic smb.conf: >> >> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=32768 >> SO_SNDBUF=32768 >> read raw = yes >> write raw = yes >> strict locking = no >> min receivefile size = 16384 >> use sendfile = yes >> aio read size = 16384 >> aio write size = 16384 >> strict allocate = yes > Just get rid of all these options, in particular the socket options. > Replace them with > > aio read size = 1 > aio write size = 1 > > and that's it. > > What't your network latency (ping time)? That's a lot more critical > than raw bandwidth. > > Volker > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > For more information please visit http://www.symanteccloud.com > ______________________________________________________________________
On Tue, Jan 10, 2017 at 08:24:35AM +1100, John Gardeniers via samba wrote:> Hi Volker, > > Thanks for the response. The clients are Win 7, Win 10 and various flavours > of Linux, with Samba client versions ranging from 3.6 to 4.5.3. There is no > perceptible difference between any of them. > > Network latency is never above 0.2s and mostly sits at around 0.1s. If > that's a problem it is so only for Samba.How does Windows server side perform?> I did as you suggested with the smb options but that just made it worse. > Using the same test environment and file set, the final speed dropped from > average of around 3.5MB/s to an average of about 3.0MB/s over a series if > test runs. Again, the OS, and therefore the client version, made no > difference. I guess I won't be using those settings in production.What is your Unix? Is that some AIX or Solaris flavor? Ok. Can you strace the server side smbd with strace -ttT -o /tmp/smbd.strace -f -p <smbd-pid> ? Volker
On Wed, Jan 11, 2017 at 08:29:25AM +1100, John Gardeniers wrote:> Hi Volker, > > On 10/01/17 16:41, Volker Lendecke via samba wrote: > >On Tue, Jan 10, 2017 at 08:24:35AM +1100, John Gardeniers via samba wrote: > >>Hi Volker, > >> > >>Thanks for the response. The clients are Win 7, Win 10 and various flavours > >>of Linux, with Samba client versions ranging from 3.6 to 4.5.3. There is no > >>perceptible difference between any of them. > >> > >>Network latency is never above 0.2s and mostly sits at around 0.1s. If > >>that's a problem it is so only for Samba. > >How does Windows server side perform? > Huh? We're using Samba, not Windows Server. However, Servers are all on aI was trying to nail down how much would be to attribute to Samba itself and how much to the client, protocol an network in between. For that a comparison to a Windows server in the same location would be helpful.> For the sake of testing I set up a Windows Server 2008 machine and ran the > same test. The overall speed is more than 3 times as fast.Ok, so Samba is a factor of 3 or more slower. That's bad, and we need to do something about it.> There are multiple smbd processes, so I'm going to assume you mean the > parent. The trace was run while copying the same test directory from a Win 7No, I mean the process doing the work for an individual client. The parent is not helpful in this respect, sorry. You can find out which smbd is responsible for your test client using the smbstatus command. There you will find a list of entries, grep for the client's IP address and you might be able to figure out a process ID.> machine to the CentOS 6 test server, which is currently running Sernet Samba > 4.5.3. Running strace killed the performance in a really major way, so I > didn't finish copying the full test set.Right, strace does kill performance. However, I am surprised that stracing the parent kills performance so badly.> The full trace I got compressed to about 18MB, which is a bit large, so I've > attached a subset which is from the start to a chunk into where the copying > took place. Please let me know if you need more than that.Yes, I need more than that. I need it while it is running full steam. Please make sure that you trace the smbd that is responsible for your client. I am afraid that this will slow down your client, but there is not much that Samba can be blamed for in this respect. But it will still gather valuable information assuming you are able to catch the main worker smbd that is responsible for your client. Volker
On Wed, Jan 11, 2017 at 08:29:25AM +1100, John Gardeniers wrote:> >strace -ttT -o /tmp/smbd.strace -f -p <smbd-pid> ? > There are multiple smbd processes, so I'm going to assume you mean theOk, you did trace the worker smbd. Can you try disabling the full_audit module? This might impose some load. Volker
Hi Volker, Sorry to take so long to respond, it's been a bit busy around here. I haven't taken another trace but I can report that with full_audit disable I'm now getting small file performance similar to what I get when transferring to/from a Windows server, so I suppose that's as good as it's going to get. There is no perceptible difference with large files. This raises an interesting question that should probably still be investigated. Although the full_audit module had previously been loaded, the Samba share I have been using for testing did not have auditing enabled. Why then did it still have such a huge impact? Also, I have commented out the full_audit module in smb.conf on the production server and reloaded the config but the performance is still the same. It would appear that Samba needs to be restarted for the change to take effect, which is not something we can easily do without impacting multiple key systems. Is there another way to unload the module that won't cause disconnections? regards, John On 11/01/17 21:00, Volker Lendecke wrote:> On Wed, Jan 11, 2017 at 08:29:25AM +1100, John Gardeniers wrote: >>> strace -ttT -o /tmp/smbd.strace -f -p <smbd-pid> ? >> There are multiple smbd processes, so I'm going to assume you mean the > Ok, you did trace the worker smbd. Can you try disabling the > full_audit module? This might impose some load. > > Volker > > ______________________________________________________________________ > This email has been scanned by the Symantec Email Security.cloud service. > For more information please visit http://www.symanteccloud.com > ______________________________________________________________________