We use SMB to transfer large files (between 1GB and 5GB) from RedHat AS4 Content Storage servers to Windows clients with 6 DVD burners and robotic arms and other cool gadgets. The servers used to be Windows based, but we're migrating to RedHat for a host of reasons. Unfortunately, the RedHat Samba servers are about 2.5 times slower than the Windows servers. Windows will copy a 1GB file in about 30 seconds, where as it takes about 70 to 75 seconds to copy the same file from a RedHat Samba server. I've asked Dr. Google and gotten all kinds of suggestions, most of which have already been applied by RedHat to the stock Samba config. I've opened a ticket with RedHat. They pointed out a couple errors in my config, but fixing those didn't have any effect. Some tweaking, however, has gotten the transfer speed to about 50 seconds for that 1GB file. But I seem to have hit a brick wall; my fastest time ever was 44 seconds, but typically it's around 50. I know it's not a problem with network or disk; if I use Apache and HTTP to transfer the same file from the same server, it transfers in about 15 to 20 seconds. Unfortunately, HTTP doesn't meet our other requirements for random access to the file. Do you folks use Samba for large file transfers at all? Have you had any luck speeding it up past about 23MBps (the 44 second transfer speed)? Any help you may have would be fantastic. Thanks. -Mark
On Tue, Sep 19, 2006 at 06:19:43PM -0700, Mark Smith wrote:> We use SMB to transfer large files (between 1GB and 5GB) from RedHat AS4 > Content Storage servers to Windows clients with 6 DVD burners and > robotic arms and other cool gadgets. The servers used to be Windows > based, but we're migrating to RedHat for a host of reasons. > > Unfortunately, the RedHat Samba servers are about 2.5 times slower than > the Windows servers. Windows will copy a 1GB file in about 30 seconds, > where as it takes about 70 to 75 seconds to copy the same file from a > RedHat Samba server. > > I've asked Dr. Google and gotten all kinds of suggestions, most of which > have already been applied by RedHat to the stock Samba config. I've > opened a ticket with RedHat. They pointed out a couple errors in my > config, but fixing those didn't have any effect. Some tweaking, > however, has gotten the transfer speed to about 50 seconds for that 1GB > file. > > But I seem to have hit a brick wall; my fastest time ever was 44 > seconds, but typically it's around 50. > > I know it's not a problem with network or disk; if I use Apache and HTTP > to transfer the same file from the same server, it transfers in about 15 > to 20 seconds. Unfortunately, HTTP doesn't meet our other requirements > for random access to the file. > > Do you folks use Samba for large file transfers at all? Have you had > any luck speeding it up past about 23MBps (the 44 second transfer > speed)? Any help you may have would be fantastic. Thanks.An interesting thing you could do is to use a port of smbclient on Windows (no I don't know where to get one :-) to copy the client to the Windows client in userspace. smbclient will use read pipelining (ie. issue more than one read at a time) whereas Windows clients issue one read, wait for response, issue the next read, wait for response etc. That would tell you if it's a client redirector issue. You could probably use cygwin to compile smbclient. Jeremy.
I wanted to follow up to my email to provide at least a partial answer to my problem. The stock RedHat AS4-U3 Samba config has SO_SNDBUF and SO_RCVBUF set to 8k. With this value, I can transfer a 1GB file in about 70-75 seconds, about 14MBps. If I increase those buffers to their max value of 64k, that same 1GB file transfers in 45-50 seconds, about 23MBps. That is the _ONLY_ configuration value I've found that made any difference in my setup. All the other tweaks I'd done, when removed, seemed to make no difference at all. I was playing with oplocks, buffers, max xmit sizes, you name it. But the socket option buffers was the only thing that made a difference. I'm still looking for more speed. I'll report if I find anything else that helps. In response to Jeremy's suggestion of using smbclient, I ran a test from a Linux client using smbclient and it reported a transfer rate of 21MBps, about the same as a normal smbfs mount. I haven't tried porting smbclient to Windows yet, and probably won't until we get more info on what the server is doing. Thanks everyone. -Mark Mark Smith wrote:> We use SMB to transfer large files (between 1GB and 5GB) from RedHat AS4 > Content Storage servers to Windows clients with 6 DVD burners and > robotic arms and other cool gadgets. The servers used to be Windows > based, but we're migrating to RedHat for a host of reasons. > > Unfortunately, the RedHat Samba servers are about 2.5 times slower than > the Windows servers. Windows will copy a 1GB file in about 30 seconds, > where as it takes about 70 to 75 seconds to copy the same file from a > RedHat Samba server. > > I've asked Dr. Google and gotten all kinds of suggestions, most of which > have already been applied by RedHat to the stock Samba config. I've > opened a ticket with RedHat. They pointed out a couple errors in my > config, but fixing those didn't have any effect. Some tweaking, > however, has gotten the transfer speed to about 50 seconds for that 1GB > file. > > But I seem to have hit a brick wall; my fastest time ever was 44 > seconds, but typically it's around 50. > > I know it's not a problem with network or disk; if I use Apache and HTTP > to transfer the same file from the same server, it transfers in about 15 > to 20 seconds. Unfortunately, HTTP doesn't meet our other requirements > for random access to the file. > > Do you folks use Samba for large file transfers at all? Have you had > any luck speeding it up past about 23MBps (the 44 second transfer > speed)? Any help you may have would be fantastic. Thanks. > > -Mark
I have not. Unfortunately, that is not a trivial process. I believe everything supports it, but it's a somewhat major change to my production systems. It might be worth trying as a data point, however. Given the iPerf tests, I really don't think it's a network bottleneck at this point. -Mark Pitti, Raul wrote:> i am not an expert, but, > do you have jumbo frame enabled on your nic and switch? > try using ethtools... > > RP > > Mark Smith wrote: >> Mark Smith wrote: >>> As a data point, I'm going to try a newer version of Samba. (RHEL4 >>> uses 3.0.10-RedHat-Heavily-Modified-Of-Course) If that makes a >>> difference, then I have to decide whether it's worth it to me to keep >>> RedHat support or not. (And when I say "I," I really mean "my >>> management.") >> >> I've just tried this. Samba v3.0.23c, built locally from the Fedora >> Source RPM as distributed at samba.org, makes no noticeable >> difference: still about 45 seconds. >> >>>> I have lots of memory. I use these configurations in sysctl.conf to >>>> up the performance of send/recieve windows on my systems. There's >>>> articles out there. I don't have historical references handy. >>>> YMMV. >>>> net.core.wmem_max = 1048576 >>>> net.core.rmem_max = 1048576 >>>> net.ipv4.tcp_wmem = 4096 65536 1048575 >>>> net.ipv4.tcp_rmem = 4096 524288 1048575 >>>> net.ipv4.tcp_window_scaling = 1 >>> >>> I have not tried tweaking the TCP stack in the OS. I'll give that a >>> shot. >> >> The person at RedHat who's handling my ticket just suggested these >> very changes, without the last one. They did not help. The values he >> gave were a little different: >> >> ----- snip! ----- >> # increase TCP maximum buffer size >> net.core.rmem_max = 16777216 >> net.core.wmem_max = 16777216 >> >> # increase Linux autotuning TCP buffer limits >> # min, default, and maximum number of bytes to use >> net.ipv4.tcp_rmem = 4096 87380 16777216 >> net.ipv4.tcp_wmem = 4096 65536 16777216 >> ----- snip! ----- >> >> I also tried your values, with the tcp_window_scaling, with no luck. >> >> -Mark >
i am not an expert, but, do you have jumbo frame enabled on your nic and switch? try using ethtools... RP Mark Smith wrote:> Mark Smith wrote: >> As a data point, I'm going to try a newer version of Samba. (RHEL4 >> uses 3.0.10-RedHat-Heavily-Modified-Of-Course) If that makes a >> difference, then I have to decide whether it's worth it to me to keep >> RedHat support or not. (And when I say "I," I really mean "my >> management.") > > I've just tried this. Samba v3.0.23c, built locally from the Fedora > Source RPM as distributed at samba.org, makes no noticeable difference: > still about 45 seconds. > >>> I have lots of memory. I use these configurations in sysctl.conf to >>> up the performance of send/recieve windows on my systems. There's >>> articles out there. I don't have historical references handy. >>> YMMV. >>> net.core.wmem_max = 1048576 >>> net.core.rmem_max = 1048576 >>> net.ipv4.tcp_wmem = 4096 65536 1048575 >>> net.ipv4.tcp_rmem = 4096 524288 1048575 >>> net.ipv4.tcp_window_scaling = 1 >> >> I have not tried tweaking the TCP stack in the OS. I'll give that a >> shot. > > The person at RedHat who's handling my ticket just suggested these very > changes, without the last one. They did not help. The values he gave > were a little different: > > ----- snip! ----- > # increase TCP maximum buffer size > net.core.rmem_max = 16777216 > net.core.wmem_max = 16777216 > > # increase Linux autotuning TCP buffer limits > # min, default, and maximum number of bytes to use > net.ipv4.tcp_rmem = 4096 87380 16777216 > net.ipv4.tcp_wmem = 4096 65536 16777216 > ----- snip! ----- > > I also tried your values, with the tcp_window_scaling, with no luck. > > -Mark-- -------------------- Ra?l Pitt? Palma, Eng. Global Engineering and Technology S.A. mobile (507)-6616-0194 office (507)-390-4338 Republic of Panama www.globaltecsa.com
Mark Smith
2006-Sep-30 00:51 UTC
Conclusion - Re: [Samba] Transfer rates faster than 23MBps?
In the interest of closing this loop for the sake of the archives... :) My company has officially thrown in the towel on this issue. Everything we're seeing is suggesting that 22MBps is about as fast as we're going to get with SMB. RedHat, our parent company's IT department (much bigger than ours), the (read: this) Samba Mailing List, and all my techie friends, all agree that we're at a limit. For the parts of our process where higher speed is critical, we'll look into replacing SMB with a more appropriate protocol (eg: FTP or SCP hacked to allow a cipher of "none"). I want to thank everyone who tried to help with suggestions of things to try, areas to look, and what-not. Every time, I'd see something and say "Huh. Haven't tried that!" and got hopeful. Alas, none of it ended up helping, but that's not y'all's fault. :) Again, thanks everyone for your help. Cheers. -Mark Mark Smith wrote:> We use SMB to transfer large files (between 1GB and 5GB) from RedHat AS4 > Content Storage servers to Windows clients with 6 DVD burners and > robotic arms and other cool gadgets. The servers used to be Windows > based, but we're migrating to RedHat for a host of reasons. > > Unfortunately, the RedHat Samba servers are about 2.5 times slower than > the Windows servers. Windows will copy a 1GB file in about 30 seconds, > where as it takes about 70 to 75 seconds to copy the same file from a > RedHat Samba server. > > I've asked Dr. Google and gotten all kinds of suggestions, most of which > have already been applied by RedHat to the stock Samba config. I've > opened a ticket with RedHat. They pointed out a couple errors in my > config, but fixing those didn't have any effect. Some tweaking, > however, has gotten the transfer speed to about 50 seconds for that 1GB > file. > > But I seem to have hit a brick wall; my fastest time ever was 44 > seconds, but typically it's around 50. > > I know it's not a problem with network or disk; if I use Apache and HTTP > to transfer the same file from the same server, it transfers in about 15 > to 20 seconds. Unfortunately, HTTP doesn't meet our other requirements > for random access to the file. > > Do you folks use Samba for large file transfers at all? Have you had > any luck speeding it up past about 23MBps (the 44 second transfer > speed)? Any help you may have would be fantastic. Thanks. > > -Mark