samba - Sep 2006 - Transfer rates faster than 23MBps?

We use SMB to transfer large files (between 1GB and 5GB) from RedHat AS4 
Content Storage servers to Windows clients with 6 DVD burners and 
robotic arms and other cool gadgets.  The servers used to be Windows 
based, but we're migrating to RedHat for a host of reasons.

Unfortunately, the RedHat Samba servers are about 2.5 times slower than 
the Windows servers.  Windows will copy a 1GB file in about 30 seconds, 
where as it takes about 70 to 75 seconds to copy the same file from a 
RedHat Samba server.

I've asked Dr. Google and gotten all kinds of suggestions, most of which 
have already been applied by RedHat to the stock Samba config.  I've 
opened a ticket with RedHat.  They pointed out a couple errors in my 
config, but fixing those didn't have any effect.  Some tweaking, 
however, has gotten the transfer speed to about 50 seconds for that 1GB 
file.

But I seem to have hit a brick wall; my fastest time ever was 44 
seconds, but typically it's around 50.

I know it's not a problem with network or disk; if I use Apache and HTTP 
to transfer the same file from the same server, it transfers in about 15 
to 20 seconds.  Unfortunately, HTTP doesn't meet our other requirements 
for random access to the file.

Do you folks use Samba for large file transfers at all?  Have you had 
any luck speeding it up past about 23MBps (the 44 second transfer 
speed)?  Any help you may have would be fantastic.  Thanks.

-Mark

On Tue, Sep 19, 2006 at 06:19:43PM -0700, Mark Smith
wrote:
> We use SMB to transfer large files (between 1GB and 5GB) from RedHat AS4 
> Content Storage servers to Windows clients with 6 DVD burners and 
> robotic arms and other cool gadgets.  The servers used to be Windows 
> based, but we're migrating to RedHat for a host of reasons.
> 
> Unfortunately, the RedHat Samba servers are about 2.5 times slower than 
> the Windows servers.  Windows will copy a 1GB file in about 30 seconds, 
> where as it takes about 70 to 75 seconds to copy the same file from a 
> RedHat Samba server.
> 
> I've asked Dr. Google and gotten all kinds of suggestions, most of
which
> have already been applied by RedHat to the stock Samba config.  I've 
> opened a ticket with RedHat.  They pointed out a couple errors in my 
> config, but fixing those didn't have any effect.  Some tweaking, 
> however, has gotten the transfer speed to about 50 seconds for that 1GB 
> file.
> 
> But I seem to have hit a brick wall; my fastest time ever was 44 
> seconds, but typically it's around 50.
> 
> I know it's not a problem with network or disk; if I use Apache and
HTTP
> to transfer the same file from the same server, it transfers in about 15 
> to 20 seconds.  Unfortunately, HTTP doesn't meet our other requirements
> for random access to the file.
> 
> Do you folks use Samba for large file transfers at all?  Have you had 
> any luck speeding it up past about 23MBps (the 44 second transfer 
> speed)?  Any help you may have would be fantastic.  Thanks.
An interesting thing you could do is to use a port of smbclient
on Windows (no I don't know where to get one :-) to copy the
client to the Windows client in userspace. smbclient will use
read pipelining (ie. issue more than one read at a time) whereas
Windows clients issue one read, wait for response, issue the next
read, wait for response etc.

That would tell you if it's a client redirector issue. You could
probably use cygwin to compile smbclient.

Jeremy.

I wanted to follow up to my email to provide at least a partial answer 
to my problem.

The stock RedHat AS4-U3 Samba config has SO_SNDBUF and SO_RCVBUF set to 
8k.  With this value, I can transfer a 1GB file in about 70-75 seconds, 
about 14MBps.  If I increase those buffers to their max value of 64k, 
that same 1GB file transfers in 45-50 seconds, about 23MBps.

That is the _ONLY_ configuration value I've found that made any 
difference in my setup.  All the other tweaks I'd done, when removed, 
seemed to make no difference at all.  I was playing with oplocks, 
buffers, max xmit sizes, you name it.  But the socket option buffers was 
the only thing that made a difference.

I'm still looking for more speed.  I'll report if I find anything else 
that helps.

In response to Jeremy's suggestion of using smbclient, I ran a test from 
a Linux client using smbclient and it reported a transfer rate of 
21MBps, about the same as a normal smbfs mount.  I haven't tried porting 
smbclient to Windows yet, and probably won't until we get more info on 
what the server is doing.

Thanks everyone.

-Mark

Mark Smith wrote:
> We use SMB to transfer large files (between 1GB and 5GB) from RedHat AS4 
> Content Storage servers to Windows clients with 6 DVD burners and 
> robotic arms and other cool gadgets.  The servers used to be Windows 
> based, but we're migrating to RedHat for a host of reasons.
> 
> Unfortunately, the RedHat Samba servers are about 2.5 times slower than 
> the Windows servers.  Windows will copy a 1GB file in about 30 seconds, 
> where as it takes about 70 to 75 seconds to copy the same file from a 
> RedHat Samba server.
> 
> I've asked Dr. Google and gotten all kinds of suggestions, most of
which
> have already been applied by RedHat to the stock Samba config.  I've 
> opened a ticket with RedHat.  They pointed out a couple errors in my 
> config, but fixing those didn't have any effect.  Some tweaking, 
> however, has gotten the transfer speed to about 50 seconds for that 1GB 
> file.
> 
> But I seem to have hit a brick wall; my fastest time ever was 44 
> seconds, but typically it's around 50.
> 
> I know it's not a problem with network or disk; if I use Apache and
HTTP
> to transfer the same file from the same server, it transfers in about 15 
> to 20 seconds.  Unfortunately, HTTP doesn't meet our other requirements
> for random access to the file.
> 
> Do you folks use Samba for large file transfers at all?  Have you had 
> any luck speeding it up past about 23MBps (the 44 second transfer 
> speed)?  Any help you may have would be fantastic.  Thanks.
> 
> -Mark

I have not.  Unfortunately, that is not a trivial process.  I believe 
everything supports it, but it's a somewhat major change to my 
production systems.  It might be worth trying as a data point, however.

Given the iPerf tests, I really don't think it's a network bottleneck at
this point.

-Mark

Pitti, Raul wrote:
> i am not an expert, but,
> do you have jumbo frame enabled on your nic and  switch?
> try using ethtools...
> 
> RP
> 
> Mark Smith wrote:
>> Mark Smith wrote:
>>> As a data point, I'm going to try a newer version of Samba. 
(RHEL4
>>> uses 3.0.10-RedHat-Heavily-Modified-Of-Course)  If that makes a 
>>> difference, then I have to decide whether it's worth it to me
to keep
>>> RedHat support or not.  (And when I say "I," I really
mean "my
>>> management.")
>>
>> I've just tried this.  Samba v3.0.23c, built locally from the
Fedora
>> Source RPM as distributed at samba.org, makes no noticeable 
>> difference:  still about 45 seconds.
>>
>>>> I have lots of memory.  I use these configurations in
sysctl.conf to
>>>> up the performance of send/recieve windows on my systems. 
There's
>>>> articles out there.  I don't have historical references
handy.
>>>> YMMV.
>>>> net.core.wmem_max = 1048576
>>>> net.core.rmem_max = 1048576
>>>> net.ipv4.tcp_wmem = 4096 65536 1048575
>>>> net.ipv4.tcp_rmem = 4096 524288 1048575
>>>> net.ipv4.tcp_window_scaling = 1
>>>
>>> I have not tried tweaking the TCP stack in the OS.  I'll give
that a
>>> shot.
>>
>> The person at RedHat who's handling my ticket just suggested these 
>> very changes, without the last one.  They did not help.  The values he 
>> gave were a little different:
>>
>> ----- snip! -----
>> # increase TCP maximum buffer size
>> net.core.rmem_max = 16777216
>> net.core.wmem_max = 16777216
>>
>> # increase Linux autotuning TCP buffer limits
>> # min, default, and maximum number of bytes to use
>> net.ipv4.tcp_rmem = 4096 87380 16777216
>> net.ipv4.tcp_wmem = 4096 65536 16777216
>> ----- snip! -----
>>
>> I also tried your values, with the tcp_window_scaling, with no luck.
>>
>> -Mark
>

i am not an expert, but,
do you have jumbo frame enabled on your nic and  switch?
try using ethtools...

RP

Mark Smith wrote:
> Mark Smith wrote:
>> As a data point, I'm going to try a newer version of Samba.  (RHEL4
>> uses 3.0.10-RedHat-Heavily-Modified-Of-Course)  If that makes a 
>> difference, then I have to decide whether it's worth it to me to
keep
>> RedHat support or not.  (And when I say "I," I really mean
"my
>> management.")
> 
> I've just tried this.  Samba v3.0.23c, built locally from the Fedora 
> Source RPM as distributed at samba.org, makes no noticeable difference: 
>  still about 45 seconds.
> 
>>> I have lots of memory.  I use these configurations in sysctl.conf
to
>>> up the performance of send/recieve windows on my systems. 
There's
>>> articles out there.  I don't have historical references handy.
>>> YMMV.
>>> net.core.wmem_max = 1048576
>>> net.core.rmem_max = 1048576
>>> net.ipv4.tcp_wmem = 4096 65536 1048575
>>> net.ipv4.tcp_rmem = 4096 524288 1048575
>>> net.ipv4.tcp_window_scaling = 1
>>
>> I have not tried tweaking the TCP stack in the OS.  I'll give that
a
>> shot.
> 
> The person at RedHat who's handling my ticket just suggested these very
> changes, without the last one.  They did not help.  The values he gave 
> were a little different:
> 
> ----- snip! -----
> # increase TCP maximum buffer size
> net.core.rmem_max = 16777216
> net.core.wmem_max = 16777216
> 
> # increase Linux autotuning TCP buffer limits
> # min, default, and maximum number of bytes to use
> net.ipv4.tcp_rmem = 4096 87380 16777216
> net.ipv4.tcp_wmem = 4096 65536 16777216
> ----- snip! -----
> 
> I also tried your values, with the tcp_window_scaling, with no luck.
> 
> -Mark
-- 
--------------------
Ra?l Pitt? Palma, Eng.

Global Engineering and Technology S.A.
mobile (507)-6616-0194
office (507)-390-4338
Republic of Panama
www.globaltecsa.com

In the interest of closing this loop for the sake of the archives...  :)

My company has officially thrown in the towel on this issue.  Everything 
we're seeing is suggesting that 22MBps is about as fast as we're going 
to get with SMB.  RedHat, our parent company's IT department (much 
bigger than ours), the (read: this) Samba Mailing List, and all my 
techie friends, all agree that we're at a limit.

For the parts of our process where higher speed is critical, we'll look 
into replacing SMB with a more appropriate protocol (eg: FTP or SCP 
hacked to allow a cipher of "none").

I want to thank everyone who tried to help with suggestions of things to 
try, areas to look, and what-not.  Every time, I'd see something and say 
"Huh.  Haven't tried that!" and got hopeful.  Alas, none of it
ended up
helping, but that's not y'all's fault.  :)

Again, thanks everyone for your help.  Cheers.

-Mark

Mark Smith wrote:
> We use SMB to transfer large files (between 1GB and 5GB) from RedHat AS4 
> Content Storage servers to Windows clients with 6 DVD burners and 
> robotic arms and other cool gadgets.  The servers used to be Windows 
> based, but we're migrating to RedHat for a host of reasons.
> 
> Unfortunately, the RedHat Samba servers are about 2.5 times slower than 
> the Windows servers.  Windows will copy a 1GB file in about 30 seconds, 
> where as it takes about 70 to 75 seconds to copy the same file from a 
> RedHat Samba server.
> 
> I've asked Dr. Google and gotten all kinds of suggestions, most of
which
> have already been applied by RedHat to the stock Samba config.  I've 
> opened a ticket with RedHat.  They pointed out a couple errors in my 
> config, but fixing those didn't have any effect.  Some tweaking, 
> however, has gotten the transfer speed to about 50 seconds for that 1GB 
> file.
> 
> But I seem to have hit a brick wall; my fastest time ever was 44 
> seconds, but typically it's around 50.
> 
> I know it's not a problem with network or disk; if I use Apache and
HTTP
> to transfer the same file from the same server, it transfers in about 15 
> to 20 seconds.  Unfortunately, HTTP doesn't meet our other requirements
> for random access to the file.
> 
> Do you folks use Samba for large file transfers at all?  Have you had 
> any luck speeding it up past about 23MBps (the 44 second transfer 
> speed)?  Any help you may have would be fantastic.  Thanks.
> 
> -Mark