samba - Oct 2007 - Unusable performance over WAN (part 2)

Hi all,
Disregard my previous posts, I've consolidated everything here.
I'm having terrible performance issues with samba over a WAN
(point-to-point T1 link).
Doing a copy of a 2MB file from a samba server to a linux client
running smbclient takes over 5 minutes.
SCPing the same file takes seconds.

The server is running samba version 3.0.25c with kernel 2.6.16.18.

I've put up a set of debugging logs at:
http://emagiccards.com/james/sambalogs.tar.bz2

Inside are 3 files:
smb.conf - the configuration of the samba server
log.agard - the level 10 debug log of the copy from samba
samba-tcpdump.log - a tcpdump log from the client side of the copy

Any help to fix this issue would be greatly appreciated since the file
server is pretty unusuable over the WAN.
If you need any more information, please let me know. It is imperative
that I find out what's happening here.

Thanks.

-- James Lamanna

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

James Lamanna wrote:
> Hi all,
> Disregard my previous posts, I've consolidated everything here.
> I'm having terrible performance issues with samba over a WAN
> (point-to-point T1 link).
> Doing a copy of a 2MB file from a samba server to a linux client
> running smbclient takes over 5 minutes.
> SCPing the same file takes seconds.
> 
> The server is running samba version 3.0.25c with kernel 2.6.16.18.
> 
> I've put up a set of debugging logs at:
> http://emagiccards.com/james/sambalogs.tar.bz2
> 
> Inside are 3 files:
> smb.conf - the configuration of the samba server
> log.agard - the level 10 debug log of the copy from samba
> samba-tcpdump.log - a tcpdump log from the client side of the copy
> 
> Any help to fix this issue would be greatly appreciated since the file
> server is pretty unusuable over the WAN.
> If you need any more information, please let me know. It is imperative
> that I find out what's happening here.
> 
Well, there's always paid support.  See the samba web site.

testparm yeilds 1 error and 1 warning.
Unknown parameter encountered: "show preserve case"
Ignoring unknown parameter "show preserve case"

should probably be "short preserve case"

Server's Role (logon server) NOT ADVISED with domain-level security
Loaded services file OK.
Server role: ROLE_DOMAIN_BDC

When I specifically want samba to use an IP address, I use the IP
address in the interfaces clause.  eth1 can change when replacing
network cards.

I've used cifs over WAN a lot over the years.  It is slower than ftp and
scp but there shouldn't be breaks waiting for ACK.  You'll not want to
hear it, but this is almost always a network issue; card, router,
switch, WAN link box, etc.

I like larger packets for most of the uses of a server.  So I add to
"socket options" SO_SNDBUF=65536 SO_RCVBUF=65536

Regards, Doug

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iD8DBQFHCAU4FqWysr/jOHMRAs5KAKCiRBH5t8Ke5QU0U9sXQ0+mtl8s7ACfa0ce
V2/foUb+PpUUiZ/YModZFFQ=AIJ8
-----END PGP SIGNATURE-----

On 10/6/07, James Lamanna <jlamanna@gmail.com>
wrote:
> Hi all,
> Disregard my previous posts, I've consolidated everything here.
> I'm having terrible performance issues with samba over a WAN
> (point-to-point T1 link).
> Doing a copy of a 2MB file from a samba server to a linux client
> running smbclient takes over 5 minutes.
> SCPing the same file takes seconds.
>
> The server is running samba version 3.0.25c with kernel 2.6.16.18.
>
> I've put up a set of debugging logs at:
> http://emagiccards.com/james/sambalogs.tar.bz2
>
> Inside are 3 files:
> smb.conf - the configuration of the samba server
> log.agard - the level 10 debug log of the copy from samba
> samba-tcpdump.log - a tcpdump log from the client side of the copy
>
> Any help to fix this issue would be greatly appreciated since the file
> server is pretty unusuable over the WAN.
> If you need any more information, please let me know. It is imperative
> that I find out what's happening here.
>
> Thanks.
>
> -- James Lamanna
>
One of the other things that I've noticed is that the pattern of Data
/ Acks is different based on where I connect from.

If I connect on the server side of the T1 link, the data / ack pattern
seems to be:

Server sends 1500 byte packet
Server sends 1500 byte packet
Server sends 1500 byte packet
etc...(to 64k)
Client sends 52 byte ACK
Client sends 52 byte ACK
Client sends 52 byte ACK
etc...

If I connect from the other side of the T1, the data and acks are interleaved:

Server sends 1500 byte packet
Client sends 52 bye ACK
Server sends 1500 byte packet
Client sends 52 byte ACK
etc..

Can anyone think of a reason for this?

Thanks.

-- James

On Sun, Oct 07, 2007 at 09:31:23AM -0700, James Lamanna wrote:
> Server sends 1500 byte packet
> Client sends 52 bye ACK
> Server sends 1500 byte packet
> Client sends 52 byte ACK
> etc..
> 
> Can anyone think of a reason for this?
I did not find a link spontaneously, but Windows sometimes
falls back to something that we call "rabbit pellet"
mode. Maybe google shows up something for you.

Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url :
http://lists.samba.org/archive/samba/attachments/20071007/fb89e054/attachment.bin

On 10/7/07, Volker Lendecke <Volker.Lendecke@sernet.de>
wrote:
> On Sun, Oct 07, 2007 at 09:31:23AM -0700, James Lamanna wrote:
>
> > Server sends 1500 byte packet
> > Client sends 52 bye ACK
> > Server sends 1500 byte packet
> > Client sends 52 byte ACK
> > etc..
> >
> > Can anyone think of a reason for this?
>
> I did not find a link spontaneously, but Windows sometimes
> falls back to something that we call "rabbit pellet"
> mode. Maybe google shows up something for you.
>
> Volker
>
>
I actually see that behavior using smbclient from a linux machine, so
its not necessarily Windows related.

-- James

James Lamanna wrote:
> On 10/7/07, Volker Lendecke <Volker.Lendecke@sernet.de> wrote:
>> On Sun, Oct 07, 2007 at 09:31:23AM -0700, James Lamanna wrote:
>>
>>> Server sends 1500 byte packet
>>> Client sends 52 bye ACK
>>> Server sends 1500 byte packet
>>> Client sends 52 byte ACK
>>> etc..
>>>
>>> Can anyone think of a reason for this?
>> I did not find a link spontaneously, but Windows sometimes
>> falls back to something that we call "rabbit pellet"
>> mode. Maybe google shows up something for you.
>>
>> Volker
>>
>>
> 
> I actually see that behavior using smbclient from a linux machine, so
> its not necessarily Windows related.
> 
> -- James
James, Volker,

See:

http://www.linuxarkivet.se/mlists/mandrake-expert/0309/msg00162.html


-- 
David C. Rankin, J.D., P.E.
Rankin Law Firm, PLLC
510 Ochiltree Street
Nacogdoches, Texas 75961
(936) 715-9333
(936) 715-9339 fax
www.rankinlawfirm.com

On 10/8/07, Mike Eggleston <mikeegg1@mac.com>
wrote:
> On Mon, 08 Oct 2007, James Lamanna might have said:
>
> > So as it turns out, apparently it was a window scaling issue.
> > Turning on an excessively large window size on the routers (thereby
> > enabling dynamic TCP window scaling) seems to have fixed the issue. I
> > now get transfer rates around 130-160k/s.
>
> Great. For hysterical porpoises please document what specific changes
> you made on the windows boxes and what specific changes you made on
> your router.
>
> Mike
>
The only change I made on the routers was I added the global
configuration command (both Cisco routers btw)
ip tcp window-size 750000

-- James

On Mon, 08 Oct 2007, James Lamanna might have said:
> On 10/8/07, Mike Eggleston <mikeegg1@mac.com> wrote:
> > On Mon, 08 Oct 2007, James Lamanna might have said:
> >
> > > So as it turns out, apparently it was a window scaling issue.
> > > Turning on an excessively large window size on the routers
(thereby
> > > enabling dynamic TCP window scaling) seems to have fixed the
issue. I
> > > now get transfer rates around 130-160k/s.
> >
> > Great. For hysterical porpoises please document what specific changes
> > you made on the windows boxes and what specific changes you made on
> > your router.
> >
> > Mike
> >
> 
> The only change I made on the routers was I added the global
> configuration command (both Cisco routers btw)
> ip tcp window-size 750000
> 
> -- James
And the change on the windows box?

Mike

On 10/8/07, Mike Eggleston <mikeegg1@mac.com>
wrote:
> On Mon, 08 Oct 2007, James Lamanna might have said:
>
> > On 10/8/07, Mike Eggleston <mikeegg1@mac.com> wrote:
> > > On Mon, 08 Oct 2007, James Lamanna might have said:
> > >
> > > > So as it turns out, apparently it was a window scaling
issue.
> > > > Turning on an excessively large window size on the routers
(thereby
> > > > enabling dynamic TCP window scaling) seems to have fixed the
issue. I
> > > > now get transfer rates around 130-160k/s.
> > >
> > > Great. For hysterical porpoises please document what specific
changes
> > > you made on the windows boxes and what specific changes you made
on
> > > your router.
> > >
> > > Mike
> > >
> >
> > The only change I made on the routers was I added the global
> > configuration command (both Cisco routers btw)
> > ip tcp window-size 750000
> >
> > -- James
>
> And the change on the windows box?
>
> Mike
>
Actually no changes to the windows box.
They seem to behaving well now.
Most of my tests were done with smbclient from a linux box anyways.

-- James

James Lamanna wrote:
> On 10/8/07, Mike Eggleston <mikeegg1@mac.com> wrote:
>> On Mon, 08 Oct 2007, James Lamanna might have said:
>>
>>> So as it turns out, apparently it was a window scaling issue.
>>> Turning on an excessively large window size on the routers (thereby
>>> enabling dynamic TCP window scaling) seems to have fixed the issue.
I
>>> now get transfer rates around 130-160k/s.
>> Great. For hysterical porpoises please document what specific changes
>> you made on the windows boxes and what specific changes you made on
>> your router.
>>
>> Mike
>>
> 
> The only change I made on the routers was I added the global
> configuration command (both Cisco routers btw)
> ip tcp window-size 750000
> 
> -- James
Is 750000 a good value.

My router says the valid range is 0-65535.

Stu