samba - Nov 2006 - Weird Samba upload performance on Gigabit network

Here's a weird one that may have nothing to do with Samba and more to  
do with network frame sizes.

I have recently upgraded the network infrastructure to support  
gigabit speeds.

- OSX Tiger to OSX Tiger file transfers operate at gigabit speeds.
- Samba 3.0.23C (Suse 10) to OSX Tiger file transfers operate at  
gigabit speeds.
- BUT OSX Tiger to Samba 3.0.23C (Suse 10) file transfers operate at  
1-2meg/second which is well down from expected performance.

The strange thing is if I begin a file transfer from OSX Tiger to  
Samba and in a terminal window on OSX begin an SCP copy operation to  
the same Suse server the file transfer speed of Samba jumps up to  
gigabit level speeds. As soon as the SCP copy operation is stopped  
the Samba file transfer process drops to 1-2meg/second again.

My guess is this has something to do with network frame sizes but I  
don't really know.

The gigabit card on the Suse box is a low-end model but the fact that  
downloads and non-Samba related uploads appear to work fine (with low  
CPU usage in all cases) suggests this is more to do with inbound SMB  
packets rather than the hardware itself.

Any hints/pointers would be appreciated.

Regards,


David


P.S. Sorry I thankfully do not have any 'real' Windows machines to  
test performance on.

On 11/9/06, David Harrison <david.harrison@stress-free.co.nz>
wrote:
>
> Here's a weird one that may have nothing to do with Samba and more to
> do with network frame sizes.
>
> I have recently upgraded the network infrastructure to support
> gigabit speeds.
>
> - OSX Tiger to OSX Tiger file transfers operate at gigabit speeds.
> - Samba 3.0.23C (Suse 10) to OSX Tiger file transfers operate at
> gigabit speeds.
> - BUT OSX Tiger to Samba 3.0.23C (Suse 10) file transfers operate at
> 1-2meg/second which is well down from expected performance.
>
> The strange thing is if I begin a file transfer from OSX Tiger to
> Samba and in a terminal window on OSX begin an SCP copy operation to
> the same Suse server the file transfer speed of Samba jumps up to
> gigabit level speeds. As soon as the SCP copy operation is stopped
> the Samba file transfer process drops to 1-2meg/second again.

Looking at your numbers I believe that you have both network and samba
problems.

Can you run netperf or do some nfs testing?

Today I ran it and found out a few of my servers (using nvidia mobo
adapters) although the gigabit light was on the nic and the switch the
adapter was not transfering at gigabit speeds.  I did a nfs test like the
following:

dd if=/dev/zero of=/mnt/remoteserver/test.dat bs=16k count=16k

and got around 10MB/s which is only 100mbit speeds. On netperf I verified
that my maximum transfer rate was around 94mbit/s so I googled a bit and
then ended up upgrading my kernel to 2.6.18 and then did the same test and I
got 109 MB/s on netperf and on the nfs test above I got 43MB/s and after
setting nfs to asynchronous mode I got 57MB/s.

John

On 10/11/2006, at 12:22 PM, John Drescher wrote:
> On 11/9/06, David Harrison <david.harrison@stress-free.co.nz> wrote:
> - BUT OSX Tiger to Samba 3.0.23C (Suse 10) file transfers operate at
> 1-2meg/second which is well down from expected performance.
>
> The strange thing is if I begin a file transfer from OSX Tiger to
> Samba and in a terminal window on OSX begin an SCP copy operation to
> the same Suse server the file transfer speed of Samba jumps up to
> gigabit level speeds. As soon as the SCP copy operation is stopped
> the Samba file transfer process drops to 1-2meg/second again.
>
> Looking at your numbers I believe that you have both network and  
> samba problems.
>
> Can you run netperf or do some nfs testing?
>
> Today I ran it and found out a few of my servers (using nvidia mobo  
> adapters) although the gigabit light was on the nic and the switch  
> the adapter was not transfering at gigabit speeds.  I did a nfs  
> test like the following:
>
> dd if=/dev/zero of=/mnt/remoteserver/test.dat bs=16k count=16k
>
> and got around 10MB/s which is only 100mbit speeds. On netperf I  
> verified that my maximum transfer rate was around 94mbit/s so I  
> googled a bit and then ended up upgrading my kernel to 2.6.18 and  
> then did the same test and I got 109 MB/s on netperf and on the nfs  
> test above I got 43MB/s and after setting nfs to asynchronous mode  
> I got 57MB/s.
>
> John
I've updated the Suse box to the 2.6.18.2 kernel with no effect.
I'm not in a position to run NFS tests on the machine but I'm  
guessing it is the network card (low-end Netgear with the r8169 driver).
To (hopefully) fix the problem for good I've ordered a decent gigabit  
card that by the looks of it has good Linux support.
I'd rather spend the money than continue messing with a card/driver  
that according to Google has some underlying performance issues :-)

Regards,


David

---------- Forwarded message ----------
From: John Drescher <drescherjm@gmail.com>
Date: Nov 10, 2006 8:41 AM
Subject: Re: [Samba] Weird Samba upload performance on Gigabit network
To: David Harrison <david.harrison@stress-free.co.nz>

I've updated the Suse box to the 2.6.18.2 kernel with no
effect.
> I'm not in a position to run NFS tests on the machine but I'm
> guessing it is the network card (low-end Netgear with the r8169 driver).
>
We have several dozen of those cards in our department (we needed around 100
cards so we bought them in bulk) and we have never had that problem.

John


-- 
John M. Drescher

On 11/11/2006, at 6:04 AM, Andrew Morgan wrote:
> On Fri, 10 Nov 2006, David Harrison wrote:
>
>> On 10/11/2006, at 12:22 PM, John Drescher wrote:
>>
>>> On 11/9/06, David Harrison <david.harrison@stress-free.co.nz>
wrote:
>>> - BUT OSX Tiger to Samba 3.0.23C (Suse 10) file transfers operate
at
>>> 1-2meg/second which is well down from expected performance.
>>> The strange thing is if I begin a file transfer from OSX Tiger to
>>> Samba and in a terminal window on OSX begin an SCP copy operation
to
>>> the same Suse server the file transfer speed of Samba jumps up to
>>> gigabit level speeds. As soon as the SCP copy operation is stopped
>>> the Samba file transfer process drops to 1-2meg/second again.
>>> Looking at your numbers I believe that you have both network and  
>>> samba problems.
>>> Can you run netperf or do some nfs testing?
>>> Today I ran it and found out a few of my servers (using nvidia  
>>> mobo adapters) although the gigabit light was on the nic and the  
>>> switch the adapter was not transfering at gigabit speeds.  I did  
>>> a nfs test like the following:
>>> dd if=/dev/zero of=/mnt/remoteserver/test.dat bs=16k count=16k
>>> and got around 10MB/s which is only 100mbit speeds. On netperf I  
>>> verified that my maximum transfer rate was around 94mbit/s so I  
>>> googled a bit and then ended up upgrading my kernel to 2.6.18 and  
>>> then did the same test and I got 109 MB/s on netperf and on the  
>>> nfs test above I got 43MB/s and after setting nfs to asynchronous  
>>> mode I got 57MB/s.
>>> John
>>
>> I've updated the Suse box to the 2.6.18.2 kernel with no effect.
>> I'm not in a position to run NFS tests on the machine but I'm  
>> guessing it is the network card (low-end Netgear with the r8169  
>> driver).
>> To (hopefully) fix the problem for good I've ordered a decent  
>> gigabit card that by the looks of it has good Linux support.
>> I'd rather spend the money than continue messing with a card/ 
>> driver that according to Google has some underlying performance  
>> issues :-)
>
> Have you verified that the duplex settings on the network switch  
> and the Suse server are the same?  If you have half duplex on the  
> Suse server and full duplex on the network switch, you could get  
> the behavior you describe.
>
> 	Andy
No the network checks out, all interfaces are running at 1000baseT  
full duplex.
Network performance in general is good for other services, it seems  
the only issue is with client upload rates to the Samba server.

It would appear to be very similar to this issue in terms of the  
hardware and symptoms:
http://lists.samba.org/archive/samba/2004-February/081668.html

Changing the NIC should resolve the problem and it will give me an  
opportunity to test the card on another system.


David

hey david,

are you by any chance running OSX 10.4.8?
we had the same problem and it was related to samba & OSX (10.4.8). OSX 
to an W2k3-Server was fine. so we thought it might be samba. but after 
downgrading OSX to 10.4.7 everything was fine again. so the apple update 
must have changed/ broken something. we did not look deeper into it. on 
google groups we found one posting with the same problem but between OSX 
and W2k. ftp, etc. was fine on OSX, only smb/ cifs was slow

hope this helps (anybody)
greez

David Harrison wrote:
> Here's a weird one that may have nothing to do with Samba and more to
do
> with network frame sizes.
> 
> I have recently upgraded the network infrastructure to support gigabit 
> speeds.
> 
> - OSX Tiger to OSX Tiger file transfers operate at gigabit speeds.
> - Samba 3.0.23C (Suse 10) to OSX Tiger file transfers operate at gigabit 
> speeds.
> - BUT OSX Tiger to Samba 3.0.23C (Suse 10) file transfers operate at 
> 1-2meg/second which is well down from expected performance.
> 
> The strange thing is if I begin a file transfer from OSX Tiger to Samba 
> and in a terminal window on OSX begin an SCP copy operation to the same 
> Suse server the file transfer speed of Samba jumps up to gigabit level 
> speeds. As soon as the SCP copy operation is stopped the Samba file 
> transfer process drops to 1-2meg/second again.
> 
> My guess is this has something to do with network frame sizes but I 
> don't really know.
> 
> The gigabit card on the Suse box is a low-end model but the fact that 
> downloads and non-Samba related uploads appear to work fine (with low 
> CPU usage in all cases) suggests this is more to do with inbound SMB 
> packets rather than the hardware itself.
> 
> Any hints/pointers would be appreciated.
> 
> Regards,
> 
> 
> David
> 
> 
> P.S. Sorry I thankfully do not have any 'real' Windows machines to
test
> performance on.
> 
> 
> 
> --To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/listinfo/samba
> 
>

On 13/11/2006, at 2:00 AM, Michael Gasch wrote:
> are you by any chance running OSX 10.4.8?
> we had the same problem and it was related to samba & OSX (10.4.8).  
> OSX to an W2k3-Server was fine. so we thought it might be samba.  
> but after downgrading OSX to 10.4.7 everything was fine again. so  
> the apple update must have changed/ broken something. we did not  
> look deeper into it. on google groups we found one posting with the  
> same problem but between OSX and W2k. ftp, etc. was fine on OSX,  
> only smb/ cifs was slow
>
> David Harrison wrote:
>> Here's a weird one that may have nothing to do with Samba and more
>> to do with network frame sizes.
>> I have recently upgraded the network infrastructure to support  
>> gigabit speeds.
>> - OSX Tiger to OSX Tiger file transfers operate at gigabit speeds.
>> - Samba 3.0.23C (Suse 10) to OSX Tiger file transfers operate at  
>> gigabit speeds.
>> - BUT OSX Tiger to Samba 3.0.23C (Suse 10) file transfers operate  
>> at 1-2meg/second which is well down from expected performance.
>> The strange thing is if I begin a file transfer from OSX Tiger to  
>> Samba and in a terminal window on OSX begin an SCP copy operation  
>> to the same Suse server the file transfer speed of Samba jumps up  
>> to gigabit level speeds. As soon as the SCP copy operation is  
>> stopped the Samba file transfer process drops to 1-2meg/second again.
Yes I am running 10.4.8 on the clients.
Just as a test I have mounted the Samba volume from the command line  
(with mount.cifs and mount.smbfs) and received the same performance  
as through the Finder so if the problem is with 10.4.8 its at a  
fairly low level.

My money's still on the gigabit network card on the Samba machine,  
once the new card arrives I'll not only be able to test it with  
10.4.8 but also run some gigabit transfer tests from Linux machine to  
Linux machine which should clearly identify where the problem lies.


David

On 13/11/2006, at 8:23 AM, David Harrison wrote:
>>> Here's a weird one that may have nothing to do with Samba and  
>>> more to do with network frame sizes.
>>> I have recently upgraded the network infrastructure to support  
>>> gigabit speeds.
>>> - OSX Tiger to OSX Tiger file transfers operate at gigabit speeds.
>>> - Samba 3.0.23C (Suse 10) to OSX Tiger file transfers operate at  
>>> gigabit speeds.
>>> - BUT OSX Tiger to Samba 3.0.23C (Suse 10) file transfers operate  
>>> at 1-2meg/second which is well down from expected performance.
>>> The strange thing is if I begin a file transfer from OSX Tiger to  
>>> Samba and in a terminal window on OSX begin an SCP copy operation  
>>> to the same Suse server the file transfer speed of Samba jumps up  
>>> to gigabit level speeds. As soon as the SCP copy operation is  
>>> stopped the Samba file transfer process drops to 1-2meg/second  
>>> again.
>
> Yes I am running 10.4.8 on the clients.
> Just as a test I have mounted the Samba volume from the command  
> line (with mount.cifs and mount.smbfs) and received the same  
> performance as through the Finder so if the problem is with 10.4.8  
> its at a fairly low level.
I replaced the network card on the Suse box from the Netgear card  
(r8169 driver) to a HP NC1020 (tg3 driver).
Samba performance is now at gigabit levels and is very consistent  
with multiple Linux (Suse) and OSX (10.4.8) clients.
The Netgear card appears to operate okay as a Samba client on another  
machine but still performs poorly when the Samba process is receiving  
large quantities of uploaded data.


David