samba - Apr 2010 - Samba 3.0.22 - slow performance - Really urgent help

Hi Samba world,

Have been struggling with this for the past 10 days, we are running Samba 
3.0.22 on VCS zone, we have end users saving files
onto Samba mapped drive, and complained that it 5mins to save 300 files, 
now it takes 30 mins.  There is recently a change in the
topology.

Before

        Enduser ----------- samba mapped -------------- server (local 
attached storage)
                                                      physical
                                                      Solaris 8
Now
 
        Enduser-------------samba mapped ------------  server 
------------------------------ SFCFS/HA
                                                     virtual VCS NFS 
cluster
                                                     Solaris 10 
 

        We have reverted the NFS from version 4 to 3

        Fine tune Samba, wb and rb are at 64k

        Is there in samba log somewhere to tell me why is the bottleneck ?
 
        If I move the Samba server which is less loaded in Memory 
consumption , will it help.
        Is the above degrading expected ?
Regards
Alan

---

This e-mail may contain confidential and/or privileged information. If you 
are not the intended recipient (or have received this e-mail in error) 
please notify the sender immediately and destroy this e-mail. Any 
unauthorized copying, disclosure or distribution of the material in this 
e-mail is strictly forbidden.

On Thu, Apr 01, 2010 at 05:24:26PM +0800, Alan Pek
wrote:
> Have been struggling with this for the past 10 days, we are running Samba 
> 3.0.22 on VCS zone, we have end users saving files
> onto Samba mapped drive, and complained that it 5mins to save 300 files, 
> now it takes 30 mins.  There is recently a change in the
> topology.
> 
> Before
> 
>         Enduser ----------- samba mapped -------------- server (local 
> attached storage)
>                                                       physical
>                                                       Solaris 8
> Now
>  
>         Enduser-------------samba mapped ------------  server 
> ------------------------------ SFCFS/HA
>                                                      virtual VCS NFS 
> cluster
>                                                      Solaris 10 
>  
> 
>         We have reverted the NFS from version 4 to 3
> 
>         Fine tune Samba, wb and rb are at 64k
> 
>         Is there in samba log somewhere to tell me why is the bottleneck ?
>  
>         If I move the Samba server which is less loaded in Memory 
> consumption , will it help.
>         Is the above degrading expected ?
Some slow-down is expected if you re-export NFS because the
data will travel over the network twice instead of once. A
factor of 6 is a bit more than is expected though. To see
where the bottleneck is, it might be interesting to look at
simultaneous network traces for the SMB and NFS side of your
Samba server to see what NFS requests are triggered by what
SMB requests. Doing a truss with high-resolution timestamps
on the smbd processes might also give hints. Next would be
to look at "top" (pstat?) if there is a local CPU bottleneck
on the Samba server. There's just many things that can go
slow, and in a moderatly complex environment (HA clusters
fall into this category), this can be an interesting hunt
:-)

Volker
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL:
<http://lists.samba.org/pipermail/samba/attachments/20100401/c5d38953/attachment.pgp>

Hi,

if you do 'truss -f -p <smbd-PID>' do you see a lot of:

19702:  fcntl(17, F_SETLKW64, 0xFFBFE640)               = 0

?

If yes this is a problem that is present in most of the recent samba versions 
(tried 5 or 6 versions off the different branches) it's a solaris related
bug
but that's all I can say. It might be fixed in 3.5.1 ...

cheers
christoph


On Thu, 1 Apr 2010, Volker Lendecke wrote:
>  On Thu, Apr 01, 2010 at 06:22:34PM +0800, Alan Pek wrote:
> >  a) Though the CPU utilisation is not high but there memory used and
> >  swapped are very high. This will slow it down further.
> > 
> >  b) Think every samba connection will consume a considerable amout of
> >  memory as well.
>
>  The rough estimate here is that Samba should have 2-3MB real
>  RAM per client. If your smbds use a lot more, you might be
>  hitting a memleak that might be fixed in later versions.
>  3.0.22 is very old, I would really recommend looking at
>  3.4.7 or 3.5.2 (next week).
> 
> >  c) We did a test, by saving a huge number of files onto local drive
(on
> >  enduser PC) and the copy into the mapped drives This is much faster,
> >  probably by factor 6 or more, then using Excel marco
> >  and writing directly into the samba share over NFS.
>
>  Do you have oplocks disabled? This would severely impact
>  performance. If you don't have concurrent NFS clients
>  accessing the same data, you might also want to try
>
>  posix locking = no
> 
> >  d) Looking at the system, and samba processes,  how should I do a
truss
> >  with high-resolution timestamps on the smbd processes ? Every smbd
process
> >  ?
> >  Just do :
> > 
> >          truss -p 20995 without any option ?
>
>  Not sure how truss really works, in Linux you would do a
>
>  strace -ttT -p 20995 -o smbd.out
>
>  You pick a single smbd with "smbstatus" and run your test.
>  The truss output will need some interpretation though.
> 
> >  e)  Network tracing , meaning application profiling by putting a
sniffer,
> >  or/and snoop on both samba and nfs side ?
>
>  Yes. Best done on the samba server box itself.
>
>  BTW, before you send stuff to the list -- both the truss and
>  the network sniffer output will probably contain sensitive
>  data.
>
>  Volker
>
best regards
 	~ christoph


-- 
/*   Christoph Beyer     |   Office: Building 2b / 23     *\
   *   DESY                |    Phone: 040-8998-2317        *
   *   - IT -              |      Fax: 040-8998-4060        *
\*   22603 Hamburg       |     http://www.desy.de         */