samba - Jun 2005 - Tracking down a Samba problem under load

We use Samba as an interop file sharing solution for the ClearCase version 
control system.  Unfortunately we have run into a problem after a recent 
upgrade to RedHat Enterprise 3.  This problem occurs both with the RedHat 
included 3.0.9-1.3E.3, and a custom compile of the latest 3.0.14a direct 
from Samba.

The symptoms are: under high load, such as during a build, new samba 
processes will spawn, but not respond.  A ps list shows many smbd 
processes active, but no new connections can be established.  Older 
connections seem to still function ok.  Shutting down samba via the 
standard RedHat method "service smb stop" only kills the older
processes,
and the new non-responding ones must be kill -9'd to get them to 
terminate.  Base level logging tells me nothing, and turning up the debug 
level just gets more of the same, but I'm not extremely well versed in 
what I'd be looking for to isolate the problem.  I have seen a few
panic's
in the log, usually one or two per day, but the not responding problem 
doesn't seem to coincide with any of those.

My basic problem is, I don't have any way of figuring out where the 
problem lies, and what to submit as a bug.  Any pointers in that direction 
would be very helpful.

Just for reference, here is the smb.conf, basically our only change from 
default is to turn off oplocks, as instructed by the ClearCase 
documentation:

[global]
        log level = 1
        log file = /var/log/samba/smbd.log
        max log size = 0
        password server = *
        server string = ClearCase VOB Server
        workgroup = SEL
        username map = /etc/samba/smbusers
        socket options = TCP_NODELAY SO_RCVBUF=8192 SO_SNDBUF=8192
        load printers = No
        wins server = x.x.x.x
        security = DOMAIN
        kernel oplocks = No
        create mask = 0775
        directory mask = 0775

[array]
        comment = VOB Storage
        path = /array
        read only = No
        oplocks = No
        level2 oplocks = No

--------------------------------------------
Brian Pickering - Brian_Pickering@selinc.com
System Administrator - Information Services
Schweitzer Engineering Laboratories, Inc.

tir, 21.06.2005 kl. 21.12 skrev Brian_Pickering@selinc.com:
> We use Samba as an interop file sharing solution for the ClearCase version 
> control system.  Unfortunately we have run into a problem after a recent 
> upgrade to RedHat Enterprise 3.  This problem occurs both with the RedHat 
> included 3.0.9-1.3E.3, and a custom compile of the latest 3.0.14a direct 
> from Samba.
> 
> The symptoms are: under high load, such as during a build, new samba 
> processes will spawn, but not respond.  A ps list shows many smbd 
> processes active, but no new connections can be established.  Older 
> connections seem to still function ok.  Shutting down samba via the 
> standard RedHat method "service smb stop" only kills the older
processes,
> and the new non-responding ones must be kill -9'd to get them to 
> terminate.  Base level logging tells me nothing, and turning up the debug 
> level just gets more of the same, but I'm not extremely well versed in 
> what I'd be looking for to isolate the problem.  I have seen a few
panic's
> in the log, usually one or two per day, but the not responding problem 
> doesn't seem to coincide with any of those.
> 
> My basic problem is, I don't have any way of figuring out where the 
> problem lies, and what to submit as a bug.  Any pointers in that direction 
> would be very helpful.
Did you install binary rpms? If so, try to build and install the 3.0.14a
srpm (source code in the BUILD directory should be patched with the
3.0.14a anti-spin patch).

The reason I write this is, that we have no problems with the srpm on a
moderately loaded (30-40 w2k workstations) RHAS3 server.

--Tonni

-- 
mail: tonye@billy.demon.nl
http://www.billy.demon.nl