samba - Jul 2007 - What are my smbd's doing ? (was Re: secrets.tdb locking fun!)

Hi all,

	We're still suffering here with odd performance issues on our
Samba 3.0.24 install here on our big chunky Solaris 9 box.


The user perceived issue is slow performance (whilst browsing in Windows
Explorer, or opening files from within applications etc.etc.)

The obvious symptom on the server is a collection of smbd's at the top
of the 'top' listing.  4 or 5 or so taking 25% or so of CPU each.


On one previous occasion, the whole thing seemed to grind to a virtual
halt, and we suspected (but couldn't prove) that a locking battle over
(something like) secrets.tdb was to blame.


gdb backtraces of the processes at the top of 'top' all show the
processes are in a 'select' call looking for all the world as if
they're
just idly waiting for the next SMB event to occur.  (given the the gdb
backtrace is snapshot, I'm not too surprised by this)

Possibly more useful is the 'truss' output we can obtain from these
running processes. (a log of every system call, in effect)

Something that stands out is a huge ammount of time (1.9 - 2.0 seconds)
spent reading a file in /var/tmp  (/var/tmp/firesun1_044_0). This seems
to happen quite often (maybe every few seconds). This file isn't terribly
(human) readable, and I don't recall Samba ever having needed
a temp file outside of 'var/locks'.

The file looks like this:-

-rw-------   1 root     root      165125 Jul 13 09:33 firesun1-044_0  

(firesun1 is the name of the server)

It seems to contain a whole load of odd-looking machine trust account
names (in amongst less readable stuff).


Does anyone know what this file is supposed to be doing?


Could this be the cause of our trouble ?  Why does it take _so_ long to
read this file?  Why is it being read so often?

                               Mac
          Assistant Systems Administrator @nibsc.ac.uk
                           mac@nibsc.ac.uk
   Work: +44 1707 641565          Everything else: +44 7956 237670 (anytime)

Greetings once again,
>	We're still suffering here with odd performance issues on our
>Samba 3.0.24 install here on our big chunky Solaris 9 box.
Still suffering, although we're now running 3.0.25b (krb5 v1.6)

>The user perceived issue is slow performance (whilst browsing in Windows
>Explorer, or opening files from within applications etc.etc.)
Where slow = (apparently) stopped.  Things do complete, but only if huge
patience is applied.

>The obvious symptom on the server is a collection of smbd's at the top
>of the 'top' listing.  4 or 5 or so taking 25% or so of CPU each.
One less obvious symptom is that the number of smbd processes is way in
excess of the number of smb sessions.

ps -ef | grep smbd | wc -l       1440

smbstatus -p | wc -l             101


This is not normal on our system.


Also, at least one client started accumulating TCP connections in the
FIN_WAIT_2 state.  (The connection were to port 445 on the Samba server
in question)
>On one previous occasion, the whole thing seemed to grind to a virtual
>halt, and we suspected (but couldn't prove) that a locking battle over
>(something like) secrets.tdb was to blame.
This still seems to be in there somewhere.  I'm coding tonight an
automated 'gdb bt' of all smbd processes.  We'll run it next time it
happens.


                               Mac
          Assistant Systems Administrator @nibsc.ac.uk
                           mac@nibsc.ac.uk
   Work: +44 1707 641565          Everything else: +44 7956 237670 (anytime)