samba - Apr 2005 - user's smbd process generates high cpu utilization

I'm probably looking for suggestions on tracing
problems rather than a specific fix, since I don't see
any links relevant to my problem.  

We have a Samba PDC with an ldap backend, serving
about 100 active users and another 400 relatively
inactive lab machines that connect once at the
beginning of the day and possibly load a file or two
from a network share.  

We have a problem where occasionally (couple of times
a day to a couple of times a week) a user smbd process
pegs the cpu, causing login failures for other users. 
This is always one of the active users, not a lab
account. Existing sessions proceed smoothly.  Until we
identify the process with top and kill it, the problem
persists.  In attempt to drown the problem in
resources, we recently upgraded the server from a
generally underworked pIII 500/ 1 gig ram /ultra 2
scsi disks to a 2.4 gHz/2 gig server with U320 disks. 
Problem still comes up.  I conclude the box is at
least adequate.  Hell, the previous one was plenty
adequate.  

Some details:
[root@student1 sbin]# rpm -q samba
samba-3.0.13-1
[root@student1 sbin]# rpm -q openldap
openldap-2.2.13-2
[root@student1 sbin]# uname -a
Linux server.redacted.edu 2.6.10-1.770_FC3smp #1 SMP
Thu Feb 24 14:20:06 EST 2005 i686 i686 i386 GNU/Linux


top - 11:05:39 up 2 days, 22:57,  3 users,  load
average: 0.08, 0.06, 0.08
Tasks: 362 total,   1 running, 361 sleeping,   0
stopped,   0 zombie
Cpu(s):  2.0% us,  4.1% sy,  0.0% ni, 91.2% id,  0.0%
wa,  0.5% hi,  2.1% si
Mem:   2074988k total,  2046280k used,    28708k free,
  353724k buffers
Swap:  4096532k total,      200k used,  4096332k free,
 1353236k cached

Here's a top exerpt taken when experiencing the
problem:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM   
TIME+  COMMAND
 6185 jesterj   22   0 17180 5008 3912 R 99.6  0.2 
17:49.13 smbd
[root@student1 /var]# pstree jesterj
smbd


I omit the config files, because the setup works, most
of the time.  It's not a basic setup problem.  I'd be
happy to provide the configs if they'll help. 

It's possible that the smbd process is actually
waiting for ldap to do something; we have a tool for
account creation that writes directly to ldap (ie, not
involved in samba).  That is affected by the slowdown,
too.  This could be explained by smbd bogging the cpu
down in general, or it could indicate that a problem
with ldap is causing a symptom in smbd.  

If nobody has seen the same thing, I'd appreciate some
pointers in debugging this.  If we turn on significant
logging on the ldap server, things slow to a crawl. 
We can do it off peak hours, though.  

I turned oplocks off because there were some oplock
break failures in a log, but that didn't fix it and I
can't confirm there was any relationship.  The user
may have been on a separate session entirely.  

So to recap:  I need to dig information out of a
running process smbd, which does not have any children
spawned, and find out what it is doing/trying to do.

Thanks for any pointers.



		
__________________________________ 
Do you Yahoo!? 
Read only the mail you want - Yahoo! Mail SpamGuard. 
http://promotions.yahoo.com/new_mail

On Wed, Apr 06, 2005 at 03:50:27PM -0700, James Affeld
wrote:
> 
> So to recap:  I need to dig information out of a
> running process smbd, which does not have any children
> spawned, and find out what it is doing/trying to do.
> 
> Thanks for any pointers.
I'm attaching a patch for a bug in 3.0.13 than can cause a 
spinning smbd (it's already in svn). But to check if this is the
cause you can use strace -p <pid>.

Jeremy.
-------------- next part --------------
--- smbd/dir.c	2005-03-23 12:30:27.000000000 -0800
+++ /home/jeremy/src/samba3.0/source/smbd/dir.c	2005-04-06 18:57:46.157484483
-0700
@@ -602,7 +603,7 @@
 		return False;
 	}
 
-	while (SearchDir(dptr->dir_hnd, name, poffset) == True) {
+	if (SearchDir(dptr->dir_hnd, name, poffset)) {
 		if (is_visible_file(dptr->conn, dptr->path, name, pst, True)) {
 			return True;
 		}

> We have a problem where occasionally (couple of times
> a day to a couple of times a week) a user smbd process
> pegs the cpu, causing login failures for other users.
If you dig a little further back in the Samba archives you 
may find a thread of mine that sounds suspiciously like
your itch. The cause of the problem (high cpu usage for
some users) was an inconsistency in the passwd/group
a slightly wrong spelling of a user name, IIRC. The thread 
ends with some shell and awk scripts to help check the files 
for possible inconcistencies. Perhaps it is not relevant 
for you since you are using ldap instead of files,
but who knows?