samba - Jun 2002 - nmbd causing very high CPU utilization

Does anyone have any idea what can cause nmbd to begin gobbling CPU?
We are running Samba 2.2.4 on HP-UX 10.20, having upgraded from Samba
2.0.10 about a month ago.  Our clients are a mixture of NT 4.0 and Win2K.
Today, we have begun to see widespread client drive disconnections, and
nmbd is averaging 60%+ CPU usage with peaks up to 100%.  We have heard
reports of network problems at our site affecting clients that do not 
connect to my HP/Samba servers, so I'm wondering if the high CPU usage 
is a reflection of nmbd trying to cope with problems or a
symptom that they might be causing them instead.  There seems to be nothing
out of the ordinary in the nmbd log, but my client smbd logs are full of the
infamous "Connection reset by peer" messages.  I have my site IS group
investigating this (I'm in an engineering group, but we normally provide
production IS support ourselves except for site-wide network issues), but
have not heard from them yet.

This particular server supports production at my manufacturing site, so
I cannot easily shut it down.

Any ideas.  Any help is greatly appreciated!

Best Regards,

Dave Windsor
AdW/MOE2.12
Team Leader, Test Systems Engineering
Robert Bosch Corporation
Anderson, SC, USA
Voice: (864) 260-8459
Fax:   (864) 260-8142
email: mailto:Dave.Windsor@us.bosch.com

On Wed, Jun 26, 2002 at 03:51:54PM -0400, Windsor Dave (AdW/MOE2.1)
wrote:
> Does anyone have any idea what can cause nmbd to begin gobbling CPU?
> We are running Samba 2.2.4 on HP-UX 10.20, having upgraded from Samba
> 2.0.10 about a month ago.  Our clients are a mixture of NT 4.0 and Win2K.
> Today, we have begun to see widespread client drive disconnections, and
> nmbd is averaging 60%+ CPU usage with peaks up to 100%.  We have heard
> reports of network problems at our site affecting clients that do not 
> connect to my HP/Samba servers, so I'm wondering if the high CPU usage 
> is a reflection of nmbd trying to cope with problems or a
> symptom that they might be causing them instead.  There seems to be nothing
> out of the ordinary in the nmbd log, but my client smbd logs are full of
the
> infamous "Connection reset by peer" messages.  I have my site IS
group
> investigating this (I'm in an engineering group, but we normally
provide
> production IS support ourselves except for site-wide network issues), but
> have not heard from them yet.
> 
> This particular server supports production at my manufacturing site, so
> I cannot easily shut it down.
Can you do the equivalent of an strace or truss onnmbd under HPUX 10.x ?
If so that may tell you what it is doing. Is the unexpected packet
database in unexpected.tdb growing ?

Jeremy.

On HP-UX 11, tusc is the truss equivilent, but on HP-UX 10.20,
trace corresponds to truss.  I'm in the process of trying to
get trace working - I keep getting bad address errors, no
matter what process I attempt to trace.  I've never tried to
use strace - on my system the man page describes it as
"strace - write STREAMS event trace messages to standard output".
Perhaps not the same thing?

As for the unexpected.tdb, the size is increasing - it will go for
long periods where the size doesn't increase but the modification
time does, then the size will increase.  It is currently at 
163840 bytes, if that's meaningful.

Best Regards,

Dave Windsor
AdW/MOE2.12
Team Leader, Test Systems Engineering
Robert Bosch Corporation
Anderson, SC, USA
Voice: (864) 260-8459
Fax:   (864) 260-8142
email: mailto:Dave.Windsor@us.bosch.com



-----Original Message-----
From: jra@samba.org [mailto:jra@samba.org]
Sent: Wednesday, June 26, 2002 4:18 PM
To: Windsor Dave (AdW/MOE2.1)
Cc: 'samba@samba.org'
Subject: Re: [Samba] nmbd causing very high CPU utilization


On Wed, Jun 26, 2002 at 03:51:54PM -0400, Windsor Dave (AdW/MOE2.1)
wrote:
> Does anyone have any idea what can cause nmbd to begin gobbling CPU?
> We are running Samba 2.2.4 on HP-UX 10.20, having upgraded from Samba
> 2.0.10 about a month ago.  Our clients are a mixture of NT 4.0 and Win2K.
> Today, we have begun to see widespread client drive disconnections, and
> nmbd is averaging 60%+ CPU usage with peaks up to 100%.  We have heard
> reports of network problems at our site affecting clients that do not 
> connect to my HP/Samba servers, so I'm wondering if the high CPU usage 
> is a reflection of nmbd trying to cope with problems or a
> symptom that they might be causing them instead.  There seems to be
nothing
> out of the ordinary in the nmbd log, but my client smbd logs are full of
the
> infamous "Connection reset by peer" messages.  I have my site IS
group
> investigating this (I'm in an engineering group, but we normally
provide
> production IS support ourselves except for site-wide network issues), but
> have not heard from them yet.
> 
> This particular server supports production at my manufacturing site, so
> I cannot easily shut it down.
Can you do the equivalent of an strace or truss onnmbd under HPUX 10.x ?
If so that may tell you what it is doing. Is the unexpected packet
database in unexpected.tdb growing ?

Jeremy.

On Wed, Jun 26, 2002 at 06:11:10PM -0400, Windsor Dave (AdW/MOE2.1)
wrote:
> OK, attached is the output of the trace command on nmbd.  I'm not sure
> how helpful it will be, but there are quite a few lseeks and reads
> taking place.
> 
> Summary of software versions:
> Samba 2.2.4
> HP-UX 10.20
> trace 1.6
> 
> Thanks and Best Regards,
Ok, the lseek/read pairs are searching within a tdb for free
space on a system that doesn't support mmap correctly.

My guess is that you are being hit with a lot of unexpected
udp packets on port 137, you may want to look into what is
causing this on your network.

To make the tdb space allocation more efficient you might
want to increase the hash size in libsmb/unexpected.c by
using the following patch :

Hope this helps,

	Jeremy.

Index: libsmb/unexpected.c
==================================================================RCS file:
/data/cvs/samba/source/libsmb/unexpected.c,v
retrieving revision 1.4.8.6
diff -u -r1.4.8.6 unexpected.c
--- libsmb/unexpected.c 2002/01/23 03:15:04     1.4.8.6
+++ libsmb/unexpected.c 2002/06/27 00:09:47
@@ -47,7 +47,7 @@
        int len=0;

        if (!tdbd) {
-               tdbd = tdb_open_log(lock_path("unexpected.tdb"), 1,
+               tdbd = tdb_open_log(lock_path("unexpected.tdb"), 1024,
                               TDB_CLEAR_IF_FIRST|TDB_DEFAULT,
                               O_RDWR | O_CREAT, 0644);
                if (!tdbd) {

Jeremy,

The patch has proven very helpful!  It has reduced the
nmbd CPU usage by approximately one third or more.  We still 
haven't found the cause of our problem, but by temporarily
isolating our production network from our site backbone, we
have proven that the packets are coming from somewhere else
in our network.  We are currently looking closely at the
configurations of all our Cisco switches.

I'll provide an update when we discover the origin of the udp
packets, for anyone that is interested.

Thanks to you and the entire Samba Team for providing such a
great product and for taking such an interest in solving
problems such as this!

Dave Windsor
AdW/MOE2.12
Team Leader, Test Systems Engineering
Robert Bosch Corporation
Anderson, SC, USA
Voice: (864) 260-8459
Fax:   (864) 260-8142
email: mailto:Dave.Windsor@us.bosch.com



-----Original Message-----
From: jra@samba.org [mailto:jra@samba.org]
Sent: Wednesday, June 26, 2002 8:10 PM
To: Windsor Dave (AdW/MOE2.1)
Cc: 'jra@samba.org'; 'samba@samba.org'
Subject: Re: [Samba] nmbd causing very high CPU utilization


On Wed, Jun 26, 2002 at 06:11:10PM -0400, Windsor Dave (AdW/MOE2.1)
wrote:
> OK, attached is the output of the trace command on nmbd.  I'm not sure
> how helpful it will be, but there are quite a few lseeks and reads
> taking place.
> 
> Summary of software versions:
> Samba 2.2.4
> HP-UX 10.20
> trace 1.6
> 
> Thanks and Best Regards,
Ok, the lseek/read pairs are searching within a tdb for free
space on a system that doesn't support mmap correctly.

My guess is that you are being hit with a lot of unexpected
udp packets on port 137, you may want to look into what is
causing this on your network.

To make the tdb space allocation more efficient you might
want to increase the hash size in libsmb/unexpected.c by
using the following patch :

Hope this helps,

	Jeremy.

Index: libsmb/unexpected.c
==================================================================RCS file:
/data/cvs/samba/source/libsmb/unexpected.c,v
retrieving revision 1.4.8.6
diff -u -r1.4.8.6 unexpected.c
--- libsmb/unexpected.c 2002/01/23 03:15:04     1.4.8.6
+++ libsmb/unexpected.c 2002/06/27 00:09:47
@@ -47,7 +47,7 @@
        int len=0;

        if (!tdbd) {
-               tdbd = tdb_open_log(lock_path("unexpected.tdb"), 1,
+               tdbd = tdb_open_log(lock_path("unexpected.tdb"), 1024,
                               TDB_CLEAR_IF_FIRST|TDB_DEFAULT,
                               O_RDWR | O_CREAT, 0644);
                if (!tdbd) {