samba - May 2003 - Performance Issues With Samba and RedHat 8.

Appologies if this has been asked before, i have searched the archives and 
while there are some almost related issues, none really answer my 
question/problem.

Recently (about 4 months ago) we purchased a new Dell 2600 (2.0Ghz Xeon 
2GB Ram), Machine (single cpu) and migrated the data and system from rh 
6.2 to 8 on this new server. By migrate i mean Installed rh8 fresh and 
moved the user data and .conf's, cron etc.

All was well and fine until one afternoon we had a massive slow down in 
the "system" (overall system, network etc. not this actual server).
upon
investigation we noticed that smbd was pulling 99% cpu time for about 
15-20 mins (before we decided to restart smbd)

restarting smbd, we had a small issue where all the old links etc tryed to 
contact the server so a little slow for a few mins then everything went 
back to normal. Very odd and certainly something we had not previously 
experienced on our older server.

so we kept an eye on it for a few weeks and all seemed well. we had a few 
reports of the system being incredibly slow at times, although this did 
not coincide with smbd using 99% cpu. using IPtraf/ethereal/ping showed 
that there wasnt a network issue to or from the server. 

Looking at the samba log file we started to notice a lot more OPLOCK 
requests/bad/failures etc around about the same time everything was going 
slow. So on a few of the shares we made them read only and put on FAKE 
OPLOCKS (netlogon, start menus etc). And we changed level2 oplocks to no. 

 # Turn off level 2 oplocks
        level2 oplocks = no

This seemed to improve things until it went slow/99% usage once again. 
(about 2 week or so later) Hair ripping time once again.

So after more research I read an article about the smbd.conf and some 
sort of windowing problem with samba and windows clients. so i modified 
"socket options" to be....

# Most people will find that this option gives better performance.
  # See speed.txt and the manual pages for details
  # You may want to add the following on a Linux system:
  #         SO_RCVBUF=8192 SO_SNDBUF=8192
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_SNDBUF=8192 SO_RCVBUF=8192

Now this seemed to work and all was well until the past 3 weeks, where we 
have had to restart the server twice (no console response) and have had 
numerous problems with 99% cpu on smbd or this incredible slowdown of 
loging on, opening files, listing dir's etc.

Other details about samba is that its version 2.2.7a (according to 
smbstatus), we have nt4 clients and on average have about 100-150 users 
at peak times. it also has the wins server option enabled (so that it acts 
like a wins server)

I have no other ideas to try or even think to search for, so any advise, 
questions and maybe answers would be gratefully received.

Faithfully

Ross McInnes

Ross:

Hi!
> Appologies if this has been asked before, i have searched the 
> archives and 
> while there are some almost related issues, none really answer my 
> question/problem.
> 
> Recently (about 4 months ago) we purchased a new Dell 2600 
> (2.0Ghz Xeon 2GB Ram), Machine
Only thing I have to input here is:  Does your server have the
"Broadcom"
NIC(s) in it?  I've seen horror stories about this card on the Dell
PowerEdge on Linux mailing list.  If you do, I would recommend that you
check out this link: http://lists.us.dell.com and search for "tg3" and
or
"broadcom" to see the troubles reported there.  Most of the problems
with
the Broadcoms actually end up leading to complete server lockups though.  I
just thought it might be worth mentioning.

Dell has since moved to the Intel brand for integrated NICs in the 2600 -
I've got one on the way right now...where is UPS anyway?  :-)

BTW, that mailing list is a VERY good source for anyone running Linux on
Dell PowerEdge servers.  There are several Dell people moderating the list
and there is always a good "flurry" of information from knowledgeable
people
being passed around.

Hope I helped a little,

Kevin L. Collins, MCSE
Systems Manager
Nesbitt Engineering, Inc.

> Ive just pulled my system service tag off and looked on the 
> dell website
> armed with that tag and it only lists Intel Drivers for the NIC. can i
> safely assume that i have an intel based nic in mine? i dont 
> really want
> to down server/pull it apart to find out for sure.
Generally speaking the Dell website is right.

To be sure do an lsmod to look for the tg3/e1000 driver.

For the Broadcom Gigabit card do 'lsmod | grep tg3'
For the Intel Gigabit card do 'lsmod | grep e1000'.

If the site is right (and it should be) you will see the later command
return information about the loaded driver.

Good luck.

Kevin

> but if thats the only way then maybe i have no choice :)
> 
> ill also try that mailing list and see if they have any info.
> 
> Thanks Again
> 
> Ross McInnes
> 
> On Tue, 6 May 2003, Collins, Kevin wrote:
> 
> > Ross:
> > 
> > Hi!
> > 
> > > Appologies if this has been asked before, i have searched the 
> > > archives and 
> > > while there are some almost related issues, none really answer my
> > > question/problem.
> > > 
> > > Recently (about 4 months ago) we purchased a new Dell 2600 
> > > (2.0Ghz Xeon 2GB Ram), Machine
> > 
> > Only thing I have to input here is:  Does your server have 
> the "Broadcom"
> > NIC(s) in it?  I've seen horror stories about this card on the
Dell
> > PowerEdge on Linux mailing list.  If you do, I would 
> recommend that you
> > check out this link: http://lists.us.dell.com and search 
> for "tg3" and or
> > "broadcom" to see the troubles reported there.  Most of the 
> problems with
> > the Broadcoms actually end up leading to complete server 
> lockups though.  I
> > just thought it might be worth mentioning.
> > 
> > Dell has since moved to the Intel brand for integrated NICs 
> in the 2600 -
> > I've got one on the way right now...where is UPS anyway?  :-)
> > 
> > BTW, that mailing list is a VERY good source for anyone 
> running Linux on
> > Dell PowerEdge servers.  There are several Dell people 
> moderating the list
> > and there is always a good "flurry" of information from 
> knowledgeable people
> > being passed around.
> > 
> > Hope I helped a little,
> > 
> > Kevin L. Collins, MCSE
> > Systems Manager
> > Nesbitt Engineering, Inc.
> > 
> > 
>

As an aditional note/something ive just noticed, during these "slow
downs"
(one of which is happening as i type) we quite often get a
"user"(random
not something they are doing thats naughty) who sits on a high cpu smbd 
thread.

like so :

19566 nm102     15   0  4116 3860  2592 S    15.1  0.1   0:02 smbd

the 15.1 % is the cpu usage, and its been sitting there now for about 10 
mins.

any ideas why or what it could be doing?

also at the same time the samba log and an increased amount of OPLOCK 
errors/requests. related?

Thanks

Ross

On Tue, 6 May 2003, Ross McInnes (Systems) wrote:
> Appologies if this has been asked before, i have searched the archives and 
> while there are some almost related issues, none really answer my 
> question/problem.
> 
> Recently (about 4 months ago) we purchased a new Dell 2600 (2.0Ghz Xeon 
> 2GB Ram), Machine (single cpu) and migrated the data and system from rh 
> 6.2 to 8 on this new server. By migrate i mean Installed rh8 fresh and 
> moved the user data and .conf's, cron etc.
> 
> All was well and fine until one afternoon we had a massive slow down in 
> the "system" (overall system, network etc. not this actual
server). upon
> investigation we noticed that smbd was pulling 99% cpu time for about 
> 15-20 mins (before we decided to restart smbd)
> 
> restarting smbd, we had a small issue where all the old links etc tryed to 
> contact the server so a little slow for a few mins then everything went 
> back to normal. Very odd and certainly something we had not previously 
> experienced on our older server.
> 
> so we kept an eye on it for a few weeks and all seemed well. we had a few 
> reports of the system being incredibly slow at times, although this did 
> not coincide with smbd using 99% cpu. using IPtraf/ethereal/ping showed 
> that there wasnt a network issue to or from the server. 
> 
> Looking at the samba log file we started to notice a lot more OPLOCK 
> requests/bad/failures etc around about the same time everything was going 
> slow. So on a few of the shares we made them read only and put on FAKE 
> OPLOCKS (netlogon, start menus etc). And we changed level2 oplocks to no. 
> 
>  # Turn off level 2 oplocks
>         level2 oplocks = no
> 
> This seemed to improve things until it went slow/99% usage once again. 
> (about 2 week or so later) Hair ripping time once again.
> 
> So after more research I read an article about the smbd.conf and some 
> sort of windowing problem with samba and windows clients. so i modified 
> "socket options" to be....
> 
> # Most people will find that this option gives better performance.
>   # See speed.txt and the manual pages for details
>   # You may want to add the following on a Linux system:
>   #         SO_RCVBUF=8192 SO_SNDBUF=8192
> socket options = TCP_NODELAY IPTOS_LOWDELAY SO_SNDBUF=8192 SO_RCVBUF=8192
> 
> Now this seemed to work and all was well until the past 3 weeks, where we 
> have had to restart the server twice (no console response) and have had 
> numerous problems with 99% cpu on smbd or this incredible slowdown of 
> loging on, opening files, listing dir's etc.
> 
> Other details about samba is that its version 2.2.7a (according to 
> smbstatus), we have nt4 clients and on average have about 100-150 users 
> at peak times. it also has the wins server option enabled (so that it acts 
> like a wins server)
> 
> I have no other ideas to try or even think to search for, so any advise, 
> questions and maybe answers would be gratefully received.
> 
> Faithfully
> 
> Ross McInnes
> 
>