Ross McInnes (Systems)
2003-May-06 13:58 UTC
[Samba] Performance Issues With Samba and RedHat 8.
Appologies if this has been asked before, i have searched the archives and while there are some almost related issues, none really answer my question/problem. Recently (about 4 months ago) we purchased a new Dell 2600 (2.0Ghz Xeon 2GB Ram), Machine (single cpu) and migrated the data and system from rh 6.2 to 8 on this new server. By migrate i mean Installed rh8 fresh and moved the user data and .conf's, cron etc. All was well and fine until one afternoon we had a massive slow down in the "system" (overall system, network etc. not this actual server). upon investigation we noticed that smbd was pulling 99% cpu time for about 15-20 mins (before we decided to restart smbd) restarting smbd, we had a small issue where all the old links etc tryed to contact the server so a little slow for a few mins then everything went back to normal. Very odd and certainly something we had not previously experienced on our older server. so we kept an eye on it for a few weeks and all seemed well. we had a few reports of the system being incredibly slow at times, although this did not coincide with smbd using 99% cpu. using IPtraf/ethereal/ping showed that there wasnt a network issue to or from the server. Looking at the samba log file we started to notice a lot more OPLOCK requests/bad/failures etc around about the same time everything was going slow. So on a few of the shares we made them read only and put on FAKE OPLOCKS (netlogon, start menus etc). And we changed level2 oplocks to no. # Turn off level 2 oplocks level2 oplocks = no This seemed to improve things until it went slow/99% usage once again. (about 2 week or so later) Hair ripping time once again. So after more research I read an article about the smbd.conf and some sort of windowing problem with samba and windows clients. so i modified "socket options" to be.... # Most people will find that this option gives better performance. # See speed.txt and the manual pages for details # You may want to add the following on a Linux system: # SO_RCVBUF=8192 SO_SNDBUF=8192 socket options = TCP_NODELAY IPTOS_LOWDELAY SO_SNDBUF=8192 SO_RCVBUF=8192 Now this seemed to work and all was well until the past 3 weeks, where we have had to restart the server twice (no console response) and have had numerous problems with 99% cpu on smbd or this incredible slowdown of loging on, opening files, listing dir's etc. Other details about samba is that its version 2.2.7a (according to smbstatus), we have nt4 clients and on average have about 100-150 users at peak times. it also has the wins server option enabled (so that it acts like a wins server) I have no other ideas to try or even think to search for, so any advise, questions and maybe answers would be gratefully received. Faithfully Ross McInnes
Ross: Hi!> Appologies if this has been asked before, i have searched the > archives and > while there are some almost related issues, none really answer my > question/problem. > > Recently (about 4 months ago) we purchased a new Dell 2600 > (2.0Ghz Xeon 2GB Ram), MachineOnly thing I have to input here is: Does your server have the "Broadcom" NIC(s) in it? I've seen horror stories about this card on the Dell PowerEdge on Linux mailing list. If you do, I would recommend that you check out this link: http://lists.us.dell.com and search for "tg3" and or "broadcom" to see the troubles reported there. Most of the problems with the Broadcoms actually end up leading to complete server lockups though. I just thought it might be worth mentioning. Dell has since moved to the Intel brand for integrated NICs in the 2600 - I've got one on the way right now...where is UPS anyway? :-) BTW, that mailing list is a VERY good source for anyone running Linux on Dell PowerEdge servers. There are several Dell people moderating the list and there is always a good "flurry" of information from knowledgeable people being passed around. Hope I helped a little, Kevin L. Collins, MCSE Systems Manager Nesbitt Engineering, Inc.
> Ive just pulled my system service tag off and looked on the > dell website > armed with that tag and it only lists Intel Drivers for the NIC. can i > safely assume that i have an intel based nic in mine? i dont > really want > to down server/pull it apart to find out for sure.Generally speaking the Dell website is right. To be sure do an lsmod to look for the tg3/e1000 driver. For the Broadcom Gigabit card do 'lsmod | grep tg3' For the Intel Gigabit card do 'lsmod | grep e1000'. If the site is right (and it should be) you will see the later command return information about the loaded driver. Good luck. Kevin> but if thats the only way then maybe i have no choice :) > > ill also try that mailing list and see if they have any info. > > Thanks Again > > Ross McInnes > > On Tue, 6 May 2003, Collins, Kevin wrote: > > > Ross: > > > > Hi! > > > > > Appologies if this has been asked before, i have searched the > > > archives and > > > while there are some almost related issues, none really answer my > > > question/problem. > > > > > > Recently (about 4 months ago) we purchased a new Dell 2600 > > > (2.0Ghz Xeon 2GB Ram), Machine > > > > Only thing I have to input here is: Does your server have > the "Broadcom" > > NIC(s) in it? I've seen horror stories about this card on the Dell > > PowerEdge on Linux mailing list. If you do, I would > recommend that you > > check out this link: http://lists.us.dell.com and search > for "tg3" and or > > "broadcom" to see the troubles reported there. Most of the > problems with > > the Broadcoms actually end up leading to complete server > lockups though. I > > just thought it might be worth mentioning. > > > > Dell has since moved to the Intel brand for integrated NICs > in the 2600 - > > I've got one on the way right now...where is UPS anyway? :-) > > > > BTW, that mailing list is a VERY good source for anyone > running Linux on > > Dell PowerEdge servers. There are several Dell people > moderating the list > > and there is always a good "flurry" of information from > knowledgeable people > > being passed around. > > > > Hope I helped a little, > > > > Kevin L. Collins, MCSE > > Systems Manager > > Nesbitt Engineering, Inc. > > > > >
Ross McInnes (Systems)
2003-May-08 13:09 UTC
[Samba] Performance Issues With Samba and RedHat 8.
As an aditional note/something ive just noticed, during these "slow downs" (one of which is happening as i type) we quite often get a "user"(random not something they are doing thats naughty) who sits on a high cpu smbd thread. like so : 19566 nm102 15 0 4116 3860 2592 S 15.1 0.1 0:02 smbd the 15.1 % is the cpu usage, and its been sitting there now for about 10 mins. any ideas why or what it could be doing? also at the same time the samba log and an increased amount of OPLOCK errors/requests. related? Thanks Ross On Tue, 6 May 2003, Ross McInnes (Systems) wrote:> Appologies if this has been asked before, i have searched the archives and > while there are some almost related issues, none really answer my > question/problem. > > Recently (about 4 months ago) we purchased a new Dell 2600 (2.0Ghz Xeon > 2GB Ram), Machine (single cpu) and migrated the data and system from rh > 6.2 to 8 on this new server. By migrate i mean Installed rh8 fresh and > moved the user data and .conf's, cron etc. > > All was well and fine until one afternoon we had a massive slow down in > the "system" (overall system, network etc. not this actual server). upon > investigation we noticed that smbd was pulling 99% cpu time for about > 15-20 mins (before we decided to restart smbd) > > restarting smbd, we had a small issue where all the old links etc tryed to > contact the server so a little slow for a few mins then everything went > back to normal. Very odd and certainly something we had not previously > experienced on our older server. > > so we kept an eye on it for a few weeks and all seemed well. we had a few > reports of the system being incredibly slow at times, although this did > not coincide with smbd using 99% cpu. using IPtraf/ethereal/ping showed > that there wasnt a network issue to or from the server. > > Looking at the samba log file we started to notice a lot more OPLOCK > requests/bad/failures etc around about the same time everything was going > slow. So on a few of the shares we made them read only and put on FAKE > OPLOCKS (netlogon, start menus etc). And we changed level2 oplocks to no. > > # Turn off level 2 oplocks > level2 oplocks = no > > This seemed to improve things until it went slow/99% usage once again. > (about 2 week or so later) Hair ripping time once again. > > So after more research I read an article about the smbd.conf and some > sort of windowing problem with samba and windows clients. so i modified > "socket options" to be.... > > # Most people will find that this option gives better performance. > # See speed.txt and the manual pages for details > # You may want to add the following on a Linux system: > # SO_RCVBUF=8192 SO_SNDBUF=8192 > socket options = TCP_NODELAY IPTOS_LOWDELAY SO_SNDBUF=8192 SO_RCVBUF=8192 > > Now this seemed to work and all was well until the past 3 weeks, where we > have had to restart the server twice (no console response) and have had > numerous problems with 99% cpu on smbd or this incredible slowdown of > loging on, opening files, listing dir's etc. > > Other details about samba is that its version 2.2.7a (according to > smbstatus), we have nt4 clients and on average have about 100-150 users > at peak times. it also has the wins server option enabled (so that it acts > like a wins server) > > I have no other ideas to try or even think to search for, so any advise, > questions and maybe answers would be gratefully received. > > Faithfully > > Ross McInnes > >