Hi List, Looking for assistance with a squid authentication problem against Samba 4. The squid proxy we're using worked fine on our old Samba 3 domain with 500+ users but keeps freezing on our new Samba 4 domain. I've joined the proxy using net ads join and the samba 4 network is a clean build as we wanted to leave any baggage from the old one behind. What we now have is a situation where Samba 4 authenticates squid using NTLM perfectly up until around 120 users are using it. Once we get above 120, it starts to down and as we approach 140 it dies altogether. At this point, we restart samba and it works perfectly well for a period of about 5 minutes with the 140+ users connected at which point it will either slow to a crawl then fall over or sometimes will just fall over. The network has three Samba 4 Domain controllers. replication works across the three and at any given time, they are running at around 25% CPU load and consuming around 500MB of RAM. All three are 3GHz, quad core Xeons with between 4 and 12GB of RAM. The odd thing is that at no point when Samba seems to be hanging, do we lose access to shares on our fileserver and I also have Owncloud authenticating via a read only LDAP proxy which is caching. The really odd thing is that I'm not seeing any obvious messages on either squid, the samba 3 install or the DCs that points towards any major problem. Given the numbers issue, I thought maybe I was hitting a ulimit wall but the hard and soft limits are both unlimited. Does anyone have a similar setup and any info on where to go from here, i.e. which logs to check, etc.? The OS details are as follows: DC1 and DC1 - centos 6.4 Samba 4.0.10 (compiled from source) with internal DNS DC3 - Debian Squeeze with Samba 4.0.10 (compiled from source) with Bind 9.8 with dlz Squid proxy - Debian squeeze with Squid 2.7 Stable 9.2 from .deb package Clients Windows 7 & XP SP3 Cheers, Julian -- Borden Grammar School, Avenue of Remembrance, Sittingbourne, Kent, ME10 4DB. Tel: 01795 424192 **************************************************************************** This e-mail is from Borden Grammar School Trust. This e-mail, together with any files transmitted with it, are confidential, and are intended solely for the use of the individual or entity to whom they are addressed. Any unauthorised dissemination or copying of this e-mail or its attachments, and any use or disclosure of any information contained in them, is strictly prohibited, and may also be illegal. If you are not the intended recipient you must not use, disclose, distribute, copy, print or relay this e-mail. Please note that any views expressed by an individual within this e-mail, do not necessarily reflect the views of the Borden Grammar School Trust. Borden Grammar School Trust has taken reasonable precautions to ensure no viruses are present in this e-mail, the Academy cannot accept responsibility for any loss or damage arising from the use of this e-mail and/or files attached. Registered office: Borden Grammar School, Avenue of Remembrance, Sittingbourne, Kent, ME10 4DB Registered in England: 07827591
On Thu, 2013-10-10 at 16:36 +0100, Julian Pilfold-Bagwell wrote:> Hi List, > > Looking for assistance with a squid authentication problem against Samba 4. > > The squid proxy we're using worked fine on our old Samba 3 domain with > 500+ users but keeps freezing on our new Samba 4 domain. I've joined > the proxy using net ads join and the samba 4 network is a clean build as > we wanted to leave any baggage from the old one behind. > > What we now have is a situation where Samba 4 authenticates squid using > NTLM perfectly up until around 120 users are using it. Once we get above > 120, it starts to down and as we approach 140 it dies altogether. At > this point, we restart samba and it works perfectly well for a period of > about 5 minutes with the 140+ users connected at which point it will > either slow to a crawl then fall over or sometimes will just fall over. > > The network has three Samba 4 Domain controllers. replication works > across the three and at any given time, they are running at around 25% > CPU load and consuming around 500MB of RAM. All three are 3GHz, quad > core Xeons with between 4 and 12GB of RAM. > > The odd thing is that at no point when Samba seems to be hanging, do we > lose access to shares on our fileserver and I also have Owncloud > authenticating via a read only LDAP proxy which is caching. The really > odd thing is that I'm not seeing any obvious messages on either squid, > the samba 3 install or the DCs that points towards any major problem. > Given the numbers issue, I thought maybe I was hitting a ulimit wall but > the hard and soft limits are both unlimited. > > Does anyone have a similar setup and any info on where to go from here, > i.e. which logs to check, etc.? > > The OS details are as follows: > > DC1 and DC1 - centos 6.4 Samba 4.0.10 (compiled from source) with > internal DNS > DC3 - Debian Squeeze with Samba 4.0.10 (compiled from source) with Bind > 9.8 with dlz > Squid proxy - Debian squeeze with Squid 2.7 Stable 9.2 from .deb packageMy guess is that the single thread that is doing the lookups in the sam.ldb and the subsequent authentication is choking on the constant barrage of NTLM authentication traffic. You might want to look into using kerberos, rather than NTLM authentication, now you have an AD domain. This will not need to place load on the DC for each page load. However, we should cope with lots of authentication, so if you have the skill, running 'perf record -g PID' on the busy PID could be quite illuminating, once analyzed with 'perf report -g'. Please don't try and mail me the perf.data output (it needs the build tree and symbols), but examine it and tell me where the CPU is being used and what callers responsible for it (screen-shots are OK in this specific instance). Also, just have a look at a wireshark trace of the success and failure modes, and see if you can show a difference. If the traces are not massive, these you can mail to me. Either way, the wireshark 'service response time' over DCE/RPC would be particularly interesting to see. I hope this helps, Andrew Bartlett Thanks, Andrew Bartlett -- Andrew Bartlett samba.org/~abartlet Authentication Developer, Samba Team samba.org Samba Developer, Catalyst IT catalyst.net.nz