Michael Hendrie
2012-Aug-18 13:33 UTC
[Samba] Unable to use more than 1000 concurrent ntlm_auth processes
Hi List, I'm running a heavily loaded squid server that uses ntlm_auth to provide NTLM authentication. As load has increased over time, I've found the need to increase the number of ntlm_auth processes available to squid as well as the "winbind max clients" value in the smb.conf file. This has worked well up until now but seems I've hit some sort of limit. If I keep the number of ntlm_auth processes under 1000, all is good. Going above continually produces the messages below in /var/log/messages and the additional helpers unusable: Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.342283, 0] utils/ntlm_auth.c:186(get_winbind_domain) Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.345335, 0] utils/ntlm_auth.c:186(get_winbind_domain) Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.353230, 0] utils/ntlm_auth.c:186(get_winbind_domain) Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.358237, 0] utils/ntlm_auth.c:186(get_winbind_domain) Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! And with winbindd log level on 9, /var/log/samba/winbindd.log shows: [2012/08/16 22:33:42.352991, 6] winbindd/winbindd.c:768(new_connection) accepted socket 1032 [2012/08/16 22:33:42.359183, 6] winbindd/winbindd.c:768(new_connection) accepted socket 1036 [2012/08/16 22:37:59.337941, 2] winbindd/winbindd.c:710(winbind_client_response_written) Could not write response[14772:INTERFACE_VERSION] to client: Broken pipe [2012/08/16 22:37:59.338755, 3] winbindd/winbindd_misc.c:352(winbindd_interface_version) [14607]: request interface version [2012/08/16 22:37:59.339035, 2] winbindd/winbindd.c:710(winbind_client_response_written) Could not write response[14607:INTERFACE_VERSION] to client: Broken pipe [2012/08/16 22:37:59.339319, 3] winbindd/winbindd_misc.c:352(winbindd_interface_version) [14777]: request interface version [2012/08/16 22:37:59.339637, 2] winbindd/winbindd.c:710(winbind_client_response_written) Could not write response[14777:INTERFACE_VERSION] to client: Broken pipe [2012/08/16 22:42:59.321236, 3] winbindd/winbindd_misc.c:352(winbindd_interface_version) [14363]: request interface version [2012/08/16 22:42:59.321588, 2] winbindd/winbindd.c:710(winbind_client_response_written) Could not write response[14363:INTERFACE_VERSION] to client: Broken pipe Running distro supplied samba versions: samba3x.x86_64: 3.5.10-0.110.el5_8 samba3x-common.x86_64: 3.5.10-0.110.el5_8 samba3x-winbind.x86_64: 3.5.10-0.110.el5_8 Does anyone have any suggestions on how to overcome this issue, I am happy to compile from source if there are any options that could help? Thanks Mick
Michael Wood
2012-Aug-18 23:34 UTC
[Samba] Unable to use more than 1000 concurrent ntlm_auth processes
Just a guess but maybe you have a limit of 1024 sockets/open files. Try increasing that and see if it makes a difference. On 8/18/12, Michael Hendrie <michael at hendrie.id.au> wrote:> Hi List, > > I'm running a heavily loaded squid server that uses ntlm_auth to provide > NTLM authentication. > > As load has increased over time, I've found the need to increase the number > of ntlm_auth processes available to squid as well as the "winbind max > clients" value in the smb.conf file. This has worked well up until now but > seems I've hit some sort of limit. > > If I keep the number of ntlm_auth processes under 1000, all is good. Going > above continually produces the messages below in /var/log/messages and the > additional helpers unusable: > > Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.342283, 0] > utils/ntlm_auth.c:186(get_winbind_domain) > Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! > Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.345335, 0] > utils/ntlm_auth.c:186(get_winbind_domain) > Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! > Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.353230, 0] > utils/ntlm_auth.c:186(get_winbind_domain) > Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! > Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.358237, 0] > utils/ntlm_auth.c:186(get_winbind_domain) > Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! > > And with winbindd log level on 9, /var/log/samba/winbindd.log shows: > > [2012/08/16 22:33:42.352991, 6] winbindd/winbindd.c:768(new_connection) > accepted socket 1032 > [2012/08/16 22:33:42.359183, 6] winbindd/winbindd.c:768(new_connection) > accepted socket 1036 > [2012/08/16 22:37:59.337941, 2] > winbindd/winbindd.c:710(winbind_client_response_written) > Could not write response[14772:INTERFACE_VERSION] to client: Broken pipe > [2012/08/16 22:37:59.338755, 3] > winbindd/winbindd_misc.c:352(winbindd_interface_version) > [14607]: request interface version > [2012/08/16 22:37:59.339035, 2] > winbindd/winbindd.c:710(winbind_client_response_written) > Could not write response[14607:INTERFACE_VERSION] to client: Broken pipe > [2012/08/16 22:37:59.339319, 3] > winbindd/winbindd_misc.c:352(winbindd_interface_version) > [14777]: request interface version > [2012/08/16 22:37:59.339637, 2] > winbindd/winbindd.c:710(winbind_client_response_written) > Could not write response[14777:INTERFACE_VERSION] to client: Broken pipe > [2012/08/16 22:42:59.321236, 3] > winbindd/winbindd_misc.c:352(winbindd_interface_version) > [14363]: request interface version > [2012/08/16 22:42:59.321588, 2] > winbindd/winbindd.c:710(winbind_client_response_written) > Could not write response[14363:INTERFACE_VERSION] to client: Broken pipe > > Running distro supplied samba versions: > > samba3x.x86_64: 3.5.10-0.110.el5_8 > samba3x-common.x86_64: 3.5.10-0.110.el5_8 > samba3x-winbind.x86_64: 3.5.10-0.110.el5_8 > > Does anyone have any suggestions on how to overcome this issue, I am happy > to compile from source if there are any options that could help? > > Thanks > Mick > > > -- > To unsubscribe from this list go to the following URL and read the > instructions: https://lists.samba.org/mailman/options/samba >-- Michael Wood <esiotrot at gmail.com>
Andrew Bartlett
2012-Sep-03 06:34 UTC
[Samba] Unable to use more than 1000 concurrent ntlm_auth processes
On Sat, 2012-08-18 at 23:03 +0930, Michael Hendrie wrote:> Hi List, > > I'm running a heavily loaded squid server that uses ntlm_auth to provide NTLM authentication. > > As load has increased over time, I've found the need to increase the > number of ntlm_auth processes available to squid as well as the > "winbind max clients" value in the smb.conf file. This has worked > well up until now but seems I've hit some sort of limit. > > If I keep the number of ntlm_auth processes under 1000, all is good. Going above continually produces the messages below in /var/log/messages and the additional helpers unusable: > > Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.342283, 0] utils/ntlm_auth.c:186(get_winbind_domain) > Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! > Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.345335, 0] utils/ntlm_auth.c:186(get_winbind_domain) > Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! > Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.353230, 0] utils/ntlm_auth.c:186(get_winbind_domain) > Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! > Aug 16 22:34:17 prox (ntlm_auth): [2012/08/16 22:34:17.358237, 0] utils/ntlm_auth.c:186(get_winbind_domain) > Aug 16 22:34:17 prox (ntlm_auth): could not obtain winbind domain name! > > And with winbindd log level on 9, /var/log/samba/winbindd.log shows: > > [2012/08/16 22:33:42.352991, 6] winbindd/winbindd.c:768(new_connection) > accepted socket 1032 > [2012/08/16 22:33:42.359183, 6] winbindd/winbindd.c:768(new_connection) > accepted socket 1036 > [2012/08/16 22:37:59.337941, 2] winbindd/winbindd.c:710(winbind_client_response_written) > Could not write response[14772:INTERFACE_VERSION] to client: Broken pipe> Running distro supplied samba versions: > > samba3x.x86_64: 3.5.10-0.110.el5_8 > samba3x-common.x86_64: 3.5.10-0.110.el5_8 > samba3x-winbind.x86_64: 3.5.10-0.110.el5_8 > > Does anyone have any suggestions on how to overcome this issue, I am happy to compile from source if there are any options that could help?In relation to a similar query, it was suggested that with master (or a Samba 4.0 beta) you could set: winbind max domain connections = <larger number than 1> This might increase the throughput, and avoid the backlog getting to 1000. I still think that you are hitting an OS limit somewhere (perhaps on the total of the ntlm_auth children, rather than winbindd?), but having 1000 outstanding authentications would be painful in any case. Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org