Hi, yesterday we experienced a heavy request flooding from multiple servers being a domain member against our Samba Sernet DCs. All those servers are domain members and allow login using PAM (Samba+Winbind). Running TCPDump we had like 400 Requests per 5 seconds like this: tcpdump -i eth0 dst port 389 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 10:43:14.195984 IP hostname.nli.local.38153 > dc.ldap: Flags [P.], seq 3322715177:3322715340, ack 2230642521, win 1444, options [nop,nop,TS val 54690397 ecr 19378130], length 163 10:43:14.199632 IP hostname.nli.local.38153 > dc.ldap: Flags [P.], seq 163:353, ack 91, win 1444, options [nop,nop,TS val 54690398 ecr 19455846], length 190 10:43:14.201426 IP hostname.nli.local.38153 > dc.ldap: Flags [.], ack 1578, win 1444, options [nop,nop,TS val 54690399 ecr 19455847], length 0 10:43:14.203080 IP hostname.nli.local.38153 > dc.ldap: Flags [P.], seq 353:586, ack 1578, win 1444, options [nop,nop,TS val 54690399 ecr 19455847], length 233 10:43:14.212292 IP hostname.nli.local.39958 > dc.ldap: Flags [P.], seq 651643402:651643565, ack 654655483, win 2888, options [nop,nop,TS val 55032891 ecr 19374126], length 163 10:43:14.288788 IP hostname.nli.local.38153 > dc.ldap: Flags [.], ack 7117, win 1422, options [nop,nop,TS val 54690420 ecr 19455869], length 0 10:43:14.291937 IP hostname.nli.local.39958 > dc.ldap: Flags [P.], seq 163:353, ack 91, win 2888, options [nop,nop,TS val 55032911 ecr 19455869], length 190 10:43:14.293453 IP hostname.nli.local.39958 > dc.ldap: Flags [.], ack 1578, win 2888, options [nop,nop,TS val 55032912 ecr 19455870], length 0 10:43:14.296267 IP hostname.nli.local.39958 > dc.ldap: Flags [P.], seq 353:586, ack 1578, win 2888, options [nop,nop,TS val 55032912 ecr 19455870], length 233 10:43:14.375968 IP hostname.nli.local.39958 > dc.ldap: Flags [.], ack 7117, win 2844, options [nop,nop,TS val 55032932 ecr 19455890], length 0 10:43:16.737485 IP hostname.nli.local.58569 > dc.ldap: Flags [P.], seq 2429997579:2429997742, ack 112410608, win 1444, options [nop,nop,TS val 54761116 ecr 19378382], length 163 10:43:16.740721 IP hostname.nli.local.58569 > dc.ldap: Flags [P.], seq 163:353, ack 91, win 1444, options [nop,nop,TS val 54761117 ecr 19456481], length 190 10:43:16.742187 IP hostname.nli.local.58569 > dc.ldap: Flags [.], ack 1578, win 1444, options [nop,nop,TS val 54761117 ecr 19456482], length 0 10:43:16.743849 IP hostname.nli.local.58569 > dc.ldap: Flags [P.], seq 353:586, ack 1578, win 1444, options [nop,nop,TS val 54761118 ecr 19456482], length 233 10:43:16.810808 IP hostname.nli.local.58569 > dc.ldap: Flags [.], ack 4474, win 1444, options [nop,nop,TS val 54761135 ecr 19456499], length 0 10:43:16.810914 IP hostname.nli.local.58569 > dc.ldap: Flags [.], ack 7117, win 1444, options [nop,nop,TS val 54761135 ecr 19456499], length 0 Due to that flooding, even logins via ssh on our servers timed out. CPU Load on both DCs went up to 95% without high I/O wait. After restarting the sernet-samba-ad service on both DCs, the Problem went away. Any conclusion on what went wrong here? * Julian Wichtiger Hinweis: Der Inhalt dieser E-Mail ist vertraulich und ausschlie?lich f?r den bezeichneten Adressaten bestimmt. Wenn Sie nicht der vorgesehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, so beachten Sie bitte, dass jede Form der Kenntnisnahme, Ver?ffentlichung, Vervielf?ltigung oder Weitergabe des Inhalts dieser E-Mail unzul?ssig ist. Wir bitten Sie, sich in diesem Fall mit dem Absender der E-Mail in Verbindung zu setzen. Wir m?chten Sie au?erdem darauf hinweisen, dass die Kommunikation per E-Mail ?ber das Internet unsicher ist, da f?r unberechtigte Dritte grunds?tzlich die M?glichkeit der Kenntnisnahme und Manipulation besteht Important Note: The information contained in this e-mail is confidential. It is intended solely for the addressee. Access to this e-mail by anyone else is unauthorized. If you are not the intended recipient, any form of disclosure, reproduction, distribution or any action taken or refrained from in reliance on it, is prohibited and may be unlawful. Please notify the sender immediately. We also would like to inform you that communication via e-mail over the internet is insecure because third parties may have the possibility to access and manipulate e-mails.
On Tue, 25 Apr 2017 09:05:28 +0000 Julian Zielke via samba <samba at lists.samba.org> wrote:> Due to that flooding, even logins via ssh on our servers timed out. > CPU Load on both DCs went up to 95% without high I/O wait. After > restarting the sernet-samba-ad service on both DCs, the Problem went > away. > > Any conclusion on what went wrong here? > >It might help if you posted your smb.conf file Rowland
On Tue, 2017-04-25 at 09:05 +0000, Julian Zielke via samba wrote:> Hi, > > yesterday we experienced a heavy request flooding from multiple > servers being a domain member against our Samba Sernet DCs. > All those servers are domain members and allow login using PAM > (Samba+Winbind).Currently we only have one process handling the LDAP traffic, which would explain why a traffic flood like this:> Running TCPDump we had like 400 Requests per 5 seconds like this:... Still only gives busy CPU, but not I/O wait:> Due to that flooding, even logins via ssh on our servers timed out. > CPU Load on both DCs went up to 95% without high I/O wait. > After restarting the sernet-samba-ad service on both DCs, the Problem > went away.My first guess is that the multiple connections caused timeouts on the clients, causing the clients to reconnect and try again, magnifying the load. However it is really hard to tell with the limited information provided. I'm working to make our LDAP server multi-process, more efficient and able to use all the available CPUs. Andrew Bartlett -- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
Hi,
our virtual DCs specs are:
1vCPU, 1GB RAM
System is running on iSCSI Storage in an Openstack Enviroment.
Smb.conf on our DC 03 and 04:
=================# Global parameters
[global]
workgroup = NLI
realm = nli.local
netbios name = dc04
server role = active directory domain controller
allow dns updates = nonsecure
dns forwarder = xxx.xxx.xxx.xxx
ldap server require strong auth = no
[netlogon]
path = /var/lib/samba/sysvol/nli.local/scripts
read only = No
[sysvol]
path = /var/lib/samba/sysvol
read only = Yes
=================
Smb.conf on our clients:
=================#Ansible managed
# global options
[global]
workgroup = NLI
realm = NLI.LOCAL
netbios name = xxxxxx
server string = Samba AD Client Version %v
security = ads
password server = dc3.nli.local, dc4.nli.local, dc2.nli.local, dc1.nli.local,
*
server role = member server
socket options = TCP_NODELAY SO_KEEPALIVE=4
deadtime = 15
# winbind options
winbind enum users = yes
winbind enum groups = yes
winbind refresh tickets = yes
winbind offline logon = true
winbind nested groups = yes
winbind use default domain = yes
winbind cache time = 300
winbind nss info = template
template shell = /bin/bash
template homedir = /home/NLI.LOCAL/%U
# local user id mapping
idmap config * : backend = tdb
idmap config * : range = 3000-7999
# domain user id mapping
idmap config NLI : backend = rid
idmap config NLI : range = 10000-999999
# log configuration
log file = /var/log/samba/log.%m
log level = 1
max log size = 1000
# root to domain admin mapping
username map = /etc/samba/user.map
=================
Dc03 and dc04 in our cloud replicate to our dc01 and dc02 in our HQ.
- Julian
-----Ursprüngliche Nachricht-----
Von: Andrew Bartlett [mailto:abartlet at samba.org]
Gesendet: Dienstag, 25. April 2017 11:45
An: Julian Zielke <jzielke at next-level-integration.com>; samba at
lists.samba.org
Betreff: Re: [Samba] Flooding Samba DC with random requests
On Tue, 2017-04-25 at 09:05 +0000, Julian Zielke via samba
wrote:> Hi,
>
> yesterday we experienced a heavy request flooding from multiple
> servers being a domain member against our Samba Sernet DCs.
> All those servers are domain members and allow login using PAM
> (Samba+Winbind).
Currently we only have one process handling the LDAP traffic, which would
explain why a traffic flood like this:
> Running TCPDump we had like 400 Requests per 5 seconds like this:
...
Still only gives busy CPU, but not I/O wait:
> Due to that flooding, even logins via ssh on our servers timed out.
> CPU Load on both DCs went up to 95% without high I/O wait.
> After restarting the sernet-samba-ad service on both DCs, the Problem
> went away.
My first guess is that the multiple connections caused timeouts on the clients,
causing the clients to reconnect and try again, magnifying the load. However it
is really hard to tell with the limited information provided.
I'm working to make our LDAP server multi-process, more efficient and able
to use all the available CPUs.
Andrew Bartlett
--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
Wichtiger Hinweis: Der Inhalt dieser E-Mail ist vertraulich und ausschließlich
für den bezeichneten Adressaten bestimmt. Wenn Sie nicht der vorgesehene
Adressat dieser E-Mail oder dessen Vertreter sein sollten, so beachten Sie
bitte, dass jede Form der Kenntnisnahme, Veröffentlichung, Vervielfältigung oder
Weitergabe des Inhalts dieser E-Mail unzulässig ist. Wir bitten Sie, sich in
diesem Fall mit dem Absender der E-Mail in Verbindung zu setzen. Wir möchten Sie
außerdem darauf hinweisen, dass die Kommunikation per E-Mail über das Internet
unsicher ist, da für unberechtigte Dritte grundsätzlich die Möglichkeit der
Kenntnisnahme und Manipulation besteht
Important Note: The information contained in this e-mail is confidential. It is
intended solely for the addressee. Access to this e-mail by anyone else is
unauthorized. If you are not the intended recipient, any form of disclosure,
reproduction, distribution or any action taken or refrained from in reliance on
it, is prohibited and may be unlawful. Please notify the sender immediately. We
also would like to inform you that communication via e-mail over the internet is
insecure because third parties may have the possibility to access and manipulate
e-mails.