Thanks for answer. I am very sad to see that.
I had examined samba behavior with tracer and had seen too many poll,
fcntl and stat system calls by samba task[ldapsrv] process
Elapsed Times for PID 1423,
SYSCALL TIME (ns)
dup 2653
flock 5811
lseek 33513
geteuid 42747
getpid 63819
getpeername 85480
getsockname 151683
socket 237632
accept 311871
fstat 404420
madvise 452120
readv 601524
mmap 858391
writev 910337
gettimeofday 929595
read 1064811
close 1270401
open 1733182
lstat 1938394
bind 1939462
unlink 2074873
munmap 2569299
mkdir 3539965
clock_gettime 5662586
stat 7543724
fcntl 74322331
poll 5393911015
I think may be samba something doing not optimal way. Like this, first
we try to create directory (mkdir), then we check it (stat). May be that
leads to impact filesystems and samba ldap works slower than openldap.
fcntl(15,F_SETLKW,0x7fffffffd118) = 0 (0x0)
fcntl(15,F_SETLKW,0x7fffffffd188) = 0 (0x0)
clock_gettime(13,{1448375653.000000000 }) = 0 (0x0)
mkdir("/var/db/samba4/private/sam.ldb.d",0700) ERR#17 'File
exists'
stat("/var/db/samba4/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=SPEECHPRO,DC=COM.ldb",{
mode=-rw------- ,inode=242034,size=4247552,blksize=32768 }) = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0)
clock_gettime(13,{1448375653.000000000 }) = 0 (0x0)
clock_gettime(13,{1448375653.000000000 }) = 0 (0x0)
fcntl(16,F_SETLK,0x7fffffffd098) = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd118) = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd188) = 0 (0x0)
clock_gettime(13,{1448375653.000000000 }) = 0 (0x0)
mkdir("/var/db/samba4/private/sam.ldb.d",0700) ERR#17 'File
exists'
stat("/var/db/samba4/private/sam.ldb.d/DC=FORESTDNSZONES,DC=SPEECHPRO,DC=COM.ldb",{
mode=-rw------- ,inode=242035,size=4247552,blksize=32768 }) = 0 (0x0)
fcntl(17,F_SETLKW,0x7fffffffd098) = 0 (0x0)
fcntl(17,F_SETLKW,0x7fffffffd138) = 0 (0x0)
fcntl(17,F_SETLKW,0x7fffffffd098)
23.11.2015 18:44, mathias dufresne пишет:> Hi,
>
> I feel like I must reply off-list: Samba 4 is not yet able to use threads.
> To work around that we are about to deploy our DCs as virtual
> machines, to be able to multiply the number of Samba running.
>
> I expect you would have already thought about that, but in case you
> didn't, I said it :p
>
> I will follow closely that topic you opened. Sorry to not help more.
>
> Cheers,
>
> mathias
>
>
>
> 2015-11-23 14:50 GMT+01:00 Yuriy Tabolin <tabolin at speechpro.com
> <mailto:tabolin at speechpro.com>>:
>
> Hi all.
> I have samba 4.2.3 on freebsd 10.1 server. There are three DC and
> about 350 PC on domain. DCs have 3 CPU and 3GB RAM. Some servers
> with services like apache, exim, dovecot, etc use samba4 ldap
> (port 389) for user authentication. Some times ago after adding
> some services to use ldap I found, that samba4 cannot serve all
> ldap requests. Every 10-30 minutes I see in DCs logs:
> dc1 kernel: sonewconn: pcb 0xfffff800753d6ab8: Listen queue
> overflow: 16 already in queue awaiting acceptance (28 occurrences)
>
> After that I have used tcpdump for recording ldap-traffic and have
> seen that after TCP handshaking, server some times suddenly send
> TCP-RST to close connection. I have enlarged DCs resources (CPU
> and RAM), kern.ipc.somaxconn, did some other system tuning but
> all that didn't help. Load average on DCs permanently near 0.9-1.0
> and samba cannot serve all ldap conncetions. ldap clients works
> well because they use as minimum as two domain controllers as ldap
> servers. Is there a some performance problem in samba4, slow
> processing ldap requests or something else?
> Thanks for any help.
>
> --
> With best regards,
>
> Tabolin Yuriy
> System administrator
> Speech Technology Center
>
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions: https://lists.samba.org/mailman/options/samba
>
>
--
With best regards,
Tabolin Yuriy
System administrator
Speech Technology Center
Andrew Bartlett
2015-Nov-25 07:59 UTC
[Samba] samba4 ldap high load and port queue overflow
On Tue, 2015-11-24 at 17:48 +0300, Yuriy Tabolin wrote:> Thanks for answer. I am very sad to see that. > I had examined samba behavior with tracer and had seen too many poll, > fcntl and stat system calls by samba task[ldapsrv] process > > Elapsed Times for PID 1423, > > SYSCALL TIME (ns) > dup 2653 > flock 5811 > lseek 33513 > geteuid 42747 > getpid 63819 > getpeername 85480 > getsockname 151683 > socket 237632 > accept 311871 > fstat 404420 > madvise 452120 > readv 601524 > mmap 858391 > writev 910337 > gettimeofday 929595 > read 1064811 > close 1270401 > open 1733182 > lstat 1938394 > bind 1939462 > unlink 2074873 > munmap 2569299 > mkdir 3539965 > clock_gettime 5662586 > stat 7543724 > fcntl 74322331 > poll 5393911015That poll() count is very high, I do wonder if something isn't working right with your system. I would have expected it to be more in line with fcntl(), which we use a lot of in TDB. If you could profile your use case on a linux host with the perf tools, that may give us some insights.> I think may be samba something doing not optimal way. Like this, > first > we try to create directory (mkdir), then we check it (stat). May be > that > leads to impact filesystems and samba ldap works slower than > openldap. > > fcntl(15,F_SETLKW,0x7fffffffd118) = 0 (0x0) > fcntl(15,F_SETLKW,0x7fffffffd188) = 0 (0x0) > clock_gettime(13,{1448375653.000000000 }) = 0 (0x0) > mkdir("/var/db/samba4/private/sam.ldb.d",0700) ERR#17 'File exists' > stat("/var/db/samba4/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=SPEECHPRO > ,DC=COM.ldb",{ > mode=-rw------- ,inode=242034,size=4247552,blksize=32768 }) = 0 (0x0) > fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0) > fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0) > fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0) > fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0) > fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0) > fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0) > clock_gettime(13,{1448375653.000000000 }) = 0 (0x0) > clock_gettime(13,{1448375653.000000000 }) = 0 (0x0) > fcntl(16,F_SETLK,0x7fffffffd098) = 0 (0x0) > fcntl(16,F_SETLKW,0x7fffffffd118) = 0 (0x0) > fcntl(16,F_SETLKW,0x7fffffffd188) = 0 (0x0) > clock_gettime(13,{1448375653.000000000 }) = 0 (0x0) > mkdir("/var/db/samba4/private/sam.ldb.d",0700) ERR#17 'File exists' > stat("/var/db/samba4/private/sam.ldb.d/DC=FORESTDNSZONES,DC=SPEECHPRO > ,DC=COM.ldb",{ > mode=-rw------- ,inode=242035,size=4247552,blksize=32768 }) = 0 (0x0) > fcntl(17,F_SETLKW,0x7fffffffd098) = 0 (0x0) > fcntl(17,F_SETLKW,0x7fffffffd138) = 0 (0x0) > fcntl(17,F_SETLKW,0x7fffffffd098) >These really should happen only once per startup of the process, how often do you see it?> 23.11.2015 18:44, mathias dufresne пишет: > > Hi, > > > > I feel like I must reply off-list: Samba 4 is not yet able to use > > threads. > > To work around that we are about to deploy our DCs as virtual > > machines, to be able to multiply the number of Samba running. > > > > I expect you would have already thought about that, but in case you > > didn't, I said it :p > > > > I will follow closely that topic you opened. Sorry to not help > > more. > > > > Cheers, > > > > mathias > > > > > > > > 2015-11-23 14:50 GMT+01:00 Yuriy Tabolin <tabolin at speechpro.com > > <mailto:tabolin at speechpro.com>>: > > > > Hi all. > > I have samba 4.2.3 on freebsd 10.1 server. There are three DC > > and > > about 350 PC on domain. DCs have 3 CPU and 3GB RAM. Some > > servers > > with services like apache, exim, dovecot, etc use samba4 ldap > > (port 389) for user authentication. Some times ago after adding > > some services to use ldap I found, that samba4 cannot serve all > > ldap requests. Every 10-30 minutes I see in DCs logs: > > dc1 kernel: sonewconn: pcb 0xfffff800753d6ab8: Listen queue > > overflow: 16 already in queue awaiting acceptance (28 > > occurrences) > > > > After that I have used tcpdump for recording ldap-traffic and > > have > > seen that after TCP handshaking, server some times suddenly > > send > > TCP-RST to close connection. I have enlarged DCs resources (CPU > > and RAM), kern.ipc.somaxconn, did some other system tuning but > > all that didn't help. Load average on DCs permanently near 0.9 > > -1.0 > > and samba cannot serve all ldap conncetions. ldap clients works > > well because they use as minimum as two domain controllers as > > ldap > > servers. Is there a some performance problem in samba4, slow > > processing ldap requests or something else? > > Thanks for any help. > > > > -- > > With best regards, > > > > Tabolin Yuriy > > System administrator > > Speech Technology Center > > > > > > -- > > To unsubscribe from this list go to the following URL and read > > the > > instructions: https://lists.samba.org/mailman/options/samba > > > > > > > -- > With best regards, > > Tabolin Yuriy > System administrator > Speech Technology Center >-- Andrew Bartlett http://samba.org/~abartlet/ Authentication Developer, Samba Team http://samba.org Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba
25.11.2015 10:59, Andrew Bartlett пишет:> On Tue, 2015-11-24 at 17:48 +0300, Yuriy Tabolin wrote: >> Thanks for answer. I am very sad to see that. >> I had examined samba behavior with tracer and had seen too many poll, >> fcntl and stat system calls by samba task[ldapsrv] process >> >> Elapsed Times for PID 1423, >> >> SYSCALL TIME (ns) >> dup 2653 >> flock 5811 >> lseek 33513 >> geteuid 42747 >> getpid 63819 >> getpeername 85480 >> getsockname 151683 >> socket 237632 >> accept 311871 >> fstat 404420 >> madvise 452120 >> readv 601524 >> mmap 858391 >> writev 910337 >> gettimeofday 929595 >> read 1064811 >> close 1270401 >> open 1733182 >> lstat 1938394 >> bind 1939462 >> unlink 2074873 >> munmap 2569299 >> mkdir 3539965 >> clock_gettime 5662586 >> stat 7543724 >> fcntl 74322331 >> poll 5393911015 > That poll() count is very high, I do wonder if something isn't working > right with your system. I would have expected it to be more in line > with fcntl(), which we use a lot of in TDB.I think that poll calls are too much too. Trace look that fcntl(13,F_SETLKW,0x7fffffffdda8) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffde48) = 0 (0x0) writev(0x85,0x81019cc90,0x1,0x1,0xffffffffea994e98,0x810197010) = 251 (0xfb) gettimeofday({1448524674.289517 },0x0) = 0 (0x0) gettimeofday({1448524674.289624 },0x0) = 0 (0x0) poll({33/POLLIN|POLLHUP 32/POLLIN|POLLHUP 31/POLLIN|POLLHUP 30/POLLIN|POLLHUP 29/POLLIN|POLLHUP 28/POLLIN|POLLHUP 27/POLLIN|POLLHUP 26/POLLIN|POLLHUP 25/POLLIN|POLLHUP 24/POLLIN|POLLHUP 23/POLLIN|POLLHUP 22/POLLIN|POLLHUP 74/POLLIN|POLLHUP 73/POLLIN|POLLHUP 130/POLLIN|POLLHUP 129/POLLIN|POLLHUP 66/POLLIN|POLLHUP 65/POLLIN|POLLHUP 35/POLLIN|POLLHUP 34/POLLIN|POLLHUP 53/POLLIN|POLLHUP 54/POLLIN|POLLHUP 166/POLLIN|POLLHUP 165/POLLIN|POLLHUP 68/POLLIN|POLLHUP 67/POLLIN|POLLHUP 56/POLLIN|POLLHUP 55/POLLIN|POLLHUP 117/POLLIN|POLLHUP 118/POLLIN|POLLHUP 175/POLLIN|POLLHUP 176/POLLIN|POLLHUP 60/POLLIN|POLLHUP 59/POLLIN|POLLHUP 184/POLLIN|POLLHUP 183/POLLIN|POLLHUP 171/POLLIN|POLLHUP 172/POLLIN|POLLHUP 88/POLLIN|POLLHUP 87/POLLIN|POLLHUP 157/POLLIN|POLLHUP 158/POLLIN|POLLHUP 193/POLLIN|POLLHUP 194/POLLIN|POLLHUP 58/POLLIN|POLLHUP 57/POLLIN|POLLHUP 77/POLLIN|POLLHUP 78/POLLIN|POLLHUP 72/POLLIN|POLLHUP 71/POLLIN|POLLHUP 80/POLLIN|POLLHUP 79/POLLIN|POLLHUP 52/POLLIN|POLLHUP 51/POLLIN|POLLHUP 48/POLLIN|POLLHUP 47/POLLIN|POLLHUP 141/POLLIN|POLLHUP 142/POLLIN|POLLHUP 46/POLLIN|POLLHUP 45/POLLIN|POLLHUP 41/POLLIN|POLLHUP 42/POLLIN|POLLHUP 127/POLLIN|POLLHUP 128/POLLIN|POLLHUP 37/POLLIN|POLLHUP 38/POLLIN|POLLHUP 50/POLLIN|POLLHUP 49/POLLIN|POLLHUP 92/POLLIN|POLLHUP 91/POLLIN|POLLHUP 116/POLLIN|POLLHUP 115/POLLIN|POLLHUP 63/POLLIN|POLLHUP 64/POLLIN|POLLHUP 76/POLLIN|POLLHUP 75/POLLIN|POLLHUP 102/POLLIN|POLLHUP 101/POLLIN|POLLHUP 104/POLLIN|POLLHUP 103/POLLIN|POLLHUP 85/POLLIN|POLLHUP 86/POLLIN|POLLHUP 121/POLLIN|POLLHUP 122/POLLIN|POLLHUP 100/POLLIN|POLLHUP 99/POLLIN|POLLHUP 93/POLLIN|POLLHUP 94/POLLIN|POLLHUP 95/POLLIN|POLLHUP 96/POLLIN|POLLHUP 114/POLLIN|POLLHUP 113/POLLIN|POLLHUP 39/POLLIN|POLLHUP 40/POLLIN|POLLHUP 98/POLLIN|POLLHUP 97/POLLIN|POLLHUP 61/POLLIN|POLLHUP 62/POLLIN|POLLHUP 138/POLLIN|POLLHUP 137/POLLIN|POLLHUP 89/POLLIN|POLLHUP 90/POLLIN|POLLHUP 111/POLLIN|POLLHUP 112/POLLIN|POLLHUP 83/POLLIN|POLLHUP 84/POLLIN|POLLHUP 108/POLLIN|POLLHUP 107/POLLIN|POLLHUP 178/POLLIN|POLLHUP 177/POLLIN|POLLHUP 123/POLLIN|POLLHUP 124/POLLIN|POLLHUP 110/POLLIN|POLLHUP 109/POLLIN|POLLHUP 81/POLLIN|POLLHUP 82/POLLIN|POLLHUP 106/POLLIN|POLLHUP 105/POLLIN|POLLHUP 164/POLLIN|POLLHUP 163/POLLIN|POLLHUP 156/POLLIN|POLLHUP 155/POLLIN|POLLHUP 125/POLLIN|POLLHUP 126/POLLIN|POLLHUP 150/POLLIN|POLLHUP 149/POLLIN|POLLHUP 173/POLLIN|POLLHUP 174/POLLIN|POLLHUP 139/POLLIN|POLLHUP 140/POLLIN|POLLHUP 161/POLLIN|POLLHUP 162/POLLIN|POLLHUP 136/POLLIN|POLLHUP 135/POLLIN|POLLHUP 160/POLLIN|POLLHUP 159/POLLIN|POLLHUP 69/POLLIN|POLLHUP 70/POLLIN|POLLHUP 168/POLLIN|POLLHUP 167/POLLIN|POLLHUP 119/POLLIN|POLLHUP 120/POLLIN|POLLHUP 182/POLLIN|POLLHUP 181/POLLIN|POLLHUP 144/POLLIN|POLLHUP 143/POLLIN|POLLHUP 43/POLLIN|POLLHUP 44/POLLIN|POLLHUP 131/POLLIN|POLLHUP 132/POLLIN|POLLHUP 133/POLLIN|POLLHUP 134/POLLIN|POLLHUP},152,5114) = 1 (0x1) readv(0x85,0x81019cc90,0x1,0x1,0x80860bd82,0x0) = 7 (0x7) gettimeofday({1448524674.290357 },0x0) = 0 (0x0) poll({33/POLLIN|POLLHUP 32/POLLIN|POLLHUP 31/POLLIN|POLLHUP 30/POLLIN|POLLHUP 29/POLLIN|POLLHUP 28/POLLIN|POLLHUP 27/POLLIN|POLLHUP 26/POLLIN|POLLHUP 25/POLLIN|POLLHUP 24/POLLIN|POLLHUP 23/POLLIN|POLLHUP 22/POLLIN|POLLHUP 74/POLLIN|POLLHUP 73/POLLIN|POLLHUP 130/POLLIN|POLLHUP 129/POLLIN|POLLHUP 66/POLLIN|POLLHUP 65/POLLIN|POLLHUP 35/POLLIN|POLLHUP 34/POLLIN|POLLHUP 53/POLLIN|POLLHUP 54/POLLIN|POLLHUP 166/POLLIN|POLLHUP 165/POLLIN|POLLHUP 68/POLLIN|POLLHUP 67/POLLIN|POLLHUP 56/POLLIN|POLLHUP 55/POLLIN|POLLHUP 117/POLLIN|POLLHUP 118/POLLIN|POLLHUP 175/POLLIN|POLLHUP 176/POLLIN|POLLHUP 60/POLLIN|POLLHUP 59/POLLIN|POLLHUP 184/POLLIN|POLLHUP 183/POLLIN|POLLHUP 171/POLLIN|POLLHUP 172/POLLIN|POLLHUP 88/POLLIN|POLLHUP 87/POLLIN|POLLHUP 157/POLLIN|POLLHUP 158/POLLIN|POLLHUP 193/POLLIN|POLLHUP 194/POLLIN|POLLHUP 58/POLLIN|POLLHUP 57/POLLIN|POLLHUP 77/POLLIN|POLLHUP 78/POLLIN|POLLHUP 72/POLLIN|POLLHUP 71/POLLIN|POLLHUP 80/POLLIN|POLLHUP 79/POLLIN|POLLHUP 52/POLLIN|POLLHUP 51/POLLIN|POLLHUP 48/POLLIN|POLLHUP 47/POLLIN|POLLHUP 141/POLLIN|POLLHUP 142/POLLIN|POLLHUP 46/POLLIN|POLLHUP 45/POLLIN|POLLHUP 41/POLLIN|POLLHUP 42/POLLIN|POLLHUP 127/POLLIN|POLLHUP 128/POLLIN|POLLHUP 37/POLLIN|POLLHUP 38/POLLIN|POLLHUP 50/POLLIN|POLLHUP 49/POLLIN|POLLHUP 92/POLLIN|POLLHUP 91/POLLIN|POLLHUP 116/POLLIN|POLLHUP 115/POLLIN|POLLHUP 63/POLLIN|POLLHUP 64/POLLIN|POLLHUP 76/POLLIN|POLLHUP 75/POLLIN|POLLHUP 102/POLLIN|POLLHUP 101/POLLIN|POLLHUP 104/POLLIN|POLLHUP 103/POLLIN|POLLHUP 85/POLLIN|POLLHUP 86/POLLIN|POLLHUP 121/POLLIN|POLLHUP 122/POLLIN|POLLHUP 100/POLLIN|POLLHUP 99/POLLIN|POLLHUP 93/POLLIN|POLLHUP 94/POLLIN|POLLHUP 95/POLLIN|POLLHUP 96/POLLIN|POLLHUP 114/POLLIN|POLLHUP 113/POLLIN|POLLHUP 39/POLLIN|POLLHUP 40/POLLIN|POLLHUP 98/POLLIN|POLLHUP 97/POLLIN|POLLHUP 61/POLLIN|POLLHUP 62/POLLIN|POLLHUP 138/POLLIN|POLLHUP 137/POLLIN|POLLHUP 89/POLLIN|POLLHUP 90/POLLIN|POLLHUP 111/POLLIN|POLLHUP 112/POLLIN|POLLHUP 83/POLLIN|POLLHUP 84/POLLIN|POLLHUP 108/POLLIN|POLLHUP 107/POLLIN|POLLHUP 178/POLLIN|POLLHUP 177/POLLIN|POLLHUP 123/POLLIN|POLLHUP 124/POLLIN|POLLHUP 110/POLLIN|POLLHUP 109/POLLIN|POLLHUP 81/POLLIN|POLLHUP 82/POLLIN|POLLHUP 106/POLLIN|POLLHUP 105/POLLIN|POLLHUP 164/POLLIN|POLLHUP 163/POLLIN|POLLHUP 156/POLLIN|POLLHUP 155/POLLIN|POLLHUP 125/POLLIN|POLLHUP 126/POLLIN|POLLHUP 150/POLLIN|POLLHUP 149/POLLIN|POLLHUP 173/POLLIN|POLLHUP 174/POLLIN|POLLHUP 139/POLLIN|POLLHUP 140/POLLIN|POLLHUP 161/POLLIN|POLLHUP 162/POLLIN|POLLHUP 136/POLLIN|POLLHUP 135/POLLIN|POLLHUP 160/POLLIN|POLLHUP 159/POLLIN|POLLHUP 69/POLLIN|POLLHUP 70/POLLIN|POLLHUP 168/POLLIN|POLLHUP 167/POLLIN|POLLHUP 119/POLLIN|POLLHUP 120/POLLIN|POLLHUP 182/POLLIN|POLLHUP 181/POLLIN|POLLHUP 144/POLLIN|POLLHUP 143/POLLIN|POLLHUP 43/POLLIN|POLLHUP 44/POLLIN|POLLHUP 131/POLLIN|POLLHUP 132/POLLIN|POLLHUP 133/POLLIN|POLLHUP 134/POLLIN|POLLHUP},152,5114) = 1 (0x1) readv(0x85,0x81019cc90,0x1,0x1,0x80860bd82,0x0) = 129 (0x81) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffcd48) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffcde8) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffcc78) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffcd18) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffccc8) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffcd68) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffcbf8) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffcc98) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffcf48) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffcfe8) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffce78) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffcf18) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffc5e8) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffc688) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffc518) = 0 (0x0) fcntl(14,F_SETLKW,0x7fffffffc5b8) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffcef8) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffcf98) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffce28) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffcec8) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffce78) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffcf18) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffcda8) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffce48) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffde78) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffdf18) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffdda8) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffde48) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffd518) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffd5b8) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffd448) = 0 (0x0) fcntl(13,F_SETLKW,0x7fffffffd4e8) = 0 (0x0) writev(0x85,0x81019cc90,0x1,0x1,0xffffffffea994e98,0x810197010) = 129 (0x81) gettimeofday({1448524674.295938 },0x0) = 0 (0x0) gettimeofday({1448524674.295982 },0x0) = 0 (0x0) poll({33/POLLIN|POLLHUP 32/POLLIN|POLLHUP 31/POLLIN|POLLHUP 30/POLLIN|POLLHUP 29/POLLIN|POLLHUP 28/POLLIN|POLLHUP 27/POLLIN|POLLHUP 26/POLLIN|POLLHUP 25/POLLIN|POLLHUP 24/POLLIN|POLLHUP 23/POLLIN|POLLHUP 22/POLLIN|POLLHUP 74/POLLIN|POLLHUP 73/POLLIN|POLLHUP 130/POLLIN|POLLHUP 129/POLLIN|POLLHUP 66/POLLIN|POLLHUP 65/POLLIN|POLLHUP 35/POLLIN|POLLHUP 34/POLLIN|POLLHUP 53/POLLIN|POLLHUP 54/POLLIN|POLLHUP 166/POLLIN|POLLHUP 165/POLLIN|POLLHUP 68/POLLIN|POLLHUP 67/POLLIN|POLLHUP 56/POLLIN|POLLHUP 55/POLLIN|POLLHUP 117/POLLIN|POLLHUP 118/POLLIN|POLLHUP 175/POLLIN|POLLHUP 176/POLLIN|POLLHUP 60/POLLIN|POLLHUP 59/POLLIN|POLLHUP 184/POLLIN|POLLHUP 183/POLLIN|POLLHUP 171/POLLIN|POLLHUP 172/POLLIN|POLLHUP 88/POLLIN|POLLHUP 87/POLLIN|POLLHUP 157/POLLIN|POLLHUP 158/POLLIN|POLLHUP 193/POLLIN|POLLHUP 194/POLLIN|POLLHUP 58/POLLIN|POLLHUP 57/POLLIN|POLLHUP 77/POLLIN|POLLHUP 78/POLLIN|POLLHUP 72/POLLIN|POLLHUP 71/POLLIN|POLLHUP 80/POLLIN|POLLHUP 79/POLLIN|POLLHUP 52/POLLIN|POLLHUP 51/POLLIN|POLLHUP 48/POLLIN|POLLHUP 47/POLLIN|POLLHUP 141/POLLIN|POLLHUP 142/POLLIN|POLLHUP 46/POLLIN|POLLHUP 45/POLLIN|POLLHUP 41/POLLIN|POLLHUP 42/POLLIN|POLLHUP 127/POLLIN|POLLHUP 128/POLLIN|POLLHUP 37/POLLIN|POLLHUP 38/POLLIN|POLLHUP 50/POLLIN|POLLHUP 49/POLLIN|POLLHUP 92/POLLIN|POLLHUP 91/POLLIN|POLLHUP 116/POLLIN|POLLHUP 115/POLLIN|POLLHUP 63/POLLIN|POLLHUP 64/POLLIN|POLLHUP 76/POLLIN|POLLHUP 75/POLLIN|POLLHUP 102/POLLIN|POLLHUP 101/POLLIN|POLLHUP 104/POLLIN|POLLHUP 103/POLLIN|POLLHUP 85/POLLIN|POLLHUP 86/POLLIN|POLLHUP 121/POLLIN|POLLHUP 122/POLLIN|POLLHUP 100/POLLIN|POLLHUP 99/POLLIN|POLLHUP 93/POLLIN|POLLHUP 94/POLLIN|POLLHUP 95/POLLIN|POLLHUP 96/POLLIN|POLLHUP 114/POLLIN|POLLHUP 113/POLLIN|POLLHUP 39/POLLIN|POLLHUP 40/POLLIN|POLLHUP 98/POLLIN|POLLHUP 97/POLLIN|POLLHUP 61/POLLIN|POLLHUP 62/POLLIN|POLLHUP 138/POLLIN|POLLHUP 137/POLLIN|POLLHUP 89/POLLIN|POLLHUP 90/POLLIN|POLLHUP 111/POLLIN|POLLHUP 112/POLLIN|POLLHUP 83/POLLIN|POLLHUP 84/POLLIN|POLLHUP 108/POLLIN|POLLHUP 107/POLLIN|POLLHUP 178/POLLIN|POLLHUP 177/POLLIN|POLLHUP 123/POLLIN|POLLHUP 124/POLLIN|POLLHUP 110/POLLIN|POLLHUP 109/POLLIN|POLLHUP 81/POLLIN|POLLHUP 82/POLLIN|POLLHUP 106/POLLIN|POLLHUP 105/POLLIN|POLLHUP 164/POLLIN|POLLHUP 163/POLLIN|POLLHUP 156/POLLIN|POLLHUP 155/POLLIN|POLLHUP 125/POLLIN|POLLHUP 126/POLLIN|POLLHUP 150/POLLIN|POLLHUP 149/POLLIN|POLLHUP 173/POLLIN|POLLHUP 174/POLLIN|POLLHUP 139/POLLIN|POLLHUP 140/POLLIN|POLLHUP 161/POLLIN|POLLHUP 162/POLLIN|POLLHUP 136/POLLIN|POLLHUP 135/POLLIN|POLLHUP 160/POLLIN|POLLHUP 159/POLLIN|POLLHUP 69/POLLIN|POLLHUP 70/POLLIN|POLLHUP 168/POLLIN|POLLHUP 167/POLLIN|POLLHUP 119/POLLIN|POLLHUP 120/POLLIN|POLLHUP 182/POLLIN|POLLHUP 181/POLLIN|POLLHUP 144/POLLIN|POLLHUP 143/POLLIN|POLLHUP 43/POLLIN|POLLHUP 44/POLLIN|POLLHUP 131/POLLIN|POLLHUP 132/POLLIN|POLLHUP 133/POLLIN|POLLHUP 134/POLLIN|POLLHUP},152,5108) = 1 (0x1) readv(0x85,0x81019cc90,0x1,0x1,0x80860bd82,0x0) = 7 (0x7) gettimeofday({1448524674.297480 },0x0) = 0 (0x0) poll({33/POLLIN|POLLHUP 32/POLLIN|POLLHUP 31/POLLIN|POLLHUP 30/POLLIN|POLLHUP 29/POLLIN|POLLHUP 28/POLLIN|POLLHUP 27/POLLIN|POLLHUP 26/POLLIN|POLLHUP 25/POLLIN|POLLHUP 24/POLLIN|POLLHUP 23/POLLIN|POLLHUP 22/POLLIN|POLLHUP 74/POLLIN|POLLHUP 73/POLLIN|POLLHUP 130/POLLIN|POLLHUP 129/POLLIN|POLLHUP 66/POLLIN|POLLHUP 65/POLLIN|POLLHUP 35/POLLIN|POLLHUP 34/POLLIN|POLLHUP 53/POLLIN|POLLHUP 54/POLLIN|POLLHUP 166/POLLIN|POLLHUP 165/POLLIN|POLLHUP 68/POLLIN|POLLHUP 67/POLLIN|POLLHUP 56/POLLIN|POLLHUP 55/POLLIN|POLLHUP 117/POLLIN|POLLHUP 118/POLLIN|POLLHUP 175/POLLIN|POLLHUP 176/POLLIN|POLLHUP 60/POLLIN|POLLHUP 59/POLLIN|POLLHUP 184/POLLIN|POLLHUP 183/POLLIN|POLLHUP 171/POLLIN|POLLHUP 172/POLLIN|POLLHUP 88/POLLIN|POLLHUP 87/POLLIN|POLLHUP 157/POLLIN|POLLHUP 158/POLLIN|POLLHUP 193/POLLIN|POLLHUP 194/POLLIN|POLLHUP 58/POLLIN|POLLHUP 57/POLLIN|POLLHUP 77/POLLIN|POLLHUP 78/POLLIN|POLLHUP 72/POLLIN|POLLHUP 71/POLLIN|POLLHUP 80/POLLIN|POLLHUP 79/POLLIN|POLLHUP 52/POLLIN|POLLHUP 51/POLLIN|POLLHUP 48/POLLIN|POLLHUP 47/POLLIN|POLLHUP 141/POLLIN|POLLHUP 142/POLLIN|POLLHUP 46/POLLIN|POLLHUP 45/POLLIN|POLLHUP 41/POLLIN|POLLHUP 42/POLLIN|POLLHUP 127/POLLIN|POLLHUP 128/POLLIN|POLLHUP 37/POLLIN|POLLHUP 38/POLLIN|POLLHUP 50/POLLIN|POLLHUP 49/POLLIN|POLLHUP 92/POLLIN|POLLHUP 91/POLLIN|POLLHUP 116/POLLIN|POLLHUP 115/POLLIN|POLLHUP 63/POLLIN|POLLHUP 64/POLLIN|POLLHUP 76/POLLIN|POLLHUP 75/POLLIN|POLLHUP 102/POLLIN|POLLHUP 101/POLLIN|POLLHUP 104/POLLIN|POLLHUP 103/POLLIN|POLLHUP 85/POLLIN|POLLHUP 86/POLLIN|POLLHUP 121/POLLIN|POLLHUP 122/POLLIN|POLLHUP 100/POLLIN|POLLHUP 99/POLLIN|POLLHUP 93/POLLIN|POLLHUP 94/POLLIN|POLLHUP 95/POLLIN|POLLHUP 96/POLLIN|POLLHUP 114/POLLIN|POLLHUP 113/POLLIN|POLLHUP 39/POLLIN|POLLHUP 40/POLLIN|POLLHUP 98/POLLIN|POLLHUP 97/POLLIN|POLLHUP 61/POLLIN|POLLHUP 62/POLLIN|POLLHUP 138/POLLIN|POLLHUP 137/POLLIN|POLLHUP 89/POLLIN|POLLHUP 90/POLLIN|POLLHUP 111/POLLIN|POLLHUP 112/POLLIN|POLLHUP 83/POLLIN|POLLHUP 84/POLLIN|POLLHUP 108/POLLIN|POLLHUP 107/POLLIN|POLLHUP 178/POLLIN|POLLHUP 177/POLLIN|POLLHUP 123/POLLIN|POLLHUP 124/POLLIN|POLLHUP 110/POLLIN|POLLHUP 109/POLLIN|POLLHUP 81/POLLIN|POLLHUP 82/POLLIN|POLLHUP 106/POLLIN|POLLHUP 105/POLLIN|POLLHUP 164/POLLIN|POLLHUP 163/POLLIN|POLLHUP 156/POLLIN|POLLHUP 155/POLLIN|POLLHUP 125/POLLIN|POLLHUP 126/POLLIN|POLLHUP 150/POLLIN|POLLHUP 149/POLLIN|POLLHUP 173/POLLIN|POLLHUP 174/POLLIN|POLLHUP 139/POLLIN|POLLHUP 140/POLLIN|POLLHUP 161/POLLIN|POLLHUP 162/POLLIN|POLLHUP 136/POLLIN|POLLHUP 135/POLLIN|POLLHUP 160/POLLIN|POLLHUP 159/POLLIN|POLLHUP 69/POLLIN|POLLHUP 70/POLLIN|POLLHUP 168/POLLIN|POLLHUP 167/POLLIN|POLLHUP 119/POLLIN|POLLHUP 120/POLLIN|POLLHUP 182/POLLIN|POLLHUP 181/POLLIN|POLLHUP 144/POLLIN|POLLHUP 143/POLLIN|POLLHUP 43/POLLIN|POLLHUP 44/POLLIN|POLLHUP 131/POLLIN|POLLHUP 132/POLLIN|POLLHUP 133/POLLIN|POLLHUP 134/POLLIN|POLLHUP},152,5106) = 1 (0x1) readv(0x85,0x81019cc90,0x1,0x1,0x80860bd82,0x0) = 190 (0xbe) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0) clock_gettime(13,{1448524674.000000000 }) = 0 (0x0)> If you could profile your use case on a linux host with the perf tools, > that may give us some insights. >Unfortunately now I don't have enought free time, but I try test it when it will.>> I think may be samba something doing not optimal way. Like this, >> first >> we try to create directory (mkdir), then we check it (stat). May be >> that >> leads to impact filesystems and samba ldap works slower than >> openldap. >> >> fcntl(15,F_SETLKW,0x7fffffffd118) = 0 (0x0) >> fcntl(15,F_SETLKW,0x7fffffffd188) = 0 (0x0) >> clock_gettime(13,{1448375653.000000000 }) = 0 (0x0) >> mkdir("/var/db/samba4/private/sam.ldb.d",0700) ERR#17 'File exists' >> stat("/var/db/samba4/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=SPEECHPRO >> ,DC=COM.ldb",{ >> mode=-rw------- ,inode=242034,size=4247552,blksize=32768 }) = 0 (0x0) >> fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0) >> fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0) >> fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0) >> fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0) >> fcntl(16,F_SETLKW,0x7fffffffd098) = 0 (0x0) >> fcntl(16,F_SETLKW,0x7fffffffd138) = 0 (0x0) >> clock_gettime(13,{1448375653.000000000 }) = 0 (0x0) >> clock_gettime(13,{1448375653.000000000 }) = 0 (0x0) >> fcntl(16,F_SETLK,0x7fffffffd098) = 0 (0x0) >> fcntl(16,F_SETLKW,0x7fffffffd118) = 0 (0x0) >> fcntl(16,F_SETLKW,0x7fffffffd188) = 0 (0x0) >> clock_gettime(13,{1448375653.000000000 }) = 0 (0x0) >> mkdir("/var/db/samba4/private/sam.ldb.d",0700) ERR#17 'File exists' >> stat("/var/db/samba4/private/sam.ldb.d/DC=FORESTDNSZONES,DC=SPEECHPRO >> ,DC=COM.ldb",{ >> mode=-rw------- ,inode=242035,size=4247552,blksize=32768 }) = 0 (0x0) >> fcntl(17,F_SETLKW,0x7fffffffd098) = 0 (0x0) >> fcntl(17,F_SETLKW,0x7fffffffd138) = 0 (0x0) >> fcntl(17,F_SETLKW,0x7fffffffd098) >> > These really should happen only once per startup of the process, how > often do you see it? >I see it about every seconds with any files in /var/db/samba4/private/sam.ldb.d directory.>> 23.11.2015 18:44, mathias dufresne пишет: >>> Hi, >>> >>> I feel like I must reply off-list: Samba 4 is not yet able to use >>> threads. >>> To work around that we are about to deploy our DCs as virtual >>> machines, to be able to multiply the number of Samba running. >>> >>> I expect you would have already thought about that, but in case you >>> didn't, I said it :p >>> >>> I will follow closely that topic you opened. Sorry to not help >>> more. >>> >>> Cheers, >>> >>> mathias >>> >>> >>> >>> 2015-11-23 14:50 GMT+01:00 Yuriy Tabolin <tabolin at speechpro.com >>> <mailto:tabolin at speechpro.com>>: >>> >>> Hi all. >>> I have samba 4.2.3 on freebsd 10.1 server. There are three DC >>> and >>> about 350 PC on domain. DCs have 3 CPU and 3GB RAM. Some >>> servers >>> with services like apache, exim, dovecot, etc use samba4 ldap >>> (port 389) for user authentication. Some times ago after adding >>> some services to use ldap I found, that samba4 cannot serve all >>> ldap requests. Every 10-30 minutes I see in DCs logs: >>> dc1 kernel: sonewconn: pcb 0xfffff800753d6ab8: Listen queue >>> overflow: 16 already in queue awaiting acceptance (28 >>> occurrences) >>> >>> After that I have used tcpdump for recording ldap-traffic and >>> have >>> seen that after TCP handshaking, server some times suddenly >>> send >>> TCP-RST to close connection. I have enlarged DCs resources (CPU >>> and RAM), kern.ipc.somaxconn, did some other system tuning but >>> all that didn't help. Load average on DCs permanently near 0.9 >>> -1.0 >>> and samba cannot serve all ldap conncetions. ldap clients works >>> well because they use as minimum as two domain controllers as >>> ldap >>> servers. Is there a some performance problem in samba4, slow >>> processing ldap requests or something else? >>> Thanks for any help. >>> >>> -- >>> With best regards, >>> >>> Tabolin Yuriy >>> System administrator >>> Speech Technology Center >>> >>> >>> -- >>> To unsubscribe from this list go to the following URL and read >>> the >>> instructions: https://lists.samba.org/mailman/options/samba >>> >>> >> >> -- >> With best regards, >> >> Tabolin Yuriy >> System administrator >> Speech Technology Center >>-- With best regards, Tabolin Yuriy System administrator Speech Technology Center
Apparently Analagous Threads
- samba4 ldap high load and port queue overflow
- Migration from 3.6.25-0ubuntu0.12.04.10 to 4.x with passdb backend = ldapsam
- Fwd: samba & Oracle ACFS Issues
- samba4 schema for openldap
- Migration from 3.6.25-0ubuntu0.12.04.10 to 4.x with passdb backend = ldapsam