thr3ads.net - samba - [Samba] samba4 ldap high load and port queue overflow [Nov 2015]

If this information is useful, please help other people find it:
Share via:

Yuriy Tabolin

2015-Nov-24 14:48 UTC

[Samba] samba4 ldap high load and port queue overflow

Thanks for answer. I am very sad to see that.
I had examined samba behavior with tracer and had seen too many poll, 
fcntl and stat system calls by samba task[ldapsrv] process

Elapsed Times for PID 1423,

          SYSCALL          TIME (ns)
              dup               2653
            flock               5811
            lseek              33513
          geteuid              42747
           getpid              63819
      getpeername              85480
      getsockname             151683
           socket             237632
           accept             311871
            fstat             404420
          madvise             452120
            readv             601524
             mmap             858391
           writev             910337
     gettimeofday             929595
             read            1064811
            close            1270401
             open            1733182
            lstat            1938394
             bind            1939462
           unlink            2074873
           munmap            2569299
            mkdir            3539965
    clock_gettime            5662586
             stat            7543724
            fcntl           74322331
             poll         5393911015


I think may be samba something doing not optimal way. Like this, first 
we try to create directory (mkdir), then we check it (stat). May be that 
leads to impact filesystems and samba ldap works slower than openldap.

fcntl(15,F_SETLKW,0x7fffffffd118)                = 0 (0x0)
fcntl(15,F_SETLKW,0x7fffffffd188)                = 0 (0x0)
clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
mkdir("/var/db/samba4/private/sam.ldb.d",0700)   ERR#17 'File
exists'
stat("/var/db/samba4/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=SPEECHPRO,DC=COM.ldb",{
mode=-rw------- ,inode=242034,size=4247552,blksize=32768 }) = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
fcntl(16,F_SETLK,0x7fffffffd098)                 = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd118)                = 0 (0x0)
fcntl(16,F_SETLKW,0x7fffffffd188)                = 0 (0x0)
clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
mkdir("/var/db/samba4/private/sam.ldb.d",0700)   ERR#17 'File
exists'
stat("/var/db/samba4/private/sam.ldb.d/DC=FORESTDNSZONES,DC=SPEECHPRO,DC=COM.ldb",{
mode=-rw------- ,inode=242035,size=4247552,blksize=32768 }) = 0 (0x0)
fcntl(17,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
fcntl(17,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
fcntl(17,F_SETLKW,0x7fffffffd098)




23.11.2015 18:44, mathias dufresne пишет:> Hi,
>
> I feel like I must reply off-list: Samba 4 is not yet able to use threads.
> To work around that we are about to deploy our DCs as virtual 
> machines, to be able to multiply the number of Samba running.
>
> I expect you would have already thought about that, but in case you 
> didn't, I said it :p
>
> I will follow closely that topic you opened. Sorry to not help more.
>
> Cheers,
>
> mathias
>
>
>
> 2015-11-23 14:50 GMT+01:00 Yuriy Tabolin <tabolin at speechpro.com 
> <mailto:tabolin at speechpro.com>>:
>
>     Hi all.
>     I have samba 4.2.3 on freebsd 10.1 server. There are three DC and
>     about 350 PC on domain. DCs have 3 CPU and 3GB RAM. Some servers
>     with services like apache, exim, dovecot, etc use samba4 ldap
>     (port 389) for user authentication. Some times ago after adding
>     some services to use ldap I found, that samba4 cannot serve all
>     ldap requests. Every 10-30 minutes I see in DCs logs:
>     dc1 kernel: sonewconn: pcb 0xfffff800753d6ab8: Listen queue
>     overflow: 16 already in queue awaiting acceptance (28 occurrences)
>
>     After that I have used tcpdump for recording ldap-traffic and have
>     seen that after TCP handshaking, server some times suddenly send
>     TCP-RST to close connection. I have enlarged DCs resources (CPU
>     and RAM),  kern.ipc.somaxconn, did some other system tuning but
>     all that didn't help. Load average on DCs permanently near 0.9-1.0
>     and samba cannot serve all ldap conncetions. ldap clients works
>     well because they use as minimum as two domain controllers as ldap
>     servers. Is there a some performance problem in samba4, slow
>     processing ldap requests or something else?
>     Thanks for any help.
>
>     -- 
>     With best regards,
>
>     Tabolin Yuriy
>     System administrator
>     Speech Technology Center
>
>
>     -- 
>     To unsubscribe from this list go to the following URL and read the
>     instructions: https://lists.samba.org/mailman/options/samba
>
>

-- 
With best regards,

Tabolin Yuriy
System administrator
Speech Technology Center

Andrew Bartlett

2015-Nov-25 07:59 UTC

head link

[Samba] samba4 ldap high load and port queue overflow

On Tue, 2015-11-24 at 17:48 +0300, Yuriy Tabolin wrote:> Thanks for answer. I am very sad to see that.
> I had examined samba behavior with tracer and had seen too many poll,
> fcntl and stat system calls by samba task[ldapsrv] process
> 
> Elapsed Times for PID 1423,
> 
>           SYSCALL          TIME (ns)
>               dup               2653
>             flock               5811
>             lseek              33513
>           geteuid              42747
>            getpid              63819
>       getpeername              85480
>       getsockname             151683
>            socket             237632
>            accept             311871
>             fstat             404420
>           madvise             452120
>             readv             601524
>              mmap             858391
>            writev             910337
>      gettimeofday             929595
>              read            1064811
>             close            1270401
>              open            1733182
>             lstat            1938394
>              bind            1939462
>            unlink            2074873
>            munmap            2569299
>             mkdir            3539965
>     clock_gettime            5662586
>              stat            7543724
>             fcntl           74322331
>              poll         5393911015
That poll() count is very high, I do wonder if something isn't working
right with your system.  I would have expected it to be more in line
with fcntl(), which we use a lot of in TDB. 

If you could profile your use case on a linux host with the perf tools,
that may give us some insights. 
> I think may be samba something doing not optimal way. Like this,
> first 
> we try to create directory (mkdir), then we check it (stat). May be
> that 
> leads to impact filesystems and samba ldap works slower than
> openldap.
> 
> fcntl(15,F_SETLKW,0x7fffffffd118)                = 0 (0x0)
> fcntl(15,F_SETLKW,0x7fffffffd188)                = 0 (0x0)
> clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
> mkdir("/var/db/samba4/private/sam.ldb.d",0700)   ERR#17 'File
exists'
> stat("/var/db/samba4/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=SPEECHPRO
> ,DC=COM.ldb",{ 
> mode=-rw------- ,inode=242034,size=4247552,blksize=32768 }) = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
> clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
> clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
> fcntl(16,F_SETLK,0x7fffffffd098)                 = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd118)                = 0 (0x0)
> fcntl(16,F_SETLKW,0x7fffffffd188)                = 0 (0x0)
> clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
> mkdir("/var/db/samba4/private/sam.ldb.d",0700)   ERR#17 'File
exists'
> stat("/var/db/samba4/private/sam.ldb.d/DC=FORESTDNSZONES,DC=SPEECHPRO
> ,DC=COM.ldb",{ 
> mode=-rw------- ,inode=242035,size=4247552,blksize=32768 }) = 0 (0x0)
> fcntl(17,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
> fcntl(17,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
> fcntl(17,F_SETLKW,0x7fffffffd098)
> 
These really should happen only once per startup of the process, how
often do you see it?
> 23.11.2015 18:44, mathias dufresne пишет:
> > Hi,
> > 
> > I feel like I must reply off-list: Samba 4 is not yet able to use
> > threads.
> > To work around that we are about to deploy our DCs as virtual 
> > machines, to be able to multiply the number of Samba running.
> > 
> > I expect you would have already thought about that, but in case you
> > didn't, I said it :p
> > 
> > I will follow closely that topic you opened. Sorry to not help
> > more.
> > 
> > Cheers,
> > 
> > mathias
> > 
> > 
> > 
> > 2015-11-23 14:50 GMT+01:00 Yuriy Tabolin <tabolin at speechpro.com 
> > <mailto:tabolin at speechpro.com>>:
> > 
> >     Hi all.
> >     I have samba 4.2.3 on freebsd 10.1 server. There are three DC
> > and
> >     about 350 PC on domain. DCs have 3 CPU and 3GB RAM. Some
> > servers
> >     with services like apache, exim, dovecot, etc use samba4 ldap
> >     (port 389) for user authentication. Some times ago after adding
> >     some services to use ldap I found, that samba4 cannot serve all
> >     ldap requests. Every 10-30 minutes I see in DCs logs:
> >     dc1 kernel: sonewconn: pcb 0xfffff800753d6ab8: Listen queue
> >     overflow: 16 already in queue awaiting acceptance (28
> > occurrences)
> > 
> >     After that I have used tcpdump for recording ldap-traffic and
> > have
> >     seen that after TCP handshaking, server some times suddenly
> > send
> >     TCP-RST to close connection. I have enlarged DCs resources (CPU
> >     and RAM),  kern.ipc.somaxconn, did some other system tuning but
> >     all that didn't help. Load average on DCs permanently near 0.9
> > -1.0
> >     and samba cannot serve all ldap conncetions. ldap clients works
> >     well because they use as minimum as two domain controllers as
> > ldap
> >     servers. Is there a some performance problem in samba4, slow
> >     processing ldap requests or something else?
> >     Thanks for any help.
> > 
> >     -- 
> >     With best regards,
> > 
> >     Tabolin Yuriy
> >     System administrator
> >     Speech Technology Center
> > 
> > 
> >     -- 
> >     To unsubscribe from this list go to the following URL and read
> > the
> >     instructions: https://lists.samba.org/mailman/options/samba
> > 
> > 
> 
> 
> -- 
> With best regards,
> 
> Tabolin Yuriy
> System administrator
> Speech Technology Center
> -- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

Yuriy Tabolin

2015-Nov-26 08:52 UTC

head link

[Samba] samba4 ldap high load and port queue overflow

25.11.2015 10:59, Andrew Bartlett пишет:> On Tue, 2015-11-24 at 17:48 +0300, Yuriy Tabolin wrote:
>> Thanks for answer. I am very sad to see that.
>> I had examined samba behavior with tracer and had seen too many poll,
>> fcntl and stat system calls by samba task[ldapsrv] process
>>
>> Elapsed Times for PID 1423,
>>
>>            SYSCALL          TIME (ns)
>>                dup               2653
>>              flock               5811
>>              lseek              33513
>>            geteuid              42747
>>             getpid              63819
>>        getpeername              85480
>>        getsockname             151683
>>             socket             237632
>>             accept             311871
>>              fstat             404420
>>            madvise             452120
>>              readv             601524
>>               mmap             858391
>>             writev             910337
>>       gettimeofday             929595
>>               read            1064811
>>              close            1270401
>>               open            1733182
>>              lstat            1938394
>>               bind            1939462
>>             unlink            2074873
>>             munmap            2569299
>>              mkdir            3539965
>>      clock_gettime            5662586
>>               stat            7543724
>>              fcntl           74322331
>>               poll         5393911015
> That poll() count is very high, I do wonder if something isn't working
> right with your system.  I would have expected it to be more in line
> with fcntl(), which we use a lot of in TDB.I think that poll calls are too much too. Trace look that

fcntl(13,F_SETLKW,0x7fffffffdda8)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffde48)                = 0 (0x0)
writev(0x85,0x81019cc90,0x1,0x1,0xffffffffea994e98,0x810197010) = 251 (0xfb)
gettimeofday({1448524674.289517 },0x0)           = 0 (0x0)
gettimeofday({1448524674.289624 },0x0)           = 0 (0x0)
poll({33/POLLIN|POLLHUP 32/POLLIN|POLLHUP 31/POLLIN|POLLHUP 
30/POLLIN|POLLHUP 29/POLLIN|POLLHUP 28/POLLIN|POLLHUP 27/POLLIN|POLLHUP 
26/POLLIN|POLLHUP 25/POLLIN|POLLHUP 24/POLLIN|POLLHUP 23/POLLIN|POLLHUP 
22/POLLIN|POLLHUP 74/POLLIN|POLLHUP 73/POLLIN|POLLHUP 130/POLLIN|POLLHUP 
129/POLLIN|POLLHUP 66/POLLIN|POLLHUP 65/POLLIN|POLLHUP 35/POLLIN|POLLHUP 
34/POLLIN|POLLHUP 53/POLLIN|POLLHUP 54/POLLIN|POLLHUP 166/POLLIN|POLLHUP 
165/POLLIN|POLLHUP 68/POLLIN|POLLHUP 67/POLLIN|POLLHUP 56/POLLIN|POLLHUP 
55/POLLIN|POLLHUP 117/POLLIN|POLLHUP 118/POLLIN|POLLHUP 
175/POLLIN|POLLHUP 176/POLLIN|POLLHUP 60/POLLIN|POLLHUP 
59/POLLIN|POLLHUP 184/POLLIN|POLLHUP 183/POLLIN|POLLHUP 
171/POLLIN|POLLHUP 172/POLLIN|POLLHUP 88/POLLIN|POLLHUP 
87/POLLIN|POLLHUP 157/POLLIN|POLLHUP 158/POLLIN|POLLHUP 
193/POLLIN|POLLHUP 194/POLLIN|POLLHUP 58/POLLIN|POLLHUP 
57/POLLIN|POLLHUP 77/POLLIN|POLLHUP 78/POLLIN|POLLHUP 72/POLLIN|POLLHUP 
71/POLLIN|POLLHUP 80/POLLIN|POLLHUP 79/POLLIN|POLLHUP 52/POLLIN|POLLHUP 
51/POLLIN|POLLHUP 48/POLLIN|POLLHUP 47/POLLIN|POLLHUP 141/POLLIN|POLLHUP 
142/POLLIN|POLLHUP 46/POLLIN|POLLHUP 45/POLLIN|POLLHUP 41/POLLIN|POLLHUP 
42/POLLIN|POLLHUP 127/POLLIN|POLLHUP 128/POLLIN|POLLHUP 
37/POLLIN|POLLHUP 38/POLLIN|POLLHUP 50/POLLIN|POLLHUP 49/POLLIN|POLLHUP 
92/POLLIN|POLLHUP 91/POLLIN|POLLHUP 116/POLLIN|POLLHUP 
115/POLLIN|POLLHUP 63/POLLIN|POLLHUP 64/POLLIN|POLLHUP 76/POLLIN|POLLHUP 
75/POLLIN|POLLHUP 102/POLLIN|POLLHUP 101/POLLIN|POLLHUP 
104/POLLIN|POLLHUP 103/POLLIN|POLLHUP 85/POLLIN|POLLHUP 
86/POLLIN|POLLHUP 121/POLLIN|POLLHUP 122/POLLIN|POLLHUP 
100/POLLIN|POLLHUP 99/POLLIN|POLLHUP 93/POLLIN|POLLHUP 94/POLLIN|POLLHUP 
95/POLLIN|POLLHUP 96/POLLIN|POLLHUP 114/POLLIN|POLLHUP 
113/POLLIN|POLLHUP 39/POLLIN|POLLHUP 40/POLLIN|POLLHUP 98/POLLIN|POLLHUP 
97/POLLIN|POLLHUP 61/POLLIN|POLLHUP 62/POLLIN|POLLHUP 138/POLLIN|POLLHUP 
137/POLLIN|POLLHUP 89/POLLIN|POLLHUP 90/POLLIN|POLLHUP 
111/POLLIN|POLLHUP 112/POLLIN|POLLHUP 83/POLLIN|POLLHUP 
84/POLLIN|POLLHUP 108/POLLIN|POLLHUP 107/POLLIN|POLLHUP 
178/POLLIN|POLLHUP 177/POLLIN|POLLHUP 123/POLLIN|POLLHUP 
124/POLLIN|POLLHUP 110/POLLIN|POLLHUP 109/POLLIN|POLLHUP 
81/POLLIN|POLLHUP 82/POLLIN|POLLHUP 106/POLLIN|POLLHUP 
105/POLLIN|POLLHUP 164/POLLIN|POLLHUP 163/POLLIN|POLLHUP 
156/POLLIN|POLLHUP 155/POLLIN|POLLHUP 125/POLLIN|POLLHUP 
126/POLLIN|POLLHUP 150/POLLIN|POLLHUP 149/POLLIN|POLLHUP 
173/POLLIN|POLLHUP 174/POLLIN|POLLHUP 139/POLLIN|POLLHUP 
140/POLLIN|POLLHUP 161/POLLIN|POLLHUP 162/POLLIN|POLLHUP 
136/POLLIN|POLLHUP 135/POLLIN|POLLHUP 160/POLLIN|POLLHUP 
159/POLLIN|POLLHUP 69/POLLIN|POLLHUP 70/POLLIN|POLLHUP 
168/POLLIN|POLLHUP 167/POLLIN|POLLHUP 119/POLLIN|POLLHUP 
120/POLLIN|POLLHUP 182/POLLIN|POLLHUP 181/POLLIN|POLLHUP 
144/POLLIN|POLLHUP 143/POLLIN|POLLHUP 43/POLLIN|POLLHUP 
44/POLLIN|POLLHUP 131/POLLIN|POLLHUP 132/POLLIN|POLLHUP 
133/POLLIN|POLLHUP 134/POLLIN|POLLHUP},152,5114) = 1 (0x1)
readv(0x85,0x81019cc90,0x1,0x1,0x80860bd82,0x0)  = 7 (0x7)
gettimeofday({1448524674.290357 },0x0)           = 0 (0x0)
poll({33/POLLIN|POLLHUP 32/POLLIN|POLLHUP 31/POLLIN|POLLHUP 
30/POLLIN|POLLHUP 29/POLLIN|POLLHUP 28/POLLIN|POLLHUP 27/POLLIN|POLLHUP 
26/POLLIN|POLLHUP 25/POLLIN|POLLHUP 24/POLLIN|POLLHUP 23/POLLIN|POLLHUP 
22/POLLIN|POLLHUP 74/POLLIN|POLLHUP 73/POLLIN|POLLHUP 130/POLLIN|POLLHUP 
129/POLLIN|POLLHUP 66/POLLIN|POLLHUP 65/POLLIN|POLLHUP 35/POLLIN|POLLHUP 
34/POLLIN|POLLHUP 53/POLLIN|POLLHUP 54/POLLIN|POLLHUP 166/POLLIN|POLLHUP 
165/POLLIN|POLLHUP 68/POLLIN|POLLHUP 67/POLLIN|POLLHUP 56/POLLIN|POLLHUP 
55/POLLIN|POLLHUP 117/POLLIN|POLLHUP 118/POLLIN|POLLHUP 
175/POLLIN|POLLHUP 176/POLLIN|POLLHUP 60/POLLIN|POLLHUP 
59/POLLIN|POLLHUP 184/POLLIN|POLLHUP 183/POLLIN|POLLHUP 
171/POLLIN|POLLHUP 172/POLLIN|POLLHUP 88/POLLIN|POLLHUP 
87/POLLIN|POLLHUP 157/POLLIN|POLLHUP 158/POLLIN|POLLHUP 
193/POLLIN|POLLHUP 194/POLLIN|POLLHUP 58/POLLIN|POLLHUP 
57/POLLIN|POLLHUP 77/POLLIN|POLLHUP 78/POLLIN|POLLHUP 72/POLLIN|POLLHUP 
71/POLLIN|POLLHUP 80/POLLIN|POLLHUP 79/POLLIN|POLLHUP 52/POLLIN|POLLHUP 
51/POLLIN|POLLHUP 48/POLLIN|POLLHUP 47/POLLIN|POLLHUP 141/POLLIN|POLLHUP 
142/POLLIN|POLLHUP 46/POLLIN|POLLHUP 45/POLLIN|POLLHUP 41/POLLIN|POLLHUP 
42/POLLIN|POLLHUP 127/POLLIN|POLLHUP 128/POLLIN|POLLHUP 
37/POLLIN|POLLHUP 38/POLLIN|POLLHUP 50/POLLIN|POLLHUP 49/POLLIN|POLLHUP 
92/POLLIN|POLLHUP 91/POLLIN|POLLHUP 116/POLLIN|POLLHUP 
115/POLLIN|POLLHUP 63/POLLIN|POLLHUP 64/POLLIN|POLLHUP 76/POLLIN|POLLHUP 
75/POLLIN|POLLHUP 102/POLLIN|POLLHUP 101/POLLIN|POLLHUP 
104/POLLIN|POLLHUP 103/POLLIN|POLLHUP 85/POLLIN|POLLHUP 
86/POLLIN|POLLHUP 121/POLLIN|POLLHUP 122/POLLIN|POLLHUP 
100/POLLIN|POLLHUP 99/POLLIN|POLLHUP 93/POLLIN|POLLHUP 94/POLLIN|POLLHUP 
95/POLLIN|POLLHUP 96/POLLIN|POLLHUP 114/POLLIN|POLLHUP 
113/POLLIN|POLLHUP 39/POLLIN|POLLHUP 40/POLLIN|POLLHUP 98/POLLIN|POLLHUP 
97/POLLIN|POLLHUP 61/POLLIN|POLLHUP 62/POLLIN|POLLHUP 138/POLLIN|POLLHUP 
137/POLLIN|POLLHUP 89/POLLIN|POLLHUP 90/POLLIN|POLLHUP 
111/POLLIN|POLLHUP 112/POLLIN|POLLHUP 83/POLLIN|POLLHUP 
84/POLLIN|POLLHUP 108/POLLIN|POLLHUP 107/POLLIN|POLLHUP 
178/POLLIN|POLLHUP 177/POLLIN|POLLHUP 123/POLLIN|POLLHUP 
124/POLLIN|POLLHUP 110/POLLIN|POLLHUP 109/POLLIN|POLLHUP 
81/POLLIN|POLLHUP 82/POLLIN|POLLHUP 106/POLLIN|POLLHUP 
105/POLLIN|POLLHUP 164/POLLIN|POLLHUP 163/POLLIN|POLLHUP 
156/POLLIN|POLLHUP 155/POLLIN|POLLHUP 125/POLLIN|POLLHUP 
126/POLLIN|POLLHUP 150/POLLIN|POLLHUP 149/POLLIN|POLLHUP 
173/POLLIN|POLLHUP 174/POLLIN|POLLHUP 139/POLLIN|POLLHUP 
140/POLLIN|POLLHUP 161/POLLIN|POLLHUP 162/POLLIN|POLLHUP 
136/POLLIN|POLLHUP 135/POLLIN|POLLHUP 160/POLLIN|POLLHUP 
159/POLLIN|POLLHUP 69/POLLIN|POLLHUP 70/POLLIN|POLLHUP 
168/POLLIN|POLLHUP 167/POLLIN|POLLHUP 119/POLLIN|POLLHUP 
120/POLLIN|POLLHUP 182/POLLIN|POLLHUP 181/POLLIN|POLLHUP 
144/POLLIN|POLLHUP 143/POLLIN|POLLHUP 43/POLLIN|POLLHUP 
44/POLLIN|POLLHUP 131/POLLIN|POLLHUP 132/POLLIN|POLLHUP 
133/POLLIN|POLLHUP 134/POLLIN|POLLHUP},152,5114) = 1 (0x1)
readv(0x85,0x81019cc90,0x1,0x1,0x80860bd82,0x0)  = 129 (0x81)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffcd48)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffcde8)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffcc78)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffcd18)                = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffccc8)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffcd68)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffcbf8)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffcc98)                = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffcf48)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffcfe8)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffce78)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffcf18)                = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffc5e8)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffc688)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffc518)                = 0 (0x0)
fcntl(14,F_SETLKW,0x7fffffffc5b8)                = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffcef8)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffcf98)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffce28)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffcec8)                = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffce78)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffcf18)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffcda8)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffce48)                = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffde78)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffdf18)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffdda8)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffde48)                = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffd518)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffd5b8)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffd448)                = 0 (0x0)
fcntl(13,F_SETLKW,0x7fffffffd4e8)                = 0 (0x0)
writev(0x85,0x81019cc90,0x1,0x1,0xffffffffea994e98,0x810197010) = 129 (0x81)
gettimeofday({1448524674.295938 },0x0)           = 0 (0x0)
gettimeofday({1448524674.295982 },0x0)           = 0 (0x0)
poll({33/POLLIN|POLLHUP 32/POLLIN|POLLHUP 31/POLLIN|POLLHUP 
30/POLLIN|POLLHUP 29/POLLIN|POLLHUP 28/POLLIN|POLLHUP 27/POLLIN|POLLHUP 
26/POLLIN|POLLHUP 25/POLLIN|POLLHUP 24/POLLIN|POLLHUP 23/POLLIN|POLLHUP 
22/POLLIN|POLLHUP 74/POLLIN|POLLHUP 73/POLLIN|POLLHUP 130/POLLIN|POLLHUP 
129/POLLIN|POLLHUP 66/POLLIN|POLLHUP 65/POLLIN|POLLHUP 35/POLLIN|POLLHUP 
34/POLLIN|POLLHUP 53/POLLIN|POLLHUP 54/POLLIN|POLLHUP 166/POLLIN|POLLHUP 
165/POLLIN|POLLHUP 68/POLLIN|POLLHUP 67/POLLIN|POLLHUP 56/POLLIN|POLLHUP 
55/POLLIN|POLLHUP 117/POLLIN|POLLHUP 118/POLLIN|POLLHUP 
175/POLLIN|POLLHUP 176/POLLIN|POLLHUP 60/POLLIN|POLLHUP 
59/POLLIN|POLLHUP 184/POLLIN|POLLHUP 183/POLLIN|POLLHUP 
171/POLLIN|POLLHUP 172/POLLIN|POLLHUP 88/POLLIN|POLLHUP 
87/POLLIN|POLLHUP 157/POLLIN|POLLHUP 158/POLLIN|POLLHUP 
193/POLLIN|POLLHUP 194/POLLIN|POLLHUP 58/POLLIN|POLLHUP 
57/POLLIN|POLLHUP 77/POLLIN|POLLHUP 78/POLLIN|POLLHUP 72/POLLIN|POLLHUP 
71/POLLIN|POLLHUP 80/POLLIN|POLLHUP 79/POLLIN|POLLHUP 52/POLLIN|POLLHUP 
51/POLLIN|POLLHUP 48/POLLIN|POLLHUP 47/POLLIN|POLLHUP 141/POLLIN|POLLHUP 
142/POLLIN|POLLHUP 46/POLLIN|POLLHUP 45/POLLIN|POLLHUP 41/POLLIN|POLLHUP 
42/POLLIN|POLLHUP 127/POLLIN|POLLHUP 128/POLLIN|POLLHUP 
37/POLLIN|POLLHUP 38/POLLIN|POLLHUP 50/POLLIN|POLLHUP 49/POLLIN|POLLHUP 
92/POLLIN|POLLHUP 91/POLLIN|POLLHUP 116/POLLIN|POLLHUP 
115/POLLIN|POLLHUP 63/POLLIN|POLLHUP 64/POLLIN|POLLHUP 76/POLLIN|POLLHUP 
75/POLLIN|POLLHUP 102/POLLIN|POLLHUP 101/POLLIN|POLLHUP 
104/POLLIN|POLLHUP 103/POLLIN|POLLHUP 85/POLLIN|POLLHUP 
86/POLLIN|POLLHUP 121/POLLIN|POLLHUP 122/POLLIN|POLLHUP 
100/POLLIN|POLLHUP 99/POLLIN|POLLHUP 93/POLLIN|POLLHUP 94/POLLIN|POLLHUP 
95/POLLIN|POLLHUP 96/POLLIN|POLLHUP 114/POLLIN|POLLHUP 
113/POLLIN|POLLHUP 39/POLLIN|POLLHUP 40/POLLIN|POLLHUP 98/POLLIN|POLLHUP 
97/POLLIN|POLLHUP 61/POLLIN|POLLHUP 62/POLLIN|POLLHUP 138/POLLIN|POLLHUP 
137/POLLIN|POLLHUP 89/POLLIN|POLLHUP 90/POLLIN|POLLHUP 
111/POLLIN|POLLHUP 112/POLLIN|POLLHUP 83/POLLIN|POLLHUP 
84/POLLIN|POLLHUP 108/POLLIN|POLLHUP 107/POLLIN|POLLHUP 
178/POLLIN|POLLHUP 177/POLLIN|POLLHUP 123/POLLIN|POLLHUP 
124/POLLIN|POLLHUP 110/POLLIN|POLLHUP 109/POLLIN|POLLHUP 
81/POLLIN|POLLHUP 82/POLLIN|POLLHUP 106/POLLIN|POLLHUP 
105/POLLIN|POLLHUP 164/POLLIN|POLLHUP 163/POLLIN|POLLHUP 
156/POLLIN|POLLHUP 155/POLLIN|POLLHUP 125/POLLIN|POLLHUP 
126/POLLIN|POLLHUP 150/POLLIN|POLLHUP 149/POLLIN|POLLHUP 
173/POLLIN|POLLHUP 174/POLLIN|POLLHUP 139/POLLIN|POLLHUP 
140/POLLIN|POLLHUP 161/POLLIN|POLLHUP 162/POLLIN|POLLHUP 
136/POLLIN|POLLHUP 135/POLLIN|POLLHUP 160/POLLIN|POLLHUP 
159/POLLIN|POLLHUP 69/POLLIN|POLLHUP 70/POLLIN|POLLHUP 
168/POLLIN|POLLHUP 167/POLLIN|POLLHUP 119/POLLIN|POLLHUP 
120/POLLIN|POLLHUP 182/POLLIN|POLLHUP 181/POLLIN|POLLHUP 
144/POLLIN|POLLHUP 143/POLLIN|POLLHUP 43/POLLIN|POLLHUP 
44/POLLIN|POLLHUP 131/POLLIN|POLLHUP 132/POLLIN|POLLHUP 
133/POLLIN|POLLHUP 134/POLLIN|POLLHUP},152,5108) = 1 (0x1)
readv(0x85,0x81019cc90,0x1,0x1,0x80860bd82,0x0)  = 7 (0x7)
gettimeofday({1448524674.297480 },0x0)           = 0 (0x0)
poll({33/POLLIN|POLLHUP 32/POLLIN|POLLHUP 31/POLLIN|POLLHUP 
30/POLLIN|POLLHUP 29/POLLIN|POLLHUP 28/POLLIN|POLLHUP 27/POLLIN|POLLHUP 
26/POLLIN|POLLHUP 25/POLLIN|POLLHUP 24/POLLIN|POLLHUP 23/POLLIN|POLLHUP 
22/POLLIN|POLLHUP 74/POLLIN|POLLHUP 73/POLLIN|POLLHUP 130/POLLIN|POLLHUP 
129/POLLIN|POLLHUP 66/POLLIN|POLLHUP 65/POLLIN|POLLHUP 35/POLLIN|POLLHUP 
34/POLLIN|POLLHUP 53/POLLIN|POLLHUP 54/POLLIN|POLLHUP 166/POLLIN|POLLHUP 
165/POLLIN|POLLHUP 68/POLLIN|POLLHUP 67/POLLIN|POLLHUP 56/POLLIN|POLLHUP 
55/POLLIN|POLLHUP 117/POLLIN|POLLHUP 118/POLLIN|POLLHUP 
175/POLLIN|POLLHUP 176/POLLIN|POLLHUP 60/POLLIN|POLLHUP 
59/POLLIN|POLLHUP 184/POLLIN|POLLHUP 183/POLLIN|POLLHUP 
171/POLLIN|POLLHUP 172/POLLIN|POLLHUP 88/POLLIN|POLLHUP 
87/POLLIN|POLLHUP 157/POLLIN|POLLHUP 158/POLLIN|POLLHUP 
193/POLLIN|POLLHUP 194/POLLIN|POLLHUP 58/POLLIN|POLLHUP 
57/POLLIN|POLLHUP 77/POLLIN|POLLHUP 78/POLLIN|POLLHUP 72/POLLIN|POLLHUP 
71/POLLIN|POLLHUP 80/POLLIN|POLLHUP 79/POLLIN|POLLHUP 52/POLLIN|POLLHUP 
51/POLLIN|POLLHUP 48/POLLIN|POLLHUP 47/POLLIN|POLLHUP 141/POLLIN|POLLHUP 
142/POLLIN|POLLHUP 46/POLLIN|POLLHUP 45/POLLIN|POLLHUP 41/POLLIN|POLLHUP 
42/POLLIN|POLLHUP 127/POLLIN|POLLHUP 128/POLLIN|POLLHUP 
37/POLLIN|POLLHUP 38/POLLIN|POLLHUP 50/POLLIN|POLLHUP 49/POLLIN|POLLHUP 
92/POLLIN|POLLHUP 91/POLLIN|POLLHUP 116/POLLIN|POLLHUP 
115/POLLIN|POLLHUP 63/POLLIN|POLLHUP 64/POLLIN|POLLHUP 76/POLLIN|POLLHUP 
75/POLLIN|POLLHUP 102/POLLIN|POLLHUP 101/POLLIN|POLLHUP 
104/POLLIN|POLLHUP 103/POLLIN|POLLHUP 85/POLLIN|POLLHUP 
86/POLLIN|POLLHUP 121/POLLIN|POLLHUP 122/POLLIN|POLLHUP 
100/POLLIN|POLLHUP 99/POLLIN|POLLHUP 93/POLLIN|POLLHUP 94/POLLIN|POLLHUP 
95/POLLIN|POLLHUP 96/POLLIN|POLLHUP 114/POLLIN|POLLHUP 
113/POLLIN|POLLHUP 39/POLLIN|POLLHUP 40/POLLIN|POLLHUP 98/POLLIN|POLLHUP 
97/POLLIN|POLLHUP 61/POLLIN|POLLHUP 62/POLLIN|POLLHUP 138/POLLIN|POLLHUP 
137/POLLIN|POLLHUP 89/POLLIN|POLLHUP 90/POLLIN|POLLHUP 
111/POLLIN|POLLHUP 112/POLLIN|POLLHUP 83/POLLIN|POLLHUP 
84/POLLIN|POLLHUP 108/POLLIN|POLLHUP 107/POLLIN|POLLHUP 
178/POLLIN|POLLHUP 177/POLLIN|POLLHUP 123/POLLIN|POLLHUP 
124/POLLIN|POLLHUP 110/POLLIN|POLLHUP 109/POLLIN|POLLHUP 
81/POLLIN|POLLHUP 82/POLLIN|POLLHUP 106/POLLIN|POLLHUP 
105/POLLIN|POLLHUP 164/POLLIN|POLLHUP 163/POLLIN|POLLHUP 
156/POLLIN|POLLHUP 155/POLLIN|POLLHUP 125/POLLIN|POLLHUP 
126/POLLIN|POLLHUP 150/POLLIN|POLLHUP 149/POLLIN|POLLHUP 
173/POLLIN|POLLHUP 174/POLLIN|POLLHUP 139/POLLIN|POLLHUP 
140/POLLIN|POLLHUP 161/POLLIN|POLLHUP 162/POLLIN|POLLHUP 
136/POLLIN|POLLHUP 135/POLLIN|POLLHUP 160/POLLIN|POLLHUP 
159/POLLIN|POLLHUP 69/POLLIN|POLLHUP 70/POLLIN|POLLHUP 
168/POLLIN|POLLHUP 167/POLLIN|POLLHUP 119/POLLIN|POLLHUP 
120/POLLIN|POLLHUP 182/POLLIN|POLLHUP 181/POLLIN|POLLHUP 
144/POLLIN|POLLHUP 143/POLLIN|POLLHUP 43/POLLIN|POLLHUP 
44/POLLIN|POLLHUP 131/POLLIN|POLLHUP 132/POLLIN|POLLHUP 
133/POLLIN|POLLHUP 134/POLLIN|POLLHUP},152,5106) = 1 (0x1)
readv(0x85,0x81019cc90,0x1,0x1,0x80860bd82,0x0)  = 190 (0xbe)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
clock_gettime(13,{1448524674.000000000 })        = 0 (0x0)
> If you could profile your use case on a linux host with the perf tools,
> that may give us some insights.
>Unfortunately now I don't have enought free time, but I try test it when 
it will.
>> I think may be samba something doing not optimal way. Like this,
>> first
>> we try to create directory (mkdir), then we check it (stat). May be
>> that
>> leads to impact filesystems and samba ldap works slower than
>> openldap.
>>
>> fcntl(15,F_SETLKW,0x7fffffffd118)                = 0 (0x0)
>> fcntl(15,F_SETLKW,0x7fffffffd188)                = 0 (0x0)
>> clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
>> mkdir("/var/db/samba4/private/sam.ldb.d",0700)   ERR#17
'File exists'
>>
stat("/var/db/samba4/private/sam.ldb.d/DC=DOMAINDNSZONES,DC=SPEECHPRO
>> ,DC=COM.ldb",{
>> mode=-rw------- ,inode=242034,size=4247552,blksize=32768 }) = 0 (0x0)
>> fcntl(16,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
>> fcntl(16,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
>> fcntl(16,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
>> fcntl(16,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
>> fcntl(16,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
>> fcntl(16,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
>> clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
>> clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
>> fcntl(16,F_SETLK,0x7fffffffd098)                 = 0 (0x0)
>> fcntl(16,F_SETLKW,0x7fffffffd118)                = 0 (0x0)
>> fcntl(16,F_SETLKW,0x7fffffffd188)                = 0 (0x0)
>> clock_gettime(13,{1448375653.000000000 })        = 0 (0x0)
>> mkdir("/var/db/samba4/private/sam.ldb.d",0700)   ERR#17
'File exists'
>>
stat("/var/db/samba4/private/sam.ldb.d/DC=FORESTDNSZONES,DC=SPEECHPRO
>> ,DC=COM.ldb",{
>> mode=-rw------- ,inode=242035,size=4247552,blksize=32768 }) = 0 (0x0)
>> fcntl(17,F_SETLKW,0x7fffffffd098)                = 0 (0x0)
>> fcntl(17,F_SETLKW,0x7fffffffd138)                = 0 (0x0)
>> fcntl(17,F_SETLKW,0x7fffffffd098)
>>
> These really should happen only once per startup of the process, how
> often do you see it?
>I see it about every seconds with any files in 
/var/db/samba4/private/sam.ldb.d directory.>> 23.11.2015 18:44, mathias dufresne пишет:
>>> Hi,
>>>
>>> I feel like I must reply off-list: Samba 4 is not yet able to use
>>> threads.
>>> To work around that we are about to deploy our DCs as virtual
>>> machines, to be able to multiply the number of Samba running.
>>>
>>> I expect you would have already thought about that, but in case you
>>> didn't, I said it :p
>>>
>>> I will follow closely that topic you opened. Sorry to not help
>>> more.
>>>
>>> Cheers,
>>>
>>> mathias
>>>
>>>
>>>
>>> 2015-11-23 14:50 GMT+01:00 Yuriy Tabolin <tabolin at
speechpro.com
>>> <mailto:tabolin at speechpro.com>>:
>>>
>>>      Hi all.
>>>      I have samba 4.2.3 on freebsd 10.1 server. There are three DC
>>> and
>>>      about 350 PC on domain. DCs have 3 CPU and 3GB RAM. Some
>>> servers
>>>      with services like apache, exim, dovecot, etc use samba4 ldap
>>>      (port 389) for user authentication. Some times ago after
adding
>>>      some services to use ldap I found, that samba4 cannot serve
all
>>>      ldap requests. Every 10-30 minutes I see in DCs logs:
>>>      dc1 kernel: sonewconn: pcb 0xfffff800753d6ab8: Listen queue
>>>      overflow: 16 already in queue awaiting acceptance (28
>>> occurrences)
>>>
>>>      After that I have used tcpdump for recording ldap-traffic and
>>> have
>>>      seen that after TCP handshaking, server some times suddenly
>>> send
>>>      TCP-RST to close connection. I have enlarged DCs resources
(CPU
>>>      and RAM),  kern.ipc.somaxconn, did some other system tuning
but
>>>      all that didn't help. Load average on DCs permanently near
0.9
>>> -1.0
>>>      and samba cannot serve all ldap conncetions. ldap clients
works
>>>      well because they use as minimum as two domain controllers as
>>> ldap
>>>      servers. Is there a some performance problem in samba4, slow
>>>      processing ldap requests or something else?
>>>      Thanks for any help.
>>>
>>>      --
>>>      With best regards,
>>>
>>>      Tabolin Yuriy
>>>      System administrator
>>>      Speech Technology Center
>>>
>>>
>>>      --
>>>      To unsubscribe from this list go to the following URL and read
>>> the
>>>      instructions: https://lists.samba.org/mailman/options/samba
>>>
>>>
>>
>> -- 
>> With best regards,
>>
>> Tabolin Yuriy
>> System administrator
>> Speech Technology Center
>>-- 
With best regards,

Tabolin Yuriy
System administrator
Speech Technology Center

Reasonably Related Threads

Search for more maybe matching threads

samba - Nov 2015 - samba4 ldap high load and port queue overflow

[Samba] samba4 ldap high load and port queue overflow

[Samba] samba4 ldap high load and port queue overflow

[Samba] samba4 ldap high load and port queue overflow

Reasonably Related Threads