samba - Jun 2016 - NT_STATUS_IO_TIMEOUT at open_socket_out_send due to firewall

Hi,

We've got a few layers of firewall between our DCs and a domain guest 
providing winbind services. What I've noticed is that on first run 
winbind tries to use the high ports to set up part of the communication. 
During that process, the winbindd process runs at 100% CPU, and it takes 
about two minutes to time out. After that time things settle down and 
winbind works perfectly.

For various reasons we're not eager to open a wide range of ports across 
all the firewalls, and would like to know if there's a way in smb.conf 
to skip the step that's hanging.

I've validated that opening the port in the below log (49155) allows 
winbind to start cleanly and work without hanging for two minutes, but 
this was only for testing purposes and the port had to be closed back up 
after.

Here's loglevel 10 from where the thing happens on a Centos7 box running 
Samba 4.2.10:

[2016/06/15 11:23:34.554465,  3, pid=6383, effective(0, 0), real(0, 0)] 
../source3/lib/util_sock.c:636(open_socket_out_send)
   Connecting to [redacted IP of DC on different segment] at port 49155
[2016/06/15 11:25:41.772517, 10, pid=6383, effective(0, 0), real(0, 0), 
class=winbind] ../source3/winbindd/winbindd_cm.c:2698(cm_connect_lsa_tcp)
   cli_rpc_pipe_open_schannel_with_key failed: NT_STATUS_IO_TIMEOUT



smb:conf:
[global]
    workgroup = TEST
    realm = TEST.EXAMPLE.COM
    security = ads
    idmap config * : range = 16777216-33554431
    template shell = /bin/false
    kerberos method = secrets and keytab
    winbind use default domain = true
    winbind offline logon = false
    log file = /var/log/samba/%m.log
    log level = 10
    netbios name = TESTWB
    server string = WB server
    invalid users = root
    socket options = TCP_NODELAY
    winbind enum users = yes
    winbind enum groups = yes
    winbind max domain connections = 5
    winbind max clients = 1000

Bump.

On 06/15/2016 11:42 AM, Mike Ely wrote:
> Hi,
>
> We've got a few layers of firewall between our DCs and a domain guest 
> providing winbind services. What I've noticed is that on first run 
> winbind tries to use the high ports to set up part of the 
> communication. During that process, the winbindd process runs at 100% 
> CPU, and it takes about two minutes to time out. After that time 
> things settle down and winbind works perfectly.
>
> For various reasons we're not eager to open a wide range of ports 
> across all the firewalls, and would like to know if there's a way in 
> smb.conf to skip the step that's hanging.
>
> I've validated that opening the port in the below log (49155) allows 
> winbind to start cleanly and work without hanging for two minutes, but 
> this was only for testing purposes and the port had to be closed back 
> up after.
>
> Here's loglevel 10 from where the thing happens on a Centos7 box 
> running Samba 4.2.10:
>
> [2016/06/15 11:23:34.554465,  3, pid=6383, effective(0, 0), real(0, 
> 0)] ../source3/lib/util_sock.c:636(open_socket_out_send)
>   Connecting to [redacted IP of DC on different segment] at port 49155
> [2016/06/15 11:25:41.772517, 10, pid=6383, effective(0, 0), real(0, 
> 0), class=winbind] 
> ../source3/winbindd/winbindd_cm.c:2698(cm_connect_lsa_tcp)
>   cli_rpc_pipe_open_schannel_with_key failed: NT_STATUS_IO_TIMEOUT
>
>
>
> smb:conf:
> [global]
>    workgroup = TEST
>    realm = TEST.EXAMPLE.COM
>    security = ads
>    idmap config * : range = 16777216-33554431
>    template shell = /bin/false
>    kerberos method = secrets and keytab
>    winbind use default domain = true
>    winbind offline logon = false
>    log file = /var/log/samba/%m.log
>    log level = 10
>    netbios name = TESTWB
>    server string = WB server
>    invalid users = root
>    socket options = TCP_NODELAY
>    winbind enum users = yes
>    winbind enum groups = yes
>    winbind max domain connections = 5
>    winbind max clients = 1000

Have you tried using IPTABLES to REJECT that port.  It will send something
back and then maybe the timeout will fail immediately instead of waiting
the timeout period?  So on the local machine(that is running winbind) try

iptables -A OUTPUT -p tcp --dport 49155 -j REJECT
iptables -A INPUT -p tcp --dport 49155 -j REJECT

Might not need it in both, but since we are checking...

Jason

On Thu, Jun 23, 2016 at 3:26 PM, Mike Ely <me at mikeely.org> wrote:
> Bump.
>
> On 06/15/2016 11:42 AM, Mike Ely wrote:
>
>> Hi,
>>
>> We've got a few layers of firewall between our DCs and a domain
guest
>> providing winbind services. What I've noticed is that on first run
winbind
>> tries to use the high ports to set up part of the communication. During
>> that process, the winbindd process runs at 100% CPU, and it takes about
two
>> minutes to time out. After that time things settle down and winbind
works
>> perfectly.
>>
>> For various reasons we're not eager to open a wide range of ports
across
>> all the firewalls, and would like to know if there's a way in
smb.conf to
>> skip the step that's hanging.
>>
>> I've validated that opening the port in the below log (49155)
allows
>> winbind to start cleanly and work without hanging for two minutes, but
this
>> was only for testing purposes and the port had to be closed back up
after.
>>
>> Here's loglevel 10 from where the thing happens on a Centos7 box
running
>> Samba 4.2.10:
>>
>> [2016/06/15 11:23:34.554465,  3, pid=6383, effective(0, 0), real(0, 0)]
>> ../source3/lib/util_sock.c:636(open_socket_out_send)
>>   Connecting to [redacted IP of DC on different segment] at port 49155
>> [2016/06/15 11:25:41.772517, 10, pid=6383, effective(0, 0), real(0, 0),
>> class=winbind]
../source3/winbindd/winbindd_cm.c:2698(cm_connect_lsa_tcp)
>>   cli_rpc_pipe_open_schannel_with_key failed: NT_STATUS_IO_TIMEOUT
>>
>>
>>
>> smb:conf:
>> [global]
>>    workgroup = TEST
>>    realm = TEST.EXAMPLE.COM
>>    security = ads
>>    idmap config * : range = 16777216-33554431
>>    template shell = /bin/false
>>    kerberos method = secrets and keytab
>>    winbind use default domain = true
>>    winbind offline logon = false
>>    log file = /var/log/samba/%m.log
>>    log level = 10
>>    netbios name = TESTWB
>>    server string = WB server
>>    invalid users = root
>>    socket options = TCP_NODELAY
>>    winbind enum users = yes
>>    winbind enum groups = yes
>>    winbind max domain connections = 5
>>    winbind max clients = 1000
>>
>
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>