samba - Jan 2016 - Stymied with samba vs openldap SSL ("Failed to issue the StartTLS instruction...")

I know this is something which should have a simple fix but I'm failing 
to see it somehow.

I'm moving samba service between a couple of FreeBSD systems (9.3 to 
10.2), and I'm stuck on getting samba on the new machine to connect to 
our openldap server over ssl - frustrating since I've been running 
samba+ldap for 15 years or so; feel sure I'm missing something basic! 
I'm getting the traditional error of "Failed to issue the StartTLS 
instruction: Connect error".

I've tried this with two versions of samba: 3.6.25 (same version as the 
working installation on the older server) and 4.2.3, and get the same 
issue with both.

My default config is using:
passdb backend = ldapsam:"ldap://ldap-server-fqdn"
ldap ssl = start_tls

If I disable ssl in smb.conf with:

ldap ssl = never

then samba does start successfully - suggesting a certificate validation 
issue.

However, all my other ldap functions work fine over ssl, including pam, 
nslcd, and a plain "ldapsearch -ZZ".

Also curious is that if I disable certificate validation in the openldap 
ldap.conf, with "TLS_REQCERT never", smbd still fails to communicate.

Now, our libldap.so is linked against the system openssl, while I 
believe samba 4.2 at least uses GnuTLS - might that cause a problem? 
However my samba 3.6 build is using openssl so this doesn't seem a 
likely cause.

gnutls-cli -p 636 ldap-server-fqdn

does also successfully print out the certificate chain and declare the 
certificate trusted.

Any ideas what I might be missing?

Thanks, Graham

BTW here's a debug level 5 snippet of log around the error:
> [2016/01/05 16:50:44.382984,  2]
../source3/passdb/pdb_ldap_util.c:280(smbldap_search_domain_info)
>   smbldap_search_domain_info: Searching
for:[(&(objectClass=sambaDomain)(sambaDomainName=SPA))]
> [2016/01/05 16:50:44.383048,  5]
../source3/lib/smbldap.c:1249(smbldap_search_ext)
>   smbldap_search_ext: base => [dc=physics,dc=umn,dc=edu], filter =>
[(&(objectClass=sambaDomain)(sambaDomainName=SPA))], scope => [2]
> [2016/01/05 16:50:44.383124,  5]
../source3/lib/smbldap.c:1114(smbldap_close)
>   The connection to the LDAP server was closed
> [2016/01/05 16:50:44.407310,  0]
../source3/lib/smbldap.c:575(smbldap_start_tls)
>   Failed to issue the StartTLS instruction: Connect error
> [2016/01/05 16:50:44.407377,  1]
../source3/lib/smbldap.c:1206(get_cached_ldap_connect)
>   Connection to LDAP server failed for the 1 try!
> [2016/01/05 16:50:45.412481,  0]
../source3/lib/smbldap.c:575(smbldap_start_tls)
>   Failed to issue the StartTLS instruction: Connect error
> [2016/01/05 16:50:45.412558,  1]
../source3/lib/smbldap.c:1206(get_cached_ldap_connect)
>   Connection to LDAP server failed for the 1 try!

On Tue, Jan 5, 2016 at 3:35 PM, Graham Allan <allan at physics.umn.edu>
wrote:
> I know this is something which should have a simple fix but I'm failing
to
> see it somehow.
>
> I'm moving samba service between a couple of FreeBSD systems (9.3 to
> 10.2), and I'm stuck on getting samba on the new machine to connect to
our
> openldap server over ssl - frustrating since I've been running
samba+ldap
> for 15 years or so; feel sure I'm missing something basic! I'm
getting the
> traditional error of "Failed to issue the StartTLS instruction:
Connect
> error".
>
> I've tried this with two versions of samba: 3.6.25 (same version as the
> working installation on the older server) and 4.2.3, and get the same issue
> with both.
>
> My default config is using:
> passdb backend = ldapsam:"ldap://ldap-server-fqdn"
> ldap ssl = start_tls
>
> If I disable ssl in smb.conf with:
>
> ldap ssl = never
>
> then samba does start successfully - suggesting a certificate validation
> issue.
>
> However, all my other ldap functions work fine over ssl, including pam,
> nslcd, and a plain "ldapsearch -ZZ".
>
> Also curious is that if I disable certificate validation in the openldap
> ldap.conf, with "TLS_REQCERT never", smbd still fails to
communicate.
>
> Now, our libldap.so is linked against the system openssl, while I believe
> samba 4.2 at least uses GnuTLS - might that cause a problem? However my
> samba 3.6 build is using openssl so this doesn't seem a likely cause.
>
> gnutls-cli -p 636 ldap-server-fqdn
>
> does also successfully print out the certificate chain and declare the
> certificate trusted.
>
> Any ideas what I might be missing?
>
> Thanks, Graham
>
> BTW here's a debug level 5 snippet of log around the error:
>
> [2016/01/05 16:50:44.382984,  2]
>> ../source3/passdb/pdb_ldap_util.c:280(smbldap_search_domain_info)
>>   smbldap_search_domain_info: Searching
>> for:[(&(objectClass=sambaDomain)(sambaDomainName=SPA))]
>> [2016/01/05 16:50:44.383048,  5]
>> ../source3/lib/smbldap.c:1249(smbldap_search_ext)
>>   smbldap_search_ext: base => [dc=physics,dc=umn,dc=edu], filter
=>
>> [(&(objectClass=sambaDomain)(sambaDomainName=SPA))], scope =>
[2]
>> [2016/01/05 16:50:44.383124,  5]
>> ../source3/lib/smbldap.c:1114(smbldap_close)
>>   The connection to the LDAP server was closed
>> [2016/01/05 16:50:44.407310,  0]
>> ../source3/lib/smbldap.c:575(smbldap_start_tls)
>>   Failed to issue the StartTLS instruction: Connect error
>> [2016/01/05 16:50:44.407377,  1]
>> ../source3/lib/smbldap.c:1206(get_cached_ldap_connect)
>>   Connection to LDAP server failed for the 1 try!
>> [2016/01/05 16:50:45.412481,  0]
>> ../source3/lib/smbldap.c:575(smbldap_start_tls)
>>   Failed to issue the StartTLS instruction: Connect error
>> [2016/01/05 16:50:45.412558,  1]
>> ../source3/lib/smbldap.c:1206(get_cached_ldap_connect)
>>   Connection to LDAP server failed for the 1 try!
>>
>
> --
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba
>
A total guess would be to use either ldaps:// and don't bother with
start_tls, or add the :636 to the end of the ldap:// specification as it
seems to me that start_tls is pretty agnostic regarding whatever protocol
it works against (SMTP, LDAP, etc.).  ie

passdb backend = ldapsam:"ldaps://ldap-server-fqdn"
#ldap ssl = start_tls

OR

passdb backend = ldapsam:"ldap://ldap-server-fqdn:636"
ldap ssl = start_tls

Otherwise I'd suggest a packet dump on the ldap machine to see what the
difference is between what works and what doesn't to provide some clue.

On 1/5/2016 7:19 PM, Lee Brown wrote:
>
> A total guess would be to use either ldaps:// and don't bother with
> start_tls, or add the :636 to the end of the ldap:// specification as it
> seems to me that start_tls is pretty agnostic regarding whatever
> protocol it works against (SMTP, LDAP, etc.).  ie
>
> passdb backend = ldapsam:"ldaps://ldap-server-fqdn"
> #ldap ssl = start_tls
>
> OR
>
> passdb backend = ldapsam:"ldap://ldap-server-fqdn:636"
> ldap ssl = start_tls
>
> Otherwise I'd suggest a packet dump on the ldap machine to see what the
> difference is between what works and what doesn't to provide some clue.
>
>
The packet dump is a good idea. I get the same failure using straight 
SSL to port 636, but wireshark might be able to decode any StartTLS 
negotiation attempt on the default port. Failing that I guess I'll 
resort to running smbd in gdb...

Graham
--

On Tue, Jan 05, 2016 at 05:35:21PM -0600, Graham Allan
wrote:
> I know this is something which should have a simple fix but I'm failing
> to see it somehow.
> 
> I'm moving samba service between a couple of FreeBSD systems (9.3 to 
> 10.2), and I'm stuck on getting samba on the new machine to connect to 
> our openldap server over ssl - frustrating since I've been running 
> samba+ldap for 15 years or so; feel sure I'm missing something basic! 
> I'm getting the traditional error of "Failed to issue the StartTLS
> instruction: Connect error".
> 
> I've tried this with two versions of samba: 3.6.25 (same version as the
> working installation on the older server) and 4.2.3, and get the same 
> issue with both.
> 
> My default config is using:
> passdb backend = ldapsam:"ldap://ldap-server-fqdn"
> ldap ssl = start_tls
> 
> If I disable ssl in smb.conf with:
> 
> ldap ssl = never
> 
> then samba does start successfully - suggesting a certificate validation 
> issue.
> 
> However, all my other ldap functions work fine over ssl, including pam, 
> nslcd, and a plain "ldapsearch -ZZ".
> 
> Also curious is that if I disable certificate validation in the openldap 
> ldap.conf, with "TLS_REQCERT never", smbd still fails to
communicate.
> 
> Now, our libldap.so is linked against the system openssl, while I 
> believe samba 4.2 at least uses GnuTLS - might that cause a problem? 
> However my samba 3.6 build is using openssl so this doesn't seem a 
> likely cause.
> 
> gnutls-cli -p 636 ldap-server-fqdn
> 
> does also successfully print out the certificate chain and declare the 
> certificate trusted.
> 
> Any ideas what I might be missing?
> 
> Thanks, Graham
> 
> BTW here's a debug level 5 snippet of log around the error:
> 
> > [2016/01/05 16:50:44.382984,  2]
../source3/passdb/pdb_ldap_util.c:280(smbldap_search_domain_info)
> >   smbldap_search_domain_info: Searching
for:[(&(objectClass=sambaDomain)(sambaDomainName=SPA))]
> > [2016/01/05 16:50:44.383048,  5]
../source3/lib/smbldap.c:1249(smbldap_search_ext)
> >   smbldap_search_ext: base => [dc=physics,dc=umn,dc=edu], filter
=> [(&(objectClass=sambaDomain)(sambaDomainName=SPA))], scope => [2]
> > [2016/01/05 16:50:44.383124,  5]
../source3/lib/smbldap.c:1114(smbldap_close)
> >   The connection to the LDAP server was closed
> > [2016/01/05 16:50:44.407310,  0]
../source3/lib/smbldap.c:575(smbldap_start_tls)
> >   Failed to issue the StartTLS instruction: Connect error
> > [2016/01/05 16:50:44.407377,  1]
../source3/lib/smbldap.c:1206(get_cached_ldap_connect)
> >   Connection to LDAP server failed for the 1 try!
> > [2016/01/05 16:50:45.412481,  0]
../source3/lib/smbldap.c:575(smbldap_start_tls)
> >   Failed to issue the StartTLS instruction: Connect error
> > [2016/01/05 16:50:45.412558,  1]
../source3/lib/smbldap.c:1206(get_cached_ldap_connect)
> >   Connection to LDAP server failed for the 1 try!
> 
I work on FreeNAS and have at least one complaint about this exact same
issue. I'm interested in a solution (or reason for this) as well.

- John
> -- 
> To unsubscribe from this list go to the following URL and read the
> instructions:  https://lists.samba.org/mailman/options/samba

On 1/6/2016 5:04 PM, John Hixson wrote:
> On Tue, Jan 05, 2016 at 05:35:21PM -0600, Graham Allan wrote:
>> I know this is something which should have a simple fix but I'm
failing
>> to see it somehow.
>>
>> I'm moving samba service between a couple of FreeBSD systems (9.3
to
>> 10.2), and I'm stuck on getting samba on the new machine to connect
to
>> our openldap server over ssl - frustrating since I've been running
>> samba+ldap for 15 years or so; feel sure I'm missing something
basic!
>> I'm getting the traditional error of "Failed to issue the
StartTLS
>> instruction: Connect error".
>>
>> I've tried this with two versions of samba: 3.6.25 (same version as
the
>> working installation on the older server) and 4.2.3, and get the same
>> issue with both.
>
> I work on FreeNAS and have at least one complaint about this exact same
> issue. I'm interested in a solution (or reason for this) as well.
>
> - John
That's interesting. Maybe it makes me feel better that I might not 
missing something stupid...

Are the complaints related to the beta version of FreeNAS (based on 
FreeBSD 10.x)? I've never had any problems on 9.x. I have the same 
version of samba on each, built in tinderbox with identical options, and 
only the 10.x version has this issue for me.

The only pertinent difference I can think of is that 9.x uses openssl 
0.9.8, while 10.x uses 1.0.1, but since the openldap client libraries 
themselves work fine with ssl on both, it's hard to point the finger at 
that.

Graham

I've followed this through with every idea I can grasp at so far without 
much luck.

I hacked up a quick 30-line c program to connect to ldap using the same 
URI and credentials as used in samba - works fine.

When I follow the execution of smbd in gdb, the problem occurs when calling:

ldap_simple_bind_s(ldap_struct, ldap_state->bind_dn, 
ldap_state->bind_secret);

in smbldap_connect_system (smbldap.c), which returns:

failed to bind to server ldaps://ldap1.spa.umn.edu with 
dn="cn=admin,dc=physics,dc=umn,dc=edu" Error: Can't contact LDAP
server
error:0D0C5006:asn1 encoding routines:ASN1_item_verify:EVP lib

I'm debugging this with samba 3.6 just because it's simpler and with 
fewer dependencies than 4.x

In gdb I can examine ldap_state->bind_dn and ldap_state->bind_secret, 
and see that they are the same as my simple test program. gdb won't show 
me ldap_struct, but it appears to be set up by:

ldap_initialize(ldap_struct, uri)

in smb_ldap_setup_conn, where I can see that uri is also the same as my 
working test program.

So I'm at a loss to explain why it can fail. My test program doesn't 
call ldap_set_option at all, so maybe something smbd does there might 
explain it?

I believe the original poster on this thread is describing the same 
issue (but no resolution):
https://forums.freebsd.org/threads/samba-openldap-tls-problems.44179/

G.
--

I also (belatedly) set "ldap debug level = 1" in smb.conf (wish
I'd
thought of that before!). At the same time I set_ldap_option for the 
same debug level in my test program. Lo! the smbd output complains of 
certificate signature failure.

smbd output:
> [LDAP] ldap_simple_bind_s
> [LDAP] ldap_sasl_bind_s
> [LDAP] ldap_sasl_bind
> [LDAP] ldap_send_initial_request
> [LDAP] ldap_new_connection 1 1 0
> [LDAP] ldap_int_open_connection
> [LDAP] ldap_connect_to_host: TCP ldap.spa.umn.edu:636
> [LDAP] ldap_new_socket: 9
> [LDAP] ldap_prepare_socket: 9
> [LDAP] ldap_connect_to_host: Trying 128.101.220.24:636
> [LDAP] ldap_pvt_connect: fd: 9 tm: -1 async: 0
> [LDAP] attempting to connect:
> [LDAP] connect success
> [LDAP] TLS trace: SSL_connect:before/connect initialization
> [LDAP] TLS trace: SSL_connect:SSLv2/v3 write client hello A
> [LDAP] TLS trace: SSL_connect:SSLv3 read server hello A
> [LDAP] TLS certificate verification: depth: 3, err: 0, subject:
/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA
Root,[LDAP]  issuer: /C=SE/O=AddTrust AB/OU=AddTrust External TTP
Network/CN=AddTrust External CA Root
> [LDAP] TLS certificate verification: depth: 2, err: 0, subject:
/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA
Certification Authority,[LDAP]  issuer: /C=SE/O=AddTrust AB/OU=AddTrust External
TTP Network/CN=AddTrust External CA Root
> [LDAP] TLS certificate verification: depth: 1, err: 0, subject:
/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA,[LDAP]
issuer: /C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust
RSA Certification Authority
> [LDAP] TLS certificate verification: depth: 0, err: 7, subject:
/C=US/postalCode=55455/ST=MN/L=Minneapolis/street=100 Union Street
SE/O=University of Minnesota/OU=School of Physics and
Astronomy/CN=ldap.spa.umn.edu,[LDAP]  issuer: /C=US/ST=MI/L=Ann
Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA
> [LDAP] TLS certificate verification: Error, certificate signature failure
> [LDAP] TLS certificate verification: depth: 0, err: 7, subject:
/C=US/postalCode=55455/ST=MN/L=Minneapolis/street=100 Union Street
SE/O=University of Minnesota/OU=School of Physics and
Astronomy/CN=ldap.spa.umn.edu,[LDAP]  issuer: /C=US/ST=MI/L=Ann
Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA
> [LDAP] TLS trace: SSL_connect:SSLv3 read server certificate A
> [LDAP] TLS trace: SSL_connect:SSLv3 read server done A
> [LDAP] TLS trace: SSL_connect:SSLv3 write client key exchange A
> [LDAP] TLS trace: SSL_connect:error in error
> [LDAP] TLS trace: SSL_connect:error in error
> [LDAP] TLS: can't connect: .
But the test program on same machine gives:
> ldap_simple_bind_s
> ldap_sasl_bind_s
> ldap_sasl_bind
> ldap_send_initial_request
> ldap_new_connection 1 1 0
> ldap_int_open_connection
> ldap_connect_to_host: TCP ldap.spa.umn.edu:636
> ldap_new_socket: 3
> ldap_prepare_socket: 3
> ldap_connect_to_host: Trying 128.101.220.24:636
> ldap_pvt_connect: fd: 3 tm: -1 async: 0
> attempting to connect:
> connect success
> TLS trace: SSL_connect:before/connect initialization
> TLS trace: SSL_connect:SSLv2/v3 write client hello A
> TLS trace: SSL_connect:SSLv3 read server hello A
> TLS certificate verification: depth: 3, err: 0, subject: /C=SE/O=AddTrust
AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root, issuer:
/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA
Root
> TLS certificate verification: depth: 2, err: 0, subject: /C=US/ST=New
Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification
Authority, issuer: /C=SE/O=AddTrust AB/OU=AddTrust External TTP
Network/CN=AddTrust External CA Root
> TLS certificate verification: depth: 1, err: 0, subject: /C=US/ST=MI/L=Ann
Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA, issuer: /C=US/ST=New
Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust RSA Certification
Authority
> TLS certificate verification: depth: 0, err: 0, subject:
/C=US/postalCode=55455/ST=MN/L=Minneapolis/street=100 Union Street
SE/O=University of Minnesota/OU=School of Physics and
Astronomy/CN=ldap.spa.umn.edu, issuer: /C=US/ST=MI/L=Ann
Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server CA
> TLS trace: SSL_connect:SSLv3 read server certificate A
> TLS trace: SSL_connect:SSLv3 read server done A
> TLS trace: SSL_connect:SSLv3 write client key exchange A
> TLS trace: SSL_connect:SSLv3 write change cipher spec A
> TLS trace: SSL_connect:SSLv3 write finished A
> TLS trace: SSL_connect:SSLv3 flush data
> TLS trace: SSL_connect:SSLv3 read server session ticket A
> TLS trace: SSL_connect:SSLv3 read finished A
> ldap_open_defconn: successful
> ldap_send_server_request
Same certificate chain, but one case verifies and the other doesn't...

G.