freebsd stable - Jul 2010 - openldap client GSSAPI authentication segfaults in fbsd8stable i386

Dear All,

I have a problem: ldapsearch results in "Segmentation fault" under 
openldap-2.4.23 with cyrus-sasl-2.1.23.

A thread for similar issues was started by George Mamalakis back in 
february:
http://lists.freebsd.org/pipermail/freebsd-stable/2010-February/055017.html
but I find no solution / conclusion from this thread, hence I post here...

I have installed FreeBSD 8.0-RELEASE-p2 on i386, updated with 
freebsd-update, and ports updated with "portsnap fetch update".

Kerberos installed from packages, configured, and seems to work OK.

It seems that there are no package for openldap server with GSSAPI/SASL, 
so I have build and installed cyrus-sasl2, openldap24-server (with sasl 
configured) and openldap24-sasl-client from ports.

Those are the port versions:
# $FreeBSD: ports/security/cyrus-sasl2/Makefile,v 1.141 2009/08/02 
19:35:25 mezz Exp $
# $FreeBSD: ports/net/openldap24-server/Makefile,v 1.181 2010/07/01 
19:04:42 delphij Exp $

According to the distinfo files, those are the upstream versions:
openldap-2.4.23
cyrus-sasl-2.1.23
which, as far as I can see, are the latest stable.

Trying LDAP I get a segfault:

$ ldapsearch
SASL/GSSAPI authentication started
Segmentation fault (core dumped)

Here is the backtrace from gdb:

#0  0x283225c7 in free () from /lib/libc.so.7
#1  0x28654b42 in gss_release_buffer () from /usr/lib/libgssapi.so.10
#2  0x28654512 in gss_release_name () from /usr/lib/libgssapi.so.10
#3  0x28650e69 in gss_init_sec_context () from /usr/lib/libgssapi.so.10
#4  0x28648a0f in gssapi_client_mech_step () from 
/usr/local/lib/sasl2/libgssapiv2.so.2
#5  0x280ef4b1 in sasl_client_step () from /usr/local/lib/libsasl2.so.2
#6  0x28440200 in ?? ()
#7  0x00000000 in ?? ()
#8  0x00000000 in ?? ()
#9  0xbfbfe208 in ?? ()
#10 0xbfbfe1f4 in ?? ()
#11 0xbfbfe204 in ?? ()
#12 0x28446860 in ?? ()
#13 0x280ef3fe in sasl_client_step () from /usr/local/lib/libsasl2.so.2
#14 0xbfbfe148 in ?? ()
#15 0x280f0135 in sasl_client_start () from /usr/local/lib/libsasl2.so.2
#16 0x00000000 in ?? ()
#17 0x00000000 in ?? ()
#18 0xbfbfe208 in ?? ()
#19 0xbfbfe1f4 in ?? ()
#20 0xbfbfe204 in ?? ()
#21 0x72408f2d in ?? ()
#22 0x283b1ad8 in ?? () from /lib/libc.so.7
#23 0x00000000 in ?? ()
#24 0x283b1730 in __stderrp () from /lib/libc.so.7
#25 0xbfbfe118 in ?? ()
#26 0x28392114 in vfprintf () from /lib/libc.so.7
Previous frame inner to this frame (corrupt stack?)

I tried "valgrind ldapsearch" which produces thousands of issues,
ending
with:

==59479== Invalid free() / delete / delete[]
==59479==    at 0x59B95: free (in 
/usr/local/lib/valgrind/vgpreload_memcheck-x86-freebsd.so)
==59479==    by 0x911B41: gss_release_buffer (in /usr/lib/libgssapi.so.10)
==59479==    by 0x911511: ??? (in /usr/lib/libgssapi.so.10)
==59479==    by 0x90DE68: gss_init_sec_context (in /usr/lib/libgssapi.so.10)
==59479==    by 0x905A0E: gssapi_client_mech_step (in 
/usr/local/lib/sasl2/libgssapiv2.so.2)
==59479==    by 0xAF4B0: sasl_client_step (in /usr/local/lib/libsasl2.so.2)
==59479==    by 0xB0134: sasl_client_start (in /usr/local/lib/libsasl2.so.2)
==59479==    by 0x70C46: ldap_int_sasl_bind (in 
/usr/local/lib/libldap-2.4.so.7)
==59479==    by 0x73935: ldap_sasl_interactive_bind_s (in 
/usr/local/lib/libldap-2.4.so.7)
==59479==    by 0x80505E6: ??? (in /usr/local/bin/ldapsearch)
==59479==    by 0x804D695: ??? (in /usr/local/bin/ldapsearch)
==59479==    by 0x804A7D8: ??? (in /usr/local/bin/ldapsearch)
==59479==  Address 0x4e2c0 is not stack'd, malloc'd or (recently)
free'd
==59479=ldap_sasl_interactive_bind_s: Local error (-2)
    additional info: SASL(-1): generic failure: GSSAPI Error:  
Miscellaneous failure (see text) (unknown mech-code 2529638944 for mech 
unknown)

/var/log/messages has:
slapd[1146]: OTP unavailable because can't read/write key database 
/etc/opiekeys: Permission denied
kernel: pid 53862 (ldapsearch), uid 1001: exited on signal 11 (core dumped)

The first message is from the LDAP server. Even if it has some problem, 
it should not lead the client to segfault.

Any comments, hints or suggestions would be most appreciated!

/Henrik

On Tue, Jul 13, 2010 at 10:10:25PM +0200, Henrik /KaarPoSoft
wrote:
> Dear All,
> 
> I have a problem: ldapsearch results in "Segmentation fault"
under
> openldap-2.4.23 with cyrus-sasl-2.1.23.
> 
> A thread for similar issues was started by George Mamalakis back in
> february:
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-February/055017.html
> but I find no solution / conclusion from this thread, hence I post here...
> 
> I have installed FreeBSD 8.0-RELEASE-p2 on i386, updated with
> freebsd-update, and ports updated with "portsnap fetch update".
> 
> Kerberos installed from packages, configured, and seems to work OK.
> 
> It seems that there are no package for openldap server with
> GSSAPI/SASL, so I have build and installed cyrus-sasl2,
> openldap24-server (with sasl configured) and openldap24-sasl-client
> from ports.
> 
> Those are the port versions:
> # $FreeBSD: ports/security/cyrus-sasl2/Makefile,v 1.141 2009/08/02
> 19:35:25 mezz Exp $
> # $FreeBSD: ports/net/openldap24-server/Makefile,v 1.181 2010/07/01
> 19:04:42 delphij Exp $
> 
> According to the distinfo files, those are the upstream versions:
> openldap-2.4.23
> cyrus-sasl-2.1.23
> which, as far as I can see, are the latest stable.
> 
> Trying LDAP I get a segfault:
> 
> $ ldapsearch
> SASL/GSSAPI authentication started
> Segmentation fault (core dumped)
I know absolutely nothing about GSSAPI and have very little experience
with LDAP.  But I'll take a shot at this:
> Here is the backtrace from gdb:
> 
> #0  0x283225c7 in free () from /lib/libc.so.7
> #1  0x28654b42 in gss_release_buffer () from /usr/lib/libgssapi.so.10
> #2  0x28654512 in gss_release_name () from /usr/lib/libgssapi.so.10
> #3  0x28650e69 in gss_init_sec_context () from /usr/lib/libgssapi.so.10
> #4  0x28648a0f in gssapi_client_mech_step () from
> /usr/local/lib/sasl2/libgssapiv2.so.2
> #5  0x280ef4b1 in sasl_client_step () from /usr/local/lib/libsasl2.so.2
The problem looks like it may be that gss_release_buffer() is passing
free() an invalid pointer (which may be coming from other functions up
the stack).  Without debugging symbols enabled this is a bit difficult
to track down.

Since free() is in the call stack, I would say buffer->value contains a
pointer that is incorrect (e.g. out of process addressing scope).  It
could also be that buffer itself points to something invalid.
> /var/log/messages has:
> slapd[1146]: OTP unavailable because can't read/write key database
> /etc/opiekeys: Permission denied
> kernel: pid 53862 (ldapsearch), uid 1001: exited on signal 11 (core dumped)
> 
> The first message is from the LDAP server. Even if it has some
> problem, it should not lead the client to segfault.
I agree.

If I was to build a test box from scratch, can you tell me how to set up
all the necessary software/etc. to mimic your environment so that I
could try to reproduce this?  Reviewing the source isn't enough, I'd
have to actually build a debug version of libgssapi to track it down.

Alternatively I can try to step you through how to debug this using gdb,
but again, lack of debugging symbols makes this annoying.

-- 
| Jeremy Chadwick                                   jdc@parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |

On Tue, 13 Jul 2010, Henrik /KaarPoSoft wrote:
> Dear All,
>
> I have a problem: ldapsearch results in "Segmentation fault"
under
> openldap-2.4.23 with cyrus-sasl-2.1.23.
>
> A thread for similar issues was started by George Mamalakis back in
february:
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-February/055017.html
> but I find no solution / conclusion from this thread, hence I post here...
>
> I have installed FreeBSD 8.0-RELEASE-p2 on i386, updated with
freebsd-update,
> and ports updated with "portsnap fetch update".
>
> Kerberos installed from packages, configured, and seems to work OK.
>
> It seems that there are no package for openldap server with GSSAPI/SASL, so
I
> have build and installed cyrus-sasl2, openldap24-server (with sasl 
> configured) and openldap24-sasl-client from ports.
>
> Those are the port versions:
> # $FreeBSD: ports/security/cyrus-sasl2/Makefile,v 1.141 2009/08/02 19:35:25
> mezz Exp $
> # $FreeBSD: ports/net/openldap24-server/Makefile,v 1.181 2010/07/01
19:04:42
> delphij Exp $
>
> According to the distinfo files, those are the upstream versions:
> openldap-2.4.23
> cyrus-sasl-2.1.23
> which, as far as I can see, are the latest stable.
>
> Trying LDAP I get a segfault:
>
> $ ldapsearch
> SASL/GSSAPI authentication started
> Segmentation fault (core dumped)
>
> Here is the backtrace from gdb:
>
> #0  0x283225c7 in free () from /lib/libc.so.7
> #1  0x28654b42 in gss_release_buffer () from /usr/lib/libgssapi.so.10
> #2  0x28654512 in gss_release_name () from /usr/lib/libgssapi.so.10
> #3  0x28650e69 in gss_init_sec_context () from /usr/lib/libgssapi.so.10
> #4  0x28648a0f in gssapi_client_mech_step () from 
> /usr/local/lib/sasl2/libgssapiv2.so.2
> #5  0x280ef4b1 in sasl_client_step () from /usr/local/lib/libsasl2.so.2
> #6  0x28440200 in ?? ()
> #7  0x00000000 in ?? ()
> #8  0x00000000 in ?? ()
> #9  0xbfbfe208 in ?? ()
> #10 0xbfbfe1f4 in ?? ()
> #11 0xbfbfe204 in ?? ()
> #12 0x28446860 in ?? ()
> #13 0x280ef3fe in sasl_client_step () from /usr/local/lib/libsasl2.so.2
> #14 0xbfbfe148 in ?? ()
> #15 0x280f0135 in sasl_client_start () from /usr/local/lib/libsasl2.so.2
> #16 0x00000000 in ?? ()
> #17 0x00000000 in ?? ()
> #18 0xbfbfe208 in ?? ()
> #19 0xbfbfe1f4 in ?? ()
> #20 0xbfbfe204 in ?? ()
> #21 0x72408f2d in ?? ()
> #22 0x283b1ad8 in ?? () from /lib/libc.so.7
> #23 0x00000000 in ?? ()
> #24 0x283b1730 in __stderrp () from /lib/libc.so.7
> #25 0xbfbfe118 in ?? ()
> #26 0x28392114 in vfprintf () from /lib/libc.so.7
> Previous frame inner to this frame (corrupt stack?)
>
> I tried "valgrind ldapsearch" which produces thousands of issues,
ending
> with:
>
> ==59479== Invalid free() / delete / delete[]
> ==59479==    at 0x59B95: free (in 
> /usr/local/lib/valgrind/vgpreload_memcheck-x86-freebsd.so)
> ==59479==    by 0x911B41: gss_release_buffer (in /usr/lib/libgssapi.so.10)
> ==59479==    by 0x911511: ??? (in /usr/lib/libgssapi.so.10)
> ==59479==    by 0x90DE68: gss_init_sec_context (in
/usr/lib/libgssapi.so.10)
> ==59479==    by 0x905A0E: gssapi_client_mech_step (in 
> /usr/local/lib/sasl2/libgssapiv2.so.2)
> ==59479==    by 0xAF4B0: sasl_client_step (in /usr/local/lib/libsasl2.so.2)
> ==59479==    by 0xB0134: sasl_client_start (in
/usr/local/lib/libsasl2.so.2)
> ==59479==    by 0x70C46: ldap_int_sasl_bind (in 
> /usr/local/lib/libldap-2.4.so.7)
> ==59479==    by 0x73935: ldap_sasl_interactive_bind_s (in 
> /usr/local/lib/libldap-2.4.so.7)
> ==59479==    by 0x80505E6: ??? (in /usr/local/bin/ldapsearch)
> ==59479==    by 0x804D695: ??? (in /usr/local/bin/ldapsearch)
> ==59479==    by 0x804A7D8: ??? (in /usr/local/bin/ldapsearch)
> ==59479==  Address 0x4e2c0 is not stack'd, malloc'd or (recently)
free'd
> ==59479=> ldap_sasl_interactive_bind_s: Local error (-2)
>   additional info: SASL(-1): generic failure: GSSAPI Error:  Miscellaneous 
> failure (see text) (unknown mech-code 2529638944 for mech unknown)
>
> /var/log/messages has:
> slapd[1146]: OTP unavailable because can't read/write key database 
> /etc/opiekeys: Permission denied
> kernel: pid 53862 (ldapsearch), uid 1001: exited on signal 11 (core dumped)
>
> The first message is from the LDAP server. Even if it has some problem, it 
> should not lead the client to segfault.
>
> Any comments, hints or suggestions would be most appreciated!
Dear Henrik,

just a guess from my side.

You said, that you installed and configured Kerberos from packages (i 
guess from ports or a prebuilt package).
Did you by any chance set HEIMDAL_HOME=/usr before building and installing 
the kerberos port?

Did you set HEIMDAL_HOME to point to the place where the package/port got 
installed (e.g. HEIMDAL_HOME=/usr/local) before building the cyrus-sasl2 
port?

Did you set HEIMDAL_HOME to anything at all? Please take a look at 
${PORTSDIR}/security/cyrus-sasl2/Makefile to see the logic behind the 
kerberos selection.

The valgrind and gdb output above shows that /usr/lib/libgssapi.so.10 is 
used at runtime which comes out of the FreeBSD base system not out of your 
installed kerberos port/package. Maybe there is something messed up that 
kerberos from ports/package was used during build of cyrus-sasl2 but the 
base kerberos libs are used at runtime or vice versa.

In any case, this is just one thing i would double check before deeper 
debugging.

Kind regards
Joerg

-- 
The beginning is the most important part of the work.
 				-Plato

On Fri, Jul 16, 2010 at 03:58:04PM +0300, Reko Turja
wrote:
> >I think we need the OP of the PR[1], Mikhail T., to chime in here
> >with his
> >setup.
> 
> While waiting, can you test the following: In the
> /usr/local/etc/imapd.conf file comment out
> 
> #sasl_pwcheck_method: saslauthd
> 
> and add below it:
> 
> sasl_mech_list: gssapi pam plain
Thanks -- I did so + restarted imapd, and now we have:

testbox# cyradm localhost
Login disabled.
cyradm: cannot authenticate to server with  as root

Jul 16 06:46:02 testbox master[11087]: about to exec /usr/local/cyrus/bin/imapd
Jul 16 06:46:02 testbox imap[11087]: executed
Jul 16 06:46:02 testbox imap[11087]: accepted connection
Jul 16 06:46:02 testbox perl: GSSAPI Error:  Miscellaneous failure (see text)
(unknown mech-code 2 for mech unknown)
Jul 16 06:46:02 testbox kernel: Jul 16 06:46:02 testbox perl: GSSAPI Error: 
Miscellaneous failure (see text) (unknown mech-code 2 for mech unknown)
Jul 16 06:46:02 testbox perl: No worthy mechs found
Jul 16 06:46:02 testbox kernel: Jul 16 06:46:02 testbox perl: No worthy mechs
found

-- 
| Jeremy Chadwick                                   jdc@parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |