George Mamalakis
2010-Feb-11 18:45 UTC
openldap client GSSAPI authentication segfaults in fbsd8stable i386
Dear all,
I am facing many instabilities in FBSD8 with openldap-client and sasl
authentication (GSSAPI in particular). I have setup an openldap 2.4.1
server with gssapi support (through cyrus-sasl-2.1.23) on a fbsd8-stable
amd64 latest sources, in a esxi host. In the same host I have setup two
fbsd8-stable i386 clients; one has latest sources, the other one is
installed via the iso-image of January's fbsd snapshot; on both systems
openldap client and sasl is installed (all ldap/cyrus versions on all
hosts mentioned in this email are the same). My laptop has fbsd8-i386
stable (sources 25 January 2010), and on my laptop I have setup an
fbsd8-stable i386 (snapshot iso image) on a virtualbox client. Lastly,
on the esxi host I have setup another fbsd8-stable amd64 system, to act
as an ldap client (latest sources).
To summarize, and put a label-number on each host, we have:
1 - esxi: fbsd8(latest) amd64 openldap server
2 - esxi: fbsd8(latest) i386 openldap client
3 - esxi: fbsd8(snapshot) i386 openldap client
4 - esxi: fbsd8(latest) amd64 openldap client
5 - laptop: fbsd8(jan 25) i386 openldap client
6 - laptop/vbox: fbsd8(snapshot) i386 openldap client
The openldap server is installed in a jail, and the client is tested in
the same jail.
Kerberos works on all machines (same /etc/krb5.conf), and ldap as well.
In all machines, line 96 of /usr/bin/krb5-config is changed to read:
lib_flags="$lib_flags -lgssapi -lgssapi_spnego -lgssapi_krb5
-lheimntlm"
instead of:
lib_flags="$lib_flags -lgssapi -lheimntlm"
which was the default, since without these lines I couldn't get gssapi
authentication to work for cyrus (and spnego). This change was made
after recommendations given from this very mailing list, and I was very
happy to see that the ldap server worked as I expected.
On all system, cat /usr/local/etc/openldap/ldap.conf:
BASE dc=ee,dc=auth,dc=gr
URI ldap://ldap.ee.auth.gr
SASL_MECH GSSAPI
without kiniting in any client I get the following outcomes when I give
ldapwhoami (with no arguments):
1:
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
additional info: SASL(-1): generic failure: GSSAPI Error:
Miscellaneous failure (see text) (unknown mech-code 2 for mech unknown)
which is expected.
2:
SASL/GSSAPI authentication started
Segmentation fault: 11 (core dumped)
which is not rational at all
3:
SASL/GSSAPI authentication started
Segmentation fault: 11 (core dumped)
4:
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
additional info: SASL(-1): generic failure: GSSAPI Error:
Miscellaneous failure (see text) (unknown mech-code 2 for mech unknown)
which is the same as 1 (as expected)
5:
SASL/GSSAPI authentication started
Segmentation fault: 11 (core dumped)
6:
SASL/GSSAPI authentication started
Segmentation fault: 11 (core dumped)
if I kinit to mamalos, ldapwhoami returns:
1:
SASL/GSSAPI authentication started
SASL username: mamalos@EE.AUTH.GR
SASL SSF: 56
SASL data security layer installed.
dn:uid=mamalos,ou=people,dc=ee,dc=auth,dc=gr
which is super!
2:
SASL/GSSAPI authentication started
Segmentation fault: 11 (core dumped)
which is dramatic.
3:
SASL/GSSAPI authentication started
Segmentation fault: 11 (core dumped)
4:
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
additional info: SASL(-1): generic failure: GSSAPI Error:
Miscellaneous failure (see text) (unknown mech-code 2529638919 for mech
unknown)
which is very strange, since mech-code seems unnaturally large.
5:
SASL/GSSAPI authentication started
SASL username: mamalos@EE.AUTH.GR
SASL SSF: 56
SASL data security layer installed.
dn:uid=mamalos,ou=people,dc=ee,dc=auth,dc=gr
which is super, but without kinit the same command segfaulted on this
machine
6:
SASL/GSSAPI authentication started
SASL username: mamalos@EE.AUTH.GR
SASL SSF: 56
SASL data security layer installed.
dn:uid=mamalos,ou=people,dc=ee,dc=auth,dc=gr
which is the exact same behavior as 5 above.
All this means that there is no single pattern!!!!!!!
If I gdb ldapwhoami in the clients that segfault, I get:
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd"...(no debugging
symbols
found)...
(gdb) run -d0
Starting program: /usr/local/bin/ldapwhoami -d0
(no debugging symbols found)...(no debugging symbols found)...(no
debugging symbols found)...(no debugging symbols found)...(no debugging
symbols found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols
found)...SASL/GSSAPI authentication started
Program received signal SIGSEGV, Segmentation fault.
0x2831e187 in free () from /lib/libc.so.7
(gdb) where
#0 0x2831e187 in free () from /lib/libc.so.7
#1 0x2850fb82 in gss_release_buffer () from /usr/lib/libgssapi.so.10
#2 0x2850f552 in gss_release_name () from /usr/lib/libgssapi.so.10
#3 0x2850bea9 in gss_init_sec_context () from /usr/lib/libgssapi.so.10
#4 0x283f9abf in gssapi_client_mech_step () from
/usr/local/lib/sasl2/libgssapiv2.so.2
#5 0x280e84b1 in sasl_client_step () from /usr/local/lib/libsasl2.so.2
#6 0x28443100 in ?? ()
#7 0x00000000 in ?? ()
#8 0x00000000 in ?? ()
#9 0xbfbfe968 in ?? ()
#10 0xbfbfe954 in ?? ()
#11 0xbfbfe964 in ?? ()
#12 0x28445860 in ?? ()
#13 0x280e83fe in sasl_client_step () from /usr/local/lib/libsasl2.so.2
#14 0xbfbfe8a8 in ?? ()
#15 0x280e9135 in sasl_client_start () from /usr/local/lib/libsasl2.so.2
#16 0x00000000 in ?? ()
#17 0x00000000 in ?? ()
#18 0xbfbfe968 in ?? ()
#19 0xbfbfe954 in ?? ()
#20 0xbfbfe964 in ?? ()
#21 0xd7a3b2da in ?? ()
#22 0x283abad8 in ?? () from /lib/libc.so.7
#23 0x00000000 in ?? ()
#24 0x283ab730 in __stderrp () from /lib/libc.so.7
#25 0xbfbfe878 in ?? ()
#26 0x2838c764 in vfprintf () from /lib/libc.so.7
Previous frame inner to this frame (corrupt stack?)
I built openldap and cyrus-sasl on this machine from sources (not
ports), and (after a long time of trying to find out how to run
configure successfully in both installations) the outcome was exactly
the same (meaning segfaults). So, one of my admins wrote a c program
that overrides gss_release_buffer returning always 0 (what is expected
after normal operation) and compiled it as a library. The program code
is nothing more than just:
int gss_release_buffer(void *a, void *b) {
return 0;
}
we ld_preloaded the library, and when we ran ldapwhoami the outcome was:
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
additional info: SASL(-1): generic failure: GSSAPI Error:
Miscellaneous failure (see text) (unknown mech-code 2529638919 for mech
unknown)
When we ran it with no kerberos tickets, we got:
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
additional info: SASL(-1): generic failure: GSSAPI Error:
Miscellaneous failure (see text) (unknown mech-code 2 for mech unknown)
The exact same errors as the aforementioned client 4 (esxi,
amd64)!!!!!!!!!!!!!
What on earth is happening?!?!!?!?!
Now one can easily see that there is a definite problem regarding memory
freeing, and after overcoming that the mech-code 2529638919 implies that
some segment in memory is overwritten by some "random" value, so
mech-code returns the number 2529638919 instead of a number of
marginality 1.
What is more definite, is that openldap doesn't work out-of-the-box if
gssapi support is required, and behaves randomly in different
architectures/virtualHosts/platforms.
The problem may have been something related to line 96 in
/usr/bin/krb5-config... I don't know.
What is sure, is that I am having second thoughts on using fbsd as my
openldap/heimdal server for my setup, which would be quite disappointing
for me, since I am using fbsd for the last many-many years, on all my
laptops and servers. The only "good part" is that the only machine
that
works seamlessly (until now, at least) is the heimdal/ldap server.
Thank you all in advance and I hope that we will find an answer to all this.
--
George Mamalakis
IT Officer
Electrical and Computer Engineer (Aristotle Un. of Thessaloniki),
MSc (Imperial College of London)
Department of Electrical and Computer Engineering
Faculty of Engineering
Aristotle University of Thessaloniki
phone number : +30 (2310) 994379
jhell
2010-Feb-12 06:48 UTC
openldap client GSSAPI authentication segfaults in fbsd8stable i386
On Thu, 11 Feb 2010 13:45, mamalos@ wrote:> Dear all, > > I am facing many instabilities in FBSD8 with openldap-client and sasl > authentication (GSSAPI in particular). I have setup an openldap 2.4.1 server > with gssapi support (through cyrus-sasl-2.1.23) on a fbsd8-stable amd64 > latest sources, in a esxi host. In the same host I have setup two > fbsd8-stable i386 clients; one has latest sources, the other one is installed > via the iso-image of January's fbsd snapshot; on both systems openldap > client and sasl is installed (all ldap/cyrus versions on all hosts mentioned > in this email are the same). My laptop has fbsd8-i386 stable (sources 25 > January 2010), and on my laptop I have setup an fbsd8-stable i386 (snapshot > iso image) on a virtualbox client. Lastly, on the esxi host I have setup > another fbsd8-stable amd64 system, to act as an ldap client (latest sources). > > To summarize, and put a label-number on each host, we have: > > 1 - esxi: fbsd8(latest) amd64 openldap server > 2 - esxi: fbsd8(latest) i386 openldap client > 3 - esxi: fbsd8(snapshot) i386 openldap client > 4 - esxi: fbsd8(latest) amd64 openldap client > 5 - laptop: fbsd8(jan 25) i386 openldap client > 6 - laptop/vbox: fbsd8(snapshot) i386 openldap client > > The openldap server is installed in a jail, and the client is tested in the > same jail. > > Kerberos works on all machines (same /etc/krb5.conf), and ldap as well. > > In all machines, line 96 of /usr/bin/krb5-config is changed to read: > > lib_flags="$lib_flags -lgssapi -lgssapi_spnego -lgssapi_krb5 -lheimntlm" > > instead of: > lib_flags="$lib_flags -lgssapi -lheimntlm" > > which was the default, since without these lines I couldn't get gssapi > authentication to work for cyrus (and spnego). This change was made after > recommendations given from this very mailing list, and I was very happy to > see that the ldap server worked as I expected. > > On all system, cat /usr/local/etc/openldap/ldap.conf: > > BASE dc=ee,dc=auth,dc=gr > URI ldap://ldap.ee.auth.gr > SASL_MECH GSSAPI > > > without kiniting in any client I get the following outcomes when I give > ldapwhoami (with no arguments): > > 1: > SASL/GSSAPI authentication started > ldap_sasl_interactive_bind_s: Local error (-2) > additional info: SASL(-1): generic failure: GSSAPI Error: Miscellaneous > failure (see text) (unknown mech-code 2 for mech unknown) > > which is expected. > > 2: > SASL/GSSAPI authentication started > Segmentation fault: 11 (core dumped) > > which is not rational at all > > 3: > SASL/GSSAPI authentication started > Segmentation fault: 11 (core dumped) > > 4: > SASL/GSSAPI authentication started > ldap_sasl_interactive_bind_s: Local error (-2) > additional info: SASL(-1): generic failure: GSSAPI Error: Miscellaneous > failure (see text) (unknown mech-code 2 for mech unknown) > > which is the same as 1 (as expected) > > 5: > SASL/GSSAPI authentication started > Segmentation fault: 11 (core dumped) > > 6: > SASL/GSSAPI authentication started > Segmentation fault: 11 (core dumped) > > if I kinit to mamalos, ldapwhoami returns: > > 1: > SASL/GSSAPI authentication started > SASL username: mamalos@EE.AUTH.GR > SASL SSF: 56 > SASL data security layer installed. > dn:uid=mamalos,ou=people,dc=ee,dc=auth,dc=gr > > which is super! > > 2: > SASL/GSSAPI authentication started > Segmentation fault: 11 (core dumped) > > which is dramatic. > > 3: > SASL/GSSAPI authentication started > Segmentation fault: 11 (core dumped) > > 4: > SASL/GSSAPI authentication started > ldap_sasl_interactive_bind_s: Local error (-2) > additional info: SASL(-1): generic failure: GSSAPI Error: Miscellaneous > failure (see text) (unknown mech-code 2529638919 for mech unknown) > > which is very strange, since mech-code seems unnaturally large. > > 5: > SASL/GSSAPI authentication started > SASL username: mamalos@EE.AUTH.GR > SASL SSF: 56 > SASL data security layer installed. > dn:uid=mamalos,ou=people,dc=ee,dc=auth,dc=gr > > which is super, but without kinit the same command segfaulted on this machine > > 6: > SASL/GSSAPI authentication started > SASL username: mamalos@EE.AUTH.GR > SASL SSF: 56 > SASL data security layer installed. > dn:uid=mamalos,ou=people,dc=ee,dc=auth,dc=gr > > which is the exact same behavior as 5 above. > > All this means that there is no single pattern!!!!!!! > > If I gdb ldapwhoami in the clients that segfault, I get: > > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-marcel-freebsd"...(no debugging symbols > found)... > (gdb) run -d0 > Starting program: /usr/local/bin/ldapwhoami -d0 > (no debugging symbols found)...(no debugging symbols found)...(no debugging > symbols found)...(no debugging symbols found)...(no debugging symbols > found)...(no debugging symbols found)...(no debugging symbols found)...(no > debugging symbols found)...(no debugging symbols found)...(no debugging > symbols found)...(no debugging symbols found)...(no debugging symbols > found)...(no debugging symbols found)...(no debugging symbols found)...(no > debugging symbols found)...(no debugging symbols found)...(no debugging > symbols found)...(no debugging symbols found)...(no debugging symbols > found)...(no debugging symbols found)...(no debugging symbols found)...(no > debugging symbols found)...(no debugging symbols found)...(no debugging > symbols found)...(no debugging symbols found)...(no debugging symbols > found)...(no debugging symbols found)...(no debugging symbols found)...(no > debugging symbols found)...SASL/GSSAPI authentication started > > Program received signal SIGSEGV, Segmentation fault. > 0x2831e187 in free () from /lib/libc.so.7 > (gdb) where > #0 0x2831e187 in free () from /lib/libc.so.7 > #1 0x2850fb82 in gss_release_buffer () from /usr/lib/libgssapi.so.10 > #2 0x2850f552 in gss_release_name () from /usr/lib/libgssapi.so.10 > #3 0x2850bea9 in gss_init_sec_context () from /usr/lib/libgssapi.so.10 > #4 0x283f9abf in gssapi_client_mech_step () from > /usr/local/lib/sasl2/libgssapiv2.so.2 > #5 0x280e84b1 in sasl_client_step () from /usr/local/lib/libsasl2.so.2 > #6 0x28443100 in ?? () > #7 0x00000000 in ?? () > #8 0x00000000 in ?? () > #9 0xbfbfe968 in ?? () > #10 0xbfbfe954 in ?? () > #11 0xbfbfe964 in ?? () > #12 0x28445860 in ?? () > #13 0x280e83fe in sasl_client_step () from /usr/local/lib/libsasl2.so.2 > #14 0xbfbfe8a8 in ?? () > #15 0x280e9135 in sasl_client_start () from /usr/local/lib/libsasl2.so.2 > #16 0x00000000 in ?? () > #17 0x00000000 in ?? () > #18 0xbfbfe968 in ?? () > #19 0xbfbfe954 in ?? () > #20 0xbfbfe964 in ?? () > #21 0xd7a3b2da in ?? () > #22 0x283abad8 in ?? () from /lib/libc.so.7 > #23 0x00000000 in ?? () > #24 0x283ab730 in __stderrp () from /lib/libc.so.7 > #25 0xbfbfe878 in ?? () > #26 0x2838c764 in vfprintf () from /lib/libc.so.7 > Previous frame inner to this frame (corrupt stack?) > > I built openldap and cyrus-sasl on this machine from sources (not ports), and > (after a long time of trying to find out how to run configure successfully in > both installations) the outcome was exactly the same (meaning segfaults). So, > one of my admins wrote a c program that overrides gss_release_buffer > returning always 0 (what is expected after normal operation) and compiled it > as a library. The program code is nothing more than just: > > int gss_release_buffer(void *a, void *b) { > return 0; > } > > we ld_preloaded the library, and when we ran ldapwhoami the outcome was: > SASL/GSSAPI authentication started > ldap_sasl_interactive_bind_s: Local error (-2) > additional info: SASL(-1): generic failure: GSSAPI Error: Miscellaneous > failure (see text) (unknown mech-code 2529638919 for mech unknown) > > When we ran it with no kerberos tickets, we got: > > SASL/GSSAPI authentication started > ldap_sasl_interactive_bind_s: Local error (-2) > additional info: SASL(-1): generic failure: GSSAPI Error: Miscellaneous > failure (see text) (unknown mech-code 2 for mech unknown) > > The exact same errors as the aforementioned client 4 (esxi, > amd64)!!!!!!!!!!!!! > > What on earth is happening?!?!!?!?! > > Now one can easily see that there is a definite problem regarding memory > freeing, and after overcoming that the mech-code 2529638919 implies that some > segment in memory is overwritten by some "random" value, so mech-code returns > the number 2529638919 instead of a number of marginality 1. > > What is more definite, is that openldap doesn't work out-of-the-box if gssapi > support is required, and behaves randomly in different > architectures/virtualHosts/platforms. > > The problem may have been something related to line 96 in > /usr/bin/krb5-config... I don't know. > > What is sure, is that I am having second thoughts on using fbsd as my > openldap/heimdal server for my setup, which would be quite disappointing for > me, since I am using fbsd for the last many-many years, on all my laptops and > servers. The only "good part" is that the only machine that works seamlessly > (until now, at least) is the heimdal/ldap server. > > Thank you all in advance and I hope that we will find an answer to all this. > >This is a lot of information to consume. Lets start with this: All of the machines in question are of some form of FreeBSD 8. You enter gdb and very clearly it starts whining about a segfault & libc.so.7 Did you happen to upgrade these clients & server from FreeBSD 7 ? AFAIK the libc version on FreeBSD 8 was bumped to libc.so.8 If you upgraded and from source did you run make delete-old and make delete-old-libs after your make install and after your mergemaster ? Will wait here for results. Best wishes. -- jhell
George Mamalakis
2010-Feb-25 11:41 UTC
openldap client GSSAPI authentication segfaults in fbsd8stable i386
On 11/02/2010 20:45, George Mamalakis wrote:> 4: > SASL/GSSAPI authentication started > ldap_sasl_interactive_bind_s: Local error (-2) > additional info: SASL(-1): generic failure: GSSAPI Error: > Miscellaneous failure (see text) (unknown mech-code 2529638919 for > mech unknown) > > which is very strange, since mech-code seems unnaturally large.This problem has been resolved. I had an issue with my /etc/hosts file, where the name of the ldap server could not be resolved correctly (via the gssapi library I assume), and openldap client gave me this reply (both ldap server and heimdal server had the same IP (two jails on the same host)). After changing the order in which the host and its IP appeared in /etc/hosts the problem stopped (which is still strange, since ldapwhoami -D 'blabla' -W worked ok, even with the old /etc/hosts). -- George Mamalakis IT Officer Electrical and Computer Engineer (Aristotle Un. of Thessaloniki), MSc (Imperial College of London) Department of Electrical and Computer Engineering Faculty of Engineering Aristotle University of Thessaloniki phone number : +30 (2310) 994379
George Mamalakis
2010-Feb-25 14:13 UTC
openldap client GSSAPI authentication segfaults in fbsd8stable i386
To sum things up.
By fixing my /etc/hosts to read as it should (this needs some work too,
the behavior with the 'wrong' /etc/hosts is unexpected), ldapwhoami
works fine IF (AND ONLY IF) someone kinits to a user principal;
otherwise it segfaults. My default binding method is GSSAPI, hence the
segfault. If I use simple bind (ldapwhoami -W -D 'blabla') it works
fine. If I LD_PRELOAD the "hacked" library lala.so, which is created
like this:
lala.c:
int gss_release_buffer(void *a, void *b) {
return 0;
}
# gcc -c -fPIC -shared lala.c -o lala.so
and if I haven't obtained any kerberos tickets, then
# ldapwhoami
SASL/GSSAPI authentication started
Segmentation fault: 11 (core dumped)
once I ldpreload the above fake-library, then:
# LD_PRELOAD=./lala.so ldapwhoami
SASL/GSSAPI authentication started
ldap_sasl_interactive_bind_s: Local error (-2)
additional info: SASL(-1): generic failure: GSSAPI Error:
Miscellaneous failure (see text) (unknown mech-code 2 for mech unknown)
which is what is expected.
This, maybe implies that something is freed by gss_release_buffer that
normally shouldn't.
amd64 won't hang in the same test (so no need to ld_preload anything),
but shares the same problem with i386 when /etc/hosts is not as expected
(to recreate the /etc/hosts problem, place in your /etc/hosts file two
fqdns for the ldap server's IP, but write the ldap server's fqdn second
in turn).
Thank you all and have a nice evening.
--
George Mamalakis
IT Officer
Electrical and Computer Engineer (Aristotle Un. of Thessaloniki),
MSc (Imperial College of London)
Department of Electrical and Computer Engineering
Faculty of Engineering
Aristotle University of Thessaloniki
phone number : +30 (2310) 994379