Good day.
Last 2 years (maybe when began using bdb backend), we get slapd crash on
read load.
System on low load work with monit monitoring and fails 1-3 in month.
When load up crashes frequency up too.
Tuning helped but not much.
load about 20-30 queryes/sec in peak.
and crashes every hour.
Problem watched on Freebsd7,7.1,7.2 i386, amd64 and openldap2.3,2.4
(bdb,hdb backends) in any combinations.
I tested openldap 2.4 on debian lenny, its work under my load without
tuning (once was crashed whole linux :), but not slapd).
Mybe some freebsd tuning needed?
Some debug:
ber_scanf fmt ({m) ber:
ber_dump: buf=0x8037161b0 ptr=0x803716248 end=0x803716274 len=44
0000: 30 84 00 00 00 26 04 16 31 2e 32 2e 38 34 30 2e
0....&..1.2.840.
0010: 31 31 33 35 35 36 2e 31 2e 34 2e 33 31 39 04 0c
113556.1.4.319..
0020: 30 84 00 00 00 06 02 02 03 e8 04 00 0...........
ber_scanf fmt (m) ber:
ber_dump: buf=0x8037161b0 ptr=0x803716266 end=0x803716274 len=14
0000: 00 0c 30 84 00 00 00 06 02 02 03 e8 04 00 ..0...........
=> get_ctrls: oid="1.2.840.113556.1.4.319" (noncritical)
ber_scanf fmt ({im}) ber:
ber_dump: buf=0x803831000 ptr=0x803831000 end=0x80383100c len=12
0000: 30 84 00 00 00 06 02 02 03 e8 04 00 0...........
<= get_ctrls: n=1 rc=0 err=""
attrs: cn userPassword memberUid uniqueMember gidNumber
conn=105 op=1 SRCH base="ou=staff,dc=ulgsm,dc=ru" scope=1 deref=0
filter="(&(objectClass=posixGroup))"
conn=105 op=1 SRCH attr=cn userPassword memberUid uniqueMember gidNumber
slap_global_control: unavailable control: 1.2.840.113556.1.4.319
==> limits_get: conn=105 op=1 dn="cn=bind,ou=staff,dc=ulgsm,dc=ru"
=> hdb_search
bdb_dn2entry("ou=staff,dc=ulgsm,dc=ru")
search_candidates: base="ou=staff,dc=ulgsm,dc=ru" (0x00000002) scope=1
=> hdb_dn2idl("ou=staff,dc=ulgsm,dc=ru")
=> bdb_filter_candidates
AND
=> bdb_list_candidates 0xa0
=> bdb_filter_candidates
OR
=> bdb_list_candidates 0xa1
=> bdb_filter_candidates
EQUALITY
=> bdb_equality_candidates (objectClass)
=> key_read
zsh: segmentation fault /usr/local/libexec/slapd -d -1
--
Email: alexs@ulgsm.ru
Email/Jabber: alexs@ulgsm.ru
???. +7 951 0985685, ??. 368
Hi, On Tue, Oct 27, 2009 at 11:25:16AM +0300, alexs@ulgsm.ru wrote:> Last 2 years (maybe when began using bdb backend), we get slapd crash on > read load. > System on low load work with monit monitoring and fails 1-3 in month. > When load up crashes frequency up too. > > Tuning helped but not much. > > load about 20-30 queryes/sec in peak. > and crashes every hour. > > Problem watched on Freebsd7,7.1,7.2 i386, amd64 and openldap2.3,2.4 > (bdb,hdb backends) in any combinations. > > I tested openldap 2.4 on debian lenny, its work under my load without > tuning (once was crashed whole linux :), but not slapd). > > Mybe some freebsd tuning needed?We have slapd running on several servers with read loads of between 50 and 200 requests per second and it runs rock stable. What comes tomind, did your server crash at some point? Have you tried to either do a db_recover on the database files (while slapd is not running of course) or slapcat/slapadd to rebuild the BDB from scratch? I get the feeling your BDB is somehow damaged. - Oliver -- | Oliver Brandmueller http://sysadm.in/ ob@sysadm.in | | Ich bin das Internet. Sowahr ich Gott helfe. |
* Oliver Brandmueller <ob@e-Gitt.NET> [2009-10-27 09:56:48 +0100]:> Hi, > > On Tue, Oct 27, 2009 at 11:25:16AM +0300, alexs@ulgsm.ru wrote: > > Last 2 years (maybe when began using bdb backend), we get slapd crash on > > read load. > > System on low load work with monit monitoring and fails 1-3 in month. > > When load up crashes frequency up too. > > > > Tuning helped but not much. > > > > load about 20-30 queryes/sec in peak. > > and crashes every hour. > > > > Problem watched on Freebsd7,7.1,7.2 i386, amd64 and openldap2.3,2.4 > > (bdb,hdb backends) in any combinations. > > > > I tested openldap 2.4 on debian lenny, its work under my load without > > tuning (once was crashed whole linux :), but not slapd). > > > > Mybe some freebsd tuning needed? > > We have slapd running on several servers with read loads of between 50 > and 200 requests per second and it runs rock stable. > > What comes tomind, did your server crash at some point? Have you tried > to either do a db_recover on the database files (while slapd is not > running of course) or slapcat/slapadd to rebuild the BDB from scratch? I > get the feeling your BDB is somehow damaged.I reinstall opneldap, remove all tunung, make slapadd < backup.ldif and get about 50 failures at the night. :(> > - Oliver > > -- > | Oliver Brandmueller http://sysadm.in/ ob@sysadm.in | > | Ich bin das Internet. Sowahr ich Gott helfe. | > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"-- Email: alexs@ulgsm.ru Email/Jabber: alexs@ulgsm.ru
* Dewayne Geraghty <dewayne.geraghty@heuristicsystems.com.au> [2009-10-28 19:02:58 +1100]:> Alexs, Would you please provide the output to the following two questions: > /usr/local/libexec/slapd -V~]>/usr/local/libexec/slapd -V @(#) $OpenLDAP: slapd 2.4.19 (Oct 28 2009 09:02:40) $ root@bazar:/usr/wrkdir/usr/ports/net/openldap24-server/work/openldap-2.4.19/servers/slapd> ldd /usr/local/libexec/slapd~]>ldd /usr/local/libexec/slapd /usr/local/libexec/slapd: libldap_r-2.4.so.7 => /usr/local/lib/libldap_r-2.4.so.7 (0x80073b000) liblber-2.4.so.7 => /usr/local/lib/liblber-2.4.so.7 (0x800881000) libltdl.so.7 => /usr/local/lib/libltdl.so.7 (0x80098e000) libdb-4.7.so.0 => /usr/local/lib/libdb-4.7.so.0 (0x800a97000) libssl.so.5 => /usr/lib/libssl.so.5 (0x800cf7000) libcrypto.so.5 => /lib/libcrypto.so.5 (0x800e41000) libfetch.so.5 => /usr/lib/libfetch.so.5 (0x8010d3000) libcom_err.so.4 => /usr/lib/libcom_err.so.4 (0x8011e1000) libcrypt.so.4 => /lib/libcrypt.so.4 (0x8012e3000) libwrap.so.5 => /usr/lib/libwrap.so.5 (0x8013fc000) libthr.so.3 => /lib/libthr.so.3 (0x801505000) libc.so.7 => /lib/libc.so.7 (0x80161d000)> It will be helpful to indicate what libraries that you are linking to; and what > version are you using. Currently openldap is 2.4.19. Please note that the > FreeBSD-stable list will be helpful to you if there are operating system > issues. I don't think that you've established an operating system problem with > the information provided. You have asked a good question which would trigger > responses if other people were experiencing the same problem. > > Similar to earlier replies, LDAP has been reliable on my 7.2Stable, and I'm > tracking 3 days behind cvs. My production machines with openldap, run on > average for 600 days without any crashes or reboots; which is what you should > expect.Maybe its my mistake in freebsd and openldap configuration. I cant find it long time. Today I tried on netbsd, (there are openldap 2.4.16 in pkgsrc), work perfect! So it work on linux and netbsd, soon i`ll try on solaris.> > From your description of your problem, you might need to contact the Openldap > mailing list; but you'll need more detail. > Kind regards, Dewayne.Yes. But thea are strong moderated. I think its my english why moderator rejected me. -- Email: alexs@ulgsm.ru Email/Jabber: alexs@ulgsm.ru
* Dewayne Geraghty <dewayne.geraghty@heuristicsystems.com.au> [2009-10-29 10:39:22 +1100]:> Alexs, > Thank-you. You seem to be using a default setup from ports. > > If you are using the same configuration files for slapd.conf and > /var/openldap-data/DB_CONFIG files on the three operating systems, and FreeBSD > isn't working, I'm struggling to think of any load induced problems that it > might be from the information provided. > > Rather than asking you a lot of questions, I think you need to rule out that > disks can handle the load, that bdb is able to service the queries, and that > the tools creating the ldap queries are using the same assumptions that the > ldap server is using, etc. I think it unlikely, but I couldn't rule out a bad > ldap query as a source of the problem.I tryed put database on RAM drive (created by mdconfig -t malloc), fails anyway. All configs in defaults, inly indexes and some acls added.> Review the "man slapd.conf" and determine the loglevel that you think > appropriate, I'd suggest 2047 as a good start. > /usr/local/libexec/slapd -d 2047 -4 -h ldapi://%2fvar%2frun%2fopenldap%2fldapi/ > ldap://alexs-ldap-server/For geting crashes (for this mail only, real crashes needs to wait) I am use small script via nss_ldap. ~]>cat test.sh #!/bin/sh while true do id test > /dev/null done With 3 running scripts get fault in 30 seconds http://pastebin.org/49205 http://pastebin.org/49211 http://pastebin.org/49213 So crashes in different places. I think need some another debuging. I tried truss slapd process, but there different places too.> > Good luck, Dewayne.-- Email: alexs@ulgsm.ru Email/Jabber: alexs@ulgsm.ru