Hi all,
this is a really long email, sorry, i try to provide all the
necessary info for you to digg it.
I got a farm with 4 6.0-STABLE boxes which were up and running
since december 2005.
All these boxes were running openldap 2.3.11 with syncrepl, 1
provider, 2 consumer and 1 test box; of these 4 boxes one (the
only dual Xeon) box is used to build userland and all 4 kernels as
described in the handbook and is used as syncrepl provider.
Some days ago i decided to switch to RELENG_6_2 and to OpenLDAP 2.3.34
trying to narrow down some syncrepl casual failures i was getting, so
did the usual procedure after reading /usr/src/UPDATING
and /usr/port/UPDATING and all went smoothly for the 2 consumers and
the test box. So proceeded with the main box, the one which actually
has compiled _all_ the sources, and then problems start coming, well
one problem.
Slapd start crashing with segfaul signal 11. It simply start then
after some (very small) amount of operation it crash.
Now i changed the farm to make act the test box as the syncrepl
provider (the master LDAP tree) and all is running fine again. Please
note that _all_ the sources (src tree and ports tree) have been
compiled from the box on which OpenLDAP is crashing.
After the blah blah... here are some infos you may find useful.
The dmesg of the box:
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights
reserved. FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.2-RELEASE-p2 #2: Thu Mar 15 11:16:59 CET 2007
root@bellerofonte.cedoc.mo.it:/usr/obj/usr/src/sys/GENERIC
ACPI APIC Table: <PTLTD APIC >
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2799.22-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0xf29 Stepping = 9
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4400<CNTX-ID,<b14>>
Logical CPUs per core: 2
real memory = 2146959360 (2047 MB)
avail memory = 2095939584 (1998 MB)
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
ioapic3 <Version 2.0> irqs 72-95 on motherboard
ioapic4 <Version 2.0> irqs 96-119 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.17.2 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413,
RF5413) acpi0: <PTLTD RSDT> on motherboard
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pci0: <unknown> at device 0.1 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <base peripheral, interrupt controller> at device 28.0 (no driver
attached) pcib2: <ACPI PCI-PCI bridge> at device 29.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pci1: <base peripheral, interrupt controller> at device 30.0 (no driver
attached) pcib3: <ACPI PCI-PCI bridge> at device 31.0 on pci1
pci3: <ACPI PCI bus> on pcib3
em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port
0x3000-0x303f mem 0xf8200000-0xf821ffff irq 28 at device 2.0 on pci3
em0: Ethernet address: 00:30:48:2d:08:32 em1: <Intel(R) PRO/1000
Network Connection Version - 6.2.9> port 0x3040-0x307f mem
0xf8220000-0xf823ffff irq 29 at device 2.1 on pci3 em1: Ethernet
address: 00:30:48:2d:08:33 pcib4: <ACPI PCI-PCI bridge> at device 3.0
on pci0 pci4: <ACPI PCI bus> on pcib4 pci4: <base peripheral, interrupt
controller> at device 28.0 (no driver attached) pcib5: <ACPI PCI-PCI
bridge> at device 29.0 on pci4
pci5: <ACPI PCI bus> on pcib5
pci4: <base peripheral, interrupt controller> at device 30.0 (no driver
attached) pcib6: <ACPI PCI-PCI bridge> at device 31.0 on pci4
pci6: <ACPI PCI bus> on pcib6
asr0: <Adaptec Caching SCSI RAID> mem
0xf8400000-0xf84fffff,0xfb000000-0xfbffffff,0xfc000000-0xfdffffff irq
72 at device 1.0 on pci6 asr0: [GIANT-LOCKED] asr0: ADAPTEC 2010S FW
Rev. 3B0A, 2 channel, 256 CCBs, Protocol I2O uhci0: <Intel 82801CA/CAM
(ICH3) USB controller USB-A> port 0x2000-0x201f irq 16 at device 29.0
on pci0 uhci0: [GIANT-LOCKED] usb0: <Intel 82801CA/CAM (ICH3) USB
controller USB-A> on uhci0 usb0: USB revision 1.0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 82801CA/CAM (ICH3) USB controller USB-B> port
0x2020-0x203f irq 19 at device 29.1 on pci0 uhci1: [GIANT-LOCKED]
usb1: <Intel 82801CA/CAM (ICH3) USB controller USB-B> on uhci1
usb1: USB revision 1.0
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 82801CA/CAM (ICH3) USB controller USB-C> port
0x2040-0x205f irq 18 at device 29.2 on pci0 uhci2: [GIANT-LOCKED]
usb2: <Intel 82801CA/CAM (ICH3) USB controller USB-C> on uhci2
usb2: USB revision 1.0
uhub2: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub2: 2 ports with 2 removable, self powered
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pci7: <display, VGA> at device 1.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH3 UDMA100 controller> port
0x170-0x177,0x376,0x2060-0x206f at device 31.1 on pci0 ata2: <ATA
channel 0> on atapci0 ata1: <ATA channel 1> on atapci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0 fdc0: [FAST]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc7fff,0xc8000-0xc8fff,0xc9000-0xcefff,0xe0000-0xe3fff on isa0
ppc0: parallel port not found. sc0: <System console> at flags 0x100 on
isa0 sc0: VGA <16 virtual consoles, flags=0x300>
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0 Timecounter "TSC" frequency 2799219532 Hz quality 800
Timecounters tick every 1.000 msec
acd0: CDROM <CD-224E/1.9A> at ata1-master UDMA33
ses0 at asr0 bus 0 target 6 lun 0
ses0: <SUPER GEM318 0> Fixed Processor SCSI-2 device
ses0: SAF-TE Compliant Device
da0 at asr0 bus 0 target 0 lun 0
da0: <ADAPTEC RAID-1 3B0A> Fixed Direct Access SCSI-2 device
da0: Tagged Queueing Enabled
da0: 70007MB (143374336 512 byte sectors: 255H 63S/T 8924C)
Trying to mount root from ufs:/dev/da0s1a
em0: link state changed to UP
pid 13548 (slapd), uid 389: exited on signal 11
The relevant port installed are:
cyrus-sasl-2.1.22
db44-4.4.20.4
libltdl-1.5.22_1
openldap-sasl-client-2.3.34
openldap-server-2.3.34
with on the server the syncrepl provider enabled.
Here i attach the result of two runs inside gdb with bt, one with
libpthread and the other with libthr changed via libmap without
specifying a particular app so all programs which use libpthread then
will use libthr.
I cut off some previous output since it would have been very long, if
you're interested i got full output for you. I also got the box idle
(for some days) for any test you would like me to conduct.
With libpthread:
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are welcome to change it and/or distribute copies of it under
certain conditions. Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details. This GDB was configured as "i386-marcel-freebsd"...(no
debugging symbols found)... (gdb) run -u ldap -g ldap -d 1
Starting program: /usr/local/libexec/slapd -u ldap -g ldap -d 1
(no debugging symbols found)...(no debugging symbols found)...(no
debugging symbols found)...(no debugging symbols found)...(no debugging
symbols found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...warning:
Unable to get location for thread creation breakpoint: generic error
[New LWP 100066] (no debugging symbols found)...(no debugging symbols
found)...[New Thread 0x8127000 (LWP 100066)] @(#) $OpenLDAP: slapd
2.3.34 (Mar 21 2007 13:04:49) $
root@freebsdvm.cedoc.mo.it:/usr/ports/net/openldap23-server/work/openldap-2.3.34/servers/slapd
daemon_init: listen on ldap:/// daemon_init: 1 listeners to open...
ldap_url_parse_ext(ldap:///) daemon: listener initialized ldap:///
daemon_init: 2 listeners opened slapd init: initiated server. (no
debugging symbols found)...(no debugging symbols found)...(no debugging
symbols found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...(no
debugging symbols found)...(no debugging symbols found)...(no debugging
symbols found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...slap_sasl_init: initialized!
(no debugging symbols found)...(no debugging symbols
found)...bdb_back_initialize: initialize BDB backend
bdb_back_initialize: Sleepycat Software: Berkeley DB 4.4.20: (January
10, 2006)>>> dnNormalize: <ou=biblioteche,o=CeDoc>
<<< dnNormalize: <ou=biblioteche,o=cedoc>>>> dnNormalize: <ou=internet-accounts,o=CeDoc>
<<< dnNormalize:
<ou=internet-accounts,o=cedoc>>>> dnNormalize:
<uid=proxy,ou=servizi,ou=CD,ou=biblioteche,o=CeDoc>
<<< dnNormalize:
<uid=proxy,ou=servizi,ou=cd,ou=biblioteche,o=cedoc>>>> dnNormalize: <ou=utenti,o=CeDoc>
<<< dnNormalize: <ou=utenti,o=cedoc>>>> dnNormalize: <ou=CD,ou=biblioteche,o=CeDoc>
<<< dnNormalize:
<ou=cd,ou=biblioteche,o=cedoc>>>> dnNormalize: <ou=mail-accounts,o=CeDoc>
<<< dnNormalize:
<ou=mail-accounts,o=cedoc>>>> dnNormalize:
<userid=postfix,ou=servizi,ou=CD,ou=biblioteche,o=CeDoc>
<<< dnNormalize:
<uid=postfix,ou=servizi,ou=cd,ou=biblioteche,o=cedoc>>>> dnNormalize: <ou=mail-biblioteche,o=CeDoc>
<<< dnNormalize:
<ou=mail-biblioteche,o=cedoc>>>> dnNormalize:
<userid=postfix,ou=servizi,ou=CD,ou=biblioteche,o=CeDoc>
<<< dnNormalize:
<uid=postfix,ou=servizi,ou=cd,ou=biblioteche,o=cedoc>>>> dnNormalize: <cn=Manager,o=CeDoc>
<<< dnNormalize: <cn=manager,o=cedoc>
bdb_db_init: Initializing BDB database>>> dnPrettyNormal: <o=CeDoc>
<<< dnPrettyNormal: <o=CeDoc>,
<o=cedoc>>>> dnPrettyNormal: <cn=Manager,o=CeDoc>
<<< dnPrettyNormal: <cn=Manager,o=CeDoc>,
<cn=manager,o=cedoc>>>> dnNormalize: <cn=Subschema>
<<< dnNormalize: <cn=subschema>
matching_rule_use_init
1.2.840.113556.1.4.8
[..]
=> entry_encode(0x0001da87): uid=prandi.anna,ou=utenti,o=CeDoc
bdb_modify: updated id=0001da87 dn="uid=prandi.anna,ou=utenti,o=CeDoc"
send_ldap_result: conn=0 op=40 p=3
bdb_dn2entry("o=cedoc")
bdb_modify_internal: 0x00000001: o=CeDoc
oc_check_required entry (o=CeDoc), objectClass "organization"
oc_check_allowed type "objectClass"
oc_check_allowed type "o"
oc_check_allowed type "structuralObjectClass"
oc_check_allowed type "entryUUID"
oc_check_allowed type "creatorsName"
oc_check_allowed type "createTimestamp"
oc_check_allowed type "entryCSN"
oc_check_allowed type "modifiersName"
oc_check_allowed type "modifyTimestamp"
oc_check_allowed type "contextCSN"
=> entry_encode(0x00000001): o=CeDoc
bdb_modify: updated id=00000001 dn="o=CeDoc"
send_ldap_result: conn=0 op=40 p=3
send_ldap_response: msgid=41 tag=103 err=0
ber_flush: 14 bytes to sd 12
[New Thread 0x81a1e00 (LWP 100067)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x81a1e00 (LWP 100067)]
0x080cc823 in avl_find ()
(gdb) bt
#0 0x080cc823 in avl_find ()
#1 0x080be6a4 in overlay_init ()
#2 0x08073cdc in slap_req2res ()
#3 0x0807443f in slap_send_ldap_result ()
#4 0x285b1e1c in bdb_modify () from
/usr/local/libexec/openldap/back_bdb-2.3.so.2
#5 0x080b37d0 in overlay_op_walk ()
#6 0x080b38c8 in overlay_op_walk ()
#7 0x0807aa7a in fe_op_modify ()
#8 0x0807b4e9 in do_modify ()
#9 0x080681ee in connection_client_stop ()
#10 0x28150371 in ldap_int_thread_pool_wrapper () from
/usr/local/lib/libldap_r-2.3.so.2
#11 0x283043a5 in pthread_create () from /lib/libpthread.so.2
#12 0x283c1137 in _ctx_start () from /lib/libc.so.6
With libthr:
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are welcome to change it and/or distribute copies of it under
certain conditions. Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details. This GDB was configured as "i386-marcel-freebsd"...(no
debugging symbols found)... (gdb) run -u ldap -g ldap -d 1
Starting program: /usr/local/libexec/slapd -u ldap -g ldap -d 1
(no debugging symbols found)...(no debugging symbols found)...(no
debugging symbols found)...(no debugging symbols found)...(no debugging
symbols found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...[New LWP
100067] (no debugging symbols found)...(no debugging symbols found)...
[New Thread 0x811f000 (LWP 100067)] @(#) $OpenLDAP: slapd 2.3.34 (Mar
21 2007 13:04:49) $
root@freebsdvm.cedoc.mo.it:/usr/ports/net/openldap23-server/work/openldap-2.3.34/servers/slapd
daemon_init: listen on ldap:/// daemon_init: 1 listeners to open...
ldap_url_parse_ext(ldap:///) daemon: listener initialized ldap:///
daemon_init: 2 listeners opened slapd init: initiated server. (no
debugging symbols found)...(no debugging symbols found)...(no debugging
symbols found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...(no
debugging symbols found)...(no debugging symbols found)...(no debugging
symbols found)...(no debugging symbols found)...(no debugging symbols
found)...(no debugging symbols found)...slap_sasl_init: initialized!
(no debugging symbols found)...(no debugging symbols
found)...bdb_back_initialize: initialize BDB backend
bdb_back_initialize: Sleepycat Software: Berkeley DB 4.4.20: (January
10, 2006)>>> dnNormalize: <ou=biblioteche,o=CeDoc>
<<< dnNormalize: <ou=biblioteche,o=cedoc>>>> dnNormalize: <ou=internet-accounts,o=CeDoc>
<<< dnNormalize:
<ou=internet-accounts,o=cedoc>>>> dnNormalize:
<uid=proxy,ou=servizi,ou=CD,ou=biblioteche,o=CeDoc>
<<< dnNormalize:
<uid=proxy,ou=servizi,ou=cd,ou=biblioteche,o=cedoc>>>> dnNormalize: <ou=utenti,o=CeDoc>
<<< dnNormalize: <ou=utenti,o=cedoc>>>> dnNormalize: <ou=CD,ou=biblioteche,o=CeDoc>
<<< dnNormalize:
<ou=cd,ou=biblioteche,o=cedoc>>>> dnNormalize: <ou=mail-accounts,o=CeDoc>
<<< dnNormalize:
<ou=mail-accounts,o=cedoc>>>> dnNormalize:
<userid=postfix,ou=servizi,ou=CD,ou=biblioteche,o=CeDoc>
<<< dnNormalize:
<uid=postfix,ou=servizi,ou=cd,ou=biblioteche,o=cedoc>>>> dnNormalize: <ou=mail-biblioteche,o=CeDoc>
<<< dnNormalize:
<ou=mail-biblioteche,o=cedoc>>>> dnNormalize:
<userid=postfix,ou=servizi,ou=CD,ou=biblioteche,o=CeDoc>
<<< dnNormalize:
<uid=postfix,ou=servizi,ou=cd,ou=biblioteche,o=cedoc>>>> dnNormalize: <cn=Manager,o=CeDoc>
<<< dnNormalize: <cn=manager,o=cedoc>
bdb_db_init: Initializing BDB database>>> dnPrettyNormal: <o=CeDoc>
<<< dnPrettyNormal: <o=CeDoc>,
<o=cedoc>>>> dnPrettyNormal: <cn=Manager,o=CeDoc>
<<< dnPrettyNormal: <cn=Manager,o=CeDoc>,
<cn=manager,o=cedoc>>>> dnNormalize: <cn=Subschema>
<<< dnNormalize: <cn=subschema>
matching_rule_use_init
1.2.840.113556.1.4.8
[..]
=> entry_encode(0x0001da87): uid=prandi.anna,ou=utenti,o=CeDoc
bdb_modify: updated id=0001da87 dn="uid=prandi.anna,ou=utenti,o=CeDoc"
send_ldap_result: conn=0 op=31 p=3
bdb_dn2entry("o=cedoc")
bdb_modify_internal: 0x00000001: o=CeDoc
oc_check_required entry (o=CeDoc), objectClass "organization"
oc_check_allowed type "objectClass"
oc_check_allowed type "o"
oc_check_allowed type "structuralObjectClass"
oc_check_allowed type "entryUUID"
oc_check_allowed type "creatorsName"
oc_check_allowed type "createTimestamp"
oc_check_allowed type "entryCSN"
oc_check_allowed type "modifiersName"
oc_check_allowed type "modifyTimestamp"
oc_check_allowed type "contextCSN"
=> entry_encode(0x00000001): o=CeDoc
bdb_modify: updated id=00000001 dn="o=CeDoc"
send_ldap_result: conn=0 op=31 p=3
send_ldap_response: msgid=32 tag=103 err=0
ber_flush: 14 bytes to sd 12
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x81aa000 (LWP 100136)]
0x080cc823 in avl_find ()
(gdb) bt
#0 0x080cc823 in avl_find ()
#1 0x080be6a4 in overlay_init ()
#2 0x08073cdc in slap_req2res ()
#3 0x0807443f in slap_send_ldap_result ()
#4 0x2859ee1c in bdb_modify () from
/usr/local/libexec/openldap/back_bdb-2.3.so.2
#5 0x080b37d0 in overlay_op_walk ()
#6 0x080b38c8 in overlay_op_walk ()
#7 0x0807aa7a in fe_op_modify ()
#8 0x0807b4e9 in do_modify ()
#9 0x080681ee in connection_client_stop ()
#10 0x28150371 in ldap_int_thread_pool_wrapper () from
/usr/local/lib/libldap_r-2.3.so.2
#11 0x282fd5cf in pthread_create () from /usr/lib/libthr.so.2
#12 0x00000000 in ?? ()
Well you rich the end and i thank you so much for reading all this
stuff, please let me know if you got any clue or any test or other
infos you wish.
With best regards.
--
Massimo.run();
You may be recognized soon. Hide.