thr3ads.net - samba - [Samba] "Random" crashes of Samba as AD DC [Jul 2018]

If this information is useful, please help other people find it:
Share via:

Mauel, Volker

2018-Jul-17 16:30 UTC

[Samba] "Random" crashes of Samba as AD DC

Hey all,

up until 2 weeks ago my samba DC was running just fine, then it began crashing
randomly.
I was using 4.8.0 before, now upgraded to a self-compiled 4.8.3 with default
build parameters on an up-to-date centos 7.5 with disabled SELinux and bind9 as
DNS backend with samba_dlz.

The log output at log level = 10 is as follows:
Jul 17 14:49:20 hostname samba[4998]: [2018/07/17 14:49:20.587655,  0, pid=4998,
effective(0, 0), real(0, 0)] ../lib/util/fault.c:79(fault_report)
Jul 17 14:49:20 hostname samba[4998]:  
==============================================================Jul 17 14:49:20
hostname samba[4998]: [2018/07/17 14:49:20.587752,  0, pid=4998, effective(0,
0), real(0, 0)] ../lib/util/fault.c:80(fault_report)
Jul 17 14:49:20 hostname samba[4998]:   INTERNAL ERROR: Signal 11 in pid 4998
(4.8.3)
Jul 17 14:49:20 hostname samba[4998]:   Please read the Trouble-Shooting section
of the Samba HOWTO
Jul 17 14:49:20 hostname samba[4998]: [2018/07/17 14:49:20.587828,  0, pid=4998,
effective(0, 0), real(0, 0)] ../lib/util/fault.c:82(fault_report)
Jul 17 14:49:20 hostname samba[4998]:  
==============================================================Jul 17 14:49:20
hostname samba[4998]: [2018/07/17 14:49:20.587871,  0, pid=4998, effective(0,
0), real(0, 0)] ../lib/util/fault.c:128(smb_panic_default)
Jul 17 14:49:20 hostname samba[4998]:   smb_panic_default: PANIC (pid 4998):
internal error
Jul 17 14:49:20 hostname samba[4998]: [2018/07/17 14:49:20.588558,  0, pid=4998,
effective(0, 0), real(0, 0)] ../lib/util/fault.c:261(log_stack_trace)
Jul 17 14:49:20 hostname samba[4998]:   BACKTRACE: 45 stack frames:
Jul 17 14:49:20 hostname samba[4998]:    #0
/usr/local/samba/lib/libsamba-util.so.0(log_stack_trace+0x1f) [0x7f81f95e9332]
Jul 17 14:49:20 hostname samba[4998]:    #1
/usr/local/samba/lib/libsamba-util.so.0(+0x230ad) [0x7f81f95e90ad]
Jul 17 14:49:20 hostname samba[4998]:    #2
/usr/local/samba/lib/libsamba-util.so.0(log_stack_trace+0) [0x7f81f95e9313]
Jul 17 14:49:20 hostname samba[4998]:    #3
/usr/local/samba/lib/libsamba-util.so.0(+0x22fad) [0x7f81f95e8fad]
Jul 17 14:49:20 hostname samba[4998]:    #4
/usr/local/samba/lib/libsamba-util.so.0(+0x22fc2) [0x7f81f95e8fc2]
Jul 17 14:49:20 hostname samba[4998]:    #5 /lib64/libpthread.so.0(+0xf6d0)
[0x7f81f98566d0]
Jul 17 14:49:20 hostname samba[4998]:    #6
/usr/local/samba/lib/libsamdb.so.0(+0xf1e0) [0x7f81f74b91e0]
Jul 17 14:49:20 hostname samba[4998]:    #7
/usr/local/samba/lib/libsamdb.so.0(DsCrackNameOneName+0xd6a) [0x7f81f74b7e6b]
Jul 17 14:49:20 hostname samba[4998]:    #8
/usr/local/samba/lib/libsamdb.so.0(dcesrv_drsuapi_CrackNamesByNameFormat+0xef)
[0x7f81f74b9b20]
Jul 17 14:49:20 hostname samba[4998]:    #9
/usr/local/samba/lib/libdcerpc-server.so.0(+0x6d8a1) [0x7f81e249d8a1]
Jul 17 14:49:20 hostname samba[4998]:    #10
/usr/local/samba/lib/libdcerpc-server.so.0(+0x70bd8) [0x7f81e24a0bd8]
Jul 17 14:49:20 hostname samba[4998]:    #11
/usr/local/samba/lib/libdcerpc-server.so.0(+0x91f98) [0x7f81e24c1f98]
Jul 17 14:49:20 hostname samba[4998]:    #12
/usr/local/samba/lib/libdcerpc-server.so.0(+0x92a1c) [0x7f81e24c2a1c]
Jul 17 14:49:20 hostname samba[4998]:    #13
/usr/local/samba/lib/libdcerpc-server.so.0(+0x943f2) [0x7f81e24c43f2]
Jul 17 14:49:20 hostname samba[4998]:    #14
/usr/local/samba/lib/private/libtevent.so.0(_tevent_req_notify_callback+0x6a)
[0x7f81f8f8421f]
Jul 17 14:49:20 hostname samba[4998]:    #15
/usr/local/samba/lib/private/libtevent.so.0(+0x72f2) [0x7f81f8f842f2]
Jul 17 14:49:20 hostname samba[4998]:    #16
/usr/local/samba/lib/private/libtevent.so.0(_tevent_req_done+0x25)
[0x7f81f8f84319]
Jul 17 14:49:20 hostname samba[4998]:    #17
/usr/local/samba/lib/libdcerpc-binding.so.0(+0x1dc3e) [0x7f81f44bec3e]
Jul 17 14:49:20 hostname samba[4998]:    #18
/usr/local/samba/lib/private/libtevent.so.0(_tevent_req_notify_callback+0x6a)
[0x7f81f8f8421f]
Jul 17 14:49:20 hostname samba[4998]:    #19
/usr/local/samba/lib/private/libtevent.so.0(+0x72f2) [0x7f81f8f842f2]
Jul 17 14:49:20 hostname samba[4998]:    #20
/usr/local/samba/lib/private/libtevent.so.0(_tevent_req_done+0x25)
[0x7f81f8f84319]
Jul 17 14:49:20 hostname samba[4998]:    #21
/usr/local/samba/lib/private/libsamba-sockets-samba4.so(+0xd705)
[0x7f81f46d8705]
Jul 17 14:49:20 hostname samba[4998]:    #22
/usr/local/samba/lib/private/libsamba-sockets-samba4.so(+0xd92f)
[0x7f81f46d892f]
Jul 17 14:49:20 hostname samba[4998]:    #23
/usr/local/samba/lib/private/libtevent.so.0(_tevent_req_notify_callback+0x6a)
[0x7f81f8f8421f]
Jul 17 14:49:20 hostname samba[4998]:    #24
/usr/local/samba/lib/private/libtevent.so.0(+0x72f2) [0x7f81f8f842f2]
Jul 17 14:49:20 hostname samba[4998]:    #25
/usr/local/samba/lib/private/libtevent.so.0(_tevent_req_done+0x25)
[0x7f81f8f84319]
Jul 17 14:49:20 hostname samba[4998]:    #26
/usr/local/samba/lib/private/libsamba-sockets-samba4.so(+0xcc55)
[0x7f81f46d7c55]
Jul 17 14:49:20 hostname samba[4998]:    #27
/usr/local/samba/lib/private/libtevent.so.0(_tevent_req_notify_callback+0x6a)
[0x7f81f8f8421f]
Jul 17 14:49:20 hostname samba[4998]:    #28
/usr/local/samba/lib/private/libtevent.so.0(+0x72f2) [0x7f81f8f842f2]
Jul 17 14:49:20 hostname samba[4998]:    #29
/usr/local/samba/lib/private/libtevent.so.0(+0x7417) [0x7f81f8f84417]
Jul 17 14:49:20 hostname samba[4998]:    #30
/usr/local/samba/lib/private/libtevent.so.0(tevent_common_loop_immediate+0x1f5)
[0x7f81f8f835cc]
Jul 17 14:49:20 hostname samba[4998]:    #31
/usr/local/samba/lib/private/libtevent.so.0(+0xedb6) [0x7f81f8f8bdb6]
Jul 17 14:49:20 hostname samba[4998]:    #32
/usr/local/samba/lib/private/libtevent.so.0(+0xbb57) [0x7f81f8f88b57]
Jul 17 14:49:20 hostname samba[4998]:    #33
/usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_once+0x10f)
[0x7f81f8f8246a]
Jul 17 14:49:20 hostname samba[4998]:    #34
/usr/local/samba/lib/private/libtevent.so.0(tevent_common_loop_wait+0x25)
[0x7f81f8f8277a]
Jul 17 14:49:20 hostname samba[4998]:    #35
/usr/local/samba/lib/private/libtevent.so.0(+0xbbf9) [0x7f81f8f88bf9]
Jul 17 14:49:20 hostname samba[4998]:    #36
/usr/local/samba/lib/private/libtevent.so.0(_tevent_loop_wait+0x2b)
[0x7f81f8f8281d]
Jul 17 14:49:20 hostname samba[4998]:    #37
/usr/local/samba/lib/process_model/standard.so(+0x2793) [0x7f81eaacf793]
Jul 17 14:49:20 hostname samba[4998]:    #38
/usr/local/samba/lib/private/libservice-samba4.so(task_server_startup+0x99)
[0x7f81f8b60b2b]
Jul 17 14:49:20 hostname samba[4998]:    #39
/usr/local/samba/lib/private/libservice-samba4.so(+0x2bb4) [0x7f81f8b5ebb4]
Jul 17 14:49:20 hostname samba[4998]:    #40
/usr/local/samba/lib/private/libservice-samba4.so(server_service_startup+0x16a)
[0x7f81f8b5ed39]
Jul 17 14:49:20 hostname samba[4998]:    #41
/usr/local/samba/sbin/samba(+0x5800) [0x557a988b4800]
Jul 17 14:49:20 hostname samba[4998]:    #42
/usr/local/samba/sbin/samba(main+0x3f) [0x557a988b4901]
Jul 17 14:49:20 hostname samba[4998]:    #43
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f81f6cf1445]
Jul 17 14:49:20 hostname samba[4998]:    #44
/usr/local/samba/sbin/samba(+0x3719) [0x557a988b2719]

Since i'm currently not at work myself i could not tell you about any
specific event causing this to happen.
One thing i did notice, is that the crash appears to happen in a process that is
previously unknown. (as in: pid 4998 does not appear in the journal any time
before this crash happens).
Another thing, that makes this situation worse, is that this process seems to
block the other samba processes. once this crash occured, there won't be any
answer from the DC until it is completely killed and restarted, but the systemd
service is configured to always keep the service running.

If you need anything else, please let me know.

Thanks for your help.

Volker Mauel


------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Andrew Bartlett

2018-Jul-24 17:46 UTC

head link

[Samba] "Random" crashes of Samba as AD DC

On Tue, 2018-07-17 at 16:30 +0000, Mauel, Volker via samba
wrote:> Hey all,
> 
> up until 2 weeks ago my samba DC was running just fine, then it began
crashing randomly.
> I was using 4.8.0 before, now upgraded to a self-compiled 4.8.3 with
default build parameters on an up-to-date centos 7.5 with disabled SELinux and
bind9 as DNS backend with samba_dlz.
It certainly shouldn't crash.  

If you can get a gdb backtrace of it (using the panic action and our
gdb_backtrace script) that would be great.  

panic action = path_to/gdb_backtrace %d

Send that in to me personally as it likely has domain-specific info in
the backtrace.  If you can also make a bugzilla account in the meantime
we can trace fixing it there without needing to put your backtrace in
the public. 

Thanks!

Andrew Bartlett

-- 
Andrew Bartlett                       http://samba.org/~abartlet/
Authentication Developer, Samba Team  http://samba.org
Samba Developer, Catalyst IT          http://catalyst.net.nz/services/samba

Mauel, Volker

2018-Jul-24 18:44 UTC

head link

[Samba] "Random" crashes of Samba as AD DC

Hey Andrew,

thanks for your help, i just changed my config to include the panic action and
will send you a log as soon as the next crash happened.

In the meantime i wrote a bash script to keep my users happy with a running
service by checking the recent log for the error message and initiating a samba
restart that way.

i switched back to an older backup since the 4.8.3 version did not change
anything regarding that problem for me, recompiled with debug symbols and
systemd support, besides that, the situation is the same.

Volker
________________________________________
Von: Andrew Bartlett [abartlet at samba.org]
Gesendet: Dienstag, 24. Juli 2018 19:46
An: Mauel, Volker; samba at lists.samba.org
Betreff: Re: [Samba] "Random" crashes of Samba as AD DC

On Tue, 2018-07-17 at 16:30 +0000, Mauel, Volker via samba
wrote:> Hey all,
>
> up until 2 weeks ago my samba DC was running just fine, then it began
crashing randomly.
> I was using 4.8.0 before, now upgraded to a self-compiled 4.8.3 with
default build parameters on an up-to-date centos 7.5 with disabled SELinux and
bind9 as DNS backend with samba_dlz.
It certainly shouldn't crash.

If you can get a gdb backtrace of it (using the panic action and our
gdb_backtrace script) that would be great.

panic action = path_to/gdb_backtrace %d

Send that in to me personally as it likely has domain-specific info in
the backtrace. If you can also make a bugzilla account in the meantime
we can trace fixing it there without needing to put your backtrace in
the public.

Thanks!

Andrew Bartlett

--
Andrew Bartlett http://samba.org/~abartlet/
Authentication Developer, Samba Team http://samba.org
Samba Developer, Catalyst IT http://catalyst.net.nz/services/samba

------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Maybe Matching Threads

Search for more apparently analagous threads

samba - Jul 2018 - "Random" crashes of Samba as AD DC

[Samba] "Random" crashes of Samba as AD DC

[Samba] "Random" crashes of Samba as AD DC

[Samba] "Random" crashes of Samba as AD DC

Maybe Matching Threads