Isaac Stone
2021-Feb-16 20:06 UTC
[Samba] NetFileEnum rpc causes lock order violation and core dump
running samba 4.12.3 as a domain member in clustered mode with ctdb 4.12.3 NetFileEnum rpc is having problems. Other things seem to work, including NetShareEnum. calling with `rpclient` produces this result> cli_api_pipe failed: NT_STATUS_CONNECTION_DISCONNECTED > result was WERR_NETNAME_DELETEDlooking at the logs and I think the smb process is crashing: [2021/02/16 19:37:34.389710, 2] ../../source3/auth/auth.c:329( auth_check_ntlm_password) check_ntlm_password: authentication for user [isaac.stone] -> [isaac.stone] -> [SOMDEV\isaac.stone] succeeded [2021/02/16 19:37:34.402327, 0] ../../lib/dbwrap/dbwrap.c:156( dbwrap_lock_order_lock) dbwrap_lock_order_lock: Lock order violation: Trying brlock.tdb at 2 while share_entries.tdb at 3 is locked [2021/02/16 19:37:34.402448, 0] ../../lib/dbwrap/dbwrap.c:129( debug_lock_order) lock order: 1:<none> 2:<none> 3:share_entries.tdb 4:<none> [2021/02/16 19:37:34.402465, 0] ../../source3/lib/util.c:830(smb_panic_s3) PANIC (pid 1653800): lock order violation [2021/02/16 19:37:34.403131, 0] ../../lib/util/fault.c:265(log_stack_trace) BACKTRACE: 49 stack frames: #0 /lib64/libsamba-util.so.0(log_stack_trace+0x34) [0x7fe835aa6264] #1 /lib64/libsmbconf.so.0(smb_panic_s3+0x29) [0x7fe835501639] #2 /lib64/libsamba-util.so.0(smb_panic+0x31) [0x7fe835aa6361] #3 /usr/lib64/samba/libdbwrap-samba4.so(+0x34c0) [0x7fe8336244c0] #4 /usr/lib64/samba/libdbwrap-samba4.so(+0x361b) [0x7fe83362461b] #5 /usr/lib64/samba/libsmbd-base-samba4.so(brl_get_locks+0x5f) [0x7fe8355d2e4f] #6 /usr/lib64/samba/libsmbd-base-samba4.so(+0x157614) [0x7fe835695614] #7 /usr/lib64/samba/libsmbd-base-samba4.so(+0x98792) [0x7fe8355d6792] #8 /usr/lib64/samba/libsmbd-base-samba4.so(+0x9a44b) [0x7fe8355d844b] #9 /usr/lib64/samba/libdbwrap-samba4.so(dbwrap_do_locked+0x106) [0x7fe833624ce6] #10 /usr/lib64/samba/libsmbd-base-samba4.so(share_mode_forall_entries+0x5a) [0x7fe8355dad6a] #11 /usr/lib64/samba/libsmbd-base-samba4.so(+0x9ce68) [0x7fe8355dae68] #12 /usr/lib64/samba/libsmbd-base-samba4.so(+0x98cc8) [0x7fe8355d6cc8] #13 /lib64/libsmbconf.so.0(+0x34078) [0x7fe8354dd078] #14 /usr/lib64/samba/libsamba-cluster-support-samba4.so(+0x39f3) [0x7fe83529e9f3] #15 /usr/lib64/samba/libsamba-cluster-support-samba4.so(ctdbd_traverse+0x130) [0x7fe8352a6b80] #16 /usr/lib64/samba/libsamba-cluster-support-samba4.so(+0x4192) [0x7fe83529f192] #17 /usr/lib64/samba/libsamba-cluster-support-samba4.so(+0x435f) [0x7fe83529f35f] #18 /usr/lib64/samba/libdbwrap-samba4.so(dbwrap_traverse_read+0xb) [0x7fe83362488b] #19 /lib64/libsmbconf.so.0(+0x3511d) [0x7fe8354de11d] #20 /usr/lib64/samba/libdbwrap-samba4.so(dbwrap_traverse_read+0xb) [0x7fe83362488b] #21 /usr/lib64/samba/libsmbd-base-samba4.so(share_mode_forall+0x46) [0x7fe8355da9f6] #22 /usr/lib64/samba/libsmbd-base-samba4.so(share_entry_forall+0x47) [0x7fe8355daa67] #23 /usr/lib64/samba/libsmbd-base-samba4.so(_srvsvc_NetFileEnum+0xa1) [0x7fe835697301] #24 /usr/lib64/samba/libsmbd-base-samba4.so(+0x163ca6) [0x7fe8356a1ca6] #25 /usr/lib64/samba/libsmbd-base-samba4.so(+0x23e86a) [0x7fe83577c86a] #26 /usr/lib64/samba/libsmbd-base-samba4.so(process_complete_pdu+0x3ca) [0x7fe83577da5a] #27 /usr/lib64/samba/libsmbd-base-samba4.so(dcerpc_ncacn_packet_process+0xf1) [0x7fe83564b3b1] #28 /lib64/libdcerpc-binding.so.0(+0xd12f) [0x7fe83324712f] #29 /usr/lib64/samba/libsamba-sockets-samba4.so(+0x6c7d) [0x7fe835495c7d] #30 /usr/lib64/samba/libsamba-sockets-samba4.so(+0x5a0f) [0x7fe835494a0f] #31 /lib64/libtevent.so.0(tevent_common_invoke_immediate_handler+0x139) [0x7fe83488c969] #32 /lib64/libtevent.so.0(tevent_common_loop_immediate+0x27) [0x7fe83488c997] #33 /lib64/libtevent.so.0(+0xd8af) [0x7fe8348928af] #34 /lib64/libtevent.so.0(+0xbadb) [0x7fe834890adb] #35 /lib64/libtevent.so.0(_tevent_loop_once+0x95) [0x7fe83488bc05] #36 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7fe83488beab] #37 /lib64/libtevent.so.0(+0xba6b) [0x7fe834890a6b] #38 /usr/lib64/samba/libsmbd-base-samba4.so(smbd_process+0x7bf) [0x7fe8357350ef] #39 /usr/sbin/smbd(+0xdd20) [0x55c4b6dabd20] #40 /lib64/libtevent.so.0(tevent_common_invoke_fd_handler+0x83) [0x7fe83488c533] #41 /lib64/libtevent.so.0(+0xdacf) [0x7fe834892acf] #42 /lib64/libtevent.so.0(+0xbadb) [0x7fe834890adb] #43 /lib64/libtevent.so.0(_tevent_loop_once+0x95) [0x7fe83488bc05] #44 /lib64/libtevent.so.0(tevent_common_loop_wait+0x1b) [0x7fe83488beab] #45 /lib64/libtevent.so.0(+0xba6b) [0x7fe834890a6b] #46 /usr/sbin/smbd(main+0x1bd2) [0x55c4b6da62c2] #47 /lib64/libc.so.6(__libc_start_main+0xf3) [0x7fe8342c57b3] #48 /usr/sbin/smbd(_start+0x2e) [0x55c4b6da667e] [2021/02/16 19:37:34.403657, 0] ../../source3/lib/dumpcore.c:318(dump_core) coredump is handled by helper binary specified at /proc/sys/kernel/core_pattern Here is our smb.conf: #======================= Global Settings ===================================[global] netbios name = C25-USW2 realm = SOMDEV.LOCAL workgroup = SOMDEV security = ads clustering = yes fake oplocks = no log level = 2 idmap config * : backend = tdb idmap config * : range = 3000-7999 idmap config SOMDEV : backend = rid idmap config SOMDEV : range = 10000-999999 dedicated keytab file = /etc/krb5.keytab kerberos method = secrets and keytab map acl inherit = yes winbind refresh tickets = yes # disables printing: load printers = no printing = bsd printcap name = /dev/null disable spoolss = yes bind interfaces only = yes interfaces = lo eth0 #============================ Share Definitions =============================[SF] path = "/mnt/usw2/SF" writeable = yes browsable = yes fileid:algorithm = fsname vfs objects = fileid acl_xattr shadow_copy2 acl_xattr:ignore system acls = yes shadow:mountpoint = /mnt/usw2 shadow:snapdir = /snapshots/usw2 shadow:snapsharepath = SF shadow:format = %Y-%m-%dT%H:%M:%SZ [LA] path = "/mnt/usw2/LA" writeable = yes browsable = yes fileid:algorithm = fsname vfs objects = fileid acl_xattr shadow_copy2 acl_xattr:ignore system acls = yes shadow:mountpoint = /mnt/usw2 shadow:snapdir = /snapshots/usw2 shadow:snapsharepath = LA shadow:format = %Y-%m-%dT%H:%M:%SZ ----- I am suspicious this might be a bug:> dbwrap_lock_order_lock: Lock order violation: Trying brlock.tdb at 2while share_entries.tdb at 3 is locked> lock order: 1:<none> 2:<none> 3:share_entries.tdb 4:<none> > PANIC (pid 1654263): lock order violationShould I go ahead and log in the samba bugzilla? Are there other things I can try to further pinpoint the cause? thanks - isaac