Hi I suddenly started getting segmentation faults in the primary DC. Does anyone know how to deal with this? This is a production environment. samba-tool user show thkam ==============================================================INTERNAL ERROR: Signal 11: Segmentation fault in pid 1229 (4.14.9) If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting ==============================================================PANIC (pid 1229): Signal 11: Segmentation fault in 4.14.9 BACKTRACE: #0 log_stack_trace + 0x3b [ip=0x7fc1ff4b2fff] [sp=0x7ffcc6d3f8e0] #1 smb_panic_log + 0x1b5 [ip=0x7fc1ff4b2f73] [sp=0x7ffcc6d401f0] #2 smb_panic + 0x1c [ip=0x7fc1ff4b2f92] [sp=0x7ffcc6d40210] #3 fault_report + 0x91 [ip=0x7fc1ff4b2a96] [sp=0x7ffcc6d40230] #4 sig_fault + 0x19 [ip=0x7fc1ff4b2aaf] [sp=0x7ffcc6d402e0] #5 killpg + 0x40 [ip=0x7fc2005a30c0] [sp=0x7ffcc6d40300] #6 _dl_rtld_di_serinfo + 0x29f0 [ip=0x7fc20076ba80] [sp=0x7ffcc6d408c0] #7 _dl_find_dso_for_object + 0xadc [ip=0x7fc200773f5c] [sp=0x7ffcc6d409d0] #8 _dl_catch_exception + 0x88 [ip=0x7fc2006c0838] [sp=0x7ffcc6d40b30] #9 _dl_find_dso_for_object + 0x17a [ip=0x7fc2007735fa] [sp=0x7ffcc6d40c50] #10 <unknown symbol> [ip=0x7fc20053834c] [sp=0x7ffcc6d40d00] #11 _dl_catch_exception + 0x88 [ip=0x7fc2006c0838] [sp=0x7ffcc6d40d20] #12 _dl_catch_error + 0x33 [ip=0x7fc2006c0903] [sp=0x7ffcc6d40e40] #13 dlerror + 0x319 [ip=0x7fc200538b59] [sp=0x7ffcc6d40e80] #14 dlopen + 0x4a [ip=0x7fc2005383da] [sp=0x7ffcc6d40ea0] #15 ldb_modules_load_path + 0x3a9 [ip=0x7fc1ffb19471] [sp=0x7ffcc6d40ee0] #16 ldb_modules_load_dir + 0x2e0 [ip=0x7fc1ffb1984a] [sp=0x7ffcc6d40fd0] #17 ldb_modules_load_path + 0x373 [ip=0x7fc1ffb1943b] [sp=0x7ffcc6d41030] #18 ldb_modules_load + 0xc1 [ip=0x7fc1ffb19a7e] [sp=0x7ffcc6d41120] #19 ldb_init + 0x49 [ip=0x7fc1ffb11f47] [sp=0x7ffcc6d41170] #20 samba_ldb_init + 0x33 [ip=0x7fc1fb971254] [sp=0x7ffcc6d411b0] #21 ldb_wrap_connect + 0x43 [ip=0x7fc1fb9717a6] [sp=0x7ffcc6d41200] #22 secrets_db_connect + 0x42 [ip=0x7fc1fc86c182] [sp=0x7ffcc6d41250] #23 cli_credentials_set_secrets_lct + 0xbf [ip=0x7fc1fc863126] [sp=0x7ffcc6d41280] #24 cli_credentials_set_machine_account_db_ctx + 0x40e [ip=0x7fc1fc863c67] [sp=0x7ffcc6d41370] #25 cli_credentials_set_machine_account + 0xd4 [ip=0x7fc1fc863857] [sp=0x7ffcc6d41440] #26 py_creds_set_machine_account + 0x142 [ip=0x7fc1ffcab019] [sp=0x7ffcc6d41480] #27 PyDescr_NewMember + 0x28fd [ip=0x504c7d] [sp=0x7ffcc6d414e0] #28 _PyEval_EvalFrameDefault + 0x85a [ip=0x56b1da] [sp=0x7ffcc6d41520] #29 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d41650] #30 PyMethod_New + 0x490 [ip=0x50aaa0] [sp=0x7ffcc6d41710] #31 _PyEval_EvalFrameDefault + 0x190c [ip=0x56c28c] [sp=0x7ffcc6d41810] #32 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d41940] #33 _PyFunction_Vectorcall + 0x393 [ip=0x5f6a13] [sp=0x7ffcc6d41a00] #34 PyMethod_New + 0x939 [ip=0x50af49] [sp=0x7ffcc6d41ab0] #35 PyObject_Call + 0x1f7 [ip=0x5f3547] [sp=0x7ffcc6d41b60] #36 _PyEval_EvalFrameDefault + 0x1f4d [ip=0x56c8cd] [sp=0x7ffcc6d41bb0] #37 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d41ce0] #38 _PyFunction_Vectorcall + 0x393 [ip=0x5f6a13] [sp=0x7ffcc6d41da0] #39 PyMethod_New + 0x41c [ip=0x50aa2c] [sp=0x7ffcc6d41e50] #40 PyObject_Call + 0x1f7 [ip=0x5f3547] [sp=0x7ffcc6d41f00] #41 _PyEval_EvalFrameDefault + 0x1f4d [ip=0x56c8cd] [sp=0x7ffcc6d41f50] #42 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d42080] #43 _PyFunction_Vectorcall + 0x393 [ip=0x5f6a13] [sp=0x7ffcc6d42140] #44 PyMethod_New + 0x41c [ip=0x50aa2c] [sp=0x7ffcc6d421f0] #45 PyObject_Call + 0x1f7 [ip=0x5f3547] [sp=0x7ffcc6d422a0] #46 _PyEval_EvalFrameDefault + 0x1f4d [ip=0x56c8cd] [sp=0x7ffcc6d422f0] #47 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d42420] #48 _PyFunction_Vectorcall + 0x393 [ip=0x5f6a13] [sp=0x7ffcc6d424e0] #49 PyMethod_New + 0x41c [ip=0x50aa2c] [sp=0x7ffcc6d42590] #50 PyObject_Call + 0x1f7 [ip=0x5f3547] [sp=0x7ffcc6d42640] #51 _PyEval_EvalFrameDefault + 0x1f4d [ip=0x56c8cd] [sp=0x7ffcc6d42690] #52 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d427c0] #53 PyEval_EvalCode + 0x27 [ip=0x68d047] [sp=0x7ffcc6d42880] #54 _PyInterpreterState_Get + 0x71 [ip=0x67e351] [sp=0x7ffcc6d428e0] #55 _PyInterpreterState_Get + 0xef [ip=0x67e3cf] [sp=0x7ffcc6d42900] #56 _PyInterpreterState_Get + 0x191 [ip=0x67e471] [sp=0x7ffcc6d42920] #57 PyRun_SimpleFileExFlags + 0x197 [ip=0x67e817] [sp=0x7ffcc6d42970] #58 Py_RunMain + 0x212 [ip=0x6b6fe2] [sp=0x7ffcc6d429f0] #59 Py_BytesMain + 0x2d [ip=0x6b736d] [sp=0x7ffcc6d42ac0] #60 __libc_start_main + 0xf3 [ip=0x7fc2005840b3] [sp=0x7ffcc6d42af0] #61 _start + 0x2e [ip=0x5fa5ce] [sp=0x7ffcc6d42bc0] Aborted (core dumped) -- Dr. Thomas Kamalakis Professor and Dean of the School of Digital Technology Department of Informatics and Telematics, Harokopio University of Athens, Greece Omirou 9, Tavros, Athens, GR17778, Tel: +302109549406, Web: https://galaxy.hua.gr/~thkam, Github: https://github.com/thomaskamalakis/
"suddenly" indicates that you probably have a hardware problem, most likely bad memory. To guarantee that it's not bit rot (file errors on your disk) you could try re-installing the Samba packages, but bad memory seems like a more likely candidate for this problem. On 4/1/22 08:48, Thomas Kamalakis via samba wrote:> Hi > > I suddenly started getting segmentation faults in the primary DC. Does > anyone know how to deal with this? This is a production environment. > > samba-tool user show thkam > ==============================================================> INTERNAL ERROR: Signal 11: Segmentation fault in pid 1229 (4.14.9) > If you are running a recent Samba version, and if you think this problem is > not yet fixed in the latest versions, please consider reporting this bug, > see https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.samba.org%2Findex.php%2FBug_Reporting&data=04%7C01%7C%7Cc45ba0c420554ec7682708da13e6803d%7C31d7e2a5bdd8414e9e97bea998ebdfe1%7C0%7C0%7C637844177970561365%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=uBu2waf%2FVxRVyZN%2B4pjSApTS6w6OTzSm4Gz0UzqUYzU%3D&reserved=0 > ==============================================================> PANIC (pid 1229): Signal 11: Segmentation fault in 4.14.9 > BACKTRACE: > #0 log_stack_trace + 0x3b [ip=0x7fc1ff4b2fff] [sp=0x7ffcc6d3f8e0] > #1 smb_panic_log + 0x1b5 [ip=0x7fc1ff4b2f73] [sp=0x7ffcc6d401f0] > #2 smb_panic + 0x1c [ip=0x7fc1ff4b2f92] [sp=0x7ffcc6d40210] > #3 fault_report + 0x91 [ip=0x7fc1ff4b2a96] [sp=0x7ffcc6d40230] > #4 sig_fault + 0x19 [ip=0x7fc1ff4b2aaf] [sp=0x7ffcc6d402e0] > #5 killpg + 0x40 [ip=0x7fc2005a30c0] [sp=0x7ffcc6d40300] > #6 _dl_rtld_di_serinfo + 0x29f0 [ip=0x7fc20076ba80] [sp=0x7ffcc6d408c0] > #7 _dl_find_dso_for_object + 0xadc [ip=0x7fc200773f5c] [sp=0x7ffcc6d409d0] > #8 _dl_catch_exception + 0x88 [ip=0x7fc2006c0838] [sp=0x7ffcc6d40b30] > #9 _dl_find_dso_for_object + 0x17a [ip=0x7fc2007735fa] [sp=0x7ffcc6d40c50] > #10 <unknown symbol> [ip=0x7fc20053834c] [sp=0x7ffcc6d40d00] > #11 _dl_catch_exception + 0x88 [ip=0x7fc2006c0838] [sp=0x7ffcc6d40d20] > #12 _dl_catch_error + 0x33 [ip=0x7fc2006c0903] [sp=0x7ffcc6d40e40] > #13 dlerror + 0x319 [ip=0x7fc200538b59] [sp=0x7ffcc6d40e80] > #14 dlopen + 0x4a [ip=0x7fc2005383da] [sp=0x7ffcc6d40ea0] > #15 ldb_modules_load_path + 0x3a9 [ip=0x7fc1ffb19471] [sp=0x7ffcc6d40ee0] > #16 ldb_modules_load_dir + 0x2e0 [ip=0x7fc1ffb1984a] [sp=0x7ffcc6d40fd0] > #17 ldb_modules_load_path + 0x373 [ip=0x7fc1ffb1943b] [sp=0x7ffcc6d41030] > #18 ldb_modules_load + 0xc1 [ip=0x7fc1ffb19a7e] [sp=0x7ffcc6d41120] > #19 ldb_init + 0x49 [ip=0x7fc1ffb11f47] [sp=0x7ffcc6d41170] > #20 samba_ldb_init + 0x33 [ip=0x7fc1fb971254] [sp=0x7ffcc6d411b0] > #21 ldb_wrap_connect + 0x43 [ip=0x7fc1fb9717a6] [sp=0x7ffcc6d41200] > #22 secrets_db_connect + 0x42 [ip=0x7fc1fc86c182] [sp=0x7ffcc6d41250] > #23 cli_credentials_set_secrets_lct + 0xbf [ip=0x7fc1fc863126] > [sp=0x7ffcc6d41280] > #24 cli_credentials_set_machine_account_db_ctx + 0x40e [ip=0x7fc1fc863c67] > [sp=0x7ffcc6d41370] > #25 cli_credentials_set_machine_account + 0xd4 [ip=0x7fc1fc863857] > [sp=0x7ffcc6d41440] > #26 py_creds_set_machine_account + 0x142 [ip=0x7fc1ffcab019] > [sp=0x7ffcc6d41480] > #27 PyDescr_NewMember + 0x28fd [ip=0x504c7d] [sp=0x7ffcc6d414e0] > #28 _PyEval_EvalFrameDefault + 0x85a [ip=0x56b1da] [sp=0x7ffcc6d41520] > #29 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d41650] > #30 PyMethod_New + 0x490 [ip=0x50aaa0] [sp=0x7ffcc6d41710] > #31 _PyEval_EvalFrameDefault + 0x190c [ip=0x56c28c] [sp=0x7ffcc6d41810] > #32 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d41940] > #33 _PyFunction_Vectorcall + 0x393 [ip=0x5f6a13] [sp=0x7ffcc6d41a00] > #34 PyMethod_New + 0x939 [ip=0x50af49] [sp=0x7ffcc6d41ab0] > #35 PyObject_Call + 0x1f7 [ip=0x5f3547] [sp=0x7ffcc6d41b60] > #36 _PyEval_EvalFrameDefault + 0x1f4d [ip=0x56c8cd] [sp=0x7ffcc6d41bb0] > #37 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d41ce0] > #38 _PyFunction_Vectorcall + 0x393 [ip=0x5f6a13] [sp=0x7ffcc6d41da0] > #39 PyMethod_New + 0x41c [ip=0x50aa2c] [sp=0x7ffcc6d41e50] > #40 PyObject_Call + 0x1f7 [ip=0x5f3547] [sp=0x7ffcc6d41f00] > #41 _PyEval_EvalFrameDefault + 0x1f4d [ip=0x56c8cd] [sp=0x7ffcc6d41f50] > #42 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d42080] > #43 _PyFunction_Vectorcall + 0x393 [ip=0x5f6a13] [sp=0x7ffcc6d42140] > #44 PyMethod_New + 0x41c [ip=0x50aa2c] [sp=0x7ffcc6d421f0] > #45 PyObject_Call + 0x1f7 [ip=0x5f3547] [sp=0x7ffcc6d422a0] > #46 _PyEval_EvalFrameDefault + 0x1f4d [ip=0x56c8cd] [sp=0x7ffcc6d422f0] > #47 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d42420] > #48 _PyFunction_Vectorcall + 0x393 [ip=0x5f6a13] [sp=0x7ffcc6d424e0] > #49 PyMethod_New + 0x41c [ip=0x50aa2c] [sp=0x7ffcc6d42590] > #50 PyObject_Call + 0x1f7 [ip=0x5f3547] [sp=0x7ffcc6d42640] > #51 _PyEval_EvalFrameDefault + 0x1f4d [ip=0x56c8cd] [sp=0x7ffcc6d42690] > #52 _PyEval_EvalCodeWithName + 0x26a [ip=0x56939a] [sp=0x7ffcc6d427c0] > #53 PyEval_EvalCode + 0x27 [ip=0x68d047] [sp=0x7ffcc6d42880] > #54 _PyInterpreterState_Get + 0x71 [ip=0x67e351] [sp=0x7ffcc6d428e0] > #55 _PyInterpreterState_Get + 0xef [ip=0x67e3cf] [sp=0x7ffcc6d42900] > #56 _PyInterpreterState_Get + 0x191 [ip=0x67e471] [sp=0x7ffcc6d42920] > #57 PyRun_SimpleFileExFlags + 0x197 [ip=0x67e817] [sp=0x7ffcc6d42970] > #58 Py_RunMain + 0x212 [ip=0x6b6fe2] [sp=0x7ffcc6d429f0] > #59 Py_BytesMain + 0x2d [ip=0x6b736d] [sp=0x7ffcc6d42ac0] > #60 __libc_start_main + 0xf3 [ip=0x7fc2005840b3] [sp=0x7ffcc6d42af0] > #61 _start + 0x2e [ip=0x5fa5ce] [sp=0x7ffcc6d42bc0] > Aborted (core dumped) > > >
hi Thomas, Because you are triggering it with samba-tool...> samba-tool user show thkam > ==============================================================> INTERNAL ERROR: Signal 11: Segmentation fault in pid 1229 (4.14.9)and the crash is in Samba Python bindings...> #56 _PyInterpreterState_Get + 0x191 [ip=0x67e471] [sp=0x7ffcc6d42920] > #57 PyRun_SimpleFileExFlags + 0x197 [ip=0x67e817] [sp=0x7ffcc6d42970] > #58 Py_RunMain + 0x212 [ip=0x6b6fe2] [sp=0x7ffcc6d429f0] > #59 Py_BytesMain + 0x2d [ip=0x6b736d] [sp=0x7ffcc6d42ac0] > #60 __libc_start_main + 0xf3 [ip=0x7fc2005840b3] [sp=0x7ffcc6d42af0] > #61 _start + 0x2e [ip=0x5fa5ce] [sp=0x7ffcc6d42bc0] > Aborted (core dumped)I don't think the DC process itself is crashing. (That's good). It looks to be caused by an attempt to load an invalid .so file:> #12 _dl_catch_error + 0x33 [ip=0x7fc2006c0903] [sp=0x7ffcc6d40e40] > #13 dlerror + 0x319 [ip=0x7fc200538b59] [sp=0x7ffcc6d40e80] > #14 dlopen + 0x4a [ip=0x7fc2005383da] [sp=0x7ffcc6d40ea0] > #15 ldb_modules_load_path + 0x3a9 [ip=0x7fc1ffb19471] [sp=0x7ffcc6d40ee0]which could mean your ldb package is very out of sync with Samba, or (more likely, I think) there is some kind of filesystem or memory corruption. Ldb makes attempts to cope with dynamic libraries being broken in normal ways (e.g. not existing), so it seems something very odd has happened. That is of course consistent with your oVirt theory. cheers, Douglas