Hi Martin: Actually, I have installed all the debuginfo for this coredump file. But the netcf debuginfo report "mismatch" as below: warning: the debug information found in "/usr/lib/debug/usr/lib64/libnetcf.so.1.4.0.debug" does not match "/lib64/libnetcf.so.1" (CRC mismatch). So I ignore this warning before. You can check what I did on the attachment "netcf_debuginfo_install.log". Another attachment is the result of "rpm -Va" to check all the system config files status, I didn't find anything wrong. Thanks for your help! -Best Regards, Andy -----邮件原件----- 发件人: Martin Kletzander [mailto:mkletzan@redhat.com] 发送时间: 26 March 2015 15:54 收件人: 梅磊 抄送: libvirt-users@redhat.com 主题: Re: 答复: [libvirt-users] libvirtd can't start On Thu, Mar 26, 2015 at 02:55:04PM +0800, 梅磊 wrote:>Thanks for your reply, Martin, I have read the coredump, but I am not familiar with libvirt, so could you help me to figure it out why it crash? >Attachment is the coredump gdb output and all the generate files when libvirtd stop. >I forgot to ask you to install debuginfo packages to further investigate. But looking at the log only netcf-debuginfo is needed. Could you install it and then run the "t a a bt f" again, please?>From the current logs I can only guess that there is some network XML mangled and netcf can't read it. When you're power went out, as you said, is it possible you were updating something in the meantime? Try running "rpm -Va" to see if there's something broken in the system.>And Do you think this issue is the same issue with the below root cause? >https://bugzilla.redhat.com/show_bug.cgi?id=1179981 >It is not, that is caused by something else.> > > >-Best Regards, >Andy > > >-----邮件原件----- >发件人: Martin Kletzander [mailto:mkletzan@redhat.com] >发送时间: 24 March 2015 21:20 >收件人: 梅磊 >抄送: libvirt-users@redhat.com >主题: Re: [libvirt-users] libvirtd can't start > >On Tue, Mar 24, 2015 at 04:29:07PM +0800, 梅磊 wrote: >>Hi experts, >> The libvirtd can’t start on my server after the server interruption >>of power supply, the status is below: >> >>[root@openstack3 libvirt]# service libvirtd status Redirecting to >>/bin/systemctl status libvirtd.service libvirtd.service - >>Virtualization daemon >> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled) >> Active: failed (Result: signal) since Tue 2015-03-24 16:10:31 CST; >>7min ago >> Process: 27372 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS >>(code=killed, >>signal=SEGV) >> Main PID: 27372 (code=killed, signal=SEGV) >> >>Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: >>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 >>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: >>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 >>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: >>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 >>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: >>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 >>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000: >>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 >>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000: >>27383: debug : virStateInitialize:847 : Running global init for netcf >>state driver Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 >>08:10:31.413+0000: >>27383: debug : virObjectNew:199 : OBJECT_NEW: obj=0x7f73f0080d00 >>classname=virNetcfDriverState Mar 24 16:10:31 openstack3 >>libvirtd[27372]: ====== end of log ===== Mar 24 16:10:31 openstack3 >>systemd[1]: libvirtd.service: main process exited, code=killed, >>status=11/SEGV Mar 24 16:10:31 openstack3 systemd[1]: Unit >>libvirtd.service entered failed state. >> >>And the attachment is the log after restart the libvirtd. >> >>After long time search on google, I didn’t get any useful information, >>so anyone can give me some hint on this issue? >> > >What did you search for? > >According to the log: >abrt-hook-ccpp: Saved core dump of pid 28928 (/usr/sbin/libvirtd) to >/var/tmp/abrt/ccpp-2015-03-24-16:19:22-28928 (97218560 bytes) > >libvirtd has segfaulted and you have the coredump. Use gdb to open >it: ># gdb -c /var/tmp/abrt/ccpp-2015-03-24-16:19:22-28928 > >and then look at what happened, if you can't make anything of it, post it as an attachment here or create a bug from it: > >(gdb) bt >(gdb) t a a bt full > >>-BR, >>Andy >> > > >>_______________________________________________ >>libvirt-users mailing list >>libvirt-users@redhat.com >>https://www.redhat.com/mailman/listinfo/libvirt-users
On 03/26/2015 05:31 AM, 梅磊 wrote:> Hi Martin: > Actually, I have installed all the debuginfo for this coredump file. But the netcf debuginfo report "mismatch" as below: > warning: the debug information found in "/usr/lib/debug/usr/lib64/libnetcf.so.1.4.0.debug" does not match "/lib64/libnetcf.so.1" (CRC mismatch). > > So I ignore this warning before. You can check what I did on the attachment "netcf_debuginfo_install.log".> Core was generated by `/usr/sbin/libvirtd --listen'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007fded87138db in ?? () from /lib64/libnetcf.so.1 > Missing separate debuginfos, use: debuginfo-install netcf-libs-0.2.4-1.el6.x86_64Notice it says that you need the debuginfo for netcf-0.2.4-1.el6> yum install netcf-debuginfo > Loaded plugins: auto-update-debuginfo, fastestmirror, langpacks, priorities > Loading mirror speeds from cached hostfile > * epel: ftp.sjtu.edu.cn > * epel-debuginfo: ftp.sjtu.edu.cn > 60 packages excluded due to repository priority protections > Resolving Dependencies > --> Running transaction check > ---> Package netcf-debuginfo.x86_64 0:0.2.3-8.el7 will be installed > --> Finished Dependency ResolutionAnd here it says that what you installed is netcf-debuginfo-0.2.3-8.el7 So are you running CentOS 6 or CentOS 7? It sounds like maybe you've got packages mixed and matched in a way that isn't going to work correctly (I'm surprised that yum would even allow it; usually it doesn't). Can you tell us the output of "uname -a; rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort" ? (and also it would be useful to know how you got a mix of el6 and el7 packages on your system)
Laine, you are right, some packages mixed: [root@openstack3 ~]# rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort libnl-1.1.4-3.el7.x86_64 libnl3-3.2.21-7.el6.x86_64 libnl3-cli-3.2.21-7.el6.x86_64 libnl3-debuginfo-3.2.21-7.el6.x86_64 libnl-debuginfo-1.1.4-3.el7.x86_64 libnl-devel-1.1.4-3.el7.x86_64 libvirt-client-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-config-network-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-config-nwfilter-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-interface-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-lxc-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-network-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-nodedev-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-nwfilter-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-qemu-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-secret-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-storage-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-kvm-1.1.1-29.el7_0.3.x86_64 libvirt-debuginfo-1.1.1-29.el7_0.3.x86_64 libvirt-devel-1.1.1-29.el7_0.3.x86_64 libvirt-docs-1.1.1-29.el7_0.3.x86_64 libvirt-gconfig-0.1.7-3.el7.x86_64 libvirt-glib-0.1.7-3.el7.x86_64 libvirt-gobject-0.1.7-3.el7.x86_64 libvirt-java-0.4.9-4.el7.noarch libvirt-java-devel-0.4.9-4.el7.noarch libvirt-python-1.1.1-29.el7_0.3.x86_64 netcf-debuginfo-0.2.3-8.el7.x86_64 netcf-libs-0.2.4-1.el6.x86_64 [root@openstack3 ~]# rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort | grep el6 libnl3-3.2.21-7.el6.x86_64 libnl3-cli-3.2.21-7.el6.x86_64 libnl3-debuginfo-3.2.21-7.el6.x86_64 netcf-libs-0.2.4-1.el6.x86_64 [root@openstack3 ~]# [root@openstack3 ~]# uname -a Linux openstack3 3.10.0-123.9.3.el7.x86_64 #1 SMP Thu Nov 6 15:06:03 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux The distribution is centos 7. I tried to update the package to el7 version, but due to the package's el6 version is newer than the el7 version, so I can't update it. Then I want to remove these packages and reinstall it, but a lot of dependencies, do you think I can do it? -BR, Andy -----邮件原件----- 发件人: sendmail [mailto:justsendmailnothingelse@gmail.com] 代表 Laine Stump 发送时间: 27 March 2015 06:35 收件人: libvirt-users@redhat.com 抄送: 梅磊; 'Martin Kletzander' 主题: Re: [libvirt-users] 答复: 答复: libvirtd can't start On 03/26/2015 05:31 AM, 梅磊 wrote:> Hi Martin: > Actually, I have installed all the debuginfo for this coredump file. But the netcf debuginfo report "mismatch" as below: > warning: the debug information found in "/usr/lib/debug/usr/lib64/libnetcf.so.1.4.0.debug" does not match "/lib64/libnetcf.so.1" (CRC mismatch). > > So I ignore this warning before. You can check what I did on the attachment "netcf_debuginfo_install.log".> Core was generated by `/usr/sbin/libvirtd --listen'. > Program terminated with signal 11, Segmentation fault. > #0 0x00007fded87138db in ?? () from /lib64/libnetcf.so.1 Missing > separate debuginfos, use: debuginfo-install > netcf-libs-0.2.4-1.el6.x86_64Notice it says that you need the debuginfo for netcf-0.2.4-1.el6> yum install netcf-debuginfo > Loaded plugins: auto-update-debuginfo, fastestmirror, langpacks, > priorities Loading mirror speeds from cached hostfile > * epel: ftp.sjtu.edu.cn > * epel-debuginfo: ftp.sjtu.edu.cn > 60 packages excluded due to repository priority protections Resolving > Dependencies > --> Running transaction check > ---> Package netcf-debuginfo.x86_64 0:0.2.3-8.el7 will be installed > --> Finished Dependency ResolutionAnd here it says that what you installed is netcf-debuginfo-0.2.3-8.el7 So are you running CentOS 6 or CentOS 7? It sounds like maybe you've got packages mixed and matched in a way that isn't going to work correctly (I'm surprised that yum would even allow it; usually it doesn't). Can you tell us the output of "uname -a; rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort" ? (and also it would be useful to know how you got a mix of el6 and el7 packages on your system)