correomm
2016-Aug-18 16:39 UTC
[CentOS] BUG: soft lockup - CPU#0 stuck for 36s! [swapper/0:0]
This bug is reported only on the VM's with CentOS 7 running on on VMware ESXi 5.1. The vSphere performance graph shows high CPU consume and disk activity only on VM's with CentOS 7. Sometimes I can not connect remotely with ssh (timeout error). The details of last issues was reported to retrace.fedoraproject.org. ?Do you have a hint? [root at vmguest ~]# abrt-cli list id c52b463b15cfa94af7a96f237e5f525332750dd3 reason: systemd-journald killed by SIGABRT time: Tue 16 Aug 2016 03:10:52 PM CLST cmdline: /usr/lib/systemd/systemd-journald package: systemd-219-19.el7_2.12 uid: 0 (root) count: 1 Directory: /var/spool/abrt/ccpp-2016-08-16-15:10:52-458 Reported: https://retrace.fedoraproject.org/faf/reports/bthash/d5f5d4f75b200eeab2f83c8340a37c5d507e29a3 id e6955aa621a0d296d4cdd05421523885e85a179b reason: BUG: soft lockup - CPU#0 stuck for 36s! [swapper/0:0] time: Tue 09 Aug 2016 04:46:33 PM CLT cmdline: BOOT_IMAGE=/vmlinuz-3.10.0-327.28.2.el7.x86_64 root=/dev/mapper/centos_vmguest-root ro crashkernel=auto rd.lvm.lv=centos_vmguest/root rd.lvm.lv=centos_vmguest/swap rhgb quiet LANG=en_US.UTF-8 package: kernel uid: 0 (root) count: 1 Directory: /var/spool/abrt/oops-2016-08-09-16:46:33-3165-0 Reported: https://retrace.fedoraproject.org/faf/reports/bthash/4e231b49f72864c3487d8985337f9d41ce43da28 id 402f22e7214ea6bddcb0db9a9315527be245f943 reason: systemd-logind killed by SIGABRT time: Wed 20 Jul 2016 06:10:55 AM CLT cmdline: /usr/lib/systemd/systemd-logind package: systemd-219-19.el7_2.9 uid: 0 (root) count: 3 Directory: /var/spool/abrt/ccpp-2016-07-20-06:10:55-32283 Reported: https://retrace.fedoraproject.org/faf/reports/bthash/307b26a77cc6d5005ce2fdf18ff010fe3dc94401 id 58a46f4a45699384ad74850f53e749c702ee7b0b reason: systemd-journald killed by SIGABRT time: Tue 02 Aug 2016 05:44:50 PM CLT cmdline: /usr/lib/systemd/systemd-journald package: systemd-219-19.el7_2.11 uid: 0 (root) count: 1 Directory: /var/spool/abrt/ccpp-2016-08-02-17:44:50-454 Reported: https://retrace.fedoraproject.org/faf/reports/bthash/1af5058d2b9650a1d91c676fe15feb5612afccb9 id f4a35ca85a046e74bf4a4382c9f9a5c8dd8be149 reason: BUG: soft lockup - CPU#0 stuck for 24s! [vmtoolsd:579] time: Tue 02 Aug 2016 06:45:29 AM CLT cmdline: BOOT_IMAGE=/vmlinuz-3.10.0-327.22.2.el7.x86_64 root=/dev/mapper/centos_vmguest-root ro crashkernel=auto rd.lvm.lv=centos_vmguest/root rd.lvm.lv=centos_vmguest/swap rhgb quiet LANG=en_US.UTF-8 package: kernel uid: 0 (root) count: 1 Directory: /var/spool/abrt/oops-2016-08-02-06:45:29-11859-0 Reported: https://retrace.fedoraproject.org/faf/reports/bthash/87298dcaf6b7dea7a92b136e3332209f3fd7c4d2 id edaec629ccce62943e9bdb514fe6e319ab320669 reason: BUG: soft lockup - CPU#0 stuck for 27s! [khugepaged:51] time: Tue 26 Jul 2016 06:00:13 PM CLT cmdline: BOOT_IMAGE=/vmlinuz-3.10.0-327.18.2.el7.x86_64 root=/dev/mapper/centos_vmguest-root ro crashkernel=auto rd.lvm.lv=centos_vmguest/root rd.lvm.lv=centos_vmguest/swap rhgb quiet LANG=en_US.UTF-8 package: kernel uid: 0 (root) count: 1 Directory: /var/spool/abrt/oops-2016-07-26-18:00:08-641-4 Reported: cannot be reported id b707fd06199e2e1edcb105878a46e238a50746f3 reason: BUG: soft lockup - CPU#3 stuck for 23s! [systemd-journal:422] time: Tue 26 Jul 2016 06:00:10 PM CLT cmdline: BOOT_IMAGE=/vmlinuz-3.10.0-327.18.2.el7.x86_64 root=/dev/mapper/centos_vmguest-root ro crashkernel=auto rd.lvm.lv=centos_vmguest/root rd.lvm.lv=centos_vmguest/swap rhgb quiet LANG=en_US.UTF-8 package: kernel uid: 0 (root) count: 1 Directory: /var/spool/abrt/oops-2016-07-26-18:00:08-641-1 Reported: cannot be reported id a39eead9c9f75c2dc94df0852cd24260f414b80b reason: BUG: soft lockup - CPU#2 stuck for 22s! [swapper/2:0] time: Tue 26 Jul 2016 06:00:08 PM CLT cmdline: BOOT_IMAGE=/vmlinuz-3.10.0-327.18.2.el7.x86_64 root=/dev/mapper/centos_vmguest-root ro crashkernel=auto rd.lvm.lv=centos_vmguest/root rd.lvm.lv=centos_vmguest/swap rhgb quiet LANG=en_US.UTF-8 package: kernel uid: 0 (root) count: 1 Directory: /var/spool/abrt/oops-2016-07-26-18:00:08-641-0 Reported: cannot be reported id fe7e2542e93848e41d9d702af5c4d12b1d833b72 reason: systemd-logind killed by SIGABRT time: Wed 20 Jul 2016 08:20:29 AM CLT cmdline: /usr/lib/systemd/systemd-logind package: systemd-219-19.el7_2.9 uid: 0 (root) count: 1 Directory: /var/spool/abrt/ccpp-2016-07-20-08:20:29-32660 id e58538a3aa1f01f384fe9bdd40a693f9d2f32889 reason: BUG: soft lockup - CPU#2 stuck for 24s! [kworker/2:0:31607] time: Mon 13 Jun 2016 04:48:05 PM CLT cmdline: BOOT_IMAGE=/vmlinuz-3.10.0-327.18.2.el7.x86_64 root=/dev/mapper/centos_vmguest-root ro crashkernel=auto rd.lvm.lv=centos_vmguest/root rd.lvm.lv=centos_vmguest/swap rhgb quiet LANG=en_US.UTF-8 package: kernel uid: 0 (root) count: 1 Directory: /var/spool/abrt/oops-2016-06-13-16:48:05-10215-0 id a4dd378d494c8eee43407b572e2b314f38f7c5b9 reason: systemd-journald killed by SIGABRT time: Sat 28 May 2016 06:24:40 PM CLT cmdline: /usr/lib/systemd/systemd-journald uid: 0 Directory: /var/spool/abrt/ccpp-2016-05-28-18:24:40-6058 Reported: cannot be reported
John R Pierce
2016-Aug-18 16:58 UTC
[CentOS] BUG: soft lockup - CPU#0 stuck for 36s! [swapper/0:0]
On 8/18/2016 9:39 AM, correomm wrote:> This bug is reported only on the VM's with CentOS 7 running on on VMware > ESXi 5.1. > The vSphere performance graph shows high CPU consume and disk activity only > on VM's with CentOS 7. Sometimes I can not connect remotely with ssh > (timeout error).FWIW, I've had no problems with CentOS 7.x VMs running in ESXi 5.5.0 GA (build 1331820) -- john r pierce, recycling bits in santa cruz
Carlos A. Carnero Delgado
2016-Aug-18 17:06 UTC
[CentOS] BUG: soft lockup - CPU#0 stuck for 36s! [swapper/0:0]
2016-08-18 12:39 GMT-04:00 correomm <correomm at gmail.com>:> This bug is reported only on the VM's with CentOS 7 running on on VMware > ESXi 5.1. > The vSphere performance graph shows high CPU consume and disk activity only > on VM's with CentOS 7. Sometimes I can not connect remotely with ssh > (timeout error). >I'm also seeing those errors in several servers, running under 5.5. Currently investigating if this <https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009996> has anything to do (the resource overcommit bit). HTH, Carlos.
> 2016-08-18 12:39 GMT-04:00 correomm <correomm at gmail.com>: > >> This bug is reported only on the VM's with CentOS 7 running on on VMware >> ESXi 5.1. >> The vSphere performance graph shows high CPU consume and disk activity only >> on VM's with CentOS 7. Sometimes I can not connect remotely with ssh >> (timeout error). >> > I'm also seeing those errors in several servers, running under 5.5. > Currently investigating if this > <https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1009996> > has anything to do (the resource overcommit bit).Does this happen (only) while taking or consolidating snapshots? The VM is suspended during these operations and the OS isn't too crazy about it, especially if you have slow storage. Jack
correomm
2016-Aug-18 17:50 UTC
[CentOS] BUG: soft lockup - CPU#0 stuck for 36s! [swapper/0:0]
Yes, I tried it, but does not exists: vmguest # cat /proc/sys/kernel/softlockup_thresh cat: /proc/sys/kernel/softlockup_thresh: No such file or directory On Thu, Aug 18, 2016 at 2:06 PM, Carlos A. Carnero Delgado < carloscarnero at gmail.com> wrote:> 2016-08-18 12:39 GMT-04:00 correomm <correomm at gmail.com>: > > > This bug is reported only on the VM's with CentOS 7 running on on VMware > > ESXi 5.1. > > The vSphere performance graph shows high CPU consume and disk activity > only > > on VM's with CentOS 7. Sometimes I can not connect remotely with ssh > > (timeout error). > > > > I'm also seeing those errors in several servers, running under 5.5. > Currently investigating if this > <https://kb.vmware.com/selfservice/microsites/search. > do?language=en_US&cmd=displayKC&externalId=1009996> > has anything to do (the resource overcommit bit). > > HTH, > Carlos. > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos >