Ok, we used to get this occasionally on cluster nodes, and we just got it
on a fileserver (very bad). The system is discovered to be unresponsive:
it doesn't ping, and plugging a console in, you can see that it's not
dead, but there nothing at all on the screen, nor does it respond to even
<ctrl-alt-del>. The only answer is to power cycle it; it comes up fine.
Nothing in /var/log/dmesg or /var/log/messages. No abrts I can find. sar
tells me it went unredponsive between 18:10 and 10:20 yesterday. Note that
there are no further entries in sar, either, for yesterday, after the
event, and nothing till I power cycled it.
Has anyone else seen this - I can't imagine it's only us - or have any
thoughts?
C 7, 7.6.1810
mark
On Wed, 22 May 2019 at 09:30, mark <m.roth at 5-cent.us> wrote:> Ok, we used to get this occasionally on cluster nodes, and we just got it > on a fileserver (very bad). The system is discovered to be unresponsive: > it doesn't ping, and plugging a console in, you can see that it's not > dead, but there nothing at all on the screen, nor does it respond to even > <ctrl-alt-del>. The only answer is to power cycle it; it comes up fine. > > Nothing in /var/log/dmesg or /var/log/messages. No abrts I can find. sar > tells me it went unredponsive between 18:10 and 10:20 yesterday. Note that > there are no further entries in sar, either, for yesterday, after the > event, and nothing till I power cycled it. > > >From the above description, I would normally say it sounds like hardware.However, why do you say the system is not dead when you plug in a console.. but there is nothing on the screen and it doesn't respond to control-alt-delete. To me that sounds like 'dead'. Usually the cpu is hardlocked or the hardware went into 'over-heat' and put everything in a deep sleep hoping it would cool down but never wake up.> Has anyone else seen this - I can't imagine it's only us - or have any > thoughts? > > C 7, 7.6.1810 > > mark > > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos >-- Stephen J Smoogen.
In the past I've found that the console may have blanked (due to time) and when the system locked up/hung it won't unblank. Booting with "consoleblank=0" on the kernel command line will ensure that whatever is printed to the console (oops, panic, etc) will be there for you to see when you connect. I've had intermittent success in that type of situation with the SysRq key(s), see https://en.wikipedia.org/wiki/Magic_SysRq_key . They do require that you have it configured/enabled ahead of time. If you access the console over a BMC/IPMI KVM session it can be very difficult, if not impossible, to enter the keystroke as well. Good luck, Scott On Wed, May 22, 2019 at 8:46 AM Stephen John Smoogen <smooge at gmail.com> wrote:> On Wed, 22 May 2019 at 09:30, mark <m.roth at 5-cent.us> wrote: > > > Ok, we used to get this occasionally on cluster nodes, and we just got it > > on a fileserver (very bad). The system is discovered to be unresponsive: > > it doesn't ping, and plugging a console in, you can see that it's not > > dead, but there nothing at all on the screen, nor does it respond to even > > <ctrl-alt-del>. The only answer is to power cycle it; it comes up fine. > > > > Nothing in /var/log/dmesg or /var/log/messages. No abrts I can find. sar > > tells me it went unredponsive between 18:10 and 10:20 yesterday. Note > that > > there are no further entries in sar, either, for yesterday, after the > > event, and nothing till I power cycled it. > > > > > From the above description, I would normally say it sounds like hardware. > However, why do you say the system is not dead when you plug in a console.. > but there is nothing on the screen and it doesn't respond to > control-alt-delete. To me that sounds like 'dead'. Usually the cpu is > hardlocked or the hardware went into 'over-heat' and put everything in a > deep sleep hoping it would cool down but never wake up. > > > > > Has anyone else seen this - I can't imagine it's only us - or have any > > thoughts? > > > > C 7, 7.6.1810 > > > > mark > > > > > > _______________________________________________ > > CentOS mailing list > > CentOS at centos.org > > https://lists.centos.org/mailman/listinfo/centos > > > > > -- > Stephen J Smoogen. > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos >-- DISCLAIMER: NOTICE REGARDING PRIVACY AND CONFIDENTIALITY? The information contained in and/or accompanying this communication is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this information, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy of any e-mail and any printout thereof. Electronic transmissions cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. Simplex Trading, LLC and its affiliates reserves the right to intercept, monitor, and retain electronic communications to and from its system as permitted by law. Simplex Trading, LLC is a registered Broker Dealer with CBOE and a Member of SIPC.
Stephen John Smoogen wrote:> On Wed, 22 May 2019 at 09:30, mark <m.roth at 5-cent.us> wrote: > >> Ok, we used to get this occasionally on cluster nodes, and we just got >> it on a fileserver (very bad). The system is discovered to be >> unresponsive: >> it doesn't ping, and plugging a console in, you can see that it's not >> dead, but there nothing at all on the screen, nor does it respond to >> even <ctrl-alt-del>. The only answer is to power cycle it; it comes up >> fine. >> >> Nothing in /var/log/dmesg or /var/log/messages. No abrts I can find. >> sar tells me it went unredponsive between 18:10 and 10:20 yesterday. >> Note that >> there are no further entries in sar, either, for yesterday, after the >> event, and nothing till I power cycled it. >> > From the above description, I would normally say it sounds like hardware. > However, why do you say the system is not dead when you plug in a > console.. but there is nothing on the screen and it doesn't respond to > control-alt-delete. To me that sounds like 'dead'. Usually the cpu is > hardlocked or the hardware went into 'over-heat' and put everything in a > deep sleep hoping it would cool down but never wake up. >It seems unlikely. It's a 4U server, with 36 disks (and the dual root disks), in a machine room, and ipmitool sel list shows nada, nor are there any warnings, as I've seen on other systems occasionally, that the CPU is overheating, and is being throttled.> > >> Has anyone else seen this - I can't imagine it's only us - or have any >> thoughts? >> >> C 7, 7.6.1810 >> >> >> mark >> >> >> _______________________________________________ >> CentOS mailing list >> CentOS at centos.org >> https://lists.centos.org/mailman/listinfo/centos >> >> > > > -- > Stephen J Smoogen. > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos > >
> Ok, we used to get this occasionally on cluster nodes, and we just got it > on a fileserver (very bad). The system is discovered to be unresponsive: > it doesn't ping, and plugging a console in, you can see that it's not > dead, but there nothing at all on the screen, nor does it respond to even > <ctrl-alt-del>. The only answer is to power cycle it; it comes up fine. > > Nothing in /var/log/dmesg or /var/log/messages. No abrts I can find. sar > tells me it went unredponsive between 18:10 and 10:20 yesterday. Note that > there are no further entries in sar, either, for yesterday, after the > event, and nothing till I power cycled it. > > Has anyone else seen this - I can't imagine it's only us - or have any > thoughts? > > C 7, 7.6.1810I saw such an issue recently and never found out what happened and why. Regards, Simon