We''ve started haveing some crashed on systems running RHEL4 w/ message "badness in i8042_panick_blink at drivers/input/serio/i8042.c:992 Puppet is version 24.8. In some cases, swap was supposed to be present and wasn''t working, on others swap was intentionally omitted. So far, no reoccurances after turning on swap. Any thoughts? --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Andrew Shafer
2009-Jul-16 19:12 UTC
[Puppet Users] Re: Info - crash on RHEL4 w/o swap space
What crashes exactly? Puppet crashes? Can you see any other pattern in the logs? Searching a bit, that looks like memory starvation. Do you have metrics on memory usage for those machines? On Thu, Jul 16, 2009 at 10:07 AM, Len Rugen <lenrugen@gmail.com> wrote:> We''ve started haveing some crashed on systems running RHEL4 w/ message > "badness in i8042_panick_blink at drivers/input/serio/i8042.c:992 Puppet is > version 24.8. > > In some cases, swap was supposed to be present and wasn''t working, on > others swap was intentionally omitted. So far, no reoccurances after > turning on swap. > > Any thoughts? > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
It''s a complete system hang, only power switch will restart. The memory appears to be OK and the crash doesn''t happen during the puppet run. But no other consistant change was made to these systems other than they were puppeted, it''s only effected RHEL4 32-bit systems w/o swap BUT those might be the only systems we have that don''t have swap. It hit a few systems more than once before we found the pattern, but it hasn''t happened on a system where swap was activated. I''m from the "always have swap" school, but some people don''t like it.... I''m winning them over! On Thu, Jul 16, 2009 at 2:12 PM, Andrew Shafer <andrew@reductivelabs.com>wrote:> > What crashes exactly? Puppet crashes? > > Can you see any other pattern in the logs? > > Searching a bit, that looks like memory starvation. Do you have metrics on > memory usage for those machines? > > > > > On Thu, Jul 16, 2009 at 10:07 AM, Len Rugen <lenrugen@gmail.com> wrote: > >> We''ve started haveing some crashed on systems running RHEL4 w/ message >> "badness in i8042_panick_blink at drivers/input/serio/i8042.c:992 Puppet is >> version 24.8. >> >> In some cases, swap was supposed to be present and wasn''t working, on >> others swap was intentionally omitted. So far, no reoccurances after >> turning on swap. >> >> Any thoughts? >> >> >> > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Larry Ludwig
2009-Jul-19 16:35 UTC
[Puppet Users] Re: Info - crash on RHEL4 w/o swap space
how much memory is on this instance? Swap should ALWAYS be available, general recommendation is 2x actual memory. Once you run out of real memory, you can have a dead machine. -L -- Larry Ludwig Reductive Labs --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
There were multiple systems, 1 - 2G at most. Most had an inactive (unintentional) swap partition, it hadn''t been a problem pre-puppet. And, it wasn''t a problem until the puppet server reached about 90 clients. All the RHEL 4 systems were in crontab as 0/30, so they all hit at the same time. We spread them out, which probably helpded too. The puppet run was sometimes finished for several minutes before the crash. On Sun, Jul 19, 2009 at 11:35 AM, Larry Ludwig <larry@reductivelabs.com>wrote:> > how much memory is on this instance? > > Swap should ALWAYS be available, general recommendation is 2x actual > memory. Once you run out of real memory, you can have a dead machine. > > -L > > -- > Larry Ludwig > Reductive Labs > > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi I had this observation on RHEL 5 systems as well. Also with encrypted swap on a luks device. With direct swap I never encountered this problem. I came to the conclusion that I can''t blame puppet for the problem as the problem occured even with puppet turned off. However I could reduce it to a problem when suddenly a lot of swap is used and the systems should swap very fast, for example when munin processes limits or when a php process uses suddenly a lot of memory. I was never able to reproduce this problem on a solid basis, however what helped was to give more memory or turn encrytped swap off (which somehow has other drawbacks). this only happened so far in virtualized systems. so imho it''s more a kernel problem related to memory usage... :-/ cheers pete --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---