-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Our site has run into some stability issues with some of our 64 bit xen hosts when running puppified guests. Our dom0 configuration: 64 bit (hardware, kernel, and userland), debian etch xen-3.0.3-1 no puppet Our domU configuration: 64 bit kernel, 32 bit userland debian etch puppet 0.24.4 Formerly reliable xen hosts have started crashing after we''ve started running puppet nodes on them, with a typical time-to-failure somewhere between 20 minutes and one week. This is almost certainly the result of a misconfiguration or a bug in xen, but it is not something that we encountered before deploying puppet machines on hosts with this build. We''re trying out a few approaches, but I thought it would be worth a shot to see if anyone on this list has experienced strange behavior with 64 bit xen hosts and puppet guests, or if anyone has an idea of what puppet might be doing that could expose weaknesses in xen. Thanks, James Bellenger -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.7 (GNU/Linux) iD8DBQFITX+a8TYLVDRcCYURAuNAAJwOMbnV6+sY/EenVD8lMDtbXjs+FgCfapal 1G3HcjEjXjYZ5yxmHebs7Xo=VeFw -----END PGP SIGNATURE----- --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
I''ve not seen this on centos domU 4.6, 5.1 with 64 bit versions, dom0 5.1 and using puppet. On Jun 9, 3:08 pm, James Bellenger <ja...@forwardcamegrendel.org> wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Our site has run into some stability issues with some of our 64 bit xen > hosts when running puppified guests. > > Our dom0 configuration: > 64 bit (hardware, kernel, and userland), > debian etch > xen-3.0.3-1 > no puppet > > Our domU configuration: > 64 bit kernel, 32 bit userland > debian etch > puppet 0.24.4 > > Formerly reliable xen hosts have started crashing after we''ve started > running puppet nodes on them, with a typical time-to-failure somewhere > between 20 minutes and one week. This is almost certainly the result of > a misconfiguration or a bug in xen, but it is not something that we > encountered before deploying puppet machines on hosts with this build. > > We''re trying out a few approaches, but I thought it would be worth a > shot to see if anyone on this list has experienced strange behavior with > 64 bit xen hosts and puppet guests, or if anyone has an idea of what > puppet might be doing that could expose weaknesses in xen. > > Thanks, > James Bellenger > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.7 (GNU/Linux) > > iD8DBQFITX+a8TYLVDRcCYURAuNAAJwOMbnV6+sY/EenVD8lMDtbXjs+FgCfapal > 1G3HcjEjXjYZ5yxmHebs7Xo> =VeFw > -----END PGP SIGNATURE-------~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Hi> I''ve not seen this on centos domU 4.6, 5.1 with 64 bit versions, dom0 > 5.1 and using puppet.I have seen this issue as well :( with dom0 and domU centos 5.1 leading to locked up domU''s. Unfortuantely I didn''t yet have time to investigate on it. I would be interested to investigate on this problem further. One problem is that at the moment I don''t see any reason why a userspace tool like puppet should screw up the kernel. As well I can''t see any abnormal things the last puppet runs. greets pete --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
How closely do you monitor them? What does ''xm top'' show? One thing to consider that puppet is a memory pig and takes up at least 50-100MB of memory, and on a 256MB domU that''s a decent amount of memory. Do you monitor swap and memory usage on your instance? With any VPS since memory is usually at a premium I suspect the issue you may be seeing is more memory usage which causes the server to become unresponsive because of swap thrashing. ''xm top'' will then show 100% CPU usage and cannot ssh in. This is just like a regular server which runs out of memory and swap. In our case I haven''t seen anything related to puppet itself but more rather more memory being used by puppet causes more swap thrashing. -L -- Larry Ludwig Empowering Media 1-866-792-0489 x600 Managed and Unmanaged Xen VPSes http://www.hostcube.com/ --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Oh we are also running Xen 3.0.3 on the dom0 with CentOS/RHEL patches. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Peter Meier wrote: | I have seen this issue as well :( with dom0 and domU centos 5.1 leading | to locked up domU''s. Not quite what I meant, the issue is that dom0, a machine that is not running puppet in any form, is crashing because of a domU that is running a puppet client. James Bellenger -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.7 (GNU/Linux) iD8DBQFITbGn8TYLVDRcCYURAumxAJ9XJFKEKAdqG1Oh0RP4oXj43Tv17gCfakze ndduzZXWaqFkkuJXne/7QJY=aD+B -----END PGP SIGNATURE----- --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---