Jeffrey Lord
2008-Jun-20 20:29 UTC
[Puppet Users] High Server Load and Configuration Compilation Times.
Hello All, We are running approx. 77 hosts (clients) and a single puppetmaster server (mongrel + apache) Lately it seems that we are experiencing quite a high load: top - 13:27:39 up 1 day, 18:17, 4 users, load average: 3.71, 3.67, 3.55 Tasks: 118 total, 6 running, 112 sleeping, 0 stopped, 0 zombie Cpu(s): 62.8%us, 37.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 2075636k total, 922164k used, 1153472k free, 269524k buffers Swap: 2096472k total, 0k used, 2096472k free, 234072k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 25747 puppet 25 0 78848 62m 1912 R 48 3.1 9:06.64 puppetmasterd 25711 puppet 25 0 60024 44m 1880 R 43 2.2 13:28.83 puppetmasterd 25650 puppet 25 0 133m 118m 1916 R 42 5.8 14:14.30 puppetmasterd 25688 puppet 25 0 107m 93m 1928 R 41 4.6 14:51.11 puppetmasterd As well as very long configuration compile times: Jun 20 13:28:11 it01-sea puppetmasterd[25650]: Compiled configuration for crawl16-sea.prod.msrch in 1650.99 seconds Any ideas or help in tracking this down would be greatly appreciated. -Jeff Lord --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Paul Lathrop
2008-Jun-20 23:53 UTC
[Puppet Users] Re: High Server Load and Configuration Compilation Times.
Are you using storedconfigs? On Fri, Jun 20, 2008 at 1:29 PM, Jeffrey Lord <jlord@mediosystems.com> wrote:> Hello All, > > We are running approx. 77 hosts (clients) and a single puppetmaster server > (mongrel + apache) > > Lately it seems that we are experiencing quite a high load: > > top - 13:27:39 up 1 day, 18:17, 4 users, load average: 3.71, 3.67, 3.55 > > Tasks: 118 total, 6 running, 112 sleeping, 0 stopped, 0 zombie > > Cpu(s): 62.8%us, 37.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.2%si, > 0.0%st > > Mem: 2075636k total, 922164k used, 1153472k free, 269524k buffers > > Swap: 2096472k total, 0k used, 2096472k free, 234072k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > 25747 puppet 25 0 78848 62m 1912 R 48 3.1 9:06.64 > puppetmasterd > > 25711 puppet 25 0 60024 44m 1880 R 43 2.2 13:28.83 > puppetmasterd > > 25650 puppet 25 0 133m 118m 1916 R 42 5.8 14:14.30 > puppetmasterd > > 25688 puppet 25 0 107m 93m 1928 R 41 4.6 14:51.11 puppetmasterd > > As well as very long configuration compile times: > > Jun 20 13:28:11 it01-sea puppetmasterd[25650]: Compiled configuration for > crawl16-sea.prod.msrch in 1650.99 seconds > > Any ideas or help in tracking this down would be greatly appreciated. > > -Jeff Lord > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Larry Ludwig
2008-Jun-21 03:15 UTC
[Puppet Users] Re: High Server Load and Configuration Compilation Times.
I think you are seeing the same thing we are seeing: http://groups.google.com/group/puppet-users/browse_thread/thread/b64a1f2502b11cea -L On Jun 20, 7:53 pm, "Paul Lathrop" <p...@tertiusfamily.net> wrote:> Are you using storedconfigs? > > On Fri, Jun 20, 2008 at 1:29 PM, Jeffrey Lord <jl...@mediosystems.com> wrote: > > Hello All, > > > We are running approx. 77 hosts (clients) and a single puppetmaster server > > (mongrel + apache) > > > Lately it seems that we are experiencing quite a high load: > > > top - 13:27:39 up 1 day, 18:17, 4 users, load average: 3.71, 3.67, 3.55 > > > Tasks: 118 total, 6 running, 112 sleeping, 0 stopped, 0 zombie > > > Cpu(s): 62.8%us, 37.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.2%si, > > 0.0%st > > > Mem: 2075636k total, 922164k used, 1153472k free, 269524k buffers > > > Swap: 2096472k total, 0k used, 2096472k free, 234072k cached > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > > COMMAND > > > 25747 puppet 25 0 78848 62m 1912 R 48 3.1 9:06.64 > > puppetmasterd > > > 25711 puppet 25 0 60024 44m 1880 R 43 2.2 13:28.83 > > puppetmasterd > > > 25650 puppet 25 0 133m 118m 1916 R 42 5.8 14:14.30 > > puppetmasterd > > > 25688 puppet 25 0 107m 93m 1928 R 41 4.6 14:51.11 puppetmasterd > > > As well as very long configuration compile times: > > > Jun 20 13:28:11 it01-sea puppetmasterd[25650]: Compiled configuration for > > crawl16-sea.prod.msrch in 1650.99 seconds > > > Any ideas or help in tracking this down would be greatly appreciated. > > > -Jeff Lord--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
udo waechter
2008-Jul-14 13:58 UTC
[Puppet Users] Re: High Server Load and Configuration Compilation Times.
Hi, we have the same problem here. We use storeconfigs: - puppetmaster is: - 4 x 2.4 Ghz Opterons, 4GB RAM - 4 puppetmasters (mongrel + apache) - database server is: - 2x Intel XEON (2 Ghz, I think), 2GB RAM, SCSI-RAID5 (pretty quick). - MySQL 5.x We currently restart the puppetmasters all 2 hours. Otherwise they simply crash. Is storeconfigs the problem here? We really need that feature. Thanks, udo. On 21.06.2008, at 01:53, Paul Lathrop wrote:> > Are you using storedconfigs? > > On Fri, Jun 20, 2008 at 1:29 PM, Jeffrey Lord > <jlord@mediosystems.com> wrote: >> Hello All, >> >> We are running approx. 77 hosts (clients) and a single puppetmaster >> server >> (mongrel + apache) >> >> Lately it seems that we are experiencing quite a high load: >> >> top - 13:27:39 up 1 day, 18:17, 4 users, load average: 3.71, >> 3.67, 3.55 >> >> Tasks: 118 total, 6 running, 112 sleeping, 0 stopped, 0 zombie >> >> Cpu(s): 62.8%us, 37.1%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, >> 0.2%si, >> 0.0%st >> >> Mem: 2075636k total, 922164k used, 1153472k free, 269524k >> buffers >> >> Swap: 2096472k total, 0k used, 2096472k free, 234072k >> cached >> >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ >> COMMAND >> >> 25747 puppet 25 0 78848 62m 1912 R 48 3.1 9:06.64 >> puppetmasterd >> >> 25711 puppet 25 0 60024 44m 1880 R 43 2.2 13:28.83 >> puppetmasterd >> >> 25650 puppet 25 0 133m 118m 1916 R 42 5.8 14:14.30 >> puppetmasterd >> >> 25688 puppet 25 0 107m 93m 1928 R 41 4.6 14:51.11 >> puppetmasterd >> >> As well as very long configuration compile times: >> >> Jun 20 13:28:11 it01-sea puppetmasterd[25650]: Compiled >> configuration for >> crawl16-sea.prod.msrch in 1650.99 seconds >> >> Any ideas or help in tracking this down would be greatly appreciated. >> >> -Jeff Lord >> >>> >> > > --~--~---------~--~----~------------~-------~--~----~ > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com > To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en > -~----------~----~----~----~------~----~------~--~--- >-- :: udo waechter - root@zoide.net :: N 52º16''30.5" E 8º3''10.1" :: genuine input for your ears: http://auriculabovinari.de :: your eyes: http://ezag.zoide.net :: your brain: http://zoide.net
Duncan Hill
2008-Jul-22 12:43 UTC
[Puppet Users] Re: High Server Load and Configuration Compilation Times.
2008/7/14 udo waechter <udo.waechter@uni-osnabrueck.de>:> Hi, > we have the same problem here. > We use storeconfigs: > - puppetmaster is: > - 4 x 2.4 Ghz Opterons, 4GB RAM > - 4 puppetmasters (mongrel + apache) > - database server is: > - 2x Intel XEON (2 Ghz, I think), 2GB RAM, SCSI-RAID5 (pretty quick). > - MySQL 5.x > > We currently restart the puppetmasters all 2 hours. Otherwise they simply > crash. > > Is storeconfigs the problem here? We really need that feature.I think storeconfigs is indeed part of the problem with puppetmaster''s ballooning memory usage and subsequent CPU load (though for us the CPU load is a factor of the memory usage - puppetmaster is running in a 384 MB VM, so when it starts using swap, CPU load spikes nastily). I turned storeconfigs off last week, and I haven''t had any reports from our monitoring system about the puppetmaster server since - normally I could expect at least 1 or 2 ''ssh on puppetmaster is down'' messages per day. --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Ohad Levy
2008-Jul-22 13:18 UTC
[Puppet Users] Re: High Server Load and Configuration Compilation Times.
we are running puppetmaster without storeconfig, but we still have a lot of memory usage (up to 2GB per puppet master process in bad days). the only way we can avoid it is by have more puppetmaster processes on the same machine, this way each process handle less clients (and then the leak is reduced). and of course the restart of the leaking processes.... Ohad On Tue, Jul 22, 2008 at 8:43 PM, Duncan Hill <bajandude@googlemail.com> wrote:> > 2008/7/14 udo waechter <udo.waechter@uni-osnabrueck.de>: > > Hi, > > we have the same problem here. > > We use storeconfigs: > > - puppetmaster is: > > - 4 x 2.4 Ghz Opterons, 4GB RAM > > - 4 puppetmasters (mongrel + apache) > > - database server is: > > - 2x Intel XEON (2 Ghz, I think), 2GB RAM, SCSI-RAID5 (pretty > quick). > > - MySQL 5.x > > > > We currently restart the puppetmasters all 2 hours. Otherwise they simply > > crash. > > > > Is storeconfigs the problem here? We really need that feature. > > I think storeconfigs is indeed part of the problem with puppetmaster''s > ballooning memory usage and subsequent CPU load (though for us the CPU > load is a factor of the memory usage - puppetmaster is running in a > 384 MB VM, so when it starts using swap, CPU load spikes nastily). I > turned storeconfigs off last week, and I haven''t had any reports from > our monitoring system about the puppetmaster server since - normally I > could expect at least 1 or 2 ''ssh on puppetmaster is down'' messages > per day. > > > >--~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---
Nigel Kersten
2008-Jul-25 03:24 UTC
[Puppet Users] Re: High Server Load and Configuration Compilation Times.
We were seeing similar symptoms on our dedicated Puppet CA server until we tracked down an issue very recently that I''ve just submitted a patch for, where each cert signing resulted in every cert being appended to inventory.txt Does the problem raise its head when you sign a new cert? Since this change, load has gone way way down. On Tue, Jul 22, 2008 at 6:18 AM, Ohad Levy <ohadlevy@gmail.com> wrote:> we are running puppetmaster without storeconfig, but we still have a lot of > memory usage (up to 2GB per puppet master process in bad days). > > the only way we can avoid it is by have more puppetmaster processes on the > same machine, this way each process handle less clients (and then the leak > is reduced). > and of course the restart of the leaking processes.... > > Ohad > > On Tue, Jul 22, 2008 at 8:43 PM, Duncan Hill <bajandude@googlemail.com> > wrote: >> >> 2008/7/14 udo waechter <udo.waechter@uni-osnabrueck.de>: >> > Hi, >> > we have the same problem here. >> > We use storeconfigs: >> > - puppetmaster is: >> > - 4 x 2.4 Ghz Opterons, 4GB RAM >> > - 4 puppetmasters (mongrel + apache) >> > - database server is: >> > - 2x Intel XEON (2 Ghz, I think), 2GB RAM, SCSI-RAID5 (pretty >> > quick). >> > - MySQL 5.x >> > >> > We currently restart the puppetmasters all 2 hours. Otherwise they >> > simply >> > crash. >> > >> > Is storeconfigs the problem here? We really need that feature. >> >> I think storeconfigs is indeed part of the problem with puppetmaster''s >> ballooning memory usage and subsequent CPU load (though for us the CPU >> load is a factor of the memory usage - puppetmaster is running in a >> 384 MB VM, so when it starts using swap, CPU load spikes nastily). I >> turned storeconfigs off last week, and I haven''t had any reports from >> our monitoring system about the puppetmaster server since - normally I >> could expect at least 1 or 2 ''ssh on puppetmaster is down'' messages >> per day. >> >> > > > > >-- Nigel Kersten Systems Administrator Tech Lead - MacOps --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com To unsubscribe from this group, send email to puppet-users-unsubscribe@googlegroups.com For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en -~----------~----~----~----~------~----~------~--~---