Hi Any one can please help me with below issue? I have setup one master(Cent OS) and 40 agent(windows) ,from last three days my all nodes are showing unresponsive in puppet -dashboard. And last got error in report is as below- err Could not retrieve catalog from remote server: Error 400 on SERVER: Failed when searching for node eqxtest01.equinoxfitness.com: Cannot allocate memory - fork(2) Puppet 2013-03-24 09:36 UTC warning Error 400 on SERVER: Cannot allocate memory - fork(2) Puppet 2013-03-24 09:36 UTC warning Unable to fetch my node definition, but the agent run will continue: Puppet Please guide. -- Thanks and Regards, Mamta Garg -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On Friday, March 29, 2013 4:25:01 AM UTC-5, Mamta Garg wrote:> > Hi Any one can please help me with below issue? > > I have setup one master(Cent OS) and 40 agent(windows) ,from last three > days my all nodes are showing unresponsive in puppet -dashboard. > And last got error in report is as below- > > > err Could not retrieve catalog from remote server: Error 400 on SERVER: > Failed when searching for node eqxtest01.equinoxfitness.com: Cannot > allocate memory - fork(2) Puppet > > 2013-03-24 09:36 UTC warning Error 400 on SERVER: Cannot allocate memory > - fork(2) Puppet > > 2013-03-24 09:36 UTC warning Unable to fetch my node definition, but the > agent run will continue: Puppet > >Have you checked memory usage on the master? The error message basically says that it''s completely out of available memory. On Linux the "free" command should confirm that, or you could perhaps get a more detailed picture via "top". Or there are GUI tools that accomplish the same thing if your master is running X. The master is notorious for being a memory hog, so you need to ensure that it has plenty of RAM to work with. Ideally it should have a good piece of swap, too. If you''re running in apache/passenger or similar, then be sure you''re not getting too many masters running at the same time, as that will multiply the memory consumption issue. For an installation your size, you really oughtn''t to need more than about two concurrent master processes, and you might even be able to get away with webrick / single master, depending on the complexity of your manifests. John John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
HI jcbollinger , Thanks for your reply! I have attached my master machine memory infor screenshot.Could you guide me with that ,whats is wrong? I am using one master at a time. Thanks, Mamta On Fri, Mar 29, 2013 at 9:04 AM, jcbollinger <John.Bollinger@stjude.org>wrote:> > > On Friday, March 29, 2013 4:25:01 AM UTC-5, Mamta Garg wrote: >> >> Hi Any one can please help me with below issue? >> >> I have setup one master(Cent OS) and 40 agent(windows) ,from last three >> days my all nodes are showing unresponsive in puppet -dashboard. >> And last got error in report is as below- >> >> >> err Could not retrieve catalog from remote server: Error 400 on SERVER: >> Failed when searching for node eqxtest01.equinoxfitness.com: Cannot >> allocate memory - fork(2) Puppet >> >> 2013-03-24 09:36 UTC warning Error 400 on SERVER: Cannot allocate >> memory - fork(2) Puppet >> >> 2013-03-24 09:36 UTC warning Unable to fetch my node definition, but >> the agent run will continue: Puppet >> >> > Have you checked memory usage on the master? The error message basically > says that it''s completely out of available memory. On Linux the "free" > command should confirm that, or you could perhaps get a more detailed > picture via "top". Or there are GUI tools that accomplish the same thing > if your master is running X. > > The master is notorious for being a memory hog, so you need to ensure that > it has plenty of RAM to work with. Ideally it should have a good piece of > swap, too. If you''re running in apache/passenger or similar, then be sure > you''re not getting too many masters running at the same time, as that will > multiply the memory consumption issue. For an installation your size, you > really oughtn''t to need more than about two concurrent master processes, > and you might even be able to get away with webrick / single master, > depending on the complexity of your manifests. > > > John > > > > John > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- Thanks and Regards, Mamta Garg -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On Monday, April 1, 2013 6:15:54 AM UTC-5, Mamta Garg wrote:> > HI jcbollinger , > > Thanks for your reply! > > I have attached my master machine memory infor screenshot.Could you guide > me with that ,whats is wrong? > I am using one master at a time.I don''t see a smoking gun, but I find it suspicious that you have so little swap. I generally want to see at least as much swap as total RAM, and I usually configure 1.5x or sometimes even 2x RAM. The 2G total RAM in your box is perhaps a bit light, too, but not so much so that I would expect to see problems for a single puppetmaster process. Unless the box is providing other memory-hungry services as well. Do check the puppetmaster process''s memory usage, as you may find that puppet needs up to as much free memory as its then-current usage to successfully fork(), which it does whenever it runs an external program. (That additional allocation is released when the external program exits.) It is not uncommon for the master to use hundreds of MB of memory. For Puppet to fail (only) occasionally with the kind of error you report, you are probably are running right on the edge of the master''s capacity. In that case, increasing swap would probably make the errors cease, at least for the time being, but you will see a slowdown somewhere whenever the system needs to page anything out. Possibly little enough that you don''t notice. Alternatively, you can probably make the error go away by adding RAM. You may be able to make it go away temporarily by restarting the puppetmaster service (or apache/passenger of that''s the way you''re running it, etc.), but then the issue is likely to reappear fairly soon. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Hi jcbollinger , I have added 1 GB more RAM on server and now my free RAM is more than 2 GB.Also started puppet ,puppetmaster,puppet-dashboard services. But nodes are still showing unresponsive.Any idea? Thanks, Mamta On Mon, Apr 1, 2013 at 9:31 AM, jcbollinger <John.Bollinger@stjude.org>wrote:> > > On Monday, April 1, 2013 6:15:54 AM UTC-5, Mamta Garg wrote: >> >> HI jcbollinger , >> >> Thanks for your reply! >> >> I have attached my master machine memory infor screenshot.Could you guide >> me with that ,whats is wrong? >> I am using one master at a time. > > > > I don''t see a smoking gun, but I find it suspicious that you have so > little swap. I generally want to see at least as much swap as total RAM, > and I usually configure 1.5x or sometimes even 2x RAM. The 2G total RAM in > your box is perhaps a bit light, too, but not so much so that I would > expect to see problems for a single puppetmaster process. Unless the box > is providing other memory-hungry services as well. > > Do check the puppetmaster process''s memory usage, as you may find that > puppet needs up to as much free memory as its then-current usage to > successfully fork(), which it does whenever it runs an external program. > (That additional allocation is released when the external program exits.) > It is not uncommon for the master to use hundreds of MB of memory. > > For Puppet to fail (only) occasionally with the kind of error you report, > you are probably are running right on the edge of the master''s capacity. > In that case, increasing swap would probably make the errors cease, at > least for the time being, but you will see a slowdown somewhere whenever > the system needs to page anything out. Possibly little enough that you > don''t notice. Alternatively, you can probably make the error go away by > adding RAM. You may be able to make it go away temporarily by restarting > the puppetmaster service (or apache/passenger of that''s the way you''re > running it, etc.), but then the issue is likely to reappear fairly soon. > > > > John > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- Thanks and Regards, Mamta Garg -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On Monday, April 8, 2013 5:28:19 AM UTC-5, Mamta Garg wrote:> > Hi jcbollinger , > > I have added 1 GB more RAM on server and now my free RAM is more than 2 > GB.Also started puppet ,puppetmaster,puppet-dashboard services. > > But nodes are still showing unresponsive.Any idea? >I already suggested that your small swap size could be part of the problem. Have you tried increasing that? For 3MB RAM, I would want to see at least 3MB of swap space. Alternatively, the message pointing to memory as the problem could be a red herring. It might be that you are running out of a different system resource instead, such as number of concurrent processes or open files. That would be unlikely if the master is running as the only service on a physical machine, but if it''s running in a VM then perhaps such a problem is occurring at the VM host. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
it could be that you have a ulimit for the amount of ram the puppetmaster process (or puppet user) can use. -- Erik Dalén On Monday 8 April 2013 at 12:28, Mamta Garg wrote:> Hi jcbollinger , > > I have added 1 GB more RAM on server and now my free RAM is more than 2 GB.Also started puppet ,puppetmaster,puppet-dashboard services. > > But nodes are still showing unresponsive.Any idea? > > Thanks, > Mamta > > > On Mon, Apr 1, 2013 at 9:31 AM, jcbollinger <John.Bollinger@stjude.org (mailto:John.Bollinger@stjude.org)> wrote: > > > > > > On Monday, April 1, 2013 6:15:54 AM UTC-5, Mamta Garg wrote: > > > HI jcbollinger , > > > > > > Thanks for your reply! > > > > > > I have attached my master machine memory infor screenshot.Could you guide me with that ,whats is wrong? > > > I am using one master at a time. > > > > > > I don''t see a smoking gun, but I find it suspicious that you have so little swap. I generally want to see at least as much swap as total RAM, and I usually configure 1.5x or sometimes even 2x RAM. The 2G total RAM in your box is perhaps a bit light, too, but not so much so that I would expect to see problems for a single puppetmaster process. Unless the box is providing other memory-hungry services as well. > > > > Do check the puppetmaster process''s memory usage, as you may find that puppet needs up to as much free memory as its then-current usage to successfully fork(), which it does whenever it runs an external program. (That additional allocation is released when the external program exits.) It is not uncommon for the master to use hundreds of MB of memory. > > > > For Puppet to fail (only) occasionally with the kind of error you report, you are probably are running right on the edge of the master''s capacity. In that case, increasing swap would probably make the errors cease, at least for the time being, but you will see a slowdown somewhere whenever the system needs to page anything out. Possibly little enough that you don''t notice. Alternatively, you can probably make the error go away by adding RAM. You may be able to make it go away temporarily by restarting the puppetmaster service (or apache/passenger of that''s the way you''re running it, etc.), but then the issue is likely to reappear fairly soon. > > > > > > > > John > > > > -- > > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com (mailto:puppet-users%2Bunsubscribe@googlegroups.com). > > To post to this group, send email to puppet-users@googlegroups.com (mailto:puppet-users@googlegroups.com). > > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > > For more options, visit https://groups.google.com/groups/opt_out. > > > > > > > > > -- > Thanks and Regards, > Mamta Garg > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com (mailto:puppet-users+unsubscribe@googlegroups.com). > To post to this group, send email to puppet-users@googlegroups.com (mailto:puppet-users@googlegroups.com). > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.