Puppet users - Mar 2013 - Nodes Could not retrieve catalog from remote server: execution expired

Almost 4-5% of the total number of nodes are not receiving catalog from the 
master on an hourly run. We have about 250 nodes. The nodes that are 
receiving this error are rather random. The hourly cron happens almost at 
the same time. Is there any configuration changes for Puppet that can be 
done to avoid this ?

Puppet (err): Could not retrieve catalog from remote server: execution 
expired

Puppet (notice): Using cached catalog

/File[secure_host] (err): Could not evaluate: Connection reset by peer - 
SSL_connect Could not retrieve file metadata for 
puppet:///modules/certs/<filename>: Connection reset by peer - SSL_connect


Thanks for any suggestion.

-----------

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Wednesday, March 27, 2013 3:39:02 PM UTC-5, iamauser
wrote:
>
> Almost 4-5% of the total number of nodes are not receiving catalog from 
> the master on an hourly run. We have about 250 nodes. The nodes that are 
> receiving this error are rather random. The hourly cron happens almost at 
> the same time. Is there any configuration changes for Puppet that can be 
> done to avoid this ?
>
> Puppet (err): Could not retrieve catalog from remote server: execution 
> expired
>
> Puppet (notice): Using cached catalog
>
> /File[secure_host] (err): Could not evaluate: Connection reset by peer - 
> SSL_connect Could not retrieve file metadata for 
> puppet:///modules/certs/<filename>: Connection reset by peer -
SSL_connect
>
>
It sounds like you are pushing your master''s capacity limit.  If your 
clients are all trying to check in at about the same time then you should 
be able to make more efficient use of your available resources by spreading 
the client check-ins more evenly over time.

Supposing that your Puppet cron jobs are managed by Puppet, you can make 
use of Puppet''s built-in fqdn_rand() function to help generate launch
times
for your cron jobs that will be consistent on each node, but randomly 
varying from node to node.  If you were running the agent in daemon mode 
then the --splay option/parameter would address the same problem.


John

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Hi John,

Thanks for your suggestion. In our setup, puppet cron is managed by system 
crontab via a script and with  no-daemonize option.For now, I will divide 
the nodes and run them bi-hourly.  In the meantime, I will follow the 
example bunch on the setting up cron for puppet from here,

http://projects.puppetlabs.com/projects/1/wiki/Cron_Patterns

-------


On Thursday, March 28, 2013 8:04:08 AM UTC-5, jcbollinger
wrote:
>
>
>
> On Wednesday, March 27, 2013 3:39:02 PM UTC-5, iamauser wrote:
>>
>> Almost 4-5% of the total number of nodes are not receiving catalog from
>> the master on an hourly run. We have about 250 nodes. The nodes that
are
>> receiving this error are rather random. The hourly cron happens almost
at
>> the same time. Is there any configuration changes for Puppet that can
be
>> done to avoid this ?
>>
>> Puppet (err): Could not retrieve catalog from remote server: execution 
>> expired
>>
>> Puppet (notice): Using cached catalog
>>
>> /File[secure_host] (err): Could not evaluate: Connection reset by peer
-
>> SSL_connect Could not retrieve file metadata for 
>> puppet:///modules/certs/<filename>: Connection reset by peer -
SSL_connect
>>
>>
> It sounds like you are pushing your master''s capacity limit.  If
your
> clients are all trying to check in at about the same time then you should 
> be able to make more efficient use of your available resources by spreading
> the client check-ins more evenly over time.
>
> Supposing that your Puppet cron jobs are managed by Puppet, you can make 
> use of Puppet''s built-in fqdn_rand() function to help generate
launch times
> for your cron jobs that will be consistent on each node, but randomly 
> varying from node to node.  If you were running the agent in daemon mode 
> then the --splay option/parameter would address the same problem.
>
>
> John
>
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Wed, Mar 27, 2013 at 8:39 PM, iamauser <tapas.sarangi@gmail.com>
wrote:
> Almost 4-5% of the total number of nodes are not receiving catalog from the
> master on an hourly run. We have about 250 nodes. The nodes that are
> receiving this error are rather random. The hourly cron happens almost at
> the same time. Is there any configuration changes for Puppet that can be
> done to avoid this ?
In addition to what John said, you didn''t mention whether you are
already running the puppet master behind Apache or Nginx.  If you''re
not, then you are most probably being constrained by Puppet''s built-in
webserver, webrick, which is only single-threaded.

Regards,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Matt,

Thanks for your reply.

On Friday, March 29, 2013 2:32:10 PM UTC-5, Matthew Burgess
wrote:
>
>
> In addition to what John said, you didn''t mention whether you are 
> already running the puppet master behind Apache or Nginx.  If
you''re
> not, then you are most probably being constrained by Puppet''s
built-in
> webserver, webrick, which is only single-threaded. 
>
>
I am not running puppetmaster behind Apache or Nginx. So far with the 
default.

I looked at some of the documentation. It seems mongrel is not the way to 
go in Puppet-3.x. I get this message while trying to start and stop 
puppetmaster after the installation.
http://projects.puppetlabs.com/projects/puppet/wiki/Using_Mongrel_On_Enterprise_Linux
#
The mongrel servertype is no longer built-in to Puppet. It appears as 
though mongrel is being used, as the number of ports is greater than one. 
Starting the puppetmaster service will not behave as expected until this is 
resolved. Only the first port has been used in the service. These settings 
are defined at /etc/sysconfig/puppetmaster
#

Do you have a pointer to the docs for using puppetmaster behind Apache ?


Thanks
--------------



 
> Regards, 
>
> Matt. 
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

On Mon, Apr 1, 2013 at 5:24 PM, iamauser <tapas.sarangi@gmail.com> wrote:
>
> Do you have a pointer to the docs for using puppetmaster behind Apache ?
>
Sure.  The official docs are at
http://docs.puppetlabs.com/guides/passenger.html.

Regards,

Matt.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks. Just got it configured for our cluster. Now it should be able to 
manage 250 nodes without a bunch of nodes complaining about timeout.

-----


On Monday, April 1, 2013 2:05:02 PM UTC-5, Matthew Burgess
wrote:
>
> On Mon, Apr 1, 2013 at 5:24 PM, iamauser
<tapas....@gmail.com<javascript:>
> > wrote:
>
>>
>> Do you have a pointer to the docs for using puppetmaster behind Apache
?
>>
>
> Sure.  The official docs are at  
> http://docs.puppetlabs.com/guides/passenger.html.
>
> Regards,
>
> Matt.
>  
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscribe@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.