Hi all, I''m working to migrate our ~500 DNS records from bind flat files to individual route53 resources. Each route53 resource definition is very slow -- checking a record exists requires 2 API lookups and adding requires 3, which means a resource can take >5seconds to run. This implies a massive slowdown for Puppet -- our current catalogs finish in 90seconds and this migration will add 40 minutes(!) to that. I can probably cache more to speed up these resources, but a significant increase in Puppet run-time seems inevitable. I''m worried that longer run-times are going to be a problem as I move more and more system config from file resources to custom Puppet types. How are people dealing with this sort of slowdown? Do I live with it, or is there a solution? Alex -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On 2013-05-07 06:28, Alex Jurkiewicz wrote: > How are people dealing with this sort of slowdown? Do I live with it, > or is there a solution? A common way to accelerate API management of external resources to implement prefetching in the provider (using a real ruby type) to fetch the current state with a single (or at least small amount of) calls. That way you''ll only take a small hit in the case that nothing has changed and will only need a single call per change. > I can probably cache more to speed up these resources, but a > significant increase in Puppet run-time seems inevitable. I''m worried > that longer run-times are going to be a problem as I move more and > more system config from file resources to custom Puppet types. The slowdown you are seeing does not come from config files vs. puppet types, but from managing files vs calling into a seemingly effing slow API. Other types like hosts or cron do not have as much overhead as you are experiencing. Regards, David -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On 05/07/2013 06:28 AM, Alex Jurkiewicz wrote:> Hi all, > > I''m working to migrate our ~500 DNS records from bind flat files to > individual route53 resources. Each route53 resource definition is very > slow -- checking a record exists requires 2 API lookups and adding > requires 3, which means a resource can take >5seconds to run. This > implies a massive slowdown for Puppet -- our current catalogs finish in > 90seconds and this migration will add 40 minutes(!) to that. > > I can probably cache more to speed up these resources, but a significant > increase in Puppet run-time seems inevitable. I''m worried that longer > run-times are going to be a problem as I move more and more system > config from file resources to custom Puppet types. > > How are people dealing with this sort of slowdown? Do I live with it, or > is there a solution?I had the same issue with my own ''cobblersystem'' type. Each parameter had at least 1 API or cli call. After that I changed the provider to use prefetching. That means 1 API call to get all the systems with all the parameters at once... Thing got ~9 times faster. I suggest you to rewrite your provider also... -- Jakov Sosic www.srce.unizg.hr -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users. For more options, visit https://groups.google.com/groups/opt_out.