Jennings, Jared L CTR USAF AFMC 46 SK/CCI
2011-Mar-22 19:08 UTC
[Puppet Users] augeas slooow
I''ve got somewhere around 130 Augeas resources in my policy (catalog, entire set of manifests, what-have-you) and 11 custom lenses, each for only one or two files, and it takes like 2 1/2 minutes to apply the whole thing. Looking at the --debug output, there are a whole lot of pauses right after "opening Augeas" sorts of lines. (That happens 79 times.) I think that''s because Augeas is populating its whole tree, each time. I know I can avoid that behavior by specifying the lens and incl parameters in my augeas resources, but I like the clarity I get by not doing that, and want to preserve it. Has anyone ever tried to make it so that only one Augeas connection is opened per Puppet run? How can I profile my Puppet catalog, to see what resources take the longest to set straight? (I''ve only got puppet apply for now - no master yet) -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
> > > How can I profile my Puppet catalog, to see what resources take the > longest to set straight? (I''ve only got puppet apply for now - no master > yet) > > --+1 for profiling feature -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On Tue, Mar 22, 2011 at 1:24 PM, saurabh verma <nitw.saurabh@gmail.com> wrote:>> >> >> How can I profile my Puppet catalog, to see what resources take the >> longest to set straight? (I''ve only got puppet apply for now - no master >> yet) >> >> -- > > +1 for profiling featureMore than --evaltrace ?> > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. > >-- Nigel Kersten Product, Puppet Labs @nigelkersten -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jennings, Jared L CTR USAF AFMC 46 SK/CCI
2011-Mar-22 22:00 UTC
RE: [Puppet Users] augeas slooow
Oho, this --evaltrace is what I was looking for. Thanks!> -----Original Message----- > From: puppet-users@googlegroups.com [mailto:puppet- > users@googlegroups.com] On Behalf Of Nigel Kersten > Sent: Tuesday, March 22, 2011 4:40 PM > To: puppet-users@googlegroups.com > Cc: saurabh verma > Subject: Re: [Puppet Users] augeas slooow > > On Tue, Mar 22, 2011 at 1:24 PM, saurabh verma<nitw.saurabh@gmail.com>> wrote: > >> > >> > >> How can I profile my Puppet catalog, to see what resources take the > >> longest to set straight? (I''ve only got puppet apply for now - no > master > >> yet) > >> > >> -- > > > > +1 for profiling feature > > More than --evaltrace ? > > > > > > > > -- > > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > > To post to this group, send email to puppet-users@googlegroups.com. > > To unsubscribe from this group, send email to puppet- > users+unsubscribe@googlegroups.com. > > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > > > > > > > > -- > Nigel Kersten > Product, Puppet Labs > @nigelkersten > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet- > users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 22/03/11 19:08, Jennings, Jared L CTR USAF AFMC 46 SK/CCI wrote:> I''ve got somewhere around 130 Augeas resources in my policy (catalog, > entire set of manifests, what-have-you) and 11 custom lenses, each for > only one or two files, and it takes like 2 1/2 minutes to apply the > whole thing. Looking at the --debug output, there are a whole lot of > pauses right after "opening Augeas" sorts of lines. (That happens 79 > times.) I think that''s because Augeas is populating its whole tree, each > time. > > I know I can avoid that behavior by specifying the lens and incl > parameters in my augeas resources, but I like the clarity I get by not > doing that, and want to preserve it. > > Has anyone ever tried to make it so that only one Augeas connection is > opened per Puppet run?I saw your suggestion here and filed it a little while back: https://projects.puppetlabs.com/issues/6920 More recently at Puppet Camp I realised there''s a slightly better way of optimising the loading, which I''ve tried to implement in this issue: https://projects.puppetlabs.com/issues/7285 If you get the chance to test it, I''d appreciate any feedback via the issue report. I suspect the method''s a bit fragile, so testing''s needed. There''s also a change that Mike Knox has been working on, which will halve the time taken by Puppet when an Augeas resource has to be applied. It''s under the guise of showing file diffs for Augeas actions, but happily speeds it up too: https://projects.puppetlabs.com/issues/2728 Hopefully this focus on Puppet+Augeas performance will help out your catalog runs in the future! Cheers, -- Dominic Cleal Red Hat Consulting m: +44 (0)7818 512168 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
From a previous email of letter@repeat.com.
" ...
To avoid that, I added ''lens'' and ''incl''
parameters to the puppet type a
while ago. If you set these, Augeas will only read a specific file, avoiding
most of the unnecessary overhead.
So, you should say something like
augeas {
add_a_host:
changes => "...",
lens => "Hosts.lns",
incl => "/etc/hosts"
}
in your manifests, and everything will be lightning fast, even without
connection caching."
On May 3, 2011 11:45 AM, "Dominic Cleal" <dcleal@redhat.com>
wrote:
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Jennings, Jared L CTR USAF AFMC 46 SK/CCI
2011-May-05 13:28 UTC
RE: [Puppet Users] augeas slooow
Hi, Aaron. Aaron (quoting letter@):> To avoid that, I added ''lens'' and ''incl'' parameters to the puppet type > a while ago. If you set these, Augeas will only read a specific file, > avoiding most of the unnecessary overhead. > ... everything will be lightning fast, even without > connection caching."Me:> I know I can avoid [Augeas populating its whole tree] by specifying > the lens and incl parameters in my augeas resources, but I like the > clarity I get by not doing that, and want to preserve it.The reason I brought the issue up is that I wanted to know why Augeas connections are not cached. I think they should be, but someone on this list may have tried to make that change, failed, and gained wisdom.> lens => "Hosts.lns",It''s nice to find that I may not have to give the full path of the lens. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On May 5, 2011, at 6:28 AM, Jennings, Jared L CTR USAF AFMC 46 SK/CCI wrote:> Hi, Aaron. > > Aaron (quoting letter@): >> To avoid that, I added ''lens'' and ''incl'' parameters to the puppet type >> a while ago. If you set these, Augeas will only read a specific file, >> avoiding most of the unnecessary overhead. >> ... everything will be lightning fast, even without >> connection caching." > > Me: >> I know I can avoid [Augeas populating its whole tree] by specifying >> the lens and incl parameters in my augeas resources, but I like the >> clarity I get by not doing that, and want to preserve it. > > The reason I brought the issue up is that I wanted to know why Augeas > connections are not cached. I think they should be, but someone on this > list may have tried to make that change, failed, and gained wisdom.I can tell you one reason why. One of the most expensive parts of starting Augeas is scanning the whole filesystem (or at least all auto-included files, of which there are a lot) and making that into a tree. It would need to rescan every file before processing each resource. I think modern versions do this by checking the mtime of every file, which is expensive, though better than parsing all from scratch. Still, I would make a random guess that caching the connections might actually be slower than using noautoinclude. Personally, as soon as all our machines are 2.6.x, I''m planning to try wrapping augeas in a define that will make "context" the the file path be the same value. (Technically I need to add "/files" at the beginning too.) -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
On 05/05/11 15:43, Patrick wrote:> > On May 5, 2011, at 6:28 AM, Jennings, Jared L CTR USAF AFMC 46 SK/CCI wrote: > >> Hi, Aaron. >> >> Aaron (quoting letter@): >>> To avoid that, I added ''lens'' and ''incl'' parameters to the puppet type >>> a while ago. If you set these, Augeas will only read a specific file, >>> avoiding most of the unnecessary overhead. >>> ... everything will be lightning fast, even without >>> connection caching." >> >> Me: >>> I know I can avoid [Augeas populating its whole tree] by specifying >>> the lens and incl parameters in my augeas resources, but I like the >>> clarity I get by not doing that, and want to preserve it. >> >> The reason I brought the issue up is that I wanted to know why Augeas >> connections are not cached. I think they should be, but someone on this >> list may have tried to make that change, failed, and gained wisdom.I''ve yet to try, but it''s a tricky request as it''s mean having ways to ensure the tree is in a sane state each time.> I can tell you one reason why. One of the most expensive parts of starting Augeas is scanning the whole filesystem (or at least all auto-included files, of which there are a lot) and making that into a tree. It would need to rescan every file before processing each resource. I think modern versions do this by checking the mtime of every file, which is expensive, though better than parsing all from scratch. > > Still, I would make a random guess that caching the connections might actually be slower than using noautoinclude.Indeed, I came to the same conclusion. The way I''ve proposed doing this (as implemented in the patch for #7285) is to use noautoload, then strip out those files that don''t match the context - if it''s supplied. You still need the autoinclude as you need to know both the lens and the file, you can''t just specify the file. -- Dominic Cleal Red Hat Consulting m: +44 (0)7818 512168 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.