Puppet users - Mar 2009 - high cpu usage and slow puppet performance..

looks like the following could cause high cpu usage and take puppet
longer time to finish a catalog run


	file { "/some/path":
		owner => "user1",
		group => "group1",
		ensure => directory,
		recurse => true
	}

especially when /some/path contain lot of files and sub-directories...

anyone have the same problem?
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On 12/03/2009, at 18:41, barrowkwan <barrowkwan@gmail.com> wrote:
>
> looks like the following could cause high cpu usage and take puppet
> longer time to finish a catalog run
>
>
>    file { "/some/path":
>        owner => "user1",
>        group => "group1",
>        ensure => directory,
>        recurse => true
>    }
>
> especially when /some/path contain lot of files and sub-directories...
>
> anyone have the same problem?

Yes, and it''s totally reasonable for it to.

Puppet has to do a stat and md5sum on every file under that path,  
which depending on your system could take forever and push the iowait  
up.

I''ve had this problem with a Rails app that stored session on the  
filesystem that was deployed through Puppet''s file type(!).

Recursion on the file type should be used in moderation. :-)

Cheers,
Lindsay
>
> >
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Thu, Mar 12, 2009 at 06:51:59PM +1100, Lindsay Holmwood
wrote:
> 
> On 12/03/2009, at 18:41, barrowkwan <barrowkwan@gmail.com> wrote:
> 
> >
> > looks like the following could cause high cpu usage and take
> > puppet longer time to finish a catalog run
> >
> >
> >    file { "/some/path":
> >        owner => "user1",
> >        group => "group1",
> >        ensure => directory,
> >        recurse => true
> >    }
> >
> > especially when /some/path contain lot of files and
> > sub-directories...
> >
> > anyone have the same problem?
> 
> 
> Yes, and it''s totally reasonable for it to.
> 
> Puppet has to do a stat and md5sum on every file under that path,
It *what*?  Just for an owner and group check?  Why?

Can that be turned off?  We got a lot of performance out of cfengine
at work by telling it to use mtime instead of checksumming?

-Robin

-- 
They say:  "The first AIs will be built by the military as weapons."
And I''m thinking:  "Does it even occur to you to try for something
other than the default outcome?" -- http://shorl.com/tydruhedufogre
http://www.digitalkingdom.org/~rlpowell/ *** http://www.lojban.org/

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

2009/3/12 Robin Lee Powell
<rlpowell@digitalkingdom.org>:
>
> On Thu, Mar 12, 2009 at 06:51:59PM +1100, Lindsay Holmwood wrote:
>>
>> Puppet has to do a stat and md5sum on every file under that path,
>
> It *what*?  Just for an owner and group check?  Why?
>
> Can that be turned off?  We got a lot of performance out of cfengine
> at work by telling it to use mtime instead of checksumming?
I should have mentioned that this was under an older version of Puppet
(almost 2 years ago). I can''t comment on how the current code works.

Lindsay

-- 
http://holmwood.id.au/~lindsay/ (me)

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Thu, Mar 12, 2009 at 12:41:40AM -0700, barrowkwan
wrote:
> 
> anyone have the same problem?
Which filesystem are you using?  Some of them perform badly for tasks
that iterate over a large directory heirarchy.  JFS is particularly bad
at this.

-- 
Bruce

Remember you''re a Womble.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Thu, 2009-03-12 at 00:41 -0700, barrowkwan wrote:
> looks like the following could cause high cpu usage and take puppet
> longer time to finish a catalog run
> 
> 
> 	file { "/some/path":
> 		owner => "user1",
> 		group => "group1",
> 		ensure => directory,
> 		recurse => true
> 	}
> 
> especially when /some/path contain lot of files and sub-directories...
Do you have checksums enabled by default?

See my comments (look for masterzen comments) in this bug:
http://projects.reductivelabs.com/issues/1973

Basically, if you have a default checksum in your manifests, then what
puppet does is computing all the checksums for all the files (and
storing them in RAM too). This can take a lots of time if you only want
to manage ownership.

The solution is to rewrite to:
	file { "/some/path":
		owner => "user1",
		group => "group1",
		ensure => directory,
		recurse => true,
		checksum => undef
	}

Also you can read my blog post which talks about this issue:
http://www.masterzen.fr/2009/02/21/help-puppetd-is-eating-my-server/
-- 
Brice Figureau
My Blog: http://www.masterzen.fr/


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

very nice explanation in your blog.  Thanks!

I am a bit confused.   sometime it said the patch is on 0.25 but
sometimes it said 0.24.8, so which version/release will have this
fix?  this problem really hurt our system... :(

thanks!

On Mar 12, 3:25 am, Brice Figureau <brice-pup...@daysofwonder.com>
wrote:
> On Thu, 2009-03-12 at 00:41 -0700, barrowkwan wrote:
> > looks like the following could cause high cpu usage and take puppet
> > longer time to finish a catalog run
>
> >    file { "/some/path":
> >            owner => "user1",
> >            group => "group1",
> >            ensure => directory,
> >            recurse => true
> >    }
>
> > especially when /some/path contain lot of files and sub-directories...
>
> Do you have checksums enabled by default?
>
> See my comments (look for masterzen comments) in this
bug:http://projects.reductivelabs.com/issues/1973
>
> Basically, if you have a default checksum in your manifests, then what
> puppet does is computing all the checksums for all the files (and
> storing them in RAM too). This can take a lots of time if you only want
> to manage ownership.
>
> The solution is to rewrite to:
>         file { "/some/path":
>                 owner => "user1",
>                 group => "group1",
>                 ensure => directory,
>                 recurse => true,
>                 checksum => undef
>         }
>
> Also you can read my blog post which talks about this
issue:http://www.masterzen.fr/2009/02/21/help-puppetd-is-eating-my-server/
> --
> Brice Figureau
> My Blog:http://www.masterzen.fr/
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On 12/03/09 17:31, barrowkwan wrote:
> very nice explanation in your blog.  Thanks!
> 
> I am a bit confused.   sometime it said the patch is on 0.25 but
> sometimes it said 0.24.8, so which version/release will have this
> fix?  this problem really hurt our system... :(
Are you talking about the fix for the bug #1469?
If yes, it will be in 0.25 only (it would be very difficult to backport 
it as it was possible to implement it for 0.25 because Luke did a huge 
refactor of the file type).

Note, that this bug improves the situation only if your file{} resource 
has the source attribute (downloading files from the puppetmaster).

In your case, I think using checksum => undef should be enough to 
tremendously speed-up this operation.

[snipped original discussion]
-- 
Brice Figureau
Days of Wonder
http://www.daysofwonder.com

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

I avoid all the puppet serdes by just calling rsync.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

Hi
> Are you talking about the fix for the bug #1469?
> If yes, it will be in 0.25 only (it would be very difficult to backport 
> it as it was possible to implement it for 0.25 because Luke did a huge 
> refactor of the file type).
> 
> Note, that this bug improves the situation only if your file{} resource 
> has the source attribute (downloading files from the puppetmaster).
> 
> In your case, I think using checksum => undef should be enough to 
> tremendously speed-up this operation.
I''m currently a bit in the same or at least similar situation as we try
to ensure that certain folders and their content of webapplications
always have the right permission. so our idea have been to use the file
resource on these folders, with recursive set to true, etc.

Speed doesn''t matter that much, what matters more is the memory usage.

So this is our test-scenario:

--- foo.pp ---

$documentroot=''/var/www/vhosts/path/to/joomla/installation/www''

file{ [ "$documentroot/administrator/backups",
	"$documentroot/administrator/components",
	"$documentroot/administrator/language",
	"$documentroot/administrator/modules",
	"$documentroot/administrator/templates",
        "$documentroot/components",
        "$documentroot/images",
        "$documentroot/language",
        "$documentroot/media",
        "$documentroot/modules",
        "$documentroot/plugins",
        "$documentroot/templates",
        "$documentroot/cache",
        "$documentroot/administrator/cache" ]:
        	owner => ''user'',
                group => ''group'',
                mode => 0660,
                recurse => true,
                backup => false,
                checksum => undef,
}
---

a run of just that manifest (so puppet foo.pp) can take up to:

VIRT  RES
273m 193m

of Memory Usage.

which is for:

# du -sch administrator/backups administrator/components
administrator/language administrator/modules administrator/templates
components images language media modules plugins templates cache
administrator/cache
16K     administrator/backups
5.8M    administrator/components
4.2M    administrator/language
664K    administrator/modules
1.9M    administrator/templates
3.2M    components
892K    images
2.6M    language
616K    media
1.8M    modules
7.8M    plugins
9.0M    templates
224K    cache
16K     administrator/cache
39M     total

# find administrator/backups administrator/components
administrator/language administrator/modules administrator/templates
components images language media modules plugins templates cache
administrator/cache | wc -l
4336

# ruby -version
ruby 1.8.6 (2007-09-24 patchlevel 111) [x86_64-linux]

in my opinion still "a bit" too much.

if I don''t set recurse (and this is the way we go currently)
everything''s fine. So it certainly has to do with recursing a file
tree.
I''m running nearly latest head, at least #1965 [1] is applied.

So I''m waiting for 0.25! :) Would the standalone client already be
usable? So I could give feedback if the same manifest runs smoother.

thanks for all the work and cheers

pete

[1] http://projects.reductivelabs.com/issues/1965

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---

On Fri, 2009-03-13 at 01:25 +0100, Peter Meier wrote:
> Hi
> 
> > Are you talking about the fix for the bug #1469?
> > If yes, it will be in 0.25 only (it would be very difficult to
backport
> > it as it was possible to implement it for 0.25 because Luke did a huge
> > refactor of the file type).
> > 
> > Note, that this bug improves the situation only if your file{}
resource
> > has the source attribute (downloading files from the puppetmaster).
> > 
> > In your case, I think using checksum => undef should be enough to 
> > tremendously speed-up this operation.
> 
> I''m currently a bit in the same or at least similar situation as
we try
> to ensure that certain folders and their content of webapplications
> always have the right permission. so our idea have been to use the file
> resource on these folders, with recursive set to true, etc.
> 
> Speed doesn''t matter that much, what matters more is the memory
usage.
> 
> So this is our test-scenario:
> 
> --- foo.pp ---
> 
>
$documentroot=''/var/www/vhosts/path/to/joomla/installation/www''
> 
> file{ [ "$documentroot/administrator/backups",
> 	"$documentroot/administrator/components",
> 	"$documentroot/administrator/language",
> 	"$documentroot/administrator/modules",
> 	"$documentroot/administrator/templates",
>         "$documentroot/components",
>         "$documentroot/images",
>         "$documentroot/language",
>         "$documentroot/media",
>         "$documentroot/modules",
>         "$documentroot/plugins",
>         "$documentroot/templates",
>         "$documentroot/cache",
>         "$documentroot/administrator/cache" ]:
>         	owner => ''user'',
>                 group => ''group'',
>                 mode => 0660,
>                 recurse => true,
>                 backup => false,
>                 checksum => undef,
> }
> ---
> 
> a run of just that manifest (so puppet foo.pp) can take up to:
> 
> VIRT  RES
> 273m 193m
> 
> of Memory Usage.
Yes, that''s a lot. Unfortunately for such manifests puppet has to
create
a child file object per file it found while recursing the hierarchy.

If your aim is just to change/maintain ownership/perms, maybe an exec of
a recusive chown or combination of find/chown would be better, something
akin to:

exec {
  "chown-joomla-files":
	command => "find -type f ! -user ''user'' | xargs chown
user"
}
...

You see the idea...
Yes, the exec will execute on each catalog run, but that shouldn''t be
an
issue.
> if I don''t set recurse (and this is the way we go currently)
> everything''s fine. So it certainly has to do with recursing a file
tree.
> I''m running nearly latest head, at least #1965 [1] is applied.
#1965 won''t help on the client side. I didn''t reread the
patch, but if I
remember correctly it helps for storing the path used in the AST tree.

I think I''ll have a closer look to this issue soon.
-- 
Brice Figureau
My Blog: http://www.masterzen.fr/


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en
-~----------~----~----~----~------~----~------~--~---