Puppet users - Nov 2011 - What's the canonical way to enforce permissions/ownership on a directory subtree?

I''ve just tried this (we assume /opt/jetty-6.1.26 already exists):

file { "/opt/jetty-6.1.26":
        owner => "jetty",
        group => "users",
        recurse => true,
}

... but it''s taking an *age*. What''s the Right Way?

Cheers, Robert.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

how big is the directory structure?

we''ve had incredibly painful experiences trying to manage directory
perms/ownerships on large directory trees...
so much so that we only set the perms on a few of the top level
directories and left the rest

it''s something to do with needing to do an md5 and stat on every file
in the tree that slows it down

how important is it that the permissions are forced?
we decided eventually that the file attr wouldn''t necessarily change
unless someone had access to the directory via ssh..
and only the sys admins do to the server in question anyway

hopefully there''s a better way of doing it ... calling all gurus?

cheers,
Andrew


On Nov 1, 11:15 am, Robert Atkins <snikta.tre...@gmail.com>
wrote:
> I''ve just tried this (we assume /opt/jetty-6.1.26 already exists):
>
> file { "/opt/jetty-6.1.26":
>         owner => "jetty",
>         group => "users",
>         recurse => true,
>
> }
>
> ... but it''s taking an *age*. What''s the Right Way?
>
> Cheers, Robert.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

It also has to do with Puppet''s implementation of File resources: it
creates in memory Ruby objects for every file and directory it finds
recursively, so combine that with the md5 summing and you''ll blow out
your CPU and memory usage very quickly. I''ve done something like this
in the past:

$path="/opt/jetty-6.1.26"
exec { "enforce ${path} permissions":
  command => "/usr/bin/find ${path} ! -uid jetty -exec chown jetty {}
\;",
  onlyif  => "test $(/usr/bin/find ${path} -uid jetty | wc -l) -gt
0",
}

I wouldn''t call it elegant, but much faster.

On Nov 1, 1:34 pm, madAndroid <andrewsta...@gmail.com>
wrote:
> how big is the directory structure?
>
> we''ve had incredibly painful experiences trying to manage
directory
> perms/ownerships on large directory trees...
> so much so that we only set the perms on a few of the top level
> directories and left the rest
>
> it''s something to do with needing to do an md5 and stat on every
file
> in the tree that slows it down
>
> how important is it that the permissions are forced?
> we decided eventually that the file attr wouldn''t necessarily
change
> unless someone had access to the directory via ssh..
> and only the sys admins do to the server in question anyway
>
> hopefully there''s a better way of doing it ... calling all gurus?
>
> cheers,
> Andrew
>
> On Nov 1, 11:15 am, Robert Atkins <snikta.tre...@gmail.com> wrote:
>
>
>
>
>
>
>
> > I''ve just tried this (we assume /opt/jetty-6.1.26 already
exists):
>
> > file { "/opt/jetty-6.1.26":
> >         owner => "jetty",
> >         group => "users",
> >         recurse => true,
>
> > }
>
> > ... but it''s taking an *age*. What''s the Right Way?
>
> > Cheers, Robert.
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Nov 1, 9:42 pm, Luke Bigum <Luke.Bi...@lmax.com>
wrote:
> It also has to do with Puppet''s implementation of File resources:
it
> creates in memory Ruby objects for every file and directory it finds
[...]

From some other reading I gathered this was what it''s doing. The jetty
dir is up to half a dozen levels deep with almost 4500 files. I''ll use
the solution you mentioned below unless there''s a better way of doing
it.

Cheers, Robert.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Thanks for your solution Luke, this is much faster. For future
reference here''s the modification I made (there were a couple of typos
and/or I''ve got a different dialect of find):

exec { "enforce ${jetty_install_dir} permissions":
  command => "/usr/bin/find ${jetty_install_dir} ! -user jetty -o ! -
group users -exec chown jetty:users {} \\;",
  onlyif  => "/usr/bin/test $(/usr/bin/find ${jetty_install_dir} ! -
user jetty -o ! -group users | wc -l) -gt 0",
}

Cheers, Robert.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

It''s worth mentioning that find -exec this way forks a separate copy of
find for each file. You''ll notice how much slower this is on a really
large set of files (possibly larger than yours). If you have a recursion-capable
chown it''s quicker to use chown -R.

On Wed, Nov 02, 2011 at 01:30:04AM -0700, Robert Atkins
wrote:
> Thanks for your solution Luke, this is much faster. For future
> reference here''s the modification I made (there were a couple of
typos
> and/or I''ve got a different dialect of find):
> 
> exec { "enforce ${jetty_install_dir} permissions":
>   command => "/usr/bin/find ${jetty_install_dir} ! -user jetty -o !
-
> group users -exec chown jetty:users {} \\;",
>   onlyif  => "/usr/bin/test $(/usr/bin/find ${jetty_install_dir} !
-
> user jetty -o ! -group users | wc -l) -gt 0",
> }
> 
> Cheers, Robert.
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
> 
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Nov 2, 7:33 pm, Christopher Wood <christopher_w...@pobox.com>
wrote:
> It''s worth mentioning that find -exec this way forks a separate
copy of find for each file. You''ll notice how much slower this is on a
really large set of files (possibly larger than yours). If you have a
recursion-capable chown it''s quicker to use chown -R.
Noted, this is what I''m using now:

exec { "enforce ${jetty_install_dir} permissions":
	command => "/bin/chown jetty:users ${jetty_install_dir}",
	onlyif  => "/usr/bin/test $(/usr/bin/find ${jetty_install_dir} ! -
user jetty -o ! -group users | wc -l) -gt 0",
	subscribe => [File["/opt/jetty/bin/jetty.sh"],
File["/opt/jetty/
resources/log4j.xml"]],
	refreshonly => true,
}

I can see it execute ("notice: /Stage[main]//Exec[enforce /opt/
jetty-6.1.26 permissions]: Triggered ''refresh'' from 1
events") but the
new files are still left with the wrong ownership. When I run the
"onlyif" command manually and echo $? I get 0, which says it should
run the chown. Can anyone see what I''m missing?

Cheers, Robert.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Nov 03, 2011 at 10:39:29PM -0700, Robert Atkins
wrote:
> On Nov 2, 7:33 pm, Christopher Wood <christopher_w...@pobox.com>
> wrote:
> > It''s worth mentioning that find -exec this way forks a
separate copy of find for each file. You''ll notice how much slower this
is on a really large set of files (possibly larger than yours). If you have a
recursion-capable chown it''s quicker to use chown -R.
> 
> Noted, this is what I''m using now:
> 
> exec { "enforce ${jetty_install_dir} permissions":
> 	command => "/bin/chown jetty:users ${jetty_install_dir}",
I think you still want chown -R here:

command => "/bin/chown -R jetty:users ${jetty_install_dir}",
> 	onlyif  => "/usr/bin/test $(/usr/bin/find ${jetty_install_dir} ! -
> user jetty -o ! -group users | wc -l) -gt 0",
> 	subscribe => [File["/opt/jetty/bin/jetty.sh"],
File["/opt/jetty/
> resources/log4j.xml"]],
> 	refreshonly => true,
> }
> 
> I can see it execute ("notice: /Stage[main]//Exec[enforce /opt/
> jetty-6.1.26 permissions]: Triggered ''refresh'' from 1
events") but the
> new files are still left with the wrong ownership. When I run the
> "onlyif" command manually and echo $? I get 0, which says it
should
> run the chown. Can anyone see what I''m missing?
> 
> Cheers, Robert.
> 
> -- 
> You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
> 
> 
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Thu, Nov 03, 2011 at 10:39:29PM -0700, Robert Atkins
wrote:
> On Nov 2, 7:33 pm, Christopher Wood <christopher_w...@pobox.com>
> wrote:
> > It''s worth mentioning that find -exec this way forks a
separate copy of find for each file. You''ll notice how much slower this
is on a really large set of files (possibly larger than yours). If you have a
recursion-capable chown it''s quicker to use chown -R.
> 
> Noted, this is what I''m using now:
> 
> exec { "enforce ${jetty_install_dir} permissions":
> 	command => "/bin/chown jetty:users ${jetty_install_dir}",
> 	onlyif  => "/usr/bin/test $(/usr/bin/find ${jetty_install_dir} ! -
> user jetty -o ! -group users | wc -l) -gt 0",
> 	subscribe => [File["/opt/jetty/bin/jetty.sh"],
File["/opt/jetty/
> resources/log4j.xml"]],
> 	refreshonly => true,
> }
> 
> I can see it execute ("notice: /Stage[main]//Exec[enforce /opt/
> jetty-6.1.26 permissions]: Triggered ''refresh'' from 1
events") but the
> new files are still left with the wrong ownership. When I run the
> "onlyif" command manually and echo $? I get 0, which says it
should
> run the chown. Can anyone see what I''m missing?
> 
> Cheers, Robert.
> 
I don''t see the need to specify refreshonly => true because you
already
have a propert onlyif statement. Refreshonly means that the command is
ONLY run when the exec resource receives a refresh event. A refresh event
is triggered when the subscribed resource has changed or a changed
resource has set the notify metaparamter.

So in your case the refreshonly renders your onlyif useless; the chown
command is only run, when /opt/jetty/bin/jetty.sh or
/opt/jetty/resources/log4j.xml changes.

-Stefan

On Nov 4, 9:42 pm, Christopher Wood <christopher_w...@pobox.com>
wrote:
> I think you still want chown -R here:
>
> command => "/bin/chown -R jetty:users ${jetty_install_dir}",
Of course I do, I fat-fingered it. That''s why it wasn''t
working.
Thanks for that (also, thanks to Stefan for the advice about
refreshonly=>true.)

Cheers, Robert.

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

Hi,

On 11/01/2011 02:42 PM, Luke Bigum wrote:
> It also has to do with Puppet''s implementation of File resources:
it
> creates in memory Ruby objects for every file and directory it finds
> recursively, so combine that with the md5 summing and you''ll blow
out
> your CPU and memory usage very quickly. I''ve done something like
this
> in the past:
seeing as this isn''t mentioned in this thread yet:

When recursing through directory trees, you most likely want to specify

checksum => "none"

in your file resource. This didn''t help in puppet 0.25, but since 2.6
I''ve used it to great benefit.

Still, large-ish trees with lots of (small) files will still take a very
long time due to the other effect noted above.

Cheers,
Felix

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Nov 30, 2011 at 6:48 AM, Felix Frank <
felix.frank@alumni.tu-berlin.de> wrote:
> Hi,
>
> On 11/01/2011 02:42 PM, Luke Bigum wrote:
> > It also has to do with Puppet''s implementation of File
resources: it
> > creates in memory Ruby objects for every file and directory it finds
> > recursively, so combine that with the md5 summing and you''ll
blow out
> > your CPU and memory usage very quickly. I''ve done something
like this
> > in the past:
>
> seeing as this isn''t mentioned in this thread yet:
>
> When recursing through directory trees, you most likely want to specify
>
> checksum => "none"
>
> in your file resource. This didn''t help in puppet 0.25, but since
2.6
> I''ve used it to great benefit.
>
> Still, large-ish trees with lots of (small) files will still take a very
> long time due to the other effect noted above.
>
Also if you are running 2.7.0 to 2.7.6, we recently fixed a performance
issue, which you''d see when recursing large numbers of files:
https://projects.puppetlabs.com/issues/9671

It''s fixed in 2.7.7

Josh

-- 
Josh Cooper
Developer, Puppet Labs

-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.

On Wed, Nov 30, 2011 at 8:40 AM, Josh Cooper <josh@puppetlabs.com>
wrote:
> On Wed, Nov 30, 2011 at 6:48 AM, Felix Frank
> <felix.frank@alumni.tu-berlin.de> wrote:
>>
>> Hi,
>>
>> On 11/01/2011 02:42 PM, Luke Bigum wrote:
>> > It also has to do with Puppet''s implementation of File
resources: it
>> > creates in memory Ruby objects for every file and directory it
finds
>> > recursively, so combine that with the md5 summing and
you''ll blow out
>> > your CPU and memory usage very quickly. I''ve done
something like this
>> > in the past:
>>
>> seeing as this isn''t mentioned in this thread yet:
>>
>> When recursing through directory trees, you most likely want to specify
>>
>> checksum => "none"
>>
>> in your file resource. This didn''t help in puppet 0.25, but
since 2.6
>> I''ve used it to great benefit.
>>
>> Still, large-ish trees with lots of (small) files will still take a
very
>> long time due to the other effect noted above.
>
> Also if you are running 2.7.0 to 2.7.6, we recently fixed a performance
> issue, which you''d see when recursing large numbers of
> files: https://projects.puppetlabs.com/issues/9671
> It''s fixed in 2.7.7
Or 2.7.8rc series (coming later this week), since 2.7.7 got nixed.
> Josh
> --
> Josh Cooper
> Developer, Puppet Labs
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
-- 
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.