Sébastien Barthélémy
2011-Jan-05 17:11 UTC
[Puppet Users] Using puppet to fix a lot of files permissions
Hello,
I store camera pictures in a git repository, which became quite big:
104 GB for the whole (non-bare) repository.
I wanted to fix the files permissions, and thought puppet might be the
good tool for this (I like its declarative way of simplifying my
life).
I gave it a try, with the following statements
node "navi" {
file {
"/Users/seb/Pictures/pictures/":
mode => 0640,
owner => "seb",
group => "staff",
recurse => true,
ignore => ".git"
}
file {
"/Users/seb/Pictures/pictures/.encfs5/":
mode => 0600,
owner => "seb",
group => "staff",
recurse => true,
}
file {
"/Users/seb/Pictures/pictures/.git/":
mode => 0600,
owner => "seb",
group => "staff",
recurse => true,
}
file {
"/Users/seb/Pictures/pictures/.git/hooks/":
mode => 0700,
owner => "seb",
group => "staff",
recurse => true,
}
}
And a call to "sudo puppet -l /tmp/puppet.log ~/statement.pp"
Well, that was 4 hours ago and since then, ruby is eating 100% of my CPU
(of one core of my 2.26GHz core 2 duo).
From the log file, I can tell that puppet is indeed fixing perms, at a
rate lower than one file per 10 seconds.
I think find, xargs and chmod would take a few minutes at most (will try
later).
Why is puppet so slow at this job? Is there any way I could improve the
speed?
Thanks for any insight.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Mark Stanislav
2011-Jan-05 17:19 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
I''ve ran into the same issue, often causing my puppet run to state it''s ''Finished'' but actually be hanging and not returning back to a shell prompt. Not to thread-jack, but I am also interested in this answer. Thanks for raising it! -Mark On Jan 5, 2011, at 12:11 PM, Sébastien Barthélémy wrote:> Hello, > > I store camera pictures in a git repository, which became quite big: > 104 GB for the whole (non-bare) repository. > > I wanted to fix the files permissions, and thought puppet might be the good tool for this (I like its declarative way of simplifying my life). > > I gave it a try, with the following statements > > node "navi" { > file { > "/Users/seb/Pictures/pictures/": > mode => 0640, > owner => "seb", > group => "staff", > recurse => true, > ignore => ".git" > } > file { > "/Users/seb/Pictures/pictures/.encfs5/": > mode => 0600, > owner => "seb", > group => "staff", > recurse => true, > } > file { > "/Users/seb/Pictures/pictures/.git/": > mode => 0600, > owner => "seb", > group => "staff", > recurse => true, > } > file { > "/Users/seb/Pictures/pictures/.git/hooks/": > mode => 0700, > owner => "seb", > group => "staff", > recurse => true, > } > } > > And a call to "sudo puppet -l /tmp/puppet.log ~/statement.pp" > > Well, that was 4 hours ago and since then, ruby is eating 100% of my CPU > (of one core of my 2.26GHz core 2 duo). From the log file, I can tell that puppet is indeed fixing perms, at a rate lower than one file per 10 seconds. > > I think find, xargs and chmod would take a few minutes at most (will try later). > > Why is puppet so slow at this job? Is there any way I could improve the speed? > > Thanks for any insight. > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2011-Jan-05 18:13 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
On 05/01/11 18:11, Sébastien Barthélémy wrote:> Hello, > > I store camera pictures in a git repository, which became quite big: > 104 GB for the whole (non-bare) repository. > > I wanted to fix the files permissions, and thought puppet might be the > good tool for this (I like its declarative way of simplifying my > life). > > I gave it a try, with the following statements > > node "navi" { > file { > "/Users/seb/Pictures/pictures/": > mode => 0640, > owner => "seb", > group => "staff", > recurse => true, > ignore => ".git" > } > file { > "/Users/seb/Pictures/pictures/.encfs5/": > mode => 0600, > owner => "seb", > group => "staff", > recurse => true, > } > file { > "/Users/seb/Pictures/pictures/.git/": > mode => 0600, > owner => "seb", > group => "staff", > recurse => true, > } > file { > "/Users/seb/Pictures/pictures/.git/hooks/": > mode => 0700, > owner => "seb", > group => "staff", > recurse => true, > } > } > > And a call to "sudo puppet -l /tmp/puppet.log ~/statement.pp" > > Well, that was 4 hours ago and since then, ruby is eating 100% of my CPU > (of one core of my 2.26GHz core 2 duo). > From the log file, I can tell that puppet is indeed fixing perms, at a > rate lower than one file per 10 seconds. > > I think find, xargs and chmod would take a few minutes at most (will try > later). > > Why is puppet so slow at this job? Is there any way I could improve the > speed?Puppet is md5 checksumming all your files and that is a loooong and slooow operation. If you run 2.6, you should add: checksum => none to your file resources, and it should be way faster. -- Brice Figureau My Blog: http://www.masterzen.fr/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Patrick
2011-Jan-05 21:25 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
On Jan 5, 2011, at 10:13 AM, Brice Figureau wrote:> On 05/01/11 18:11, Sébastien Barthélémy wrote: >> Hello, >> >> I store camera pictures in a git repository, which became quite big: >> 104 GB for the whole (non-bare) repository. >> >> I wanted to fix the files permissions, and thought puppet might be the >> good tool for this (I like its declarative way of simplifying my >> life). >> >> I gave it a try, with the following statements >> >> node "navi" { >> file { >> "/Users/seb/Pictures/pictures/": >> mode => 0640, >> owner => "seb", >> group => "staff", >> recurse => true, >> ignore => ".git" >> } >> file { >> "/Users/seb/Pictures/pictures/.encfs5/": >> mode => 0600, >> owner => "seb", >> group => "staff", >> recurse => true, >> } >> file { >> "/Users/seb/Pictures/pictures/.git/": >> mode => 0600, >> owner => "seb", >> group => "staff", >> recurse => true, >> } >> file { >> "/Users/seb/Pictures/pictures/.git/hooks/": >> mode => 0700, >> owner => "seb", >> group => "staff", >> recurse => true, >> } >> } >> >> And a call to "sudo puppet -l /tmp/puppet.log ~/statement.pp" >> >> Well, that was 4 hours ago and since then, ruby is eating 100% of my CPU >> (of one core of my 2.26GHz core 2 duo). >> From the log file, I can tell that puppet is indeed fixing perms, at a >> rate lower than one file per 10 seconds. >> >> I think find, xargs and chmod would take a few minutes at most (will try >> later). >> >> Why is puppet so slow at this job? Is there any way I could improve the >> speed? > > Puppet is md5 checksumming all your files and that is a loooong and > slooow operation. > > If you run 2.6, you should add: > checksum => none > > to your file resources, and it should be way faster.I''m finding that with my version of puppet (2.6.4), the checksum line has no effect on the run time when run on a directory containing 10 files that total 7.5GB. I am not using the source parameter and it takes just over 2 seconds to run with "checksum => none" or "checksum => md5". My test computer is a Core 2 Duo running on a laptop. I am not using a puppetmaster to test. I''m still interested to see if adding that line helps though. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Patrick
2011-Jan-05 21:34 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
On Jan 5, 2011, at 10:13 AM, Brice Figureau wrote:> On 05/01/11 18:11, Sébastien Barthélémy wrote: >> Hello, >> >> I store camera pictures in a git repository, which became quite big: >> 104 GB for the whole (non-bare) repository. >> >> I wanted to fix the files permissions, and thought puppet might be the >> good tool for this (I like its declarative way of simplifying my >> life). >> >> I gave it a try, with the following statements >> >> node "navi" { >> file { >> "/Users/seb/Pictures/pictures/": >> mode => 0640, >> owner => "seb", >> group => "staff", >> recurse => true, >> ignore => ".git" >> } >> file { >> "/Users/seb/Pictures/pictures/.encfs5/": >> mode => 0600, >> owner => "seb", >> group => "staff", >> recurse => true, >> } >> file { >> "/Users/seb/Pictures/pictures/.git/": >> mode => 0600, >> owner => "seb", >> group => "staff", >> recurse => true, >> } >> file { >> "/Users/seb/Pictures/pictures/.git/hooks/": >> mode => 0700, >> owner => "seb", >> group => "staff", >> recurse => true, >> } >> } >> >> And a call to "sudo puppet -l /tmp/puppet.log ~/statement.pp" >> >> Well, that was 4 hours ago and since then, ruby is eating 100% of my CPU >> (of one core of my 2.26GHz core 2 duo). >> From the log file, I can tell that puppet is indeed fixing perms, at a >> rate lower than one file per 10 seconds. >> >> I think find, xargs and chmod would take a few minutes at most (will try >> later). >> >> Why is puppet so slow at this job? Is there any way I could improve the >> speed? > > Puppet is md5 checksumming all your files and that is a loooong and > slooow operation. > > If you run 2.6, you should add: > checksum => none > > to your file resources, and it should be way faster.I''m finding that with my version of puppet (2.6.4), the checksum line has no effect on the run time when run on a directory containing 10 files that total 7.5GB. I am not using the source parameter and it takes just over 2 seconds to run with "checksum => none" or "checksum => md5". My test computer is a Core 2 Duo running on a laptop. I am not using a puppetmaster to test. I''m still interested to see if adding that line helps though. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2011-Jan-06 12:45 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
On Wed, 2011-01-05 at 13:25 -0800, Patrick wrote:> > On Jan 5, 2011, at 10:13 AM, Brice Figureau wrote: > > > On 05/01/11 18:11, Sébastien Barthélémy wrote: > > > Hello, > > > > > > I store camera pictures in a git repository, which became quite > > > big: > > > 104 GB for the whole (non-bare) repository. > > > > > > I wanted to fix the files permissions, and thought puppet might be > > > the > > > good tool for this (I like its declarative way of simplifying my > > > life). > > > > > > I gave it a try, with the following statements > > > > > > node "navi" { > > > file { > > > "/Users/seb/Pictures/pictures/": > > > mode => 0640, > > > owner => "seb", > > > group => "staff", > > > recurse => true, > > > ignore => ".git" > > > } > > > file { > > > "/Users/seb/Pictures/pictures/.encfs5/": > > > mode => 0600, > > > owner => "seb", > > > group => "staff", > > > recurse => true, > > > } > > > file { > > > "/Users/seb/Pictures/pictures/.git/": > > > mode => 0600, > > > owner => "seb", > > > group => "staff", > > > recurse => true, > > > } > > > file { > > > "/Users/seb/Pictures/pictures/.git/hooks/": > > > mode => 0700, > > > owner => "seb", > > > group => "staff", > > > recurse => true, > > > } > > > } > > > > > > And a call to "sudo puppet -l /tmp/puppet.log ~/statement.pp" > > > > > > Well, that was 4 hours ago and since then, ruby is eating 100% of > > > my CPU > > > (of one core of my 2.26GHz core 2 duo). > > > From the log file, I can tell that puppet is indeed fixing perms, > > > at a > > > rate lower than one file per 10 seconds. > > > > > > I think find, xargs and chmod would take a few minutes at most > > > (will try > > > later). > > > > > > Why is puppet so slow at this job? Is there any way I could > > > improve the > > > speed? > > > > Puppet is md5 checksumming all your files and that is a loooong and > > slooow operation. > > > > If you run 2.6, you should add: > > checksum => none > > > > to your file resources, and it should be way faster. > > > > > > > I''m finding that with my version of puppet (2.6.4), the checksum line > has no effect on the run time when run on a directory containing 10 > files that total 7.5GB. I am not using the source parameter and it > takes just over 2 seconds to run with "checksum => none" or "checksum > => md5". My test computer is a Core 2 Duo running on a laptop. I am > not using a puppetmaster to test.Then puppet is not checksumming your files (or it would take longer than 2s). It''s possible that now we default to checksum none when there is no source or content.> I''m still interested to see if adding that line helps though.Clearly no. Do you think 2s is too long? -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Patrick
2011-Jan-06 15:05 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
On Jan 6, 2011, at 4:45 AM, Brice Figureau wrote:> On Wed, 2011-01-05 at 13:25 -0800, Patrick wrote: >> >> On Jan 5, 2011, at 10:13 AM, Brice Figureau wrote: >> >>> On 05/01/11 18:11, Sébastien Barthélémy wrote: >>>> >>>> Why is puppet so slow at this job? Is there any way I could >>>> improve the >>>> speed? >>> >>> Puppet is md5 checksumming all your files and that is a loooong and >>> slooow operation. >>> >>> If you run 2.6, you should add: >>> checksum => none >>> >>> to your file resources, and it should be way faster. >>> >> >> >> >> >> I''m finding that with my version of puppet (2.6.4), the checksum line >> has no effect on the run time when run on a directory containing 10 >> files that total 7.5GB. I am not using the source parameter and it >> takes just over 2 seconds to run with "checksum => none" or "checksum >> => md5". My test computer is a Core 2 Duo running on a laptop. I am >> not using a puppetmaster to test. > > Then puppet is not checksumming your files (or it would take longer than > 2s). It''s possible that now we default to checksum none when there is no > source or content. > >> I''m still interested to see if adding that line helps though. > > Clearly no. Do you think 2s is too long?That wasn''t what I meant. I was wondering if somehow it was defaulting to on with all those photos the original poster had. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Mark Stanislav
2011-Jan-06 15:12 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
>> Clearly no. Do you think 2s is too long? > > > That wasn''t what I meant. I was wondering if somehow it was defaulting to on with all those photos the original poster had.Perhaps they have a site.pp default? Wouldn''t be the first time someone ran into that. -Mark> > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2011-Jan-06 15:34 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
On Thu, 2011-01-06 at 10:12 -0500, Mark Stanislav wrote:> >> Clearly no. Do you think 2s is too long? > > > > > > That wasn''t what I meant. I was wondering if somehow it was > defaulting to on with all those photos the original poster had. > > Perhaps they have a site.pp default? Wouldn''t be the first time > someone ran into that.Or the OP is using puppet 0.25 or 0.24 which didn''t support the none checksum. -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Sébastien Barthélémy
2011-Jan-07 14:33 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
Hello again, thank you for your answers. I use puppet 2.6.4. I don''t think I have a site.pp On Wed, 5 Jan 2011, Patrick wrote:> I''m finding that with my version of puppet (2.6.4), the checksum linehas no effect on the run time when run on a directory containing 10 files that total 7.5GB. I am not> using the source parameter and it takes just over 2 seconds to run with"checksum => none" or "checksum => md5". My test computer is a Core 2 Duo running on a laptop. I am> not using a puppetmaster to test. > I''m still interested to see if adding that line helps though.I think the line does not help. Here are subsequent calls to puppet. The 3 first runs are with the line, the last line without. The first call is really long because it had work to do (perms to fix), the subsequents were no-ops. $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp 38779.74 real 38230.77 user 153.85 sys $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp 388.47 real 380.06 user 4.00 sys $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp 398.06 real 390.21 user 4.08 sys $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp 385.04 real 376.70 user 4.00 sys Cheers -- Sébastien -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2011-Jan-07 14:51 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
On Fri, 2011-01-07 at 15:33 +0100, Sébastien Barthélémy wrote:> Hello again, > > thank you for your answers. I use puppet 2.6.4. > I don''t think I have a site.pp > > On Wed, 5 Jan 2011, Patrick wrote: > > I''m finding that with my version of puppet (2.6.4), the checksum line > has no effect on the run time when run on a directory containing 10 files > that total 7.5GB. I am not > > using the source parameter and it takes just over 2 seconds to run with > "checksum => none" or "checksum => md5". My test computer is a Core 2 Duo > running on a laptop. I am > > not using a puppetmaster to test. > > I''m still interested to see if adding that line helps though. > > I think the line does not help. > > Here are subsequent calls to puppet. > The 3 first runs are with the line, the last line without. > > The first call is really long because it had work to do > (perms to fix), the subsequents were no-ops. > > $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp > 38779.74 real 38230.77 user 153.85 sys > $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp > 388.47 real 380.06 user 4.00 sys > $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp > 398.06 real 390.21 user 4.08 sys > $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp > 385.04 real 376.70 user 4.00 sysIt would be interesting to try with 0.25.5 (and still use checksum => none), to compare times and see if we have a regression or not. How many files do you have in total? And if you run with --debug, is there a part of the log where it looks like it is spending most of its time? We used to have some problems with recursive file resources back in the 0.24/0.25 days that we fixed in the latest 0.25 releases. It is possible some of them re-surfaced in 2.6... -- Brice Figureau Follow the latest Puppet Community evolutions on www.planetpuppet.org! -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jeff McCune
2011-Jan-07 19:44 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
-Jeff On Jan 7, 2011, at 6:51 AM, Brice Figureau <brice-puppet@daysofwonder.com> wrote:> On Fri, 2011-01-07 at 15:33 +0100, Sébastien Barthélémy wrote: >> Hello again, >> >> thank you for your answers. I use puppet 2.6.4. >> I don''t think I have a site.pp >> >> On Wed, 5 Jan 2011, Patrick wrote: >>> I''m finding that with my version of puppet (2.6.4), the checksum line >> has no effect on the run time when run on a directory containing 10 files >> that total 7.5GB. I am not >>> using the source parameter and it takes just over 2 seconds to run with >> "checksum => none" or "checksum => md5". My test computer is a Core 2 Duo >> running on a laptop. I am >>> not using a puppetmaster to test. >>> I''m still interested to see if adding that line helps though. >> >> I think the line does not help. >> >> Here are subsequent calls to puppet. >> The 3 first runs are with the line, the last line without. >> >> The first call is really long because it had work to do >> (perms to fix), the subsequents were no-ops. >> >> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >> 38779.74 real 38230.77 user 153.85 sys >> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >> 388.47 real 380.06 user 4.00 sys >> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >> 398.06 real 390.21 user 4.08 sys >> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >> 385.04 real 376.70 user 4.00 sys > > It would be interesting to try with 0.25.5 (and still use checksum => > none), to compare times and see if we have a regression or not.I think checksumming is a red herring.> How many files do you have in total?This is the primary issue think. Puppet models all of the files and Directories as resources and adds dependency relationships among them. It then sorts the graph. I suspect the sorting of the resource graph is your performance issue here. -Jeff> > And if you run with --debug, is there a part of the log where it looks > like it is spending most of its time? > > We used to have some problems with recursive file resources back in the > 0.24/0.25 days that we fixed in the latest 0.25 releases. It is possible > some of them re-surfaced in 2.6... > -- > Brice Figureau > Follow the latest Puppet Community evolutions on www.planetpuppet.org! > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Patrick
2011-Jan-07 21:29 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
On Jan 7, 2011, at 6:33 AM, Sébastien Barthélémy wrote:> Hello again, > > thank you for your answers. I use puppet 2.6.4. > I don''t think I have a site.pp > > On Wed, 5 Jan 2011, Patrick wrote: >> I''m finding that with my version of puppet (2.6.4), the checksum line > has no effect on the run time when run on a directory containing 10 files that total 7.5GB. I am not >> using the source parameter and it takes just over 2 seconds to run with > "checksum => none" or "checksum => md5". My test computer is a Core 2 Duo running on a laptop. I am >> not using a puppetmaster to test. >> I''m still interested to see if adding that line helps though. > > I think the line does not help. > > Here are subsequent calls to puppet. > The 3 first runs are with the line, the last line without. > > The first call is really long because it had work to do > (perms to fix), the subsequents were no-ops. > > $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp > 38779.74 real 38230.77 user 153.85 sys > $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp > 388.47 real 380.06 user 4.00 sys > $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp > 398.06 real 390.21 user 4.08 sys > $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp > 385.04 real 376.70 user 4.00 sysIn that case, what filesystem are you using? What OS? How many files and folders? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2011-Jan-07 22:33 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
On 07/01/11 20:44, Jeff McCune wrote:> -Jeff > > On Jan 7, 2011, at 6:51 AM, Brice Figureau > <brice-puppet@daysofwonder.com> wrote: > >> On Fri, 2011-01-07 at 15:33 +0100, Sébastien Barthélémy wrote: >>> Hello again, >>> >>> thank you for your answers. I use puppet 2.6.4. >>> I don''t think I have a site.pp >>> >>> On Wed, 5 Jan 2011, Patrick wrote: >>>> I''m finding that with my version of puppet (2.6.4), the checksum line >>> has no effect on the run time when run on a directory containing 10 files >>> that total 7.5GB. I am not >>>> using the source parameter and it takes just over 2 seconds to run with >>> "checksum => none" or "checksum => md5". My test computer is a Core 2 Duo >>> running on a laptop. I am >>>> not using a puppetmaster to test. >>>> I''m still interested to see if adding that line helps though. >>> >>> I think the line does not help. >>> >>> Here are subsequent calls to puppet. >>> The 3 first runs are with the line, the last line without. >>> >>> The first call is really long because it had work to do >>> (perms to fix), the subsequents were no-ops. >>> >>> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >>> 38779.74 real 38230.77 user 153.85 sys >>> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >>> 388.47 real 380.06 user 4.00 sys >>> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >>> 398.06 real 390.21 user 4.08 sys >>> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >>> 385.04 real 376.70 user 4.00 sys >> >> It would be interesting to try with 0.25.5 (and still use checksum => >> none), to compare times and see if we have a regression or not. > > I think checksumming is a red herring. > >> How many files do you have in total? > > This is the primary issue think. Puppet models all of the files and > Directories as resources and adds dependency relationships among them. > It then sorts the graph. I suspect the sorting of the resource graph > is your performance issue here.To my knowledge the sub-child resources are spawned during the transaction evaluation so even though they''re part of the graph they''re never really sorted (the sort phase happens before). But you''re right the graph becomes deep and large, which consumes memory. What might happen is what happened pre 0.25.5: too many events are generated (at least one for every file change) and propagated to all the other nodes (ie a n square problem). -- Brice Figureau My Blog: http://www.masterzen.fr/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Sébastien Barthélémy
2011-Jan-08 15:46 UTC
Re: [Puppet Users] Using puppet to fix a lot of files permissions
On Fri, 7 Jan 2011, Patrick wrote:> On Jan 7, 2011, at 6:33 AM, Sébastien Barthélémy wrote: >> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >> 38779.74 real 38230.77 user 153.85 sys >> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >> 388.47 real 380.06 user 4.00 sys >> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >> 398.06 real 390.21 user 4.08 sys >> $ sudo time puppet -l /tmp/puppet.log ~/puppet/respeer.pp >> 385.04 real 376.70 user 4.00 sys > > In that case, what filesystem are you using?HFS> What OS?Mac OS 10.6.6> How many files and folders?$ find ./ -type f | wc -l 27721 $ find ./ -type d | wc -l 1316 -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.