John Cesario
2010-Jul-15 17:27 UTC
[Puppet Users] puppet doing full read on files with replace => no
Hey all,
This is on puppet 0.25.4:
The manifest for one of the directories looks like this. This class/object
does not have anything below it (nothing inherits this).
file {
"/var/lib/data/$name":
ensure => directory,
owner => "data",
group => "data",
recurse => true,
mode => 750,
ignore => ".svn",
replace => false,
checksum => undef,
backup => false,
source => "puppet:///app/app_data/$seed",
require => File["/var/lib/app_data"]
}
We use this to initialize an applications data directory, and then dump
~500GB of data into it.
The problem is that on subsequent puppet runs when the directory is
populated, strace still shows puppet doing a full read of all the files in
there.
Any way to stop this besides doing recurselimit => 0. I would like the
permissions to be managed, and obviously with recurse => 0 shipping the seed
files over there becomes difficult.
Thanks,
-JC
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Patrick Mohr
2010-Jul-15 18:37 UTC
Re: [Puppet Users] puppet doing full read on files with replace => no
On Jul 15, 2010, at 10:27 AM, John Cesario wrote:> Hey all, > > This is on puppet 0.25.4: > > The manifest for one of the directories looks like this. This class/object does not have anything below it (nothing inherits this). > > file { > "/var/lib/data/$name": > ensure => directory, > owner => "data", > group => "data", > recurse => true, > mode => 750, > ignore => ".svn", > replace => false, > checksum => undef, > backup => false, > source => "puppet:///app/app_data/$seed", > require => File["/var/lib/app_data"] > } > > > We use this to initialize an applications data directory, and then dump ~500GB of data into it. > > The problem is that on subsequent puppet runs when the directory is populated, strace still shows puppet doing a full read of all the files in there. > > Any way to stop this besides doing recurselimit => 0. I would like the permissions to be managed, and obviously with recurse => 0 shipping the seed files over there becomes difficult.Frankly, I would use something other than puppet for big files. I''d suggest rsync, nfs, deb packages, or wget+tar. If you still want to do it: *) Is puppet still doing this read on the client? *) Is puppet still doing this read on the server? *) Are you pretty sure it''s a full read and not just puppet running stat on each file to make sure the file(s) are there? -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
John Cesario
2010-Jul-15 18:49 UTC
Re: [Puppet Users] puppet doing full read on files with replace => no
I dont want to ship big files through puppet. I only want to manage them. We
seed the app data and then it just naturally grows to 500GB+
What puppet ships through (souce =>) is a dataseed which is at most a few
MB.
If you still want to do it:
*) Is puppet still doing this read on the client? yes
*) Is puppet still doing this read on the server? I have not checked, but
the files on the server are tiny.
*) Are you pretty sure it''s a full read and not just puppet running
stat on
each file to make sure the file(s) are there? Yes, positive its a full read,
the lstats/fstats return quickly. Puppetd starts calling read() on some of
the files under the aforementioned resource.
open("/var/lib/app_data/app_instance/app_id/app_file.foo", O_RDONLY) =
6
fstat(6, {st_mode=S_IFREG|0750, st_size=1048576, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
0x2b13991b6000
read(6,
"O`:\370\0\0\0\0\377\377\377\377\377\377\377\377\0\0\0/\320l{uE\277\0\0\0\0\0\0"...,
4096) = 4096
read(6,
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
4096)
= 4096
read(6,
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
4096)
= 4096
read(6,
"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"...,
4096)
= 4096
On Thu, Jul 15, 2010 at 11:37 AM, Patrick Mohr <kc7zzv@gmail.com> wrote:
>
> On Jul 15, 2010, at 10:27 AM, John Cesario wrote:
>
> > Hey all,
> >
> > This is on puppet 0.25.4:
> >
> > The manifest for one of the directories looks like this. This
> class/object does not have anything below it (nothing inherits this).
> >
> > file {
> > "/var/lib/data/$name":
> > ensure => directory,
> > owner => "data",
> > group => "data",
> > recurse => true,
> > mode => 750,
> > ignore => ".svn",
> > replace => false,
> > checksum => undef,
> > backup => false,
> > source => "puppet:///app/app_data/$seed",
> > require => File["/var/lib/app_data"]
> > }
> >
> >
> > We use this to initialize an applications data directory, and then
dump
> ~500GB of data into it.
> >
> > The problem is that on subsequent puppet runs when the directory is
> populated, strace still shows puppet doing a full read of all the files in
> there.
> >
> > Any way to stop this besides doing recurselimit => 0. I would like
the
> permissions to be managed, and obviously with recurse => 0 shipping the
seed
> files over there becomes difficult.
>
> Frankly, I would use something other than puppet for big files.
I''d
> suggest rsync, nfs, deb packages, or wget+tar.
>
> If you still want to do it:
> *) Is puppet still doing this read on the client?
> *) Is puppet still doing this read on the server?
> *) Are you pretty sure it''s a full read and not just puppet
running stat on
> each file to make sure the file(s) are there?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
>
puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Peter Meier
2010-Jul-15 19:38 UTC
Re: [Puppet Users] puppet doing full read on files with replace => no
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/15/2010 08:49 PM, John Cesario wrote:> I dont want to ship big files through puppet. I only want to manage them. We > seed the app data and then it just naturally grows to 500GB+ > > What puppet ships through (souce =>) is a dataseed which is at most a few > MB.There was in improvement regarding that in 0.25.5 see http://projects.puppetlabs.com/issues/3533 for more information. cheers pete -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkw/Y5YACgkQbwltcAfKi39egACgoi364dRTHnr5TfqFlZmdILwq HysAn3X6P+18A5FIkgIToc6nABCfz9jE =NNCN -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
John Cesario
2010-Jul-15 19:43 UTC
Re: [Puppet Users] puppet doing full read on files with replace => no
Will give that a shot tonight. Thanks for your help Pete, -JC On Thu, Jul 15, 2010 at 12:38 PM, Peter Meier <peter.meier@immerda.ch>wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 07/15/2010 08:49 PM, John Cesario wrote: > > I dont want to ship big files through puppet. I only want to manage them. > We > > seed the app data and then it just naturally grows to 500GB+ > > > > What puppet ships through (souce =>) is a dataseed which is at most a > few > > MB. > > There was in improvement regarding that in 0.25.5 see > http://projects.puppetlabs.com/issues/3533 for more information. > > cheers pete > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.10 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ > > iEYEARECAAYFAkw/Y5YACgkQbwltcAfKi39egACgoi364dRTHnr5TfqFlZmdILwq > HysAn3X6P+18A5FIkgIToc6nABCfz9jE > =NNCN > -----END PGP SIGNATURE----- > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2010-Jul-15 21:50 UTC
Re: [Puppet Users] puppet doing full read on files with replace => no
On 15/07/10 21:43, John Cesario wrote:> Will give that a shot tonight.I''m not sure it will help. The issue here is that puppet is checksumming your files to compare them with the source (even though you said: replace => no). BTW, in 0.25.5 we implemented checksum => none, which makes sure Puppet doesn''t checksum files. I''m not sure it will work correctly with replace => no, so YMMV, but it is worth checking. -- Brice Figureau My Blog: http://www.masterzen.fr/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
John Cesario
2010-Jul-16 16:06 UTC
Re: [Puppet Users] puppet doing full read on files with replace => no
Could someone help me throw some puts in the right place to see if it is being checksummed? I attempted this myself but Im unsure I got the right spot. Thanks, -JC On Thu, Jul 15, 2010 at 2:50 PM, Brice Figureau < brice-puppet@daysofwonder.com> wrote:> On 15/07/10 21:43, John Cesario wrote: > > Will give that a shot tonight. > > I''m not sure it will help. The issue here is that puppet is checksumming > your files to compare them with the source (even though you said: > replace => no). > > BTW, in 0.25.5 we implemented checksum => none, which makes sure Puppet > doesn''t checksum files. I''m not sure it will work correctly with replace > => no, so YMMV, but it is worth checking. > -- > Brice Figureau > My Blog: http://www.masterzen.fr/ > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com<puppet-users%2Bunsubscribe@googlegroups.com> > . > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en. > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Brice Figureau
2010-Jul-16 18:45 UTC
Re: [Puppet Users] puppet doing full read on files with replace => no
On 16/07/10 18:06, John Cesario wrote:> Could someone help me throw some puts in the right place to see if it is > being checksummed? I attempted this myself but Im unsure I got the right > spot.Have a look in puppet/util/checksums.rb Put your debug statements in the md5_file or checksum_file methods. Did using 0.25.5 change anything to your specific problem? -- Brice Figureau My Blog: http://www.masterzen.fr/ -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.