This really bogs down the system for several minutes each run, using 100% cpu
and 50% of ram (this system has 3 gigs of memory!). What am I doing wrong
here?
file { "/etc/asterisk":
links => "ignore",
recurse => "true",
replace => "false",
source => "/home/global/asterisk"
}
Thanks.
On Aug 11, 2007, at 2:30 PM, Cameron wrote:> This really bogs down the system for several minutes each run, > using 100% cpu > and 50% of ram (this system has 3 gigs of memory!). What am I doing > wrong > here? > > file { "/etc/asterisk": > links => "ignore", > recurse => "true", > replace => "false", > source => "/home/global/asterisk" > }How big is that fileset you''re recursing over? You certainly shouldn''t be using 1.5gb of RAM, but Puppet does load information on all of the files into RAM during the run, so if you''ve got thousands of files in that filesystem, you''re probably never going to get the results you want. I haven''t thought about trying to dump each file from memory as soon as we''re done with it, which could possibly get you there. At this point, file recursion basically works for simple file operations, but if you''re doing hundreds or thousands of files, Puppet''s going to unfortunately be pretty slow until someone has the time to spend focusing on performance. -- I respect faith, but doubt is what gets you an education. -- Wilson Mizner --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Is there a better way to figure out if a change in the folder has occurred than to check every file? On Saturday 11 August 2007 12:29, Luke Kanies wrote:> On Aug 11, 2007, at 2:30 PM, Cameron wrote: > > This really bogs down the system for several minutes each run, > > using 100% cpu > > and 50% of ram (this system has 3 gigs of memory!). What am I doing > > wrong > > here? > > > > file { "/etc/asterisk": > > links => "ignore", > > recurse => "true", > > replace => "false", > > source => "/home/global/asterisk" > > } > > How big is that fileset you''re recursing over? > > You certainly shouldn''t be using 1.5gb of RAM, but Puppet does load > information on all of the files into RAM during the run, so if you''ve > got thousands of files in that filesystem, you''re probably never > going to get the results you want. > > I haven''t thought about trying to dump each file from memory as soon > as we''re done with it, which could possibly get you there. > > At this point, file recursion basically works for simple file > operations, but if you''re doing hundreds or thousands of files, > Puppet''s going to unfortunately be pretty slow until someone has the > time to spend focusing on performance. > > -- > I respect faith, but doubt is what gets you an education. > -- Wilson Mizner > --------------------------------------------------------------------- > Luke Kanies | http://reductivelabs.com | http://madstop.com > > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users
FWIW, had a similar problem with a client where we had 30K or so files we wanted to ensure where a particular owner/permission. Not very fancy hardware, but puppet consumed pretty much all the ram and cpu it could. Ended up using cfengine for that particular case (yeah, I know.. would have rather just had a cron script myself... such is life) .r'' On 8/11/07, Luke Kanies <luke@madstop.com> wrote:> > On Aug 11, 2007, at 2:30 PM, Cameron wrote: > > > This really bogs down the system for several minutes each run, > > using 100% cpu > > and 50% of ram (this system has 3 gigs of memory!). What am I doing > > wrong > > here? > > > > file { "/etc/asterisk": > > links => "ignore", > > recurse => "true", > > replace => "false", > > source => "/home/global/asterisk" > > } > > How big is that fileset you''re recursing over? > > You certainly shouldn''t be using 1.5gb of RAM, but Puppet does load > information on all of the files into RAM during the run, so if you''ve > got thousands of files in that filesystem, you''re probably never > going to get the results you want. > > I haven''t thought about trying to dump each file from memory as soon > as we''re done with it, which could possibly get you there. > > At this point, file recursion basically works for simple file > operations, but if you''re doing hundreds or thousands of files, > Puppet''s going to unfortunately be pretty slow until someone has the > time to spend focusing on performance. > > -- > I respect faith, but doubt is what gets you an education. > -- Wilson Mizner > --------------------------------------------------------------------- > Luke Kanies | http://reductivelabs.com | http://madstop.com > > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >-- If you include a signature keep it short. Rule of thumb is no longer than 4 lines. -RFC 1855 _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I haven''t contributed anything yet to puppet, it just occurs to me that what this discussion is about is afaik handled best by the rsync tool. Now, I''m not advising to use rsync for that case, but when it comes to implementation maybe someone can take a look at rsync how they implement things (because obviously on can sync gigabyts of data without consuming all the memory on machine). HTH RijilV wrote:> FWIW, had a similar problem with a client where we had 30K or so files > we wanted to ensure where a particular owner/permission. Not very fancy > hardware, but puppet consumed pretty much all the ram and cpu it could. > Ended up using cfengine for that particular case (yeah, I know.. would > have rather just had a cron script myself... such is life) > > .r'' > > On 8/11/07, *Luke Kanies* <luke@madstop.com <mailto:luke@madstop.com>> > wrote: > > On Aug 11, 2007, at 2:30 PM, Cameron wrote: > > > This really bogs down the system for several minutes each run, > > using 100% cpu > > and 50% of ram (this system has 3 gigs of memory!). What am I doing > > wrong > > here? > > > > file { "/etc/asterisk": > > links => "ignore", > > recurse => "true", > > replace => "false", > > source => "/home/global/asterisk" > > } > > How big is that fileset you''re recursing over? > > You certainly shouldn''t be using 1.5gb of RAM, but Puppet does load > information on all of the files into RAM during the run, so if you''ve > got thousands of files in that filesystem, you''re probably never > going to get the results you want. > > I haven''t thought about trying to dump each file from memory as soon > as we''re done with it, which could possibly get you there. > > At this point, file recursion basically works for simple file > operations, but if you''re doing hundreds or thousands of files, > Puppet''s going to unfortunately be pretty slow until someone has the > time to spend focusing on performance. > > -- > I respect faith, but doubt is what gets you an education. > -- Wilson Mizner > --------------------------------------------------------------------- > Luke Kanies | http://reductivelabs.com | http://madstop.com > > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com <mailto:Puppet-users@madstop.com> > https://mail.madstop.com/mailman/listinfo/puppet-users > > > > > -- > If you include a signature keep it short. Rule of thumb is no longer > than 4 lines. > -RFC 1855 > > > ------------------------------------------------------------------------ > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFGvj4i1nS0RcInK9ARAv9ZAKDUKhunO0D/Tcr72gpZ2SHDzjZvtwCg1pqF kccMTH4XW93zvehXrp2CYiY=HE36 -----END PGP SIGNATURE-----
I am using rsync to copy the files, but puppet to check if the files have changed. I''m starting to think I''ll do a cron at this point myself. Thanks for the help so far, everyone. -Cameron On Sun, 12 Aug 2007, Markus Fischer wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi, > > I haven''t contributed anything yet to puppet, it just occurs to me that > what this discussion is about is afaik handled best by the rsync tool. > Now, I''m not advising to use rsync for that case, but when it comes to > implementation maybe someone can take a look at rsync how they implement > things (because obviously on can sync gigabyts of data without consuming > all the memory on machine). > > HTH > > RijilV wrote: >> FWIW, had a similar problem with a client where we had 30K or so files >> we wanted to ensure where a particular owner/permission. Not very fancy >> hardware, but puppet consumed pretty much all the ram and cpu it could. >> Ended up using cfengine for that particular case (yeah, I know.. would >> have rather just had a cron script myself... such is life) >> >> .r'' >> >> On 8/11/07, *Luke Kanies* <luke@madstop.com <mailto:luke@madstop.com>> >> wrote: >> >> On Aug 11, 2007, at 2:30 PM, Cameron wrote: >> >> > This really bogs down the system for several minutes each run, >> > using 100% cpu >> > and 50% of ram (this system has 3 gigs of memory!). What am I doing >> > wrong >> > here? >> > >> > file { "/etc/asterisk": >> > links => "ignore", >> > recurse => "true", >> > replace => "false", >> > source => "/home/global/asterisk" >> > } >> >> How big is that fileset you''re recursing over? >> >> You certainly shouldn''t be using 1.5gb of RAM, but Puppet does load >> information on all of the files into RAM during the run, so if you''ve >> got thousands of files in that filesystem, you''re probably never >> going to get the results you want. >> >> I haven''t thought about trying to dump each file from memory as soon >> as we''re done with it, which could possibly get you there. >> >> At this point, file recursion basically works for simple file >> operations, but if you''re doing hundreds or thousands of files, >> Puppet''s going to unfortunately be pretty slow until someone has the >> time to spend focusing on performance. >> >> -- >> I respect faith, but doubt is what gets you an education. >> -- Wilson Mizner >> --------------------------------------------------------------------- >> Luke Kanies | http://reductivelabs.com | http://madstop.com >> >> >> _______________________________________________ >> Puppet-users mailing list >> Puppet-users@madstop.com <mailto:Puppet-users@madstop.com> >> https://mail.madstop.com/mailman/listinfo/puppet-users >> >> >> >> >> -- >> If you include a signature keep it short. Rule of thumb is no longer >> than 4 lines. >> -RFC 1855 >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Puppet-users mailing list >> Puppet-users@madstop.com >> https://mail.madstop.com/mailman/listinfo/puppet-users > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.6 (MingW32) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iD8DBQFGvj4i1nS0RcInK9ARAv9ZAKDUKhunO0D/Tcr72gpZ2SHDzjZvtwCg1pqF > kccMTH4XW93zvehXrp2CYiY> =HE36 > -----END PGP SIGNATURE----- > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >
On Aug 11, 2007, at 8:12 PM, Cameron wrote:> I am using rsync to copy the files, but puppet to check if the > files have > changed. > > I''m starting to think I''ll do a cron at this point myself.Just so it''s clear, we are working on this, and it looks like it''s important enough to a couple of organizations that they''re starting to sponsor work on it, so hopefully this will be much better soon. We haven''t looked at the ram consumption yet, but that will be in the list. -- To get back my youth I would do anything in the world, except take exercise, get up early, or be respectable. -- Oscar Wilde --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Cameron wrote:> I am using rsync to copy the files, but puppet to check if the files have > changed. > > I''m starting to think I''ll do a cron at this point myself.If it fits your workflow, you might consider what I do with the template root filetree for my xen machines. I have an exec that rsyncs the filetree, subscribed to a "log" file (which puppet syncs out to the machines). I edit the logfile (on the puppetmaster) by hand when I make changes to the filetree (putting in a small comment about what changed). The logfile then gets pushed out, triggering the exec, which rsyncs out the actual changes to the filetree. It''s a bit convoluted and may only be viable because I''m not changing that filesystem very often (and because typically I am the only one changing it), but it works ok for me. Jordan
This won''t work in my situation I''m afraid. Is it possible to run some command? (like perhaps a quiet diff) and if the command shows that there are differences, execute rsync? Clearly I could just do this with a shell script (and that''s what I''ll do if I have to), but I''d like to keep as much "puppetized" as possible at this point. -Cameron On Sat, 11 Aug 2007, Jordan Share wrote:> Cameron wrote: >> I am using rsync to copy the files, but puppet to check if the files have >> changed. >> >> I''m starting to think I''ll do a cron at this point myself. > > If it fits your workflow, you might consider what I do with the template > root filetree for my xen machines. > > I have an exec that rsyncs the filetree, subscribed to a "log" file > (which puppet syncs out to the machines). I edit the logfile (on the > puppetmaster) by hand when I make changes to the filetree (putting in a > small comment about what changed). > > The logfile then gets pushed out, triggering the exec, which rsyncs out > the actual changes to the filetree. > > It''s a bit convoluted and may only be viable because I''m not changing > that filesystem very often (and because typically I am the only one > changing it), but it works ok for me. > > Jordan > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >
On Sat, Aug 11, 2007 at 11:15:42PM -0700, Cameron wrote:> This won''t work in my situation I''m afraid. > > Is it possible to run some command? (like perhaps a quiet diff) and if the > command shows that there are differences, execute rsync? Clearly I could > just do this with a shell script (and that''s what I''ll do if I have to), > but I''d like to keep as much "puppetized" as possible at this point.Why not just call rsync every time? Practically speaking, any sort of "remote diff" command isn''t going to be any faster than an rsync. If you want to trigger other resources off your rsync call, then you''ll need to wrap it up in an rsync type and use --stats -n to detect whether you need to do the transfer (examine the "Number of files transferred" line) then run without -n to actually do the transfer. Note that rsync''s idea of permission management is very, very different to Puppet''s, so you''ll have a world of fun making that part of the whole dance work. - Matt -- English is about as pure as a cribhouse whore. We don''t just borrow words; on occasion, English has pursued other languages down alleyways to beat them unconscious and rifle their pockets for new vocabulary." -- James D. Nicoll, resident of rec.arts.sf.written
That might work. Does the return value of rsync depeng on whether any files were copied? Also, we''re doing this over NFS currently so the remote thing isn''t too much of an issue. NFS is deprecated right now, but it will be with us for a while at least. -Cameron On Sun, 12 Aug 2007, Matthew Palmer wrote:> On Sat, Aug 11, 2007 at 11:15:42PM -0700, Cameron wrote: >> This won''t work in my situation I''m afraid. >> >> Is it possible to run some command? (like perhaps a quiet diff) and if the >> command shows that there are differences, execute rsync? Clearly I could >> just do this with a shell script (and that''s what I''ll do if I have to), >> but I''d like to keep as much "puppetized" as possible at this point. > > Why not just call rsync every time? Practically speaking, any sort of > "remote diff" command isn''t going to be any faster than an rsync. If you > want to trigger other resources off your rsync call, then you''ll need to > wrap it up in an rsync type and use --stats -n to detect whether you need to > do the transfer (examine the "Number of files transferred" line) then run > without -n to actually do the transfer. > > Note that rsync''s idea of permission management is very, very different to > Puppet''s, so you''ll have a world of fun making that part of the whole dance > work. > > - Matt > > -- > English is about as pure as a cribhouse whore. We don''t just borrow > words; on occasion, English has pursued other languages down alleyways > to beat them unconscious and rifle their pockets for new vocabulary." > -- James D. Nicoll, resident of rec.arts.sf.written > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >
On Sun, Aug 12, 2007 at 02:48:35PM -0700, Cameron wrote:> On Sun, 12 Aug 2007, Matthew Palmer wrote: > > On Sat, Aug 11, 2007 at 11:15:42PM -0700, Cameron wrote: > >> Is it possible to run some command? (like perhaps a quiet diff) and if the > >> command shows that there are differences, execute rsync? Clearly I could > >> just do this with a shell script (and that''s what I''ll do if I have to), > >> but I''d like to keep as much "puppetized" as possible at this point. > > > > Why not just call rsync every time? Practically speaking, any sort of > > "remote diff" command isn''t going to be any faster than an rsync. If you > > want to trigger other resources off your rsync call, then you''ll need to > > wrap it up in an rsync type and use --stats -n to detect whether you need to > > do the transfer (examine the "Number of files transferred" line) then run > > without -n to actually do the transfer. > > > > Note that rsync''s idea of permission management is very, very different to > > Puppet''s, so you''ll have a world of fun making that part of the whole dance > > work. > > That might work. Does the return value of rsync depeng on whether any > files were copied?No, that''s why I suggested parsing the output provided by --stats, which *does* tell you how many files were copied.> Also, we''re doing this over NFS currently so the remote thing isn''t too > much of an issue. NFS is deprecated right now, but it will be with us for > a while at least.So your file distribution is via local files over NFS rather than puppet://? That probably makes your Puppet runs a lot faster than mine already. <grin> - Matt
We had to switch to NFS because puppet was too slow and doesn''t copy over links. On Mon, 13 Aug 2007, Matthew Palmer wrote:> On Sun, Aug 12, 2007 at 02:48:35PM -0700, Cameron wrote: >> On Sun, 12 Aug 2007, Matthew Palmer wrote: >>> On Sat, Aug 11, 2007 at 11:15:42PM -0700, Cameron wrote: >>>> Is it possible to run some command? (like perhaps a quiet diff) and if the >>>> command shows that there are differences, execute rsync? Clearly I could >>>> just do this with a shell script (and that''s what I''ll do if I have to), >>>> but I''d like to keep as much "puppetized" as possible at this point. >>> >>> Why not just call rsync every time? Practically speaking, any sort of >>> "remote diff" command isn''t going to be any faster than an rsync. If you >>> want to trigger other resources off your rsync call, then you''ll need to >>> wrap it up in an rsync type and use --stats -n to detect whether you need to >>> do the transfer (examine the "Number of files transferred" line) then run >>> without -n to actually do the transfer. >>> >>> Note that rsync''s idea of permission management is very, very different to >>> Puppet''s, so you''ll have a world of fun making that part of the whole dance >>> work. >> >> That might work. Does the return value of rsync depeng on whether any >> files were copied? > > No, that''s why I suggested parsing the output provided by --stats, which > *does* tell you how many files were copied. > >> Also, we''re doing this over NFS currently so the remote thing isn''t too >> much of an issue. NFS is deprecated right now, but it will be with us for >> a while at least. > > So your file distribution is via local files over NFS rather than puppet://? > That probably makes your Puppet runs a lot faster than mine already. <grin> > > - Matt > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >
Cameron wrote:> We had to switch to NFS because puppet was too slow and doesn''t copy over > links. >FWIW, I''ve found subversion working copies to be suitable alternatives to rsync or puppet recursive file copying. Subversion minimizes the load on the server, and scalling http/https is already a solved problem. Once I have the local working copy updated, I copy files from it into place. Cheers, -- Jeff McCune Systems Manager The Ohio State University Department of Mathematics _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Sun, 12 Aug 2007, Markus Fischer wrote:> I haven''t contributed anything yet to puppet, it just occurs to me that > what this discussion is about is afaik handled best by the rsync tool. > Now, I''m not advising to use rsync for that case, but when it comes to > implementation maybe someone can take a look at rsync how they implement > things (because obviously on can sync gigabyts of data without consuming > all the memory on machine).rsync can use also use a lot of memory, from the FAQ ---- memory usage Yes, rsync uses a lot of memory. The majority of the memory is used to hold the list of files being transferred. This takes about 100 bytes per file, so if you are transferring 800,000 files then rsync will consume about 80M of memory. It will be higher if you use -H or --delete. To fix this requires a major rewrite of rsync, which my or may not happen. ---- which I guess is better than what puppet does natively but doesn''t scale forever. Perhaps optionally use a log file (created by whatever method) to hint which files may have changed. Even run "rsync -n" to spot the changed files and then carefully sync them. -- Simon J. Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ "To stay awake all night adds a day to your life" - Stilgar | eMT.
I switched to a shell script that runs rsync, and the problems no longer occur. -Cameron On Tue, 14 Aug 2007, Simon Lyall wrote:> On Sun, 12 Aug 2007, Markus Fischer wrote: >> I haven''t contributed anything yet to puppet, it just occurs to me that >> what this discussion is about is afaik handled best by the rsync tool. >> Now, I''m not advising to use rsync for that case, but when it comes to >> implementation maybe someone can take a look at rsync how they implement >> things (because obviously on can sync gigabyts of data without consuming >> all the memory on machine). > > rsync can use also use a lot of memory, from the FAQ > > ---- > memory usage > > Yes, rsync uses a lot of memory. The majority of the memory is used to > hold the list of files being transferred. This takes about 100 bytes per > file, so if you are transferring 800,000 files then rsync will consume > about 80M of memory. It will be higher if you use -H or --delete. > > To fix this requires a major rewrite of rsync, which my or may not happen. > ---- > > which I guess is better than what puppet does natively but doesn''t scale > forever. Perhaps optionally use a log file (created by whatever method) > to hint which files may have changed. > > Even run "rsync -n" to spot the changed files and then carefully sync them. > > -- > Simon J. Lyall | Very Busy | Web: http://www.darkmere.gen.nz/ > "To stay awake all night adds a day to your life" - Stilgar | eMT. > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users >