On 8/1/07, Pedro Sim?es <pedro.a.simoes at gmail.com> wrote:> > I have two Vmware images, with 256MB of RAM memory each one, running > puppetmasterd for the server and puppetd for the client. It''s normal > that my puppetmasterd and puppetd occupies almost 10% of RAM memory > each one? They aren''t doing anything especial, just listening on the > ports. >Puppet is a good sized Ruby application. Ruby code uses more memory than C code and the data also tends to be larger. One thing I have noticed that causes large memory consumption with puppet is serving large files. Puppet reads the whole file into memory, base64 encodes it, and then sends it as an XML-RPC message. This uses lots of RAM and CPU temporarily. Depending on your OS, the process may not give back the RAM to the OS. - Ian
Hello, I have two Vmware images, with 256MB of RAM memory each one, running puppetmasterd for the server and puppetd for the client. It''s normal that my puppetmasterd and puppetd occupies almost 10% of RAM memory each one? They aren''t doing anything especial, just listening on the ports. Thanks.
Pedro Simões wrote:> Hello, > > I have two Vmware images, with 256MB of RAM memory each one, running > puppetmasterd for the server and puppetd for the client. It''s normal > that my puppetmasterd and puppetd occupies almost 10% of RAM memory > each one? They aren''t doing anything especial, just listening on the > ports. > > Thanks.I think it''s normal. My puppet master is 34m resident, which seems great considering cfservd is 217m. "too much" is certainly subjective. Cheers, -- Jeff McCune The Ohio State University Department of Mathematics Systems Manager _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On 8/1/07 9:36 AM, "Jeff McCune" <mccune@math.ohio-state.edu> wrote: Pedro Simões wrote:> Hello, > > I have two Vmware images, with 256MB of RAM memory each one, running > puppetmasterd for the server and puppetd for the client. It''s normal > that my puppetmasterd and puppetd occupies almost 10% of RAM memory > each one? They aren''t doing anything especial, just listening on the > ports. > > Thanks.I think it''s normal. My puppet master is 34m resident, which seems great considering cfservd is 217m. "too much" is certainly subjective. Indeed... I''ve 4 mongrel instances and one webrick running each consuming about ~70MB of resident memory. And that is really light weight compared to a cfengine that I had in the past. Cheers, Ryan _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Aug 1, 2007, at 1:19 PM, Ian Burrell wrote:> One thing I have noticed that causes large memory consumption with > puppet is serving large files. Puppet reads the whole file into > memory, base64 encodes it, and then sends it as an XML-RPC message. > This uses lots of RAM and CPU temporarily. Depending on your OS, the > process may not give back the RAM to the OS.Note that this problem is the reason why we''re moving to REST (see the archives of the dev list), so hopefully we''ll be able to use straight http soon, meaning no encoding. -- ''Tis better to be silent and be thought a fool, than to speak and remove all doubt. --Abraham Lincoln --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Ian Burrell wrote:> On 8/1/07, Pedro Simões <pedro.a.simoes@gmail.com> wrote: >> I have two Vmware images, with 256MB of RAM memory each one, running >> puppetmasterd for the server and puppetd for the client. It''s normal >> that my puppetmasterd and puppetd occupies almost 10% of RAM memory >> each one? They aren''t doing anything especial, just listening on the >> ports. >> > > Puppet is a good sized Ruby application. Ruby code uses more memory > than C code and the data also tends to be larger. > > One thing I have noticed that causes large memory consumption with > puppet is serving large files. Puppet reads the whole file into > memory, base64 encodes it, and then sends it as an XML-RPC message. > This uses lots of RAM and CPU temporarily. Depending on your OS, the > process may not give back the RAM to the OS. >There''s no interface to mmap(2) in ruby? -- Russell A. Jackson <raj@csub.edu> Network Analyst California State University, Bakersfield He who findeth sensuous pleasures in the bodies of lush, hot, pink damsels is not righteous, but he can have a lot more fun. _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Aug 1, 2007, at 3:06 PM, Russell Jackson wrote:> There''s no interface to mmap(2) in ruby?Would that be something that could reduce the number of copies of a string in memory while escaping that string? Either way, it at least looks like if there is one and that''s what it''s for, it''s not being used in this case, because escaping appears to create 2 new copies of each string. It''s only being done because we''re unnecessarily using XML, so switching to REST should make it just go away, and it should additionally make it easy to just do your file serving out of Apache or something similar. -- The first time I see a jogger smiling, I''ll consider it. --Joan Rivers --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
Luke Kanies wrote:> On Aug 1, 2007, at 3:06 PM, Russell Jackson wrote: > >> There''s no interface to mmap(2) in ruby? > > Would that be something that could reduce the number of copies of a > string in memory while escaping that string? > > Either way, it at least looks like if there is one and that''s what > it''s for, it''s not being used in this case, because escaping appears > to create 2 new copies of each string. It''s only being done because > we''re unnecessarily using XML, so switching to REST should make it > just go away, and it should additionally make it easy to just do your > file serving out of Apache or something similar. >With mmap(), reads the memory mapped region are read off the disk/buffers instead of copying the entire file into memory. The pointer to the region should be able to be used anywhere a string could be used (in C). I''m sure the same should be true in Ruby (it would inane otherwise). Additionally, multiple mmap() calls to the same file share reference to the same memory region. Now, I''m not sure how to get around the problem of having to copy the entire string to escape it, but surely we would be getting rid of half of the copies by having the source file mmap()ed. I agree that doing the file serving via http would be ideal. -- Russell A. Jackson <raj@csub.edu> Network Analyst California State University, Bakersfield He who findeth sensuous pleasures in the bodies of lush, hot, pink damsels is not righteous, but he can have a lot more fun. _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
Russell Jackson wrote:> Luke Kanies wrote: >> On Aug 1, 2007, at 3:06 PM, Russell Jackson wrote: >> >>> There''s no interface to mmap(2) in ruby? >> Would that be something that could reduce the number of copies of a >> string in memory while escaping that string? >> >> Either way, it at least looks like if there is one and that''s what >> it''s for, it''s not being used in this case, because escaping appears >> to create 2 new copies of each string. It''s only being done because >> we''re unnecessarily using XML, so switching to REST should make it >> just go away, and it should additionally make it easy to just do your >> file serving out of Apache or something similar. >> > > With mmap(), reads the memory mapped region are read off the disk/buffers instead ofSorry, that should read: With mmap(), reads _to_ the memory mapped region...> copying the entire file into memory. The pointer to the region should be able to be used > anywhere a string could be used (in C). I''m sure the same should be true in Ruby (it would > inane otherwise). Additionally, multiple mmap() calls to the same file share reference to > the same memory region. > > Now, I''m not sure how to get around the problem of having to copy the entire string to > escape it, but surely we would be getting rid of half of the copies by having the source > file mmap()ed. > > I agree that doing the file serving via http would be ideal. >-- Russell A. Jackson <raj@csub.edu> Network Analyst California State University, Bakersfield He who findeth sensuous pleasures in the bodies of lush, hot, pink damsels is not righteous, but he can have a lot more fun. _______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Wed, Aug 01, 2007 at 03:56:26PM -0500, Luke Kanies wrote:> On Aug 1, 2007, at 3:06 PM, Russell Jackson wrote: > > > There''s no interface to mmap(2) in ruby? > > Would that be something that could reduce the number of copies of a > string in memory while escaping that string? > > Either way, it at least looks like if there is one and that''s what > it''s for, it''s not being used in this case, because escaping appears > to create 2 new copies of each string. It''s only being done because > we''re unnecessarily using XML, so switching to REST should make it > just go away, and it should additionally make it easy to just do your > file serving out of Apache or something similar.Do you really need to escape base64 strings for XML? Considering that the characters used in base64 are: "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/" I don''t think it needs any escaping. Am I missing something? Cheers, Kostas
On Aug 2, 2007, at 3:33 PM, Kostas Georgiou wrote:> Do you really need to escape base64 strings for XML? Considering that > the characters used in base64 are: > "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/" > I don''t think it needs any escaping. Am I missing something?It could be that I''m supposed to be CGI escaping them or XML escaping them or something; I''m pretty confident I need to do some kind of escaping, since (from what I remember) I only added this when I started having trouble sending the files down. This is all wrapped up in an XML envelope, so it can''t mess with that envelope that much. -- Get forgiveness now -- tomorrow you may no longer feel guilty. --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com
On 8/2/07, Kostas Georgiou <k.georgiou@imperial.ac.uk> wrote:> > Do you really need to escape base64 strings for XML? Considering that > the characters used in base64 are: > "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/" > I don''t think it needs any escaping. Am I missing something? >The base64 strings probably don''t need to be XML escaped since they can''t contain dangerous characters. However, the XML-RPC library probably does not have a way to say that a string is safe and does not need to be XML escaped. It likely XML escapes all the strings. Stopping the XML escaping the probably will save some CPU time and maybe some space but the second copy likely comes from composing the XML message. Streaming XML-RPC is hard and tends to require making the entire response XML message before sending it back. - Ian
On Aug 2, 2007, at 5:00 PM, Ian Burrell wrote:> The base64 strings probably don''t need to be XML escaped since they > can''t contain dangerous characters. However, the XML-RPC library > probably does not have a way to say that a string is safe and does not > need to be XML escaped. It likely XML escapes all the strings. > Stopping the XML escaping the probably will save some CPU time and > maybe some space but the second copy likely comes from composing the > XML message. Streaming XML-RPC is hard and tends to require making > the entire response XML message before sending it back.If anyone would like to experiment with removing or changing the escaping mechanisms, I''m all for it. I''m not going to bother, at this point, because I''m replacing the need for that mechanism anyway. -- Neonle will continue to be rude, and will nretend that you had a small stroke which makes you unable to say or see the letter "n". Stunid nractical joke, if you ask me. Bunch of noon-heads, huh? -- Fred Barling, Humorscope --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com