Hi,
We have dedicated puppet master boxes, which basically only run a
mongrel/puppetmasterd (6 instances), an apache proxy for mongrel and
puppetd.
Recently puppetmasterds seemed to have leaked almost all of the RAM, and
the thrashing made the box painfully slow.
We restarted the puppetmasterd processes, and started collecting some
statistics. Attached is a graph of memory usage on one of the boxes over
two weeks or so. Remarks:
* The restart happened on the 21st.
* "free" and "used" are in megabytes, and OS cache/buffers
are counted
as free.
* "Time" is the cumulative CPU usage of all puppetmasterd processes,
and is in minutes, but I was to lazy to add another axis, so it''s
just stuck on top of the memory usage. Anyway, it''s just there to
show that the load on the box generated by the clients was
relatively constant.
The question is: why is the memory usage rising so steeply during the
first 4 days, and then stays almost flat?
At first I thought this may be puppetmasterd caching some data about the
clients, and it takes time to propagate to all load-balanced
puppetmasterd processes. But we have about 15 minute run interval, so
this would happen much quicker than 4 days, I think.
--
Marcin Owsiany
Web Systems Integrator - Guardian Unlimited
------------------------------------------------------------------
Visit Guardian Unlimited - the UK''s most popular newspaper website
http://guardian.co.uk http://observer.co.uk
------------------------------------------------------------------
The Newspaper Marketing Agency
Opening Up Newspapers
http://www.nmauk.co.uk
------------------------------------------------------------------
This e-mail and all attachments are confidential and may also
be privileged. If you are not the named recipient, please notify
the sender and delete the e-mail and all attachments immediately.
Do not disclose the contents to another person. You may not use
the information for any purpose, or store, or copy, it in any way.
Guardian News & Media Limited is not liable for any computer
viruses or other material transmitted with or as part of this
e-mail. You should employ virus checking software.
Guardian News & Media Limited
A member of Guardian Media Group PLC
Registered Office
Number 1 Scott Place, Manchester M3 3GG
Registered in England Number 908396
_______________________________________________
Puppet-users mailing list
Puppet-users@madstop.com
https://mail.madstop.com/mailman/listinfo/puppet-users
its too hard to say from your stats, could it be that this include buffers/cached memory? how did you measure the memory consumption? Ohad On 9/7/07, Marcin Owsiany <marcin.owsiany@guardian.co.uk> wrote:> > Hi, > > We have dedicated puppet master boxes, which basically only run a > mongrel/puppetmasterd (6 instances), an apache proxy for mongrel and > puppetd. > > Recently puppetmasterds seemed to have leaked almost all of the RAM, and > the thrashing made the box painfully slow. > > We restarted the puppetmasterd processes, and started collecting some > statistics. Attached is a graph of memory usage on one of the boxes over > two weeks or so. Remarks: > * The restart happened on the 21st. > > * "free" and "used" are in megabytes, and OS cache/buffers are counted > as free. > > * "Time" is the cumulative CPU usage of all puppetmasterd processes, > and is in minutes, but I was to lazy to add another axis, so it''s > just stuck on top of the memory usage. Anyway, it''s just there to > show that the load on the box generated by the clients was > relatively constant. > > The question is: why is the memory usage rising so steeply during the > first 4 days, and then stays almost flat? > > At first I thought this may be puppetmasterd caching some data about the > clients, and it takes time to propagate to all load-balanced > puppetmasterd processes. But we have about 15 minute run interval, so > this would happen much quicker than 4 days, I think. > > -- > Marcin Owsiany > Web Systems Integrator - Guardian Unlimited > > ------------------------------------------------------------------ > Visit Guardian Unlimited - the UK''s most popular newspaper website > http://guardian.co.uk http://observer.co.uk > ------------------------------------------------------------------ > The Newspaper Marketing Agency > Opening Up Newspapers > http://www.nmauk.co.uk > ------------------------------------------------------------------ > > This e-mail and all attachments are confidential and may also > be privileged. If you are not the named recipient, please notify > the sender and delete the e-mail and all attachments immediately. > Do not disclose the contents to another person. You may not use > the information for any purpose, or store, or copy, it in any way. > > Guardian News & Media Limited is not liable for any computer > viruses or other material transmitted with or as part of this > e-mail. You should employ virus checking software. > > Guardian News & Media Limited > A member of Guardian Media Group PLC > Registered Office > Number 1 Scott Place, Manchester M3 3GG > Registered in England Number 908396 > > _______________________________________________ > Puppet-users mailing list > Puppet-users@madstop.com > https://mail.madstop.com/mailman/listinfo/puppet-users > > >_______________________________________________ Puppet-users mailing list Puppet-users@madstop.com https://mail.madstop.com/mailman/listinfo/puppet-users
On Fri, Sep 07, 2007 at 08:50:08PM +0200, Ohad Levy wrote:> its too hard to say from your stats, could it be that this include > buffers/cached memory?I actually said that in the initial email:> > * "free" and "used" are in megabytes, and OS cache/buffers are counted > > as free.> how did you measure the memory consumption?Taking the second line of data from "free": total used free shared buffers cached Mem: 2073996 1153024 920972 0 387644 376608 -/+ buffers/cache: 388772 1685224 ^^^^^^ ^^^^^^^ Swap: 6225912 0 6225912 -- Marcin Owsiany Web Systems Integrator - Guardian Unlimited ------------------------------------------------------------------ Visit Guardian Unlimited - the UK''s most popular newspaper website http://guardian.co.uk http://observer.co.uk ------------------------------------------------------------------ The Newspaper Marketing Agency Opening Up Newspapers http://www.nmauk.co.uk ------------------------------------------------------------------ This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News & Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News & Media Limited A member of Guardian Media Group PLC Registered Office Number 1 Scott Place, Manchester M3 3GG Registered in England Number 908396