Hi all, I''ve been looking at a potential problem, as documented here: http://projects.puppetlabs.com/issues/19241 To do with a leak within the KahaDB persistence layer of ActiveMQ. Specifically, there are reports of the db.data file growing unbounded: https://issues.apache.org/jira/browse/AMQ-3956 I''m hoping to find out information from other PuppetDB users to see if this is happening in the wild. What I''m hoping is that users can provide me with information around the size of the files in /var/lib/puppetdb/localhost/KahaDB. For example, here is the size on my test machine: root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# pwd /var/lib/puppetdb/mq/localhost/KahaDB root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# du -sk * 5552 db-1.log 32 db.data 32 db.redo 0 lock root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# And an indication of how long its been running: $ ps auxw | grep java puppetdb 52606 0.3 3.6 1536236 216136 ? Sl 13:34 0:55 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d And the version of PuppetDB: # dpkg -l puppetdb ... ii puppetdb 1.1.1-1puppetlab all PuppetDB Centralized Storage. (rpm -qi puppetdb on Redhat based machines). I''ve so far had 1 suspected case of this (the directory was reported at 17 GB), but no data to back it up - if you use PuppetDB and have the time and inclination, I''d appreciate some outputs from your PuppetDB hosts like the ones shown above (scrubbing private information of course) - even if it looks fine - as it will help us decide on the priority of this bug. Thanks in advance! ken. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
On Wednesday, February 13, 2013 12:00:40 PM UTC-6, Ken Barber wrote:> > Hi all, > > I''ve been looking at a potential problem, as documented here: > > http://projects.puppetlabs.com/issues/19241 > > To do with a leak within the KahaDB persistence layer of ActiveMQ. > Specifically, there are reports of the db.data file growing unbounded: > > https://issues.apache.org/jira/browse/AMQ-3956 > > I''m hoping to find out information from other PuppetDB users to see if > this is happening in the wild. What I''m hoping is that users can > provide me with information around the size of the files in > /var/lib/puppetdb/localhost/KahaDB. For example, here is the size on > my test machine: > > root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# pwd > /var/lib/puppetdb/mq/localhost/KahaDB > root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# du -sk * > 5552 db-1.log > 32 db.data > 32 db.redo > 0 lock > root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# > > And an indication of how long its been running: > > $ ps auxw | grep java > puppetdb 52606 0.3 3.6 1536236 216136 ? Sl 13:34 0:55 > /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1g > -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar > /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d > > And the version of PuppetDB: > > # dpkg -l puppetdb > ... > ii puppetdb 1.1.1-1puppetlab all PuppetDB > Centralized Storage. > > (rpm -qi puppetdb on Redhat based machines). > > I''ve so far had 1 suspected case of this (the directory was reported > at 17 GB), but no data to back it up - if you use PuppetDB and have > the time and inclination, I''d appreciate some outputs from your > PuppetDB hosts like the ones shown above (scrubbing private > information of course) - even if it looks fine - as it will help us > decide on the priority of this bug. > > Thanks in advance! > >/var/lib/puppetdb/mq/localhost/KahaDB$ du -sk * 7800 db-1366.log 32 db.data 32 db.redo 0 lock $ ps auxw | grep java puppetdb 10913 1.5 9.1 1639876 370180 ? Sl Jan16 622:58 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d $ dpkg -l puppetdb ii puppetdb 1.0.1-1puppetlabs1 PuppetDB Centralized Storage.> ken. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Stefan Goethals
2013-Feb-14 11:26 UTC
Re: [Puppet Users] Re: PuppetDB & KahaDB db.data leak
[root@kangaroo ~]$cd /var/lib/puppetdb/mq/localhost/KahaDB [root@kangaroo KahaDB]$ls -al total 9508 drwxr-xr-x 2 puppetdb puppetdb 4096 Feb 14 11:08 . drwxr-xr-x 4 puppetdb puppetdb 4096 Jan 14 09:38 .. -rw-r--r-- 1 puppetdb puppetdb 33030144 Feb 14 12:23 db-168.log -rw-r--r-- 1 puppetdb puppetdb 32768 Feb 14 12:23 db.data -rw-r--r-- 1 puppetdb puppetdb 28720 Feb 14 12:23 db.redo -rw-r--r-- 1 puppetdb puppetdb 0 Jan 22 10:56 lock [root@kangaroo KahaDB]$du -sk * 9464 db-168.log 32 db.data 32 db.redo 0 lock [root@kangaroo KahaDB]$ps auxw | grep java puppetdb 30154 0.3 7.7 1643468 304732 ? Sl Jan22 108:09 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d [root@kangaroo KahaDB]$rpm -qi puppetdb Name : puppetdb Relocations: (not relocatable) Version : 1.0.5 Vendor: (none) Release : 1.el6 Build Date: Thu 20 Dec 2012 01:17:23 AM CET Install Date: Mon 14 Jan 2013 09:37:27 AM CET Build Host: rpm-builder.puppetlabs.lan Group : System Environment/Daemons Source RPM: puppetdb-1.0.5-1.el6.src.rpm Size : 18326834 License: ASL 2.0 Signature : RSA/SHA1, Thu 20 Dec 2012 10:08:52 PM CET, Key ID 1054b7a24bd6ec30 URL : http://github.com/puppetlabs/puppetdb Summary : Puppet Centralized Storage Daemon Description : Puppet Centralized Storage. On Wed, Feb 13, 2013 at 7:34 PM, llowder <llowderiv@gmail.com> wrote:> > > On Wednesday, February 13, 2013 12:00:40 PM UTC-6, Ken Barber wrote: >> >> Hi all, >> >> I''ve been looking at a potential problem, as documented here: >> >> http://projects.puppetlabs.**com/issues/19241<http://projects.puppetlabs.com/issues/19241> >> >> To do with a leak within the KahaDB persistence layer of ActiveMQ. >> Specifically, there are reports of the db.data file growing unbounded: >> >> https://issues.apache.org/**jira/browse/AMQ-3956<https://issues.apache.org/jira/browse/AMQ-3956> >> >> I''m hoping to find out information from other PuppetDB users to see if >> this is happening in the wild. What I''m hoping is that users can >> provide me with information around the size of the files in >> /var/lib/puppetdb/localhost/**KahaDB. For example, here is the size on >> my test machine: >> >> root@puppetdb1:/var/lib/**puppetdb/mq/localhost/KahaDB# pwd >> /var/lib/puppetdb/mq/**localhost/KahaDB >> root@puppetdb1:/var/lib/**puppetdb/mq/localhost/KahaDB# du -sk * >> 5552 db-1.log >> 32 db.data >> 32 db.redo >> 0 lock >> root@puppetdb1:/var/lib/**puppetdb/mq/localhost/KahaDB# >> >> And an indication of how long its been running: >> >> $ ps auxw | grep java >> puppetdb 52606 0.3 3.6 1536236 216136 ? Sl 13:34 0:55 >> /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1g >> -XX:+**HeapDumpOnOutOfMemoryError >> -XX:HeapDumpPath=/var/log/**puppetdb/puppetdb-oom.hprof -jar >> /usr/share/puppetdb/puppetdb.**jar services -c /etc/puppetdb/conf.d >> >> And the version of PuppetDB: >> >> # dpkg -l puppetdb >> ... >> ii puppetdb 1.1.1-1puppetlab all PuppetDB >> Centralized Storage. >> >> (rpm -qi puppetdb on Redhat based machines). >> >> I''ve so far had 1 suspected case of this (the directory was reported >> at 17 GB), but no data to back it up - if you use PuppetDB and have >> the time and inclination, I''d appreciate some outputs from your >> PuppetDB hosts like the ones shown above (scrubbing private >> information of course) - even if it looks fine - as it will help us >> decide on the priority of this bug. >> >> Thanks in advance! >> >> > /var/lib/puppetdb/mq/localhost/KahaDB$ du -sk * > 7800 db-1366.log > 32 db.data > 32 db.redo > 0 lock > > $ ps auxw | grep java > puppetdb 10913 1.5 9.1 1639876 370180 ? Sl Jan16 622:58 > /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m > -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar > /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d > > > $ dpkg -l puppetdb > ii puppetdb 1.0.1-1puppetlabs1 PuppetDB > Centralized Storage. > > > >> ken. >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Stefan Goethals
2013-Feb-14 11:29 UTC
Re: [Puppet Users] Re: PuppetDB & KahaDB db.data leak
and on my test VM root@puppetdb01 KahaDB]# du -sk * 704 db-63.log 32 db.data 32 db.redo 0 lock [root@puppetdb01 KahaDB]# ps auxw | grep java puppetdb 11110 0.8 29.5 1237920 301552 ? Sl 03:54 3:58 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d [root@puppetdb01 KahaDB]# rpm -qi puppetdb Name : puppetdb Relocations: (not relocatable) Version : 1.0.5 Vendor: (none) Release : 1.el6 Build Date: Thu 20 Dec 2012 12:17:23 AM UTC Install Date: Thu 03 Jan 2013 02:42:03 PM UTC Build Host: rpm-builder.puppetlabs.lan Group : System Environment/Daemons Source RPM: puppetdb-1.0.5-1.el6.src.rpm Size : 18326834 License: ASL 2.0 Signature : RSA/SHA1, Thu 20 Dec 2012 09:08:52 PM UTC, Key ID 1054b7a24bd6ec30 URL : http://github.com/puppetlabs/puppetdb Summary : Puppet Centralized Storage Daemon Description : Puppet Centralized Storage. On Thu, Feb 14, 2013 at 12:26 PM, Stefan Goethals <stefan@zipkid.eu> wrote:> [root@kangaroo ~]$cd /var/lib/puppetdb/mq/localhost/KahaDB > [root@kangaroo KahaDB]$ls -al > total 9508 > drwxr-xr-x 2 puppetdb puppetdb 4096 Feb 14 11:08 . > drwxr-xr-x 4 puppetdb puppetdb 4096 Jan 14 09:38 .. > -rw-r--r-- 1 puppetdb puppetdb 33030144 Feb 14 12:23 db-168.log > -rw-r--r-- 1 puppetdb puppetdb 32768 Feb 14 12:23 db.data > -rw-r--r-- 1 puppetdb puppetdb 28720 Feb 14 12:23 db.redo > -rw-r--r-- 1 puppetdb puppetdb 0 Jan 22 10:56 lock > > [root@kangaroo KahaDB]$du -sk * > 9464 db-168.log > 32 db.data > 32 db.redo > 0 lock > > [root@kangaroo KahaDB]$ps auxw | grep java > puppetdb 30154 0.3 7.7 1643468 304732 ? Sl Jan22 108:09 > /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m > -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar > /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d > > [root@kangaroo KahaDB]$rpm -qi puppetdb > Name : puppetdb Relocations: (not relocatable) > Version : 1.0.5 Vendor: (none) > Release : 1.el6 Build Date: Thu 20 Dec 2012 > 01:17:23 AM CET > Install Date: Mon 14 Jan 2013 09:37:27 AM CET Build Host: > rpm-builder.puppetlabs.lan > Group : System Environment/Daemons Source RPM: > puppetdb-1.0.5-1.el6.src.rpm > Size : 18326834 License: ASL 2.0 > Signature : RSA/SHA1, Thu 20 Dec 2012 10:08:52 PM CET, Key ID > 1054b7a24bd6ec30 > URL : http://github.com/puppetlabs/puppetdb > Summary : Puppet Centralized Storage Daemon > Description : > Puppet Centralized Storage. > > > On Wed, Feb 13, 2013 at 7:34 PM, llowder <llowderiv@gmail.com> wrote: > >> >> >> On Wednesday, February 13, 2013 12:00:40 PM UTC-6, Ken Barber wrote: >>> >>> Hi all, >>> >>> I''ve been looking at a potential problem, as documented here: >>> >>> http://projects.puppetlabs.**com/issues/19241<http://projects.puppetlabs.com/issues/19241> >>> >>> To do with a leak within the KahaDB persistence layer of ActiveMQ. >>> Specifically, there are reports of the db.data file growing unbounded: >>> >>> https://issues.apache.org/**jira/browse/AMQ-3956<https://issues.apache.org/jira/browse/AMQ-3956> >>> >>> I''m hoping to find out information from other PuppetDB users to see if >>> this is happening in the wild. What I''m hoping is that users can >>> provide me with information around the size of the files in >>> /var/lib/puppetdb/localhost/**KahaDB. For example, here is the size on >>> my test machine: >>> >>> root@puppetdb1:/var/lib/**puppetdb/mq/localhost/KahaDB# pwd >>> /var/lib/puppetdb/mq/**localhost/KahaDB >>> root@puppetdb1:/var/lib/**puppetdb/mq/localhost/KahaDB# du -sk * >>> 5552 db-1.log >>> 32 db.data >>> 32 db.redo >>> 0 lock >>> root@puppetdb1:/var/lib/**puppetdb/mq/localhost/KahaDB# >>> >>> And an indication of how long its been running: >>> >>> $ ps auxw | grep java >>> puppetdb 52606 0.3 3.6 1536236 216136 ? Sl 13:34 0:55 >>> /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1g >>> -XX:+**HeapDumpOnOutOfMemoryError >>> -XX:HeapDumpPath=/var/log/**puppetdb/puppetdb-oom.hprof -jar >>> /usr/share/puppetdb/puppetdb.**jar services -c /etc/puppetdb/conf.d >>> >>> And the version of PuppetDB: >>> >>> # dpkg -l puppetdb >>> ... >>> ii puppetdb 1.1.1-1puppetlab all PuppetDB >>> Centralized Storage. >>> >>> (rpm -qi puppetdb on Redhat based machines). >>> >>> I''ve so far had 1 suspected case of this (the directory was reported >>> at 17 GB), but no data to back it up - if you use PuppetDB and have >>> the time and inclination, I''d appreciate some outputs from your >>> PuppetDB hosts like the ones shown above (scrubbing private >>> information of course) - even if it looks fine - as it will help us >>> decide on the priority of this bug. >>> >>> Thanks in advance! >>> >>> >> /var/lib/puppetdb/mq/localhost/KahaDB$ du -sk * >> 7800 db-1366.log >> 32 db.data >> 32 db.redo >> 0 lock >> >> $ ps auxw | grep java >> puppetdb 10913 1.5 9.1 1639876 370180 ? Sl Jan16 622:58 >> /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m >> -XX:+HeapDumpOnOutOfMemoryError >> -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar >> /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d >> >> >> $ dpkg -l puppetdb >> ii puppetdb 1.0.1-1puppetlabs1 PuppetDB >> Centralized Storage. >> >> >> >>> ken. >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscribe@googlegroups.com. >> To post to this group, send email to puppet-users@googlegroups.com. >> Visit this group at http://groups.google.com/group/puppet-users?hl=en. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
[root@mo-puppetmaster ~]# cd /var/lib/puppetdb/mq/localhost/KahaDB [root@mo-puppetmaster KahaDB]# pwd /var/lib/puppetdb/mq/localhost/KahaDB [root@mo-puppetmaster KahaDB]# du -sk * 22036 db-95.log 32 db.data 32 db.redo 0 lock [root@mo-puppetmaster KahaDB]# ps auxw | grep java root 9969 0.0 0.0 103300 812 pts/0 S+ 14:25 0:00 grep java puppetdb 29242 0.8 16.0 1708184 330716 ? Sl Feb05 112:26 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d [root@mo-puppetmaster KahaDB]# rpm -qi puppetdb Name : puppetdb Relocations: (not relocatable) Version : 1.0.1 Vendor: (none) Release : 1.el6 Build Date: Tue 09 Oct 2012 02:26:45 AM CEST Install Date: Wed 17 Oct 2012 03:30:11 PM CEST Build Host: rpm-builder.puppetlabs.lan Group : System Environment/Daemons Source RPM: puppetdb-1.0.1-1.el6.src.rpm Size : 18533978 License: ASL 2.0 Signature : RSA/SHA1, Wed 10 Oct 2012 10:42:55 PM CEST, Key ID 1054b7a24bd6ec30 URL : http://github.com/puppetlabs/puppetdb Summary : Puppet Centralized Storage Daemon Description : Puppet Centralized Storage. On Wednesday, February 13, 2013 7:00:40 PM UTC+1, Ken Barber wrote:> > Hi all, > > I''ve been looking at a potential problem, as documented here: > > http://projects.puppetlabs.com/issues/19241 > > To do with a leak within the KahaDB persistence layer of ActiveMQ. > Specifically, there are reports of the db.data file growing unbounded: > > https://issues.apache.org/jira/browse/AMQ-3956 > > I''m hoping to find out information from other PuppetDB users to see if > this is happening in the wild. What I''m hoping is that users can > provide me with information around the size of the files in > /var/lib/puppetdb/localhost/KahaDB. For example, here is the size on > my test machine: > > root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# pwd > /var/lib/puppetdb/mq/localhost/KahaDB > root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# du -sk * > 5552 db-1.log > 32 db.data > 32 db.redo > 0 lock > root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# > > And an indication of how long its been running: > > $ ps auxw | grep java > puppetdb 52606 0.3 3.6 1536236 216136 ? Sl 13:34 0:55 > /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1g > -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar > /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d > > And the version of PuppetDB: > > # dpkg -l puppetdb > ... > ii puppetdb 1.1.1-1puppetlab all PuppetDB > Centralized Storage. > > (rpm -qi puppetdb on Redhat based machines). > > I''ve so far had 1 suspected case of this (the directory was reported > at 17 GB), but no data to back it up - if you use PuppetDB and have > the time and inclination, I''d appreciate some outputs from your > PuppetDB hosts like the ones shown above (scrubbing private > information of course) - even if it looks fine - as it will help us > decide on the priority of this bug. > > Thanks in advance! > > ken. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Martin Willemsma
2013-Feb-14 18:14 UTC
Re: [Puppet Users] Re: PuppetDB & KahaDB db.data leak
[root@ ~]# cd /var/lib/puppetdb/mq/localhost/KahaDB [root@ KahaDB]# du -sk * 32828 db-9947.log 432 db-9948.log 32 db.data 32 db.redo 0 lock [root@ KahaDB]# ps auxw | grep java root 5695 0.0 0.0 103232 824 pts/0 R+ 19:09 0:00 grep java puppetdb 12088 6.8 7.7 4574712 309892 ? Sl Feb05 916:30 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx2048m -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d activemq 31704 0.5 40.5 5655332 1611528 ? Sl 2012 1279:27 java -Dactivemq.home=/opt/activemq -Dactivemq.base=/opt/activemq -Djavax.net.ssl.keyStorePassword=password -Djavax.net.ssl.trustStorePassword=password -Djavax.net.ssl.keyStore=/opt/activemq/conf/broker.ks -Djavax.net.ssl.trustStore=/opt/activemq/conf/broker.ts -Dcom.sun.management.jmxremote -Dorg.apache.activemq.UseDedicatedTaskRunner=false -Djava.util.logging.config.file=logging.properties -Dactivemq.conf=/opt/activemq/conf -Dactivemq.data=/opt/activemq/data -XX:+UseCompressedOops -Xms3072m -Xmx3072m -Djava.library.path=/opt/activemq/bin/linux/ -classpath /opt/activemq/bin/wrapper.jar:/opt/activemq/bin/run.jar -Dwrapper.key=IxFTP8aU4zzVoBh_ -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=31701 -Dwrapper.version=3.2.3 -Dwrapper.native_library=wrapper -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp org.apache.activemq.console.Main start [root@ KahaDB]# rpm -qi puppetdb Name : puppetdb Relocations: (not relocatable) Version : 1.1.1 Vendor: (none) Release : 1.el6 Build Date: Fri 01 Feb 2013 11:21:27 AM CET Install Date: Tue 05 Feb 2013 12:20:17 PM CET Build Host: rpm-builder.delivery.puppetlabs.net Group : System Environment/Daemons Source RPM: puppetdb-1.1.1-1.el6.src.rpm Size : 19736313 License: ASL 2.0 Signature : RSA/SHA1, Mon 04 Feb 2013 07:51:33 PM CET, Key ID 1054b7a24bd6ec30 URL : http://github.com/puppetlabs/puppetdb Summary : Puppet Centralized Storage Daemon Description : Puppet Centralized Storage. 2013/2/14 Julien Pivotto <roidelapluie@gmail.com>> > [root@mo-puppetmaster ~]# cd /var/lib/puppetdb/mq/localhost/KahaDB > [root@mo-puppetmaster KahaDB]# pwd > /var/lib/puppetdb/mq/localhost/KahaDB > [root@mo-puppetmaster KahaDB]# du -sk * > 22036 db-95.log > > 32 db.data > 32 db.redo > 0 lock > [root@mo-puppetmaster KahaDB]# ps auxw | grep java > root 9969 0.0 0.0 103300 812 pts/0 S+ 14:25 0:00 grep java > puppetdb 29242 0.8 16.0 1708184 330716 ? Sl Feb05 112:26 > /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m > -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar > /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d > [root@mo-puppetmaster KahaDB]# rpm -qi puppetdb > > Name : puppetdb Relocations: (not relocatable) > Version : 1.0.1 Vendor: (none) > Release : 1.el6 Build Date: Tue 09 Oct 2012 > 02:26:45 AM CEST > Install Date: Wed 17 Oct 2012 03:30:11 PM CEST Build Host: > rpm-builder.puppetlabs.lan > Group : System Environment/Daemons Source RPM: > puppetdb-1.0.1-1.el6.src.rpm > Size : 18533978 License: ASL 2.0 > Signature : RSA/SHA1, Wed 10 Oct 2012 10:42:55 PM CEST, Key ID > 1054b7a24bd6ec30 > > URL : http://github.com/puppetlabs/puppetdb > Summary : Puppet Centralized Storage Daemon > Description : > Puppet Centralized Storage. > > > On Wednesday, February 13, 2013 7:00:40 PM UTC+1, Ken Barber wrote: >> >> Hi all, >> >> I''ve been looking at a potential problem, as documented here: >> >> http://projects.puppetlabs.**com/issues/19241<http://projects.puppetlabs.com/issues/19241> >> >> To do with a leak within the KahaDB persistence layer of ActiveMQ. >> Specifically, there are reports of the db.data file growing unbounded: >> >> https://issues.apache.org/**jira/browse/AMQ-3956<https://issues.apache.org/jira/browse/AMQ-3956> >> >> I''m hoping to find out information from other PuppetDB users to see if >> this is happening in the wild. What I''m hoping is that users can >> provide me with information around the size of the files in >> /var/lib/puppetdb/localhost/**KahaDB. For example, here is the size on >> my test machine: >> >> root@puppetdb1:/var/lib/**puppetdb/mq/localhost/KahaDB# pwd >> /var/lib/puppetdb/mq/**localhost/KahaDB >> root@puppetdb1:/var/lib/**puppetdb/mq/localhost/KahaDB# du -sk * >> 5552 db-1.log >> 32 db.data >> 32 db.redo >> 0 lock >> root@puppetdb1:/var/lib/**puppetdb/mq/localhost/KahaDB# >> >> And an indication of how long its been running: >> >> $ ps auxw | grep java >> puppetdb 52606 0.3 3.6 1536236 216136 ? Sl 13:34 0:55 >> /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1g >> -XX:+**HeapDumpOnOutOfMemoryError >> -XX:HeapDumpPath=/var/log/**puppetdb/puppetdb-oom.hprof -jar >> /usr/share/puppetdb/puppetdb.**jar services -c /etc/puppetdb/conf.d >> >> And the version of PuppetDB: >> >> # dpkg -l puppetdb >> ... >> ii puppetdb 1.1.1-1puppetlab all PuppetDB >> Centralized Storage. >> >> (rpm -qi puppetdb on Redhat based machines). >> >> I''ve so far had 1 suspected case of this (the directory was reported >> at 17 GB), but no data to back it up - if you use PuppetDB and have >> the time and inclination, I''d appreciate some outputs from your >> PuppetDB hosts like the ones shown above (scrubbing private >> information of course) - even if it looks fine - as it will help us >> decide on the priority of this bug. >> >> Thanks in advance! >> >> ken. >> > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to puppet-users+unsubscribe@googlegroups.com. > To post to this group, send email to puppet-users@googlegroups.com. > Visit this group at http://groups.google.com/group/puppet-users?hl=en. > For more options, visit https://groups.google.com/groups/opt_out. > > >-- Met vriendelijke groet, Kind Regards, Martin Willemsma -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
My log file is cycling at 32 MB $ date;du -sk * Thu Feb 14 19:35:38 UTC 2013 24288 db-12257.log 360 db.data 124 db.redo 0 lock $ date;du -sk * Thu Feb 14 19:37:29 UTC 2013 32816 db-12257.log 1820 db-12258.log 360 db.data 124 db.redo 0 lock $ date;du -sk * Thu Feb 14 19:37:47 UTC 2013 3660 db-12258.log 360 db.data 124 db.redo 0 lock $ date;du -sk * Thu Feb 14 19:40:00 UTC 2013 15520 db-12258.log 360 db.data 124 db.redo 0 lock $ date;du -sk * Thu Feb 14 19:42:41 UTC 2013 33024 db-12258.log 364 db-12259.log 360 db.data 124 db.redo 0 lock $ ps auxw | grep puppetdb |grep -v grep puppetdb 1486 24.6 11.7 2406524 459636 ? Sl Feb09 2050:43 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1024m -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d $ rpm -qi puppetdb Name : puppetdb Relocations: (not relocatable) Version : 1.1.1 Vendor: (none) Release : 1.el6 Build Date: Fri 01 Feb 2013 10:21:27 AM UTC Install Date: Sat 09 Feb 2013 12:08:35 AM UTC Build Host: rpm-builder.delivery.puppetlabs.net Group : System Environment/Daemons Source RPM: puppetdb-1.1.1-1.el6.src.rpm Size : 19736313 License: ASL 2.0 Signature : RSA/SHA1, Mon 04 Feb 2013 06:51:33 PM UTC, Key ID 1054b7a24bd6ec30 URL : http://github.com/puppetlabs/puppetdb Summary : Puppet Centralized Storage Daemon Description : Puppet Centralized Storage. On Wednesday, February 13, 2013 12:00:40 PM UTC-6, Ken Barber wrote:> > Hi all, > > I''ve been looking at a potential problem, as documented here: > > http://projects.puppetlabs.com/issues/19241 > > To do with a leak within the KahaDB persistence layer of ActiveMQ. > Specifically, there are reports of the db.data file growing unbounded: > > https://issues.apache.org/jira/browse/AMQ-3956 > > I''m hoping to find out information from other PuppetDB users to see if > this is happening in the wild. What I''m hoping is that users can > provide me with information around the size of the files in > /var/lib/puppetdb/localhost/KahaDB. For example, here is the size on > my test machine: > > root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# pwd > /var/lib/puppetdb/mq/localhost/KahaDB > root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# du -sk * > 5552 db-1.log > 32 db.data > 32 db.redo > 0 lock > root@puppetdb1:/var/lib/puppetdb/mq/localhost/KahaDB# > > And an indication of how long its been running: > > $ ps auxw | grep java > puppetdb 52606 0.3 3.6 1536236 216136 ? Sl 13:34 0:55 > /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx1g > -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar > /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d > > And the version of PuppetDB: > > # dpkg -l puppetdb > ... > ii puppetdb 1.1.1-1puppetlab all PuppetDB > Centralized Storage. > > (rpm -qi puppetdb on Redhat based machines). > > I''ve so far had 1 suspected case of this (the directory was reported > at 17 GB), but no data to back it up - if you use PuppetDB and have > the time and inclination, I''d appreciate some outputs from your > PuppetDB hosts like the ones shown above (scrubbing private > information of course) - even if it looks fine - as it will help us > decide on the priority of this bug. > > Thanks in advance! > > ken. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.
Ken, here''s my data from our (small) production environment: root@puppet:/var/lib/puppetdb/mq/localhost/KahaDB# du -sk * 23648 db-996.log 32 db.data 36 db.redo 0 lock root@puppet:/var/lib/puppetdb/mq/localhost/KahaDB# ps auxw | grep java USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND puppetdb 5320 0.5 16.0 1144504 163876 ? Sl Feb11 30:05 /usr/bin/java -XX:OnOutOfMemoryError=kill -9 %p -Xmx192m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/puppetdb/puppetdb-oom.hprof -jar /usr/share/puppetdb/puppetdb.jar services -c /etc/puppetdb/conf.d root@puppet:/var/lib/puppetdb/mq/localhost/KahaDB# dpkg -s puppetdb | grep Version Version: 1.1.1-1puppetlabs1 Regards, Martijn Op woensdag 13 februari 2013 19:00:40 UTC+1 schreef Ken Barber het volgende:> > Hi all, > > I''ve been looking at a potential problem, as documented here: > > http://projects.puppetlabs.com/issues/19241 > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To post to this group, send email to puppet-users@googlegroups.com. Visit this group at http://groups.google.com/group/puppet-users?hl=en. For more options, visit https://groups.google.com/groups/opt_out.