Hi everyone, We are using collectd collecting system information, I am trying to setup the postgres plugin for puppetdb ( http://collectd.org/wiki/index.php/Plugin:PostgreSQL), what could be the relevant value to get? I am trying to see what is going on because lately sometime I get some "Sun Aug 05 10:39:12 +0200 2012 Puppet (err): Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit ''replace catalog'' command for XXXXXXXXXX to PuppetDB at puppetmaster.fqdn:8081: [500 Server Error] " I am using puppet: ii puppet 2.7.18-1puppetlabs1 Centralized configuration management - agent startup and compatibility scripts ii puppet-common 2.7.18-1puppetlabs1 Centralized configuration management ii puppet-dashboard 1.2.10-1puppetlabs2 Dashboard for Puppet ii puppetdb 0.9.2-1puppetlabs2 PuppetDB Centralized Storage. ii puppetdb-terminus 0.9.2-1puppetlabs2 Connect Puppet to PuppetDB by setting up a terminus for PuppetDB. ii puppetmaster 2.7.18-1puppetlabs1 Centralized configuration management - master startup and compatibility scripts ii puppetmaster-common 2.7.18-1puppetlabs1 Puppet master common scripts On debian squeeze for handling 120 server and the puppetmaster is a 1950 with 4go of memory and it doesn''t seems to be under heavy load. Regards, JM -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Hello A_SAAS, - Do you see any messages in the file "/var/log/puppetdb/puppetdb.log"? - If not, can you try editing the file "/etc/puppetdb/log4j.properties" and replace "INFO" with "DEBUG", and then see if you get any messages in the aforementioned log file? - What version of java are you using? We''ve encountered some SSL-related problems between very early versions of the Oracle JDK 7 and puppetdb. Thanks! Chris On Monday, August 6, 2012 1:51:07 AM UTC-7, A_SAAS wrote:> > Hi everyone, > > > We are using collectd collecting system information, I am trying to setup > the postgres plugin for puppetdb ( > http://collectd.org/wiki/index.php/Plugin:PostgreSQL), what could be the > relevant value to get? I am trying to see what is going on because lately > sometime I get some "Sun Aug 05 10:39:12 +0200 2012 Puppet (err): Could not > retrieve catalog from remote server: Error 400 on SERVER: Failed to submit > ''replace catalog'' command for XXXXXXXXXX to PuppetDB at > puppetmaster.fqdn:8081: [500 Server Error] " > > > I am using puppet: > ii puppet 2.7.18-1puppetlabs1 > Centralized configuration management - agent startup and compatibility > scripts > ii puppet-common 2.7.18-1puppetlabs1 Centralized > configuration management > ii puppet-dashboard 1.2.10-1puppetlabs2 Dashboard for > Puppet > ii puppetdb 0.9.2-1puppetlabs2 PuppetDB > Centralized Storage. > ii puppetdb-terminus 0.9.2-1puppetlabs2 Connect > Puppet to PuppetDB by setting up a terminus for PuppetDB. > ii puppetmaster 2.7.18-1puppetlabs1 Centralized > configuration management - master startup and compatibility scripts > ii puppetmaster-common 2.7.18-1puppetlabs1 Puppet master > common scripts > > > On debian squeeze for handling 120 server and the puppetmaster is a 1950 > with 4go of memory and it doesn''t seems to be under heavy load. > > Regards, > JM >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/JHdLQ5zvobAJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Here is the puppetdb.log:
2012-08-05 10:39:12,431 WARN [qtp2101818118-104743]
[server.AbstractHttpConnection] /commands
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2894)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:589)
at java.lang.StringBuffer.append(StringBuffer.java:337)
at java.net.URLDecoder.decode(URLDecoder.java:199)
at ring.util.codec$form_decode_str.doInvoke(codec.clj:97)
at clojure.lang.RestFn.invoke(RestFn.java:423)
at ring.util.codec$form_decode$fn__5979.invoke(codec.clj:110)
at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:60)
at clojure.core.protocols$fn__5565.invoke(protocols.clj:30)
at
clojure.core.protocols$fn__5543$G__5538__5552.invoke(protocols.clj:11)
at clojure.core$reduce.invoke(core.clj:5995)
at ring.util.codec$form_decode.doInvoke(codec.clj:113)
at clojure.lang.RestFn.invoke(RestFn.java:423)
at ring.middleware.params$parse_params.invoke(params.clj:6)
at ring.middleware.params$assoc_form_params.invoke(params.clj:29)
at ring.middleware.params$wrap_params$fn__5994.invoke(params.clj:51)
at
com.puppetlabs.middleware$wrap_with_authorization$fn__7935.invoke(middleware.clj:20)
at
com.puppetlabs.middleware$wrap_with_certificate_cn$fn__7939.invoke(middleware.clj:35)
at
com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__7945$fn__7946.invoke(middleware.clj:60)
at
com.puppetlabs.middleware.proxy$java.lang.Object$Callable$f8c5758f.call(Unknown
Source)
at com.yammer.metrics.core.Timer.time(Timer.java:91)
at
com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__7945.invoke(middleware.clj:58)
at
com.puppetlabs.middleware$wrap_with_globals$fn__7942.invoke(middleware.clj:43)
at ring.adapter.jetty$proxy_handler$fn__2028.invoke(jetty.clj:18)
at
ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$0.handle(Unknown
Source)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
at org.eclipse.jetty.server.Server.handle(Server.java:349)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
at
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
Regarding the java version:
ii openjdk-6-jre-headless
6b18-1.8.13-0+squeeze2 OpenJDK Java runtime, using Hotspot JIT
(headless)
ii openjdk-6-jre-lib
6b18-1.8.13-0+squeeze2 OpenJDK Java runtime (architecture
independent libraries)
Does it help?
Regards,
JM
On Mon, Aug 6, 2012 at 6:40 PM, Chris Price <chris@puppetlabs.com> wrote:
> Hello A_SAAS,
>
> - Do you see any messages in the file
"/var/log/puppetdb/puppetdb.log"?
> - If not, can you try editing the file
> "/etc/puppetdb/log4j.properties" and replace "INFO"
with "DEBUG", and then
> see if you get any messages in the aforementioned log file?
> - What version of java are you using? We''ve encountered some
> SSL-related problems between very early versions of the Oracle JDK 7 and
> puppetdb.
>
> Thanks!
> Chris
>
>
> On Monday, August 6, 2012 1:51:07 AM UTC-7, A_SAAS wrote:
>>
>> Hi everyone,
>>
>>
>> We are using collectd collecting system information, I am trying to
setup
>> the postgres plugin for puppetdb (http://collectd.org/wiki/**
>>
index.php/Plugin:PostgreSQL<http://collectd.org/wiki/index.php/Plugin:PostgreSQL>),
>> what could be the relevant value to get? I am trying to see what is
going
>> on because lately sometime I get some "Sun Aug 05 10:39:12 +0200
2012
>> Puppet (err): Could not retrieve catalog from remote server: Error 400
on
>> SERVER: Failed to submit ''replace catalog'' command
for XXXXXXXXXX to
>> PuppetDB at puppetmaster.fqdn:8081: [500 Server Error] "
>>
>>
>> I am using puppet:
>> ii puppet 2.7.18-1puppetlabs1
>> Centralized configuration management - agent startup and compatibility
>> scripts
>> ii puppet-common 2.7.18-1puppetlabs1
Centralized
>> configuration management
>> ii puppet-dashboard 1.2.10-1puppetlabs2 Dashboard
>> for Puppet
>> ii puppetdb 0.9.2-1puppetlabs2
>> PuppetDB Centralized Storage.
>> ii puppetdb-terminus 0.9.2-1puppetlabs2 Connect
>> Puppet to PuppetDB by setting up a terminus for PuppetDB.
>> ii puppetmaster 2.7.18-1puppetlabs1
>> Centralized configuration management - master startup and
compatibility
>> scripts
>> ii puppetmaster-common 2.7.18-1puppetlabs1 Puppet master
>> common scripts
>>
>>
>> On debian squeeze for handling 120 server and the puppetmaster is a
1950
>> with 4go of memory and it doesn''t seems to be under heavy
load.
>>
>> Regards,
>> JM
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/JHdLQ5zvobAJ.
> To post to this group, send email to puppet-users@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
In fact I am starting to have more and more errors regarding java exception: 2012-08-07 16:39:23,864 ERROR [command-proc-77] [puppetdb.command] Fatal error parsing command java.lang.OutOfMemoryError: Java heap space 2012-08-07 16:39:26,628 WARN [command-proc-77] [listener.DefaultMessageListenerContainer] Execution of JMS message listener failed, and no ErrorHandler has been set. java.lang.OutOfMemoryError: Java heap space Regards, JM On Tue, Aug 7, 2012 at 9:58 AM, Antidot SAS <antidotsas@gmail.com> wrote:> Here is the puppetdb.log: > 2012-08-05 10:39:12,431 WARN [qtp2101818118-104743] > [server.AbstractHttpConnection] /commands > java.lang.OutOfMemoryError: Java heap space > at java.util.Arrays.copyOf(Arrays.java:2894) > at > java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117) > at > java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:589) > at java.lang.StringBuffer.append(StringBuffer.java:337) > at java.net.URLDecoder.decode(URLDecoder.java:199) > at ring.util.codec$form_decode_str.doInvoke(codec.clj:97) > at clojure.lang.RestFn.invoke(RestFn.java:423) > at ring.util.codec$form_decode$fn__5979.invoke(codec.clj:110) > at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:60) > at clojure.core.protocols$fn__5565.invoke(protocols.clj:30) > at > clojure.core.protocols$fn__5543$G__5538__5552.invoke(protocols.clj:11) > at clojure.core$reduce.invoke(core.clj:5995) > at ring.util.codec$form_decode.doInvoke(codec.clj:113) > at clojure.lang.RestFn.invoke(RestFn.java:423) > at ring.middleware.params$parse_params.invoke(params.clj:6) > at ring.middleware.params$assoc_form_params.invoke(params.clj:29) > at > ring.middleware.params$wrap_params$fn__5994.invoke(params.clj:51) > at > com.puppetlabs.middleware$wrap_with_authorization$fn__7935.invoke(middleware.clj:20) > at > com.puppetlabs.middleware$wrap_with_certificate_cn$fn__7939.invoke(middleware.clj:35) > at > com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__7945$fn__7946.invoke(middleware.clj:60) > at > com.puppetlabs.middleware.proxy$java.lang.Object$Callable$f8c5758f.call(Unknown > Source) > at com.yammer.metrics.core.Timer.time(Timer.java:91) > at > com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__7945.invoke(middleware.clj:58) > at > com.puppetlabs.middleware$wrap_with_globals$fn__7942.invoke(middleware.clj:43) > at ring.adapter.jetty$proxy_handler$fn__2028.invoke(jetty.clj:18) > at > ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$0.handle(Unknown > Source) > at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) > at org.eclipse.jetty.server.Server.handle(Server.java:349) > at > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) > at > org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894) > at > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857) > > > Regarding the java version: > ii openjdk-6-jre-headless > 6b18-1.8.13-0+squeeze2 OpenJDK Java runtime, using Hotspot JIT > (headless) > ii openjdk-6-jre-lib > 6b18-1.8.13-0+squeeze2 OpenJDK Java runtime (architecture > independent libraries) > > Does it help? > > Regards, > JM > > > On Mon, Aug 6, 2012 at 6:40 PM, Chris Price <chris@puppetlabs.com> wrote: > >> Hello A_SAAS, >> >> - Do you see any messages in the file >> "/var/log/puppetdb/puppetdb.log"? >> - If not, can you try editing the file >> "/etc/puppetdb/log4j.properties" and replace "INFO" with "DEBUG", and then >> see if you get any messages in the aforementioned log file? >> - What version of java are you using? We''ve encountered some >> SSL-related problems between very early versions of the Oracle JDK 7 and >> puppetdb. >> >> Thanks! >> Chris >> >> >> On Monday, August 6, 2012 1:51:07 AM UTC-7, A_SAAS wrote: >>> >>> Hi everyone, >>> >>> >>> We are using collectd collecting system information, I am trying to >>> setup the postgres plugin for puppetdb (http://collectd.org/wiki/** >>> index.php/Plugin:PostgreSQL<http://collectd.org/wiki/index.php/Plugin:PostgreSQL>), >>> what could be the relevant value to get? I am trying to see what is going >>> on because lately sometime I get some "Sun Aug 05 10:39:12 +0200 2012 >>> Puppet (err): Could not retrieve catalog from remote server: Error 400 on >>> SERVER: Failed to submit ''replace catalog'' command for XXXXXXXXXX to >>> PuppetDB at puppetmaster.fqdn:8081: [500 Server Error] " >>> >>> >>> I am using puppet: >>> ii puppet 2.7.18-1puppetlabs1 >>> Centralized configuration management - agent startup and compatibility >>> scripts >>> ii puppet-common 2.7.18-1puppetlabs1 >>> Centralized configuration management >>> ii puppet-dashboard 1.2.10-1puppetlabs2 Dashboard >>> for Puppet >>> ii puppetdb 0.9.2-1puppetlabs2 >>> PuppetDB Centralized Storage. >>> ii puppetdb-terminus 0.9.2-1puppetlabs2 Connect >>> Puppet to PuppetDB by setting up a terminus for PuppetDB. >>> ii puppetmaster 2.7.18-1puppetlabs1 >>> Centralized configuration management - master startup and compatibility >>> scripts >>> ii puppetmaster-common 2.7.18-1puppetlabs1 Puppet master >>> common scripts >>> >>> >>> On debian squeeze for handling 120 server and the puppetmaster is a 1950 >>> with 4go of memory and it doesn''t seems to be under heavy load. >>> >>> Regards, >>> JM >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To view this discussion on the web visit >> https://groups.google.com/d/msg/puppet-users/-/JHdLQ5zvobAJ. >> To post to this group, send email to puppet-users@googlegroups.com. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
If you wanna see the memory of the puppet server here it is. Regards, JM On Tue, Aug 7, 2012 at 5:03 PM, Antidot SAS <antidotsas@gmail.com> wrote:> In fact I am starting to have more and more errors regarding java > exception: > 2012-08-07 16:39:23,864 ERROR [command-proc-77] [puppetdb.command] Fatal > error parsing command java.lang.OutOfMemoryError: Java heap space > 2012-08-07 16:39:26,628 WARN [command-proc-77] > [listener.DefaultMessageListenerContainer] Execution of JMS message > listener failed, and no ErrorHandler has been > set. java.lang.OutOfMemoryError: Java heap space > > Regards, > JM > > > > On Tue, Aug 7, 2012 at 9:58 AM, Antidot SAS <antidotsas@gmail.com> wrote: > >> Here is the puppetdb.log: >> 2012-08-05 10:39:12,431 WARN [qtp2101818118-104743] >> [server.AbstractHttpConnection] /commands >> java.lang.OutOfMemoryError: Java heap space >> at java.util.Arrays.copyOf(Arrays.java:2894) >> at >> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117) >> at >> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:589) >> at java.lang.StringBuffer.append(StringBuffer.java:337) >> at java.net.URLDecoder.decode(URLDecoder.java:199) >> at ring.util.codec$form_decode_str.doInvoke(codec.clj:97) >> at clojure.lang.RestFn.invoke(RestFn.java:423) >> at ring.util.codec$form_decode$fn__5979.invoke(codec.clj:110) >> at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:60) >> at clojure.core.protocols$fn__5565.invoke(protocols.clj:30) >> at >> clojure.core.protocols$fn__5543$G__5538__5552.invoke(protocols.clj:11) >> at clojure.core$reduce.invoke(core.clj:5995) >> at ring.util.codec$form_decode.doInvoke(codec.clj:113) >> at clojure.lang.RestFn.invoke(RestFn.java:423) >> at ring.middleware.params$parse_params.invoke(params.clj:6) >> at ring.middleware.params$assoc_form_params.invoke(params.clj:29) >> at >> ring.middleware.params$wrap_params$fn__5994.invoke(params.clj:51) >> at >> com.puppetlabs.middleware$wrap_with_authorization$fn__7935.invoke(middleware.clj:20) >> at >> com.puppetlabs.middleware$wrap_with_certificate_cn$fn__7939.invoke(middleware.clj:35) >> at >> com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__7945$fn__7946.invoke(middleware.clj:60) >> at >> com.puppetlabs.middleware.proxy$java.lang.Object$Callable$f8c5758f.call(Unknown >> Source) >> at com.yammer.metrics.core.Timer.time(Timer.java:91) >> at >> com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__7945.invoke(middleware.clj:58) >> at >> com.puppetlabs.middleware$wrap_with_globals$fn__7942.invoke(middleware.clj:43) >> at ring.adapter.jetty$proxy_handler$fn__2028.invoke(jetty.clj:18) >> at >> ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$0.handle(Unknown >> Source) >> at >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) >> at org.eclipse.jetty.server.Server.handle(Server.java:349) >> at >> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) >> at >> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894) >> at >> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948) >> at >> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857) >> >> >> Regarding the java version: >> ii openjdk-6-jre-headless >> 6b18-1.8.13-0+squeeze2 OpenJDK Java runtime, using Hotspot JIT >> (headless) >> ii openjdk-6-jre-lib >> 6b18-1.8.13-0+squeeze2 OpenJDK Java runtime (architecture >> independent libraries) >> >> Does it help? >> >> Regards, >> JM >> >> >> On Mon, Aug 6, 2012 at 6:40 PM, Chris Price <chris@puppetlabs.com> wrote: >> >>> Hello A_SAAS, >>> >>> - Do you see any messages in the file >>> "/var/log/puppetdb/puppetdb.log"? >>> - If not, can you try editing the file >>> "/etc/puppetdb/log4j.properties" and replace "INFO" with "DEBUG", and then >>> see if you get any messages in the aforementioned log file? >>> - What version of java are you using? We''ve encountered some >>> SSL-related problems between very early versions of the Oracle JDK 7 and >>> puppetdb. >>> >>> Thanks! >>> Chris >>> >>> >>> On Monday, August 6, 2012 1:51:07 AM UTC-7, A_SAAS wrote: >>>> >>>> Hi everyone, >>>> >>>> >>>> We are using collectd collecting system information, I am trying to >>>> setup the postgres plugin for puppetdb (http://collectd.org/wiki/** >>>> index.php/Plugin:PostgreSQL<http://collectd.org/wiki/index.php/Plugin:PostgreSQL>), >>>> what could be the relevant value to get? I am trying to see what is going >>>> on because lately sometime I get some "Sun Aug 05 10:39:12 +0200 2012 >>>> Puppet (err): Could not retrieve catalog from remote server: Error 400 on >>>> SERVER: Failed to submit ''replace catalog'' command for XXXXXXXXXX to >>>> PuppetDB at puppetmaster.fqdn:8081: [500 Server Error] " >>>> >>>> >>>> I am using puppet: >>>> ii puppet 2.7.18-1puppetlabs1 >>>> Centralized configuration management - agent startup and compatibility >>>> scripts >>>> ii puppet-common 2.7.18-1puppetlabs1 >>>> Centralized configuration management >>>> ii puppet-dashboard 1.2.10-1puppetlabs2 Dashboard >>>> for Puppet >>>> ii puppetdb 0.9.2-1puppetlabs2 >>>> PuppetDB Centralized Storage. >>>> ii puppetdb-terminus 0.9.2-1puppetlabs2 Connect >>>> Puppet to PuppetDB by setting up a terminus for PuppetDB. >>>> ii puppetmaster 2.7.18-1puppetlabs1 >>>> Centralized configuration management - master startup and compatibility >>>> scripts >>>> ii puppetmaster-common 2.7.18-1puppetlabs1 Puppet master >>>> common scripts >>>> >>>> >>>> On debian squeeze for handling 120 server and the puppetmaster is a >>>> 1950 with 4go of memory and it doesn''t seems to be under heavy load. >>>> >>>> Regards, >>>> JM >>>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Puppet Users" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/puppet-users/-/JHdLQ5zvobAJ. >>> To post to this group, send email to puppet-users@googlegroups.com. >>> To unsubscribe from this group, send email to >>> puppet-users+unsubscribe@googlegroups.com. >>> For more options, visit this group at >>> http://groups.google.com/group/puppet-users?hl=en. >>> >> >> >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
JM, Hmm... so, those versions of Java seem fine--those are probably the ones we''ve done the most testing with. So, the error message that you sent indicates that the JVM is running out of RAM. This could possibly indicate that you are still using the embedded database instead of postgres--the embedded database uses a lot of RAM. Can we see what your database.ini file looks like? In fact, perhaps we can take a look at all of the files in your /etc/puppetdb/conf.d directory? You could also connect directly to your postgres database using psql or pgadmin, and check to see if the puppetdb tables exist and contain data. The only other possibility that I can think of would be if one or more of your nodes has an extremely large catalog, and that the puppetdb JVM instance doesn''t have enough memory to process the catalog. If that''s the case, we can try increasing your JVM max heap space in /etc/default/puppetdb... but we would be really interested in collecting some data from you about what that catalog looks like (for our own testing and debugging purposes) if it turns out that that is the culprit. Thanks! Chris On Tuesday, August 7, 2012 8:13:15 AM UTC-7, A_SAAS wrote:> > If you wanna see the memory of the puppet server here it is. > > > Regards, > JM > > On Tue, Aug 7, 2012 at 5:03 PM, Antidot SAS <antidotsas@gmail.com> wrote: > >> In fact I am starting to have more and more errors regarding java >> exception: >> 2012-08-07 16:39:23,864 ERROR [command-proc-77] [puppetdb.command] Fatal >> error parsing command java.lang.OutOfMemoryError: Java heap space >> 2012-08-07 16:39:26,628 WARN [command-proc-77] >> [listener.DefaultMessageListenerContainer] Execution of JMS message >> listener failed, and no ErrorHandler has been >> set. java.lang.OutOfMemoryError: Java heap space >> >> Regards, >> JM >> >> >> >> On Tue, Aug 7, 2012 at 9:58 AM, Antidot SAS <antidotsas@gmail.com> wrote: >> >>> Here is the puppetdb.log: >>> 2012-08-05 10:39:12,431 WARN [qtp2101818118-104743] >>> [server.AbstractHttpConnection] /commands >>> java.lang.OutOfMemoryError: Java heap space >>> at java.util.Arrays.copyOf(Arrays.java:2894) >>> at >>> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:117) >>> at >>> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:589) >>> at java.lang.StringBuffer.append(StringBuffer.java:337) >>> at java.net.URLDecoder.decode(URLDecoder.java:199) >>> at ring.util.codec$form_decode_str.doInvoke(codec.clj:97) >>> at clojure.lang.RestFn.invoke(RestFn.java:423) >>> at ring.util.codec$form_decode$fn__5979.invoke(codec.clj:110) >>> at clojure.lang.ArrayChunk.reduce(ArrayChunk.java:60) >>> at clojure.core.protocols$fn__5565.invoke(protocols.clj:30) >>> at >>> clojure.core.protocols$fn__5543$G__5538__5552.invoke(protocols.clj:11) >>> at clojure.core$reduce.invoke(core.clj:5995) >>> at ring.util.codec$form_decode.doInvoke(codec.clj:113) >>> at clojure.lang.RestFn.invoke(RestFn.java:423) >>> at ring.middleware.params$parse_params.invoke(params.clj:6) >>> at ring.middleware.params$assoc_form_params.invoke(params.clj:29) >>> at >>> ring.middleware.params$wrap_params$fn__5994.invoke(params.clj:51) >>> at >>> com.puppetlabs.middleware$wrap_with_authorization$fn__7935.invoke(middleware.clj:20) >>> at >>> com.puppetlabs.middleware$wrap_with_certificate_cn$fn__7939.invoke(middleware.clj:35) >>> at >>> com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__7945$fn__7946.invoke(middleware.clj:60) >>> at >>> com.puppetlabs.middleware.proxy$java.lang.Object$Callable$f8c5758f.call(Unknown >>> Source) >>> at com.yammer.metrics.core.Timer.time(Timer.java:91) >>> at >>> com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__7945.invoke(middleware.clj:58) >>> at >>> com.puppetlabs.middleware$wrap_with_globals$fn__7942.invoke(middleware.clj:43) >>> at ring.adapter.jetty$proxy_handler$fn__2028.invoke(jetty.clj:18) >>> at >>> ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$0.handle(Unknown >>> Source) >>> at >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) >>> at org.eclipse.jetty.server.Server.handle(Server.java:349) >>> at >>> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) >>> at >>> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894) >>> at >>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948) >>> at >>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857) >>> >>> >>> Regarding the java version: >>> ii openjdk-6-jre-headless >>> 6b18-1.8.13-0+squeeze2 OpenJDK Java runtime, using Hotspot JIT >>> (headless) >>> ii openjdk-6-jre-lib >>> 6b18-1.8.13-0+squeeze2 OpenJDK Java runtime (architecture >>> independent libraries) >>> >>> Does it help? >>> >>> Regards, >>> JM >>> >>> >>> On Mon, Aug 6, 2012 at 6:40 PM, Chris Price <chris@puppetlabs.com>wrote: >>> >>>> Hello A_SAAS, >>>> >>>> - Do you see any messages in the file >>>> "/var/log/puppetdb/puppetdb.log"? >>>> - If not, can you try editing the file >>>> "/etc/puppetdb/log4j.properties" and replace "INFO" with "DEBUG", and then >>>> see if you get any messages in the aforementioned log file? >>>> - What version of java are you using? We''ve encountered some >>>> SSL-related problems between very early versions of the Oracle JDK 7 and >>>> puppetdb. >>>> >>>> Thanks! >>>> Chris >>>> >>>> >>>> On Monday, August 6, 2012 1:51:07 AM UTC-7, A_SAAS wrote: >>>>> >>>>> Hi everyone, >>>>> >>>>> >>>>> We are using collectd collecting system information, I am trying to >>>>> setup the postgres plugin for puppetdb (http://collectd.org/wiki/** >>>>> index.php/Plugin:PostgreSQL<http://collectd.org/wiki/index.php/Plugin:PostgreSQL>), >>>>> what could be the relevant value to get? I am trying to see what is going >>>>> on because lately sometime I get some "Sun Aug 05 10:39:12 +0200 2012 >>>>> Puppet (err): Could not retrieve catalog from remote server: Error 400 on >>>>> SERVER: Failed to submit ''replace catalog'' command for XXXXXXXXXX to >>>>> PuppetDB at puppetmaster.fqdn:8081: [500 Server Error] " >>>>> >>>>> >>>>> I am using puppet: >>>>> ii puppet 2.7.18-1puppetlabs1 >>>>> Centralized configuration management - agent startup and compatibility >>>>> scripts >>>>> ii puppet-common 2.7.18-1puppetlabs1 >>>>> Centralized configuration management >>>>> ii puppet-dashboard 1.2.10-1puppetlabs2 Dashboard >>>>> for Puppet >>>>> ii puppetdb 0.9.2-1puppetlabs2 >>>>> PuppetDB Centralized Storage. >>>>> ii puppetdb-terminus 0.9.2-1puppetlabs2 Connect >>>>> Puppet to PuppetDB by setting up a terminus for PuppetDB. >>>>> ii puppetmaster 2.7.18-1puppetlabs1 >>>>> Centralized configuration management - master startup and compatibility >>>>> scripts >>>>> ii puppetmaster-common 2.7.18-1puppetlabs1 Puppet master >>>>> common scripts >>>>> >>>>> >>>>> On debian squeeze for handling 120 server and the puppetmaster is a >>>>> 1950 with 4go of memory and it doesn''t seems to be under heavy load. >>>>> >>>>> Regards, >>>>> JM >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Puppet Users" group. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msg/puppet-users/-/JHdLQ5zvobAJ. >>>> To post to this group, send email to puppet-users@googlegroups.com. >>>> To unsubscribe from this group, send email to >>>> puppet-users+unsubscribe@googlegroups.com. >>>> For more options, visit this group at >>>> http://groups.google.com/group/puppet-users?hl=en. >>>> >>> >>> >> >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/eoIR0s_RTDQJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Hi,
The configuration files are attached.
Here is the postgres login:
[root@puppetmaster]:/data/local/postgresql/dumps # psql -h localhost
puppetdb puppetdb
psql (8.4.12)
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)
Type "help" for help.
puppetdb=> \l
List of databases
Name | Owner | Encoding | Collation | Ctype | Access privileges
-----------+----------+-----------+-----------+-------+-----------------------
postgres | postgres | SQL_ASCII | C | C |
puppetdb | puppetdb | SQL_ASCII | C | C |
template0 | postgres | SQL_ASCII | C | C | =c/postgres
:
postgres=CTc/postgres
template1 | postgres | SQL_ASCII | C | C | =c/postgres
:
postgres=CTc/postgres
(4 rows)
puppetdb=> \d
List of relations
Schema | Name | Type | Owner
--------+-------------------------+-------+----------
public | catalog_resources | table | puppetdb
public | catalogs | table | puppetdb
public | certname_catalogs | table | puppetdb
public | certname_facts | table | puppetdb
public | certname_facts_metadata | table | puppetdb
public | certnames | table | puppetdb
public | classes | table | puppetdb
public | edges | table | puppetdb
public | resource_params | table | puppetdb
public | schema_migrations | table | puppetdb
public | tags | table | puppetdb
And by the way I changed the JVM max heap at 256M and since then no issue.
Regarding the postgres dump file, I''ll be willing to give it to you so
contact me in private to get the URL or let me know where I can put it as
long as nobody could access it publicly.
Regards,
JM
On Tue, Aug 7, 2012 at 6:59 PM, Chris Price <chris@puppetlabs.com> wrote:
> JM,
>
> Hmm... so, those versions of Java seem fine--those are probably the ones
> we''ve done the most testing with.
>
> So, the error message that you sent indicates that the JVM is running out
> of RAM. This could possibly indicate that you are still using the embedded
> database instead of postgres--the embedded database uses a lot of RAM. Can
> we see what your database.ini file looks like? In fact, perhaps we can
> take a look at all of the files in your /etc/puppetdb/conf.d directory?
>
> You could also connect directly to your postgres database using psql or
> pgadmin, and check to see if the puppetdb tables exist and contain data.
>
> The only other possibility that I can think of would be if one or more of
> your nodes has an extremely large catalog, and that the puppetdb JVM
> instance doesn''t have enough memory to process the catalog. If
that''s the
> case, we can try increasing your JVM max heap space in
> /etc/default/puppetdb... but we would be really interested in collecting
> some data from you about what that catalog looks like (for our own testing
> and debugging purposes) if it turns out that that is the culprit.
>
> Thanks!
> Chris
>
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Thanks JM. Your config files look good and the existence of all of those tables in your puppetdb database certainly makes it look as though puppetdb is communicating with postgres properly. Since increasing the heap size seems to have gotten you past the issues for now, my next guess is that it is some combination of catalog size and number of concurrent agent requests, and perhaps number of cores on your machine. We would love to investigate further, so if you are willing to share a postgres dump with us that would be fantastic! I''ll e-mail you off-list to follow up on that. I''d also be interested in seeing the output of ''cat /proc/cpuinfo''. Thanks! Chris On Wednesday, August 8, 2012 1:08:09 AM UTC-7, A_SAAS wrote:> > Hi, > > The configuration files are attached. > > Here is the postgres login: > [root@puppetmaster]:/data/local/postgresql/dumps # psql -h localhost > puppetdb puppetdb > psql (8.4.12) > SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256) > Type "help" for help. > > puppetdb=> \l > List of databases > Name | Owner | Encoding | Collation | Ctype | Access privileges > > -----------+----------+-----------+-----------+-------+----------------------- > postgres | postgres | SQL_ASCII | C | C | > puppetdb | puppetdb | SQL_ASCII | C | C | > template0 | postgres | SQL_ASCII | C | C | =c/postgres > : > postgres=CTc/postgres > template1 | postgres | SQL_ASCII | C | C | =c/postgres > : > postgres=CTc/postgres > (4 rows) > puppetdb=> \d > List of relations > Schema | Name | Type | Owner > --------+-------------------------+-------+---------- > public | catalog_resources | table | puppetdb > public | catalogs | table | puppetdb > public | certname_catalogs | table | puppetdb > public | certname_facts | table | puppetdb > public | certname_facts_metadata | table | puppetdb > public | certnames | table | puppetdb > public | classes | table | puppetdb > public | edges | table | puppetdb > public | resource_params | table | puppetdb > public | schema_migrations | table | puppetdb > public | tags | table | puppetdb > > > And by the way I changed the JVM max heap at 256M and since then no issue. > Regarding the postgres dump file, I''ll be willing to give it to you so > contact me in private to get the URL or let me know where I can put it as > long as nobody could access it publicly. > > > Regards, > JM > > > > > On Tue, Aug 7, 2012 at 6:59 PM, Chris Price <ch...@puppetlabs.com<javascript:> > > wrote: > >> JM, >> >> Hmm... so, those versions of Java seem fine--those are probably the ones >> we''ve done the most testing with. >> >> So, the error message that you sent indicates that the JVM is running out >> of RAM. This could possibly indicate that you are still using the embedded >> database instead of postgres--the embedded database uses a lot of RAM. Can >> we see what your database.ini file looks like? In fact, perhaps we can >> take a look at all of the files in your /etc/puppetdb/conf.d directory? >> >> You could also connect directly to your postgres database using psql or >> pgadmin, and check to see if the puppetdb tables exist and contain data. >> >> The only other possibility that I can think of would be if one or more of >> your nodes has an extremely large catalog, and that the puppetdb JVM >> instance doesn''t have enough memory to process the catalog. If that''s the >> case, we can try increasing your JVM max heap space in >> /etc/default/puppetdb... but we would be really interested in collecting >> some data from you about what that catalog looks like (for our own testing >> and debugging purposes) if it turns out that that is the culprit. >> >> Thanks! >> Chris >> >>On Wednesday, August 8, 2012 1:08:09 AM UTC-7, A_SAAS wrote:> > Hi, > > The configuration files are attached. > > Here is the postgres login: > [root@puppetmaster]:/data/local/postgresql/dumps # psql -h localhost > puppetdb puppetdb > psql (8.4.12) > SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256) > Type "help" for help. > > puppetdb=> \l > List of databases > Name | Owner | Encoding | Collation | Ctype | Access privileges > > -----------+----------+-----------+-----------+-------+----------------------- > postgres | postgres | SQL_ASCII | C | C | > puppetdb | puppetdb | SQL_ASCII | C | C | > template0 | postgres | SQL_ASCII | C | C | =c/postgres > : > postgres=CTc/postgres > template1 | postgres | SQL_ASCII | C | C | =c/postgres > : > postgres=CTc/postgres > (4 rows) > puppetdb=> \d > List of relations > Schema | Name | Type | Owner > --------+-------------------------+-------+---------- > public | catalog_resources | table | puppetdb > public | catalogs | table | puppetdb > public | certname_catalogs | table | puppetdb > public | certname_facts | table | puppetdb > public | certname_facts_metadata | table | puppetdb > public | certnames | table | puppetdb > public | classes | table | puppetdb > public | edges | table | puppetdb > public | resource_params | table | puppetdb > public | schema_migrations | table | puppetdb > public | tags | table | puppetdb > > > And by the way I changed the JVM max heap at 256M and since then no issue. > Regarding the postgres dump file, I''ll be willing to give it to you so > contact me in private to get the URL or let me know where I can put it as > long as nobody could access it publicly. > > > Regards, > JM > > > > > On Tue, Aug 7, 2012 at 6:59 PM, Chris Price <ch...@puppetlabs.com<javascript:> > > wrote: > >> JM, >> >> Hmm... so, those versions of Java seem fine--those are probably the ones >> we''ve done the most testing with. >> >> So, the error message that you sent indicates that the JVM is running out >> of RAM. This could possibly indicate that you are still using the embedded >> database instead of postgres--the embedded database uses a lot of RAM. Can >> we see what your database.ini file looks like? In fact, perhaps we can >> take a look at all of the files in your /etc/puppetdb/conf.d directory? >> >> You could also connect directly to your postgres database using psql or >> pgadmin, and check to see if the puppetdb tables exist and contain data. >> >> The only other possibility that I can think of would be if one or more of >> your nodes has an extremely large catalog, and that the puppetdb JVM >> instance doesn''t have enough memory to process the catalog. If that''s the >> case, we can try increasing your JVM max heap space in >> /etc/default/puppetdb... but we would be really interested in collecting >> some data from you about what that catalog looks like (for our own testing >> and debugging purposes) if it turns out that that is the culprit. >> >> Thanks! >> Chris >> >>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/zwXstXYZfFgJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.