Guillaume Blairon
2014-Sep-03 14:17 UTC
[Puppet Users] PuppetDB throwing tons of exceptions
Hi, We're using puppetdb 2.1.0 and puppet 3.6.2 on CentOS. PuppetDB has been working fine for a few weeks. Yesterday we added a new puppet master to handle our ~400 servers. When we added the new master in the SRV records, PuppetDB began to show an erratic behaviour : - the number of iops on /var has raised from 50/sec to 500+/sec - the /var/lib/puppetdb/mq/localhost/scheduler is being filled with db-*.log files until filesystem gets full - the logs are filled with this type of messages (~1 per second) : --- 2014-09-03 15:48:51,006 ERROR [c.p.p.command] [24506b0f-6fcd-4009-9761-bfe3652ee1bd] [replace facts] Retrying after attempt 5, due to: java.lang.IllegalArgumentException: No method in multimethod 'process-command!' for dispatch value: ["replace facts" 3] java.lang.IllegalArgumentException: No method in multimethod 'process-command!' for dispatch value: ["replace facts" 3] at clojure.lang.MultiFn.getFn(MultiFn.java:160) ~[puppetdb.jar:na] at clojure.lang.MultiFn.invoke(MultiFn.java:231) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$produce_message_handler$fn__10782.invoke(command.clj:630) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__10731$fn__10735.invoke(command.clj:537) ~[na:na] at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__10731.invoke(command.clj:536) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__10716$fn__10717.invoke(command.clj:490) ~[na:na] at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na] at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na] at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__10716.invoke(command.clj:489) ~[na:na] at com.puppetlabs.puppetdb.command$wrap_with_command_parser$fn__10726.invoke(command.clj:512) [na:na] at com.puppetlabs.puppetdb.command$wrap_with_meter$fn__10706.invoke(command.clj:450) [na:na] at com.puppetlabs.puppetdb.command$wrap_with_thread_name$fn__10740.invoke(command.clj:552) [na:na] at com.puppetlabs.mq$create_message_listener$reify__9909.onMessage(mq.clj:270) [na:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) [puppetdb.jar:na] at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) [puppetdb.jar:na] at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) [puppetdb.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] --- Until now, the only solution we have found is to stop puppetdb, remove /var/lib/puppetdb/mq/localhost and restart puppetdb. But it's only buying us a few hours before the filesystem gets full again. I'm not actually sure that the problem has a relation with the new master, but that's more or less when the problem started. Do you have an idea of what's happening here ? Is there something else I could check ? Cheers, -- Guillaume -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/85fb831e-4bf4-4d3d-acd3-a9d8892839e4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.