Cody Robertson
2013-Jan-09 01:37 UTC
[Puppet Users] Difficulty debugging crashing PuppetDB
Hello! How is everyone this splendid evening? I''ve recently migrated to the latest Puppet and PuppetDB (using the build in database) however I''m noticing PuppetDB keeps crashing without any errors that I can find in the logs. I''ve ran it in the foreground using the puppetdb-foreground command however it simply exits after awhile. The only thing I can consistently do is see it crash - I can''t get any useful debug information beyond that. Can anyone shed some light on how I should go about this? Thank you! puppet-3.0.2-1.el6.noarch puppet-dashboard-1.2.17-1.el6.noarch puppetdb-1.0.5-1.el6.noarch puppetdb-terminus-1.0.5-1.el6.noarch puppetlabs-release-6-6.noarch puppet-server-3.0.2-1.el6.noarch -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/iT9Xyi0toh0J. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Matthew Burgess
2013-Jan-09 11:11 UTC
Re: [Puppet Users] Difficulty debugging crashing PuppetDB
On Wed, Jan 9, 2013 at 1:37 AM, Cody Robertson <codyhawkhost@gmail.com> wrote:> Hello! How is everyone this splendid evening? > > I''ve recently migrated to the latest Puppet and PuppetDB (using the build in > database) however I''m noticing PuppetDB keeps crashing without any errors > that I can find in the logs. I''ve ran it in the foreground using the > puppetdb-foreground command however it simply exits after awhile. The only > thing I can consistently do is see it crash - I can''t get any useful debug > information beyond that. > > Can anyone shed some light on how I should go about this? Thank you!What does /var/log/messages say. Just a stab in the dark, but if your server is short of memory, then the kernel''s oom killer may be targetting the puppetdb process; that would certainly be evident in your /var/log/messages output. If that''s not the culprit, then attaching ''strace'' to the puppetdb process might be informative (strace -p <pid>). Regards, Matt. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Chris Price
2013-Jan-09 19:53 UTC
Re: [Puppet Users] Difficulty debugging crashing PuppetDB
You could also try `puppetdb-foreground --debug` for some more verbose logging. And, is there anything at all in /var/log/puppetdb? On Wednesday, January 9, 2013 3:11:19 AM UTC-8, Matthew Burgess wrote:> > On Wed, Jan 9, 2013 at 1:37 AM, Cody Robertson <codyha...@gmail.com<javascript:>> > wrote: > > Hello! How is everyone this splendid evening? > > > > I''ve recently migrated to the latest Puppet and PuppetDB (using the > build in > > database) however I''m noticing PuppetDB keeps crashing without any > errors > > that I can find in the logs. I''ve ran it in the foreground using the > > puppetdb-foreground command however it simply exits after awhile. The > only > > thing I can consistently do is see it crash - I can''t get any useful > debug > > information beyond that. > > > > Can anyone shed some light on how I should go about this? Thank you! > > What does /var/log/messages say. Just a stab in the dark, but if your > server is short of memory, then the kernel''s oom killer may be > targetting the puppetdb process; that would certainly be evident in > your /var/log/messages output. > > If that''s not the culprit, then attaching ''strace'' to the puppetdb > process might be informative (strace -p <pid>). > > Regards, > > Matt. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/r4X9JqrxEDwJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Cody Robertson
2013-Jan-09 23:07 UTC
Re: [Puppet Users] Difficulty debugging crashing PuppetDB
There is nothing in the logs as previously noted. It simply crashed quietly.
This is the same for when I''m running it in the foreground with --debug
or
when it''s a daemon. It simply quietly crashes.
--
013-01-09 18:00:15,841 DEBUG [command-proc-89]
[bonecp.PreparedStatementHandle] SELECT timestamp FROM certname_catalogs
WHERE certname=''typhoon.xxx.com'' ORDER BY timestamp DESC LIMIT
1
2013-01-09 18:00:16,185 DEBUG [command-proc-89]
[bonecp.PreparedStatementHandle] SELECT 1 FROM catalogs WHERE
hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923'' LIMIT 1
2013-01-09 18:00:16,185 DEBUG [command-proc-89] [bonecp.StatementHandle]
UPDATE catalogs SET api_version=1,
catalog_version=''1357754062'' WHERE
hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923''
2013-01-09 18:00:16,185 DEBUG [command-proc-89] [bonecp.StatementHandle]
DELETE FROM certname_catalogs WHERE certname=''typhoon.xxx.com''
2013-01-09 18:00:16,186 DEBUG [command-proc-89]
[bonecp.PreparedStatementHandle] INSERT INTO certname_catalogs
(certname,catalog,timestamp) VALUES
(''typhoon.xxx.com'',''b9915aef874b1a291e32f1b7cbe0efa9848fb923'',2013-01-09
18:00:15.815)
2013-01-09 18:00:16,186 INFO [command-proc-89] [puppetdb.command]
[7779017b-5be2-415d-afd6-264d6d4d789e] [replace catalog] typhoon.xxx.com
sh-4.1#
--
I''ll attempt to attach a strace to it however it''s so
remarkably verbose
it''s always a treat to sift through it. Is there an easy way to
increase
the verbosity of puppetDB perhaps?
-Cody
On Wednesday, January 9, 2013 6:11:19 AM UTC-5, Matthew Burgess
wrote:>
> On Wed, Jan 9, 2013 at 1:37 AM, Cody Robertson
<codyha...@gmail.com<javascript:>>
> wrote:
> > Hello! How is everyone this splendid evening?
> >
> > I''ve recently migrated to the latest Puppet and PuppetDB
(using the
> build in
> > database) however I''m noticing PuppetDB keeps crashing
without any
> errors
> > that I can find in the logs. I''ve ran it in the foreground
using the
> > puppetdb-foreground command however it simply exits after awhile. The
> only
> > thing I can consistently do is see it crash - I can''t get any
useful
> debug
> > information beyond that.
> >
> > Can anyone shed some light on how I should go about this? Thank you!
>
> What does /var/log/messages say. Just a stab in the dark, but if your
> server is short of memory, then the kernel''s oom killer may be
> targetting the puppetdb process; that would certainly be evident in
> your /var/log/messages output.
>
> If that''s not the culprit, then attaching
''strace'' to the puppetdb
> process might be informative (strace -p <pid>).
>
> Regards,
>
> Matt.
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To view this discussion on the web visit
https://groups.google.com/d/msg/puppet-users/-/vADEbpPDw7IJ.
To post to this group, send email to puppet-users@googlegroups.com.
To unsubscribe from this group, send email to
puppet-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/puppet-users?hl=en.
Ken Barber
2013-Jan-09 23:43 UTC
Re: [Puppet Users] Difficulty debugging crashing PuppetDB
Do you get a core dump? Does it seriously just silently ''stop'' with no SEGV or anything - even in the forground? On Wed, Jan 9, 2013 at 11:07 PM, Cody Robertson <codyhawkhost@gmail.com> wrote:> There is nothing in the logs as previously noted. It simply crashed quietly. > > This is the same for when I''m running it in the foreground with --debug or > when it''s a daemon. It simply quietly crashes. > > -- > 013-01-09 18:00:15,841 DEBUG [command-proc-89] > [bonecp.PreparedStatementHandle] SELECT timestamp FROM certname_catalogs > WHERE certname=''typhoon.xxx.com'' ORDER BY timestamp DESC LIMIT 1 > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] > [bonecp.PreparedStatementHandle] SELECT 1 FROM catalogs WHERE > hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923'' LIMIT 1 > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] [bonecp.StatementHandle] > UPDATE catalogs SET api_version=1, catalog_version=''1357754062'' WHERE > hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923'' > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] [bonecp.StatementHandle] > DELETE FROM certname_catalogs WHERE certname=''typhoon.xxx.com'' > 2013-01-09 18:00:16,186 DEBUG [command-proc-89] > [bonecp.PreparedStatementHandle] INSERT INTO certname_catalogs > (certname,catalog,timestamp) VALUES > (''typhoon.xxx.com'',''b9915aef874b1a291e32f1b7cbe0efa9848fb923'',2013-01-09 > 18:00:15.815) > 2013-01-09 18:00:16,186 INFO [command-proc-89] [puppetdb.command] > [7779017b-5be2-415d-afd6-264d6d4d789e] [replace catalog] typhoon.xxx.com > sh-4.1# > -- > > I''ll attempt to attach a strace to it however it''s so remarkably verbose > it''s always a treat to sift through it. Is there an easy way to increase the > verbosity of puppetDB perhaps? > > -Cody > > On Wednesday, January 9, 2013 6:11:19 AM UTC-5, Matthew Burgess wrote: >> >> On Wed, Jan 9, 2013 at 1:37 AM, Cody Robertson <codyha...@gmail.com> >> wrote: >> > Hello! How is everyone this splendid evening? >> > >> > I''ve recently migrated to the latest Puppet and PuppetDB (using the >> > build in >> > database) however I''m noticing PuppetDB keeps crashing without any >> > errors >> > that I can find in the logs. I''ve ran it in the foreground using the >> > puppetdb-foreground command however it simply exits after awhile. The >> > only >> > thing I can consistently do is see it crash - I can''t get any useful >> > debug >> > information beyond that. >> > >> > Can anyone shed some light on how I should go about this? Thank you! >> >> What does /var/log/messages say. Just a stab in the dark, but if your >> server is short of memory, then the kernel''s oom killer may be >> targetting the puppetdb process; that would certainly be evident in >> your /var/log/messages output. >> >> If that''s not the culprit, then attaching ''strace'' to the puppetdb >> process might be informative (strace -p <pid>). >> >> Regards, >> >> Matt. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/puppet-users/-/vADEbpPDw7IJ. > > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Cody Robertson
2013-Jan-10 01:30 UTC
Re: [Puppet Users] Difficulty debugging crashing PuppetDB
I have no core dumps however I need to make sure I have it set to allow them. It literally just goes kaput - very strange. I''ve yet to have time to strace it yet today however I did it briefly and it was merely doing a bunch of waits. On Wednesday, January 9, 2013 6:43:05 PM UTC-5, Ken Barber wrote:> > Do you get a core dump? Does it seriously just silently ''stop'' with no > SEGV or anything - even in the forground? > > On Wed, Jan 9, 2013 at 11:07 PM, Cody Robertson <codyha...@gmail.com<javascript:>> > wrote: > > There is nothing in the logs as previously noted. It simply crashed > quietly. > > > > This is the same for when I''m running it in the foreground with --debug > or > > when it''s a daemon. It simply quietly crashes. > > > > -- > > 013-01-09 18:00:15,841 DEBUG [command-proc-89] > > [bonecp.PreparedStatementHandle] SELECT timestamp FROM certname_catalogs > > WHERE certname=''typhoon.xxx.com'' ORDER BY timestamp DESC LIMIT 1 > > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] > > [bonecp.PreparedStatementHandle] SELECT 1 FROM catalogs WHERE > > hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923'' LIMIT 1 > > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] [bonecp.StatementHandle] > > UPDATE catalogs SET api_version=1, catalog_version=''1357754062'' WHERE > > hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923'' > > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] [bonecp.StatementHandle] > > DELETE FROM certname_catalogs WHERE certname=''typhoon.xxx.com'' > > 2013-01-09 18:00:16,186 DEBUG [command-proc-89] > > [bonecp.PreparedStatementHandle] INSERT INTO certname_catalogs > > (certname,catalog,timestamp) VALUES > > (''typhoon.xxx.com'',''b9915aef874b1a291e32f1b7cbe0efa9848fb923'',2013-01-09 > > > 18:00:15.815) > > 2013-01-09 18:00:16,186 INFO [command-proc-89] [puppetdb.command] > > [7779017b-5be2-415d-afd6-264d6d4d789e] [replace catalog] typhoon.xxx.com > > sh-4.1# > > -- > > > > I''ll attempt to attach a strace to it however it''s so remarkably verbose > > it''s always a treat to sift through it. Is there an easy way to increase > the > > verbosity of puppetDB perhaps? > > > > -Cody > > > > On Wednesday, January 9, 2013 6:11:19 AM UTC-5, Matthew Burgess wrote: > >> > >> On Wed, Jan 9, 2013 at 1:37 AM, Cody Robertson <codyha...@gmail.com> > >> wrote: > >> > Hello! How is everyone this splendid evening? > >> > > >> > I''ve recently migrated to the latest Puppet and PuppetDB (using the > >> > build in > >> > database) however I''m noticing PuppetDB keeps crashing without any > >> > errors > >> > that I can find in the logs. I''ve ran it in the foreground using the > >> > puppetdb-foreground command however it simply exits after awhile. The > >> > only > >> > thing I can consistently do is see it crash - I can''t get any useful > >> > debug > >> > information beyond that. > >> > > >> > Can anyone shed some light on how I should go about this? Thank you! > >> > >> What does /var/log/messages say. Just a stab in the dark, but if your > >> server is short of memory, then the kernel''s oom killer may be > >> targetting the puppetdb process; that would certainly be evident in > >> your /var/log/messages output. > >> > >> If that''s not the culprit, then attaching ''strace'' to the puppetdb > >> process might be informative (strace -p <pid>). > >> > >> Regards, > >> > >> Matt. > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "Puppet Users" group. > > To view this discussion on the web visit > > https://groups.google.com/d/msg/puppet-users/-/vADEbpPDw7IJ. > > > > To post to this group, send email to puppet...@googlegroups.com<javascript:>. > > > To unsubscribe from this group, send email to > > puppet-users...@googlegroups.com <javascript:>. > > For more options, visit this group at > > http://groups.google.com/group/puppet-users?hl=en. >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/EcXclwzpWOwJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ken Barber
2013-Jan-10 02:39 UTC
Re: [Puppet Users] Difficulty debugging crashing PuppetDB
> I have no core dumps however I need to make sure I have it set to allow > them.Yeah check ulimit -a for the puppetdb user, might need ulimit -c unlimited or some such.> It literally just goes kaput - very strange. I''ve yet to have time to > strace it yet today however I did it briefly and it was merely doing a bunch > of waits.This was an strace while it was running yeah? Just to be clear ... when you say ''crash'' you mean - the java process totally exits right? Some people use the term to mean different things and I just want to be sure :-). You don''t mean something like - "process is still running, but stops processing stuff do you?" I guess you really want the strace when it ''crashes'' or stops running. What doesn''t make sense really is that a java process either dies and throws an exception (which should get logged) or dies and throws a core dump ... usually with a crash related message as well. I think somehow we don''t have all the nobs on to get the right information ... alas we need more data :-). For it to just stop silently would mean the program itself has called a System.exit() or some such almost as if it was a graceful shutdown, which although not impossible seems really weird :-). ken. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Chris Price
2013-Jan-10 17:15 UTC
Re: [Puppet Users] Difficulty debugging crashing PuppetDB
Cody, Can you provide some details on your OS and JVM flavors / versions? Also, you mentioned there was "nothing in the logs"--does this include the syslog? And are there any other *files* in the /var/log/puppetdb directory? On Wednesday, January 9, 2013 5:30:03 PM UTC-8, Cody Robertson wrote:> > I have no core dumps however I need to make sure I have it set to allow > them. It literally just goes kaput - very strange. I''ve yet to have time to > strace it yet today however I did it briefly and it was merely doing a > bunch of waits. > > On Wednesday, January 9, 2013 6:43:05 PM UTC-5, Ken Barber wrote: >> >> Do you get a core dump? Does it seriously just silently ''stop'' with no >> SEGV or anything - even in the forground? >> >> On Wed, Jan 9, 2013 at 11:07 PM, Cody Robertson <codyha...@gmail.com> >> wrote: >> > There is nothing in the logs as previously noted. It simply crashed >> quietly. >> > >> > This is the same for when I''m running it in the foreground with --debug >> or >> > when it''s a daemon. It simply quietly crashes. >> > >> > -- >> > 013-01-09 18:00:15,841 DEBUG [command-proc-89] >> > [bonecp.PreparedStatementHandle] SELECT timestamp FROM >> certname_catalogs >> > WHERE certname=''typhoon.xxx.com'' ORDER BY timestamp DESC LIMIT 1 >> > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] >> > [bonecp.PreparedStatementHandle] SELECT 1 FROM catalogs WHERE >> > hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923'' LIMIT 1 >> > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] >> [bonecp.StatementHandle] >> > UPDATE catalogs SET api_version=1, catalog_version=''1357754062'' WHERE >> > hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923'' >> > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] >> [bonecp.StatementHandle] >> > DELETE FROM certname_catalogs WHERE certname=''typhoon.xxx.com'' >> > 2013-01-09 18:00:16,186 DEBUG [command-proc-89] >> > [bonecp.PreparedStatementHandle] INSERT INTO certname_catalogs >> > (certname,catalog,timestamp) VALUES >> > (''typhoon.xxx.com'',''b9915aef874b1a291e32f1b7cbe0efa9848fb923'',2013-01-09 >> >> > 18:00:15.815) >> > 2013-01-09 18:00:16,186 INFO [command-proc-89] [puppetdb.command] >> > [7779017b-5be2-415d-afd6-264d6d4d789e] [replace catalog] >> typhoon.xxx.com >> > sh-4.1# >> > -- >> > >> > I''ll attempt to attach a strace to it however it''s so remarkably >> verbose >> > it''s always a treat to sift through it. Is there an easy way to >> increase the >> > verbosity of puppetDB perhaps? >> > >> > -Cody >> > >> > On Wednesday, January 9, 2013 6:11:19 AM UTC-5, Matthew Burgess wrote: >> >> >> >> On Wed, Jan 9, 2013 at 1:37 AM, Cody Robertson <codyha...@gmail.com> >> >> wrote: >> >> > Hello! How is everyone this splendid evening? >> >> > >> >> > I''ve recently migrated to the latest Puppet and PuppetDB (using the >> >> > build in >> >> > database) however I''m noticing PuppetDB keeps crashing without any >> >> > errors >> >> > that I can find in the logs. I''ve ran it in the foreground using the >> >> > puppetdb-foreground command however it simply exits after awhile. >> The >> >> > only >> >> > thing I can consistently do is see it crash - I can''t get any useful >> >> > debug >> >> > information beyond that. >> >> > >> >> > Can anyone shed some light on how I should go about this? Thank you! >> >> >> >> What does /var/log/messages say. Just a stab in the dark, but if your >> >> server is short of memory, then the kernel''s oom killer may be >> >> targetting the puppetdb process; that would certainly be evident in >> >> your /var/log/messages output. >> >> >> >> If that''s not the culprit, then attaching ''strace'' to the puppetdb >> >> process might be informative (strace -p <pid>). >> >> >> >> Regards, >> >> >> >> Matt. >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Puppet Users" group. >> > To view this discussion on the web visit >> > https://groups.google.com/d/msg/puppet-users/-/vADEbpPDw7IJ. >> > >> > To post to this group, send email to puppet...@googlegroups.com. >> > To unsubscribe from this group, send email to >> > puppet-users...@googlegroups.com. >> > For more options, visit this group at >> > http://groups.google.com/group/puppet-users?hl=en. >> >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/4eL0sJzc1IoJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Cody Robertson
2013-Jan-16 18:36 UTC
Re: [Puppet Users] Difficulty debugging crashing PuppetDB
I''m not sure how I missed it but it appears the OOM was killing it (how embarassing). Once I raised the memory thresholds on the VM it was fine. Even though I somehow missed it in the log the catalyst for me finding it was looking at the PuppetDB command line - in particular the "-XX:OnOutOfMemoryError=kill" switch. Sorry for the confusion / time wasting. -Cody On Thursday, January 10, 2013 12:15:16 PM UTC-5, Chris Price wrote:> > Cody, > > Can you provide some details on your OS and JVM flavors / versions? Also, > you mentioned there was "nothing in the logs"--does this include the > syslog? And are there any other *files* in the /var/log/puppetdb directory? > > > On Wednesday, January 9, 2013 5:30:03 PM UTC-8, Cody Robertson wrote: >> >> I have no core dumps however I need to make sure I have it set to allow >> them. It literally just goes kaput - very strange. I''ve yet to have time to >> strace it yet today however I did it briefly and it was merely doing a >> bunch of waits. >> >> On Wednesday, January 9, 2013 6:43:05 PM UTC-5, Ken Barber wrote: >>> >>> Do you get a core dump? Does it seriously just silently ''stop'' with no >>> SEGV or anything - even in the forground? >>> >>> On Wed, Jan 9, 2013 at 11:07 PM, Cody Robertson <codyha...@gmail.com> >>> wrote: >>> > There is nothing in the logs as previously noted. It simply crashed >>> quietly. >>> > >>> > This is the same for when I''m running it in the foreground with >>> --debug or >>> > when it''s a daemon. It simply quietly crashes. >>> > >>> > -- >>> > 013-01-09 18:00:15,841 DEBUG [command-proc-89] >>> > [bonecp.PreparedStatementHandle] SELECT timestamp FROM >>> certname_catalogs >>> > WHERE certname=''typhoon.xxx.com'' ORDER BY timestamp DESC LIMIT 1 >>> > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] >>> > [bonecp.PreparedStatementHandle] SELECT 1 FROM catalogs WHERE >>> > hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923'' LIMIT 1 >>> > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] >>> [bonecp.StatementHandle] >>> > UPDATE catalogs SET api_version=1, catalog_version=''1357754062'' WHERE >>> > hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923'' >>> > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] >>> [bonecp.StatementHandle] >>> > DELETE FROM certname_catalogs WHERE certname=''typhoon.xxx.com'' >>> > 2013-01-09 18:00:16,186 DEBUG [command-proc-89] >>> > [bonecp.PreparedStatementHandle] INSERT INTO certname_catalogs >>> > (certname,catalog,timestamp) VALUES >>> > (''typhoon.xxx.com'',''b9915aef874b1a291e32f1b7cbe0efa9848fb923'',2013-01-09 >>> >>> > 18:00:15.815) >>> > 2013-01-09 18:00:16,186 INFO [command-proc-89] [puppetdb.command] >>> > [7779017b-5be2-415d-afd6-264d6d4d789e] [replace catalog] >>> typhoon.xxx.com >>> > sh-4.1# >>> > -- >>> > >>> > I''ll attempt to attach a strace to it however it''s so remarkably >>> verbose >>> > it''s always a treat to sift through it. Is there an easy way to >>> increase the >>> > verbosity of puppetDB perhaps? >>> > >>> > -Cody >>> > >>> > On Wednesday, January 9, 2013 6:11:19 AM UTC-5, Matthew Burgess wrote: >>> >> >>> >> On Wed, Jan 9, 2013 at 1:37 AM, Cody Robertson <codyha...@gmail.com> >>> >> wrote: >>> >> > Hello! How is everyone this splendid evening? >>> >> > >>> >> > I''ve recently migrated to the latest Puppet and PuppetDB (using the >>> >> > build in >>> >> > database) however I''m noticing PuppetDB keeps crashing without any >>> >> > errors >>> >> > that I can find in the logs. I''ve ran it in the foreground using >>> the >>> >> > puppetdb-foreground command however it simply exits after awhile. >>> The >>> >> > only >>> >> > thing I can consistently do is see it crash - I can''t get any >>> useful >>> >> > debug >>> >> > information beyond that. >>> >> > >>> >> > Can anyone shed some light on how I should go about this? Thank >>> you! >>> >> >>> >> What does /var/log/messages say. Just a stab in the dark, but if >>> your >>> >> server is short of memory, then the kernel''s oom killer may be >>> >> targetting the puppetdb process; that would certainly be evident in >>> >> your /var/log/messages output. >>> >> >>> >> If that''s not the culprit, then attaching ''strace'' to the puppetdb >>> >> process might be informative (strace -p <pid>). >>> >> >>> >> Regards, >>> >> >>> >> Matt. >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups >>> > "Puppet Users" group. >>> > To view this discussion on the web visit >>> > https://groups.google.com/d/msg/puppet-users/-/vADEbpPDw7IJ. >>> > >>> > To post to this group, send email to puppet...@googlegroups.com. >>> > To unsubscribe from this group, send email to >>> > puppet-users...@googlegroups.com. >>> > For more options, visit this group at >>> > http://groups.google.com/group/puppet-users?hl=en. >>> >>-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To view this discussion on the web visit https://groups.google.com/d/msg/puppet-users/-/uPYvHTKzgHAJ. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Ken Barber
2013-Jan-16 19:57 UTC
Re: [Puppet Users] Difficulty debugging crashing PuppetDB
So the log had enough information and it was just an oversight? Thats good to hear. We probably chose to kill the process as apposed to continually running as OOM''s in Java often leave things in an unknown state. We could consider tanuki wrapper/java service wrapper around the process. At least then it would restart, but you might not see there is a problem ;-). We do this currently for ActiveMQ in PE for example. On Wed, Jan 16, 2013 at 6:36 PM, Cody Robertson <codyhawkhost@gmail.com> wrote:> I''m not sure how I missed it but it appears the OOM was killing it (how > embarassing). Once I raised the memory thresholds on the VM it was fine. > Even though I somehow missed it in the log the catalyst for me finding it > was looking at the PuppetDB command line - in particular the > "-XX:OnOutOfMemoryError=kill" switch. > > Sorry for the confusion / time wasting. > > -Cody > > > On Thursday, January 10, 2013 12:15:16 PM UTC-5, Chris Price wrote: >> >> Cody, >> >> Can you provide some details on your OS and JVM flavors / versions? Also, >> you mentioned there was "nothing in the logs"--does this include the syslog? >> And are there any other *files* in the /var/log/puppetdb directory? >> >> >> On Wednesday, January 9, 2013 5:30:03 PM UTC-8, Cody Robertson wrote: >>> >>> I have no core dumps however I need to make sure I have it set to allow >>> them. It literally just goes kaput - very strange. I''ve yet to have time to >>> strace it yet today however I did it briefly and it was merely doing a bunch >>> of waits. >>> >>> On Wednesday, January 9, 2013 6:43:05 PM UTC-5, Ken Barber wrote: >>>> >>>> Do you get a core dump? Does it seriously just silently ''stop'' with no >>>> SEGV or anything - even in the forground? >>>> >>>> On Wed, Jan 9, 2013 at 11:07 PM, Cody Robertson <codyha...@gmail.com> >>>> wrote: >>>> > There is nothing in the logs as previously noted. It simply crashed >>>> > quietly. >>>> > >>>> > This is the same for when I''m running it in the foreground with >>>> > --debug or >>>> > when it''s a daemon. It simply quietly crashes. >>>> > >>>> > -- >>>> > 013-01-09 18:00:15,841 DEBUG [command-proc-89] >>>> > [bonecp.PreparedStatementHandle] SELECT timestamp FROM >>>> > certname_catalogs >>>> > WHERE certname=''typhoon.xxx.com'' ORDER BY timestamp DESC LIMIT 1 >>>> > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] >>>> > [bonecp.PreparedStatementHandle] SELECT 1 FROM catalogs WHERE >>>> > hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923'' LIMIT 1 >>>> > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] >>>> > [bonecp.StatementHandle] >>>> > UPDATE catalogs SET api_version=1, catalog_version=''1357754062'' WHERE >>>> > hash=''b9915aef874b1a291e32f1b7cbe0efa9848fb923'' >>>> > 2013-01-09 18:00:16,185 DEBUG [command-proc-89] >>>> > [bonecp.StatementHandle] >>>> > DELETE FROM certname_catalogs WHERE certname=''typhoon.xxx.com'' >>>> > 2013-01-09 18:00:16,186 DEBUG [command-proc-89] >>>> > [bonecp.PreparedStatementHandle] INSERT INTO certname_catalogs >>>> > (certname,catalog,timestamp) VALUES >>>> > >>>> > (''typhoon.xxx.com'',''b9915aef874b1a291e32f1b7cbe0efa9848fb923'',2013-01-09 >>>> > 18:00:15.815) >>>> > 2013-01-09 18:00:16,186 INFO [command-proc-89] [puppetdb.command] >>>> > [7779017b-5be2-415d-afd6-264d6d4d789e] [replace catalog] >>>> > typhoon.xxx.com >>>> > sh-4.1# >>>> > -- >>>> > >>>> > I''ll attempt to attach a strace to it however it''s so remarkably >>>> > verbose >>>> > it''s always a treat to sift through it. Is there an easy way to >>>> > increase the >>>> > verbosity of puppetDB perhaps? >>>> > >>>> > -Cody >>>> > >>>> > On Wednesday, January 9, 2013 6:11:19 AM UTC-5, Matthew Burgess wrote: >>>> >> >>>> >> On Wed, Jan 9, 2013 at 1:37 AM, Cody Robertson <codyha...@gmail.com> >>>> >> wrote: >>>> >> > Hello! How is everyone this splendid evening? >>>> >> > >>>> >> > I''ve recently migrated to the latest Puppet and PuppetDB (using the >>>> >> > build in >>>> >> > database) however I''m noticing PuppetDB keeps crashing without any >>>> >> > errors >>>> >> > that I can find in the logs. I''ve ran it in the foreground using >>>> >> > the >>>> >> > puppetdb-foreground command however it simply exits after awhile. >>>> >> > The >>>> >> > only >>>> >> > thing I can consistently do is see it crash - I can''t get any >>>> >> > useful >>>> >> > debug >>>> >> > information beyond that. >>>> >> > >>>> >> > Can anyone shed some light on how I should go about this? Thank >>>> >> > you! >>>> >> >>>> >> What does /var/log/messages say. Just a stab in the dark, but if >>>> >> your >>>> >> server is short of memory, then the kernel''s oom killer may be >>>> >> targetting the puppetdb process; that would certainly be evident in >>>> >> your /var/log/messages output. >>>> >> >>>> >> If that''s not the culprit, then attaching ''strace'' to the puppetdb >>>> >> process might be informative (strace -p <pid>). >>>> >> >>>> >> Regards, >>>> >> >>>> >> Matt. >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> > Groups >>>> > "Puppet Users" group. >>>> > To view this discussion on the web visit >>>> > https://groups.google.com/d/msg/puppet-users/-/vADEbpPDw7IJ. >>>> > >>>> > To post to this group, send email to puppet...@googlegroups.com. >>>> > To unsubscribe from this group, send email to >>>> > puppet-users...@googlegroups.com. >>>> > For more options, visit this group at >>>> > http://groups.google.com/group/puppet-users?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Puppet Users" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/puppet-users/-/uPYvHTKzgHAJ. > > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.