Mario Lassnig
2011-Dec-05 08:33 UTC
[Puppet Users] puppet kick problem: host is already running
Hi, I’m managing several machines with a 2.7.6 master, and 2.7.1 and 2.7.6 mixed, not yet all fully upgraded clients. On the puppet master, I want to kick a node but it exits with error code 3, which doesn’t seem right. The agent seems to return the wrong message. <hidden>:~$ puppet kick <hidden> --trace --debug --foreground Triggering <hidden> Getting status status is running Host <hidden> is already running <hidden> finished with exit code 3 Failed: <hidden> ( And I know that the host is not doing anything because I''ve got both node and puppet agent monitored ) An example of a good kick on another node: <hidden>:~$ puppet kick <hidden> --trace --debug --foreground Triggering <hidden> Getting status status is success <hidden> finished with exit code 0 Finished The really weird thing is, that I have several other nodes where this works perfectly (2.7.1, 2.7.6, doesn’t matter), and a few others where it doesn’t (2.7.1, 2.7.6, doesn’t matter). The nodes are configured exactly the same (auth.conf, puppet.conf, empty namespace.auth), and there really is nothing fancy going on. Several reinstalls of the gems and wiping of /var/lib/puppet didn’t help. It’s a mystery to me. That’s the auth.conf I’m using for testing: path / auth any allow * And here’s the puppet.conf (the same on master and all agents) [main] logdir = /var/log/puppet rundir = /var/run/puppet ssldir = $vardir/ssl [agent] server = <hidden> classfile = $vardir/classes.txt report = true listen = true localconfig = $vardir/localconfig preferred_serialization_format = yaml [master] certname = <hidden> reports = http, store reporturl = http://<hidden>:3000/reports/upload pluginsync = true storeconfigs = true dbadapter = mysql dbuser = root dbpassword dbserver = localhost dbsocket = /var/lib/mysql/mysql.sock I don''t know how to go about fixing/debugging this. I already tried to get the raw output with curl, but it wasn''t helpful (In one case I get the ruby object telling me it''s in status running, in the other case the successful report.) Please advice. Thanks, Mario -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Pavel Weber
2012-Jan-02 15:51 UTC
Re: [Puppet Users] puppet kick problem: host is already running
Hi Mario, I have exactly the same problem you described. Have you got the solution for this? I didn''t find any reply for your question. What we found that there is the puppetdlock file in directory: /var/lib/puppet/state on the host. If the file is there then it shows the message while running kick from master: Getting status status is running Host<hidden> is already running <hidden> finished with exit code 3 Failed:<hidden> So we have another hosts where everything works for the same puppet version, but slightly different ruby packages. best regards, Pavel On 12/05/2011 09:33 AM, Mario Lassnig wrote:> Hi, > > I’m managing several machines with a 2.7.6 master, and 2.7.1 and 2.7.6 > mixed, not yet all fully upgraded clients. > > On the puppet master, I want to kick a node but it exits with error > code 3, which doesn’t seem right. The agent seems to return the wrong > message. > > <hidden>:~$ puppet kick<hidden> --trace --debug --foreground > Triggering<hidden> > Getting status > status is running > Host<hidden> is already running > <hidden> finished with exit code 3 > Failed:<hidden> > > ( And I know that the host is not doing anything because I''ve got both > node and puppet agent monitored ) > > An example of a good kick on another node: > > <hidden>:~$ puppet kick<hidden> --trace --debug --foreground > Triggering<hidden> > Getting status > status is success > <hidden> finished with exit code 0 > Finished > > > The really weird thing is, that I have several other nodes where this > works perfectly (2.7.1, 2.7.6, doesn’t matter), and a few others where > it doesn’t (2.7.1, 2.7.6, doesn’t matter). The nodes are configured > exactly the same (auth.conf, puppet.conf, empty namespace.auth), and > there really is nothing fancy going on. Several reinstalls of the gems > and wiping of /var/lib/puppet didn’t help. It’s a mystery to me. > > That’s the auth.conf I’m using for testing: > > path / > auth any > allow * > > > And here’s the puppet.conf (the same on master and all agents) > > [main] > logdir = /var/log/puppet > rundir = /var/run/puppet > ssldir = $vardir/ssl > [agent] > server =<hidden> > classfile = $vardir/classes.txt > report = true > listen = true > localconfig = $vardir/localconfig > preferred_serialization_format = yaml > [master] > certname =<hidden> > reports = http, store > reporturl = http://<hidden>:3000/reports/upload > pluginsync = true > storeconfigs = true > dbadapter = mysql > dbuser = root > dbpassword > dbserver = localhost > dbsocket = /var/lib/mysql/mysql.sock > > > I don''t know how to go about fixing/debugging this. I already tried to > get the raw output with curl, but it wasn''t helpful (In one case I get > the ruby object telling me it''s in status running, in the other case > the successful report.) > > Please advice. > > Thanks, > > Mario >-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Jo Rhett
2012-Jan-02 18:46 UTC
Re: [Puppet Users] puppet kick problem: host is already running
Check your kernel and compare against this bug: https://projects.puppetlabs.com/issues/10418 It sounds to me like your puppet client is hung, which is why the kick isn''t working. So the problem isn''t the kick, it''s the client state. On Jan 2, 2012, at 7:51 AM, Pavel Weber wrote:> I have exactly the same problem you described. Have you got the solution for this? I didn''t find any reply for your question. What we found that there is the puppetdlock file in directory: /var/lib/puppet/state on the host. If the file is there then it shows the message while running kick from master: > > Getting status > status is running > Host<hidden> is already running > <hidden> finished with exit code 3 > Failed:<hidden> > > So we have another hosts where everything works for the same puppet version, but slightly different ruby packages. > > best regards, > > Pavel > > > > On 12/05/2011 09:33 AM, Mario Lassnig wrote: >> Hi, >> >> I’m managing several machines with a 2.7.6 master, and 2.7.1 and 2.7.6 >> mixed, not yet all fully upgraded clients. >> >> On the puppet master, I want to kick a node but it exits with error >> code 3, which doesn’t seem right. The agent seems to return the wrong >> message. >> >> <hidden>:~$ puppet kick<hidden> --trace --debug --foreground >> Triggering<hidden> >> Getting status >> status is running >> Host<hidden> is already running >> <hidden> finished with exit code 3 >> Failed:<hidden> >> >> ( And I know that the host is not doing anything because I''ve got both >> node and puppet agent monitored ) >> >> An example of a good kick on another node: >> >> <hidden>:~$ puppet kick<hidden> --trace --debug --foreground >> Triggering<hidden> >> Getting status >> status is success >> <hidden> finished with exit code 0 >> Finished >> >> >> The really weird thing is, that I have several other nodes where this >> works perfectly (2.7.1, 2.7.6, doesn’t matter), and a few others where >> it doesn’t (2.7.1, 2.7.6, doesn’t matter). The nodes are configured >> exactly the same (auth.conf, puppet.conf, empty namespace.auth), and >> there really is nothing fancy going on. Several reinstalls of the gems >> and wiping of /var/lib/puppet didn’t help. It’s a mystery to me. >> >> That’s the auth.conf I’m using for testing: >> >> path / >> auth any >> allow * >> >> >> And here’s the puppet.conf (the same on master and all agents) >> >> [main] >> logdir = /var/log/puppet >> rundir = /var/run/puppet >> ssldir = $vardir/ssl >> [agent] >> server =<hidden> >> classfile = $vardir/classes.txt >> report = true >> listen = true >> localconfig = $vardir/localconfig >> preferred_serialization_format = yaml >> [master] >> certname =<hidden> >> reports = http, store >> reporturl = http://<hidden>:3000/reports/upload >> pluginsync = true >> storeconfigs = true >> dbadapter = mysql >> dbuser = root >> dbpassword >> dbserver = localhost >> dbsocket = /var/lib/mysql/mysql.sock >> >> >> I don''t know how to go about fixing/debugging this. I already tried to >> get the raw output with curl, but it wasn''t helpful (In one case I get >> the ruby object telling me it''s in status running, in the other case >> the successful report.) >> >> Please advice. >> >> Thanks, >> >> Mario >> > > -- > You received this message because you are subscribed to the Google Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en. >-- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.
Pavel Weber
2012-Jan-03 09:04 UTC
Re: [Puppet Users] puppet kick problem: host is already running
Hi Jo, Thanks a lot for prompt reply. Exactly the problem is the kernel 2.6.18-274.12.1.el5. Very annoying. best regards, Pavel On 01/02/2012 07:46 PM, Jo Rhett wrote:> Check your kernel and compare against this bug: > https://projects.puppetlabs.com/issues/10418 > * > * > It sounds to me like your puppet client is hung, which is why the kick > isn''t working. So the problem isn''t the kick, it''s the client state. > > On Jan 2, 2012, at 7:51 AM, Pavel Weber wrote: >> I have exactly the same problem you described. Have you got the >> solution for this? I didn''t find any reply for your question. What we >> found that there is the puppetdlock file in directory: >> /var/lib/puppet/state on the host. If the file is there then it >> shows the message while running kick from master: >> >> Getting status >> status is running >> Host<hidden> is already running >> <hidden> finished with exit code 3 >> Failed:<hidden> >> >> So we have another hosts where everything works for the same puppet >> version, but slightly different ruby packages. >> >> best regards, >> >> Pavel >> >> >> >> On 12/05/2011 09:33 AM, Mario Lassnig wrote: >>> Hi, >>> >>> I’m managing several machines with a 2.7.6 master, and 2.7.1 and 2.7.6 >>> mixed, not yet all fully upgraded clients. >>> >>> On the puppet master, I want to kick a node but it exits with error >>> code 3, which doesn’t seem right. The agent seems to return the wrong >>> message. >>> >>> <hidden>:~$ puppet kick<hidden> --trace --debug --foreground >>> Triggering<hidden> >>> Getting status >>> status is running >>> Host<hidden> is already running >>> <hidden> finished with exit code 3 >>> Failed:<hidden> >>> >>> ( And I know that the host is not doing anything because I''ve got both >>> node and puppet agent monitored ) >>> >>> An example of a good kick on another node: >>> >>> <hidden>:~$ puppet kick<hidden> --trace --debug --foreground >>> Triggering<hidden> >>> Getting status >>> status is success >>> <hidden> finished with exit code 0 >>> Finished >>> >>> >>> The really weird thing is, that I have several other nodes where this >>> works perfectly (2.7.1, 2.7.6, doesn’t matter), and a few others where >>> it doesn’t (2.7.1, 2.7.6, doesn’t matter). The nodes are configured >>> exactly the same (auth.conf, puppet.conf, empty namespace.auth), and >>> there really is nothing fancy going on. Several reinstalls of the gems >>> and wiping of /var/lib/puppet didn’t help. It’s a mystery to me. >>> >>> That’s the auth.conf I’m using for testing: >>> >>> path / >>> auth any >>> allow * >>> >>> >>> And here’s the puppet.conf (the same on master and all agents) >>> >>> [main] >>> logdir = /var/log/puppet >>> rundir = /var/run/puppet >>> ssldir = $vardir/ssl >>> [agent] >>> server =<hidden> >>> classfile = $vardir/classes.txt >>> report = true >>> listen = true >>> localconfig = $vardir/localconfig >>> preferred_serialization_format = yaml >>> [master] >>> certname =<hidden> >>> reports = http, store >>> reporturl = http://<hidden>:3000/reports/upload >>> pluginsync = true >>> storeconfigs = true >>> dbadapter = mysql >>> dbuser = root >>> dbpassword >>> dbserver = localhost >>> dbsocket = /var/lib/mysql/mysql.sock >>> >>> >>> I don''t know how to go about fixing/debugging this. I already tried to >>> get the raw output with curl, but it wasn''t helpful (In one case I get >>> the ruby object telling me it''s in status running, in the other case >>> the successful report.) >>> >>> Please advice. >>> >>> Thanks, >>> >>> Mario >>> >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Puppet Users" group. >> To post to this group, send email to puppet-users@googlegroups.com >> <mailto:puppet-users@googlegroups.com>. >> To unsubscribe from this group, send email to >> puppet-users+unsubscribe@googlegroups.com >> <mailto:puppet-users+unsubscribe@googlegroups.com>. >> For more options, visit this group at >> http://groups.google.com/group/puppet-users?hl=en. >> > > -- > Jo Rhett > Net Consonance : consonant endings by net philanthropy, open source > and other randomness > > -- > You received this message because you are subscribed to the Google > Groups "Puppet Users" group. > To post to this group, send email to puppet-users@googlegroups.com. > To unsubscribe from this group, send email to > puppet-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/puppet-users?hl=en.-- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To post to this group, send email to puppet-users@googlegroups.com. To unsubscribe from this group, send email to puppet-users+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.