----- Original Message -----
From: "James Hogarth" <james.hogarth at gmail.com>
To: "centos" <centos at centos.org>
Sent: Tuesday, August 29, 2017 2:03:44 PM
Subject: Re: [CentOS] Syncronize systemctl status with reality?
On 29 Aug 2017 17:58, "Leroy Tennison" <leroy at
datavoiceint.com> wrote:
The particular issue is with puppetmaster (which admittedly takes 4 minutes
to actually start, setting TimeoutStartSec=300 in it's unit file stopped
the false timeout report) but I have seen it one other time (don't remember
the details).
systemctl status puppetmaster
? puppetmaster.service - Puppet master
Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor
preset: enabled)
Active: failed (Result: resources) since Tue 2017-08-29 11:24:36 CDT; 22min
ago
Process: 897 ExecStart=/usr/bin/puppet master (code=exited,
status=0/SUCCESS)
Aug 29 11:22:39 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Reopening log files
Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Starting Puppet master
version 3.8.5
Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Could not run: Address
already in use - listen(2)
Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: PID 1233
read from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 11:24:36 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: Unit
entered failed state.
Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: Failed
with result 'resources'.
However, ps -ef | grep puppet (run just after the above) returns
puppet 1380 1 0 11:26 ? 00:00:08 Passenger RubyApp: /usr/share/puppet/rack/
puppetmasterd
root 2015 1341 0 11:48 pts/0 00:00:00 grep --color=auto puppet
Earlier ps .. also reported
puppet 1355 1166 3 11:26 ? 00:00:01 Passenger AppPreloader:
/usr/share/puppet/rack/puppetmasterd
And, the "bottom line", puppet agent -t on a client works. It reports
finishing the catalog run and the client's yaml files on puppetmaster are
up to date.
Is there a command to tell systemd to re-scan running state and update its
understanding on what it finds? I tried systemctl daemon-reload just to be
sure that didn't solve the problem before posting this.
_______________________________________________
CentOS mailing list
CentOS at centos.org
https://lists.centos.org/mailman/listinfo/centos
First glance ity looks like someone has started that puppetmaster manually
at some point. As such it's not in a cgroup systemd is tracking so it
isn't
aware of it.
Your attempts to start the service are failing because that manually
started instance already has the port open.
Kill it with pkill -f puppet and then use ss -tnp to check for the port
being freed (wait for any time_wait states to go... which is why I'm not
filtering by listen).
Once it's clear then try starting with systemctl
_______________________________________________
CentOS mailing list
CentOS at centos.org
https://lists.centos.org/mailman/listinfo/centos
OK, something weird is definitely going on here, I have the luxury of rebooting
this system so it did. Here's what I got, note the time stamps.
ps -ef | grep puppet
root 932 1 0 15:23 ? 00:00:00 /usr/bin/ruby /usr/bin/puppet
master
root 1343 1327 0 15:24 pts/0 00:00:00 grep --color=auto puppet
(immediately afterward as fast as I could type:) uptime
15:24:56 up 1 min, 1 user, load average: 0.16, 0.07, 0.02
systemctl status puppetmaster
? puppetmaster.service - Puppet master
Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor
preset: enabled)
Active: activating (start) since Tue 2017-08-29 15:23:44 CDT; 1min 24s ago
Control: 932 (puppet)
Tasks: 1
Memory: 2.4M
CPU: 4ms
CGroup: /system.slice/puppetmaster.service
??932 /usr/bin/ruby /usr/bin/puppet master
Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
After a short delay:
systemctl status puppetmaster
? puppetmaster.service - Puppet master
Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor
preset: enabled)
Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; 11s ago
Process: 932 ExecStart=/usr/bin/puppet master (code=exited, status=0/SUCCESS)
Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master
version 3.8.5
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address
already in use - listen(2)
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 read
from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit entered
failed state.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed with
result 'resources'.
ps -ef | grep puppet
root 1373 1327 0 15:25 pts/0 00:00:00 grep --color=auto puppet
date
Tue Aug 29 15:26:04 CDT 2017
uptime
15:26:08 up 2 min, 1 user, load average: 0.08, 0.06, 0.02
(Getting a little smarter...)
date;uptime;ps -ef | grep puppet note the AppPreloader
Tue Aug 29 15:27:41 CDT 2017
15:27:41 up 4 min, 1 user, load average: 0.04, 0.05, 0.02
puppet 1377 1167 1 15:26 ? 00:00:01 Passenger AppPreloader:
/usr/share/puppet/rack/puppetmasterd
puppet 1402 1377 4 15:26 ? 00:00:03 Passenger RubyApp:
/usr/share/puppet/rack/puppetmasterd
root 1457 1327 0 15:27 pts/0 00:00:00 grep --color=auto puppet
date;uptime;systemctl status puppetmaster
Tue Aug 29 15:28:48 CDT 2017
15:28:48 up 5 min, 1 user, load average: 0.15, 0.10, 0.03
? puppetmaster.service - Puppet master
Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor
preset: enabled)
Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; 3min
37s ago
Process: 932 ExecStart=/usr/bin/puppet master (code=exited, status=0/SUCCESS)
Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master
version 3.8.5
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address
already in use - listen(2)
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 read
from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit entered
failed state.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed with
result 'resources'.
(And finally)
date;uptime;ps -ef | grep puppet
Tue Aug 29 15:36:48 CDT 2017
15:36:48 up 13 min, 1 user, load average: 0.00, 0.01, 0.00
puppet 1402 1 0 15:26 ? 00:00:03 Passenger RubyApp:
/usr/share/puppet/rack/puppetmasterd
root 1704 1327 0 15:36 pts/0 00:00:00 grep --color=auto puppet
date;uptime;systemctl status puppetmaster
Tue Aug 29 15:36:57 CDT 2017
15:36:57 up 13 min, 1 user, load average: 0.00, 0.01, 0.00
? puppetmaster.service - Puppet master
Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor
preset: enabled)
Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; 11min
ago
Process: 932 ExecStart=/usr/bin/puppet master (code=exited, status=0/SUCCESS)
Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master...
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master
version 3.8.5
Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address
already in use - listen(2)
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 read
from file /run/puppet/master.pid does not exist or is a zombie.
Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit entered
failed state.
Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed with
result 'resources'.
The AppPreloader is doing things (and probably confusing systemd in the process)
but I didn't start that, it was a part of the reboot. I looked at
/etc/init.d/puppetmaster to see if something strange was being done and it was
one of the simpler init scripts I've seen, uses start-stop-daemon.
I don't have any control over this, 'systemctl list-unit-files | grep
puppet' shows the puppetmaster.service is enabled, I just need a
"cleanup" solution, any ideas? And, BTW, thanks for any feedback.
Am 29.08.2017 um 22:52 schrieb Leroy Tennison:> The AppPreloader is doing things (and probably confusing systemd in the process) but I didn't start that, it was a part of the reboot. I looked at /etc/init.d/puppetmaster to see if something strange was being done and it was one of the simpler init scripts I've seen, uses start-stop-daemon. > > I don't have any control over this, 'systemctl list-unit-files | grep puppet' shows the puppetmaster.service is enabled, I just need a "cleanup" solution, any ideas? And, BTW, thanks for any feedback.Remove the Rack Puppet master server. https://docs.puppet.com/puppet/5.1/passenger.html#install-the-puppet-master-rack-application Alexander
Hmmm, that's an interesting option, I'll have to look into it. ----- Original Message ----- From: "Alexander Dalloz" <ad+lists at uni-x.org> To: "centos" <centos at centos.org> Sent: Tuesday, August 29, 2017 4:17:37 PM Subject: Re: [CentOS] Syncronize systemctl status with reality? Am 29.08.2017 um 22:52 schrieb Leroy Tennison:> The AppPreloader is doing things (and probably confusing systemd in the process) but I didn't start that, it was a part of the reboot. I looked at /etc/init.d/puppetmaster to see if something strange was being done and it was one of the simpler init scripts I've seen, uses start-stop-daemon. > > I don't have any control over this, 'systemctl list-unit-files | grep puppet' shows the puppetmaster.service is enabled, I just need a "cleanup" solution, any ideas? And, BTW, thanks for any feedback.Remove the Rack Puppet master server. https://docs.puppet.com/puppet/5.1/passenger.html#install-the-puppet-master-rack-application Alexander _______________________________________________ CentOS mailing list CentOS at centos.org https://lists.centos.org/mailman/listinfo/centos
On 29 August 2017 at 21:52, Leroy Tennison <leroy at datavoiceint.com> wrote:> ----- Original Message ----- > From: "James Hogarth" <james.hogarth at gmail.com> > To: "centos" <centos at centos.org> > Sent: Tuesday, August 29, 2017 2:03:44 PM > Subject: Re: [CentOS] Syncronize systemctl status with reality? > > On 29 Aug 2017 17:58, "Leroy Tennison" <leroy at datavoiceint.com> wrote: > > The particular issue is with puppetmaster (which admittedly takes 4 minutes > to actually start, setting TimeoutStartSec=300 in it's unit file stopped > the false timeout report) but I have seen it one other time (don't remember > the details). > > systemctl status puppetmaster > ? puppetmaster.service - Puppet master > Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; vendor > preset: enabled) > Active: failed (Result: resources) since Tue 2017-08-29 11:24:36 CDT; 22min > ago > Process: 897 ExecStart=/usr/bin/puppet master (code=exited, > status=0/SUCCESS) > > Aug 29 11:22:39 puppetmaster02 systemd[1]: Starting Puppet master... > Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Reopening log files > Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Starting Puppet master > version 3.8.5 > Aug 29 11:24:36 puppetmaster02 puppet-master[1233]: Could not run: Address > already in use - listen(2) > Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: PID 1233 > read from file /run/puppet/master.pid does not exist or is a zombie. > Aug 29 11:24:36 puppetmaster02 systemd[1]: Failed to start Puppet master. > Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: Unit > entered failed state. > Aug 29 11:24:36 puppetmaster02 systemd[1]: puppetmaster.service: Failed > with result 'resources'. > > However, ps -ef | grep puppet (run just after the above) returns > puppet 1380 1 0 11:26 ? 00:00:08 Passenger RubyApp: /usr/share/puppet/rack/ > puppetmasterd > root 2015 1341 0 11:48 pts/0 00:00:00 grep --color=auto puppet > > Earlier ps .. also reported > puppet 1355 1166 3 11:26 ? 00:00:01 Passenger AppPreloader: > /usr/share/puppet/rack/puppetmasterd > > And, the "bottom line", puppet agent -t on a client works. It reports > finishing the catalog run and the client's yaml files on puppetmaster are > up to date. > > Is there a command to tell systemd to re-scan running state and update its > understanding on what it finds? I tried systemctl daemon-reload just to be > sure that didn't solve the problem before posting this. > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos > > > First glance ity looks like someone has started that puppetmaster manually > at some point. As such it's not in a cgroup systemd is tracking so it isn't > aware of it. > > Your attempts to start the service are failing because that manually > started instance already has the port open. > > Kill it with pkill -f puppet and then use ss -tnp to check for the port > being freed (wait for any time_wait states to go... which is why I'm not > filtering by listen). > > Once it's clear then try starting with systemctl > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos > > OK, something weird is definitely going on here, I have the luxury of > rebooting this system so it did. Here's what I got, note the time stamps. > > ps -ef | grep puppet > root 932 1 0 15:23 ? 00:00:00 /usr/bin/ruby > /usr/bin/puppet master > root 1343 1327 0 15:24 pts/0 00:00:00 grep --color=auto puppet > (immediately afterward as fast as I could type:) uptime > 15:24:56 up 1 min, 1 user, load average: 0.16, 0.07, 0.02 > systemctl status puppetmaster > ? puppetmaster.service - Puppet master > Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; > vendor preset: enabled) > Active: activating (start) since Tue 2017-08-29 15:23:44 CDT; 1min 24s > ago > Control: 932 (puppet) > Tasks: 1 > Memory: 2.4M > CPU: 4ms > CGroup: /system.slice/puppetmaster.service > ??932 /usr/bin/ruby /usr/bin/puppet master > > Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master... > > > > After a short delay: > systemctl status puppetmaster > ? puppetmaster.service - Puppet master > Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; > vendor preset: enabled) > Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; > 11s ago > Process: 932 ExecStart=/usr/bin/puppet master (code=exited, > status=0/SUCCESS) > > Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master... > Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files > Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master > version 3.8.5 > Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address > already in use - listen(2) > Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 > read from file /run/puppet/master.pid does not exist or is a zombie. > Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master. > Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit > entered failed state. > Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed > with result 'resources'. > > ps -ef | grep puppet > root 1373 1327 0 15:25 pts/0 00:00:00 grep --color=auto puppet > > date > Tue Aug 29 15:26:04 CDT 2017 > > uptime > 15:26:08 up 2 min, 1 user, load average: 0.08, 0.06, 0.02 > > > > (Getting a little smarter...) > > date;uptime;ps -ef | grep puppet note the AppPreloader > Tue Aug 29 15:27:41 CDT 2017 > 15:27:41 up 4 min, 1 user, load average: 0.04, 0.05, 0.02 > puppet 1377 1167 1 15:26 ? 00:00:01 Passenger AppPreloader: > /usr/share/puppet/rack/puppetmasterd > puppet 1402 1377 4 15:26 ? 00:00:03 Passenger RubyApp: > /usr/share/puppet/rack/puppetmasterd > root 1457 1327 0 15:27 pts/0 00:00:00 grep --color=auto puppet > > date;uptime;systemctl status puppetmaster > Tue Aug 29 15:28:48 CDT 2017 > 15:28:48 up 5 min, 1 user, load average: 0.15, 0.10, 0.03 > ? puppetmaster.service - Puppet master > Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; > vendor preset: enabled) > Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; > 3min 37s ago > Process: 932 ExecStart=/usr/bin/puppet master (code=exited, > status=0/SUCCESS) > > Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master... > Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files > Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master > version 3.8.5 > Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address > already in use - listen(2) > Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 > read from file /run/puppet/master.pid does not exist or is a zombie. > Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master. > Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit > entered failed state. > Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed > with result 'resources'. > > > > (And finally) > date;uptime;ps -ef | grep puppet > Tue Aug 29 15:36:48 CDT 2017 > 15:36:48 up 13 min, 1 user, load average: 0.00, 0.01, 0.00 > puppet 1402 1 0 15:26 ? 00:00:03 Passenger RubyApp: > /usr/share/puppet/rack/puppetmasterd > root 1704 1327 0 15:36 pts/0 00:00:00 grep --color=auto puppet > > date;uptime;systemctl status puppetmaster > Tue Aug 29 15:36:57 CDT 2017 > 15:36:57 up 13 min, 1 user, load average: 0.00, 0.01, 0.00 > ? puppetmaster.service - Puppet master > Loaded: loaded (/lib/systemd/system/puppetmaster.service; enabled; > vendor preset: enabled) > Active: failed (Result: resources) since Tue 2017-08-29 15:25:11 CDT; > 11min ago > Process: 932 ExecStart=/usr/bin/puppet master (code=exited, > status=0/SUCCESS) > > Aug 29 15:23:44 puppetmaster02 systemd[1]: Starting Puppet master... > Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Reopening log files > Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Starting Puppet master > version 3.8.5 > Aug 29 15:25:11 puppetmaster02 puppet-master[1360]: Could not run: Address > already in use - listen(2) > Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: PID 1360 > read from file /run/puppet/master.pid does not exist or is a zombie. > Aug 29 15:25:11 puppetmaster02 systemd[1]: Failed to start Puppet master. > Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Unit > entered failed state. > Aug 29 15:25:11 puppetmaster02 systemd[1]: puppetmaster.service: Failed > with result 'resources'. > > > > The AppPreloader is doing things (and probably confusing systemd in the > process) but I didn't start that, it was a part of the reboot. I looked at > /etc/init.d/puppetmaster to see if something strange was being done and it > was one of the simpler init scripts I've seen, uses start-stop-daemon. > > I don't have any control over this, 'systemctl list-unit-files | grep > puppet' shows the puppetmaster.service is enabled, I just need a "cleanup" > solution, any ideas? And, BTW, thanks for any feedback. > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos >Note that since systemctl shows puppetmaster.service as a unit file the init script wont' be used at all so you can ignore the file in /etc/init.d ... that's just there because puppetlabs are being lazy and using the same RPM on EL6 as EL7 rather than having the right files for the right subsystems on the right distribution versions ... ./sigh Otherwise you're down to standard troubleshooting process - use ss -tlnp to identify the process that is holding the port open, systemctl cat puppetmaster.service and see what it's trying to do etc etc