CentOS - May 2019 - Bypassing 'A stop job is running' when rebooting CentOS 7

James Pearson wrote:
> James Pearson wrote:
>
>>
>> I'm currently trying to reboot a CentOS 7.5 workstation (to
complete an
>>  upgrade to 7.6), but it is 'stuck' while shutting down with
'A stop
>> job is running for ...' - the counter initially gave a limit of
'1min
>> 30s' -
>> but each time it reaches that limit, it just adds on ~90 seconds to the
>> limit ...
>>
>> Currently the limit is '25min 33s'
>>
>>
>> I'm in no hurry to have this workstation operational, but I guess
at
>> some point I will have to power cycle it ...
>>
>> Does anyone know how to bypass this? - or at least stop it increasing
>> the limit each time it is reached?
>>
>> It does seems rather pointless to keep increasing the limit like this
>> ...
>>
> It _finally_ gave up at 30 mins and rebooted
One question: did it have a mounted nfs filesystem?

The joys of systemd....

         mark

     mark

> James Pearson wrote:
>> James Pearson wrote:
>>
>>>
>>> I'm currently trying to reboot a CentOS 7.5 workstation (to
complete an
>>>  upgrade to 7.6), but it is 'stuck' while shutting down
with 'A stop
>>> job is running for ...' - the counter initially gave a limit of
'1min
>>> 30s' -
>>> but each time it reaches that limit, it just adds on ~90 seconds to
the
>>> limit ...
>>>
>>> Currently the limit is '25min 33s'
>>>
>>>
>>> I'm in no hurry to have this workstation operational, but I
guess at
>>> some point I will have to power cycle it ...
>>>
>>> Does anyone know how to bypass this? - or at least stop it
increasing
>>> the limit each time it is reached?
>>>
>>> It does seems rather pointless to keep increasing the limit like
this
>>> ...
>>>
>> It _finally_ gave up at 30 mins and rebooted
>
> One question: did it have a mounted nfs filesystem?
>
> The joys of systemd....
Yes, NFS integration with systemd is broken by default, at least it was
still the case when I last checked.
If you want it to work correctly, you have to add
'x-systemd.requires=network-online.target' as NFS mount option.

Clearly, how should systemd know that NFS won't work without network? I
knew you agree :-)

Regards,
Simon

On 22/05/2019 14:43, mark wrote:
> James Pearson wrote:
>> James Pearson wrote:
>>
>>>
>>> I'm currently trying to reboot a CentOS 7.5 workstation (to
complete an
>>>  upgrade to 7.6), but it is 'stuck' while shutting down
with 'A stop
>>> job is running for ...' - the counter initially gave a limit of
'1min
>>> 30s' -
>>> but each time it reaches that limit, it just adds on ~90 seconds to
the
>>> limit ...
>>>
>>> Currently the limit is '25min 33s'
>>>
>>>
>>> I'm in no hurry to have this workstation operational, but I
guess at
>>> some point I will have to power cycle it ...
>>>
>>> Does anyone know how to bypass this? - or at least stop it
increasing
>>> the limit each time it is reached?
>>>
>>> It does seems rather pointless to keep increasing the limit like
this
>>> ...
>>>
>> It _finally_ gave up at 30 mins and rebooted
> 
> One question: did it have a mounted nfs filesystem?
> 
> The joys of systemd....
> 
>          mark
> 
>      mark
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
> 
"Anything Windows can do, systemd can do better" (with apologies to
Irving Berlin).

-- 
J Martin Rushton MBCS

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20190522/a9ffda39/attachment-0002.sig>

mark wrote:
> James Pearson wrote:
>> James Pearson wrote:
>>
>>>
>>> I'm currently trying to reboot a CentOS 7.5 workstation (to
complete an
>>>   upgrade to 7.6), but it is 'stuck' while shutting down
with 'A stop
>>> job is running for ...' - the counter initially gave a limit of
'1min
>>> 30s' -
>>> but each time it reaches that limit, it just adds on ~90 seconds to
the
>>> limit ...
>>>
>>> Currently the limit is '25min 33s'
>>>
>>>
>>> I'm in no hurry to have this workstation operational, but I
guess at
>>> some point I will have to power cycle it ...
>>>
>>> Does anyone know how to bypass this? - or at least stop it
increasing
>>> the limit each time it is reached?
>>>
>>> It does seems rather pointless to keep increasing the limit like
this
>>> ...
>>>
>> It _finally_ gave up at 30 mins and rebooted
> 
> One question: did it have a mounted nfs filesystem?
All our boxes have NFS mounted files systems - and usually this isn't a 
problem - reboots work without an issue

In this case, it appeared to be 'stuck' on a local file bind mounted 
over a file on an NFS mounted file system

But that isn't really the point - I don't really want to have to wait a 
maximum of 30 minutes for the reboot to give up waiting for 'whatever'

Poking about a bit, I see that /usr/lib/systemd/system/reboot.target has 
the line:

  JobTimeoutSec=30min

(there is a similar JobTimeoutSec=30min in poweroff.target)

I'm guessing I could create something like 
/etc/systemd/system/reboot.target.d/override.conf containing something like:

  [Unit]
  JobTimeoutSec=3min

Now I need to see if I can reproduce the issue and see if this setting 
works ...

James Pearson

On Wed, May 22, 2019 at 7:44 AM mark <m.roth at 5-cent.us>
wrote:
>
> The joys of systemd....
I'm not sure it's right to blame systemd.  Systemd asked nicely for
the service to shutdown.  The service didn't, probably because the
update change something and pulled the rug out from beneath it.
Systemd then waited a bit to make sure the service wasn't just being
slow, and finally gave up and forcibly killed it.  I think this is a
reasonable approach to killing a misbehaving service while trying to
minimize data-loss, and the timeout can be configured.

This hasn't happened to me recently, but I think I've tried Ctl-C and
Ctl-Alt-Del without much success.  That leaves the Big Red Switch
(which is mostly small and black these days).

Jim

Once upon a time, James Szinger <jszinger at gmail.com>
said:
> On Wed, May 22, 2019 at 7:44 AM mark <m.roth at 5-cent.us> wrote:
> > The joys of systemd....
> 
> I'm not sure it's right to blame systemd.  Systemd asked nicely for
> the service to shutdown.  The service didn't, probably because the
> update change something and pulled the rug out from beneath it.
Right - before systemd, any old init script could also block shutdown.
> This hasn't happened to me recently, but I think I've tried Ctl-C
and
> Ctl-Alt-Del without much success.  That leaves the Big Red Switch
> (which is mostly small and black these days).
There's a "magic" thing systemd does now - hit C-A-D seven times
in two
seconds and it'll stop what it is waiting for and just go ahead and
reboot.  Will kill anything not shut down, but at least it'll still try
to cleanly unmount filesystems and such I believe.
-- 
Chris Adams <linux at cmadams.net>

On Wed, May 22, 2019 at 09:07:32AM -0600, James Szinger
wrote:
> On Wed, May 22, 2019 at 7:44 AM mark <m.roth at 5-cent.us> wrote:
> >
> > The joys of systemd....
> 
> I'm not sure it's right to blame systemd.  Systemd asked nicely for
> the service to shutdown.  
But we can blame systemd for the cryptic message

  A stop job is running

Surely systemd knows what service it is waiting for,
why doesn't it tell us?

  The stop job XYZ is running

jon
-- 
Jon H. LaBadie                 jcu at jgcomp.com
 11226 South Shore Rd.          (703) 787-0688 (H)
 Reston, VA  20190              (703) 935-6720 (C)