CentOS virt - Apr 2016 - Xen4CentOS 6 64bit - domUs don't shutdown on dom0 after "yum upgrade" to 4.6.1

Hello all,

I'm addressing the mailing list following the suggestion of gwd on
#centos-virt @freenode IRC.

The subject says most of it. Here are the details :

- Was running 4.4.x and configured such that /etc/xen/auto domUs would be
saved/restored on dom0 shutdown/bootup.
- Installation was based on "centos-release-xen" such that "yum
upgrade"
brough Xen to 4.6.1.

After upgrade observed behavior:
- dom0 boot starts up the domUs in /etc/xen/auto.
- dom0 shutdown does not save/shutdown domUs (the command is "shutdown -h
now" via ssh).

Diagnostics:
- Running /etc/init.d/xendomains stop/start does the right thing.
- chkconfig --list shows "xendomains     0:off 1:off 2:on 3:on 4:on 5:on
6:off"
which looks nice.
- The rc link is in place: /etc/rc.d/rc0.d/K00xendomains ->
../init.d/xendomains
- Edited /etc/init.d/xendomains such that the lines:

    start)
        $LIBEXEC_BIN/xendomains start
        ;;
    stop)
        $LIBEXEC_BIN/xendomains stop
        ;;

become (added "2>&1 | tee FILE" to the underlying start/stop
command)

    start)
        $LIBEXEC_BIN/xendomains start 2>&1 | /usr/bin/tee -a
/tmp/xd-start.log
        ;;
    stop)
        $LIBEXEC_BIN/xendomains stop 2>&1 | /usr/bin/tee -a
/tmp/xd-stop.log
        ;;

- The /tmp/xd-stop.log file is not created on dom0 shutdown (make sense,
but why?!).
- The /tmp/xd-stop.log is created on dom0 boot, as expected.
- Both files are created on manual /etc/init.d/xendomains start/stop
- Manually added an rc link: /etc/rc.d/rc0.d/K01xendomains ->
../init.d/xendomains
- The behavior remains unchanged.

This is looking much more of an upstart/sysvinit issue that a Xen 4.6 issue.
However, I believe others may have been through this experience.

Can anyone provide any pointers in further diagnosing this?
Thanks in advance.

Regards,
--
exvito
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos-virt/attachments/20160407/2abcd774/attachment.html>

On Thu, Apr 7, 2016 at 5:46 PM, exvito here <ex.vitorino at gmail.com>
wrote:
> Hello all,
>
> I'm addressing the mailing list following the suggestion of gwd on
> #centos-virt @freenode IRC.
>
> The subject says most of it. Here are the details :
>
> - Was running 4.4.x and configured such that /etc/xen/auto domUs would be
> saved/restored on dom0 shutdown/bootup.
> - Installation was based on "centos-release-xen" such that
"yum upgrade"
> brough Xen to 4.6.1.
>
> After upgrade observed behavior:
> - dom0 boot starts up the domUs in /etc/xen/auto.
> - dom0 shutdown does not save/shutdown domUs (the command is "shutdown
-h
> now" via ssh).
>
> Diagnostics:
> - Running /etc/init.d/xendomains stop/start does the right thing.
> - chkconfig --list shows "xendomains     0:off 1:off 2:on 3:on 4:on
5:on
> 6:off" which looks nice.
> - The rc link is in place: /etc/rc.d/rc0.d/K00xendomains ->
> ../init.d/xendomains
> - Edited /etc/init.d/xendomains such that the lines:
>
>     start)
>         $LIBEXEC_BIN/xendomains start
>         ;;
>     stop)
>         $LIBEXEC_BIN/xendomains stop
>         ;;
>
> become (added "2>&1 | tee FILE" to the underlying
start/stop command)
>
>     start)
>         $LIBEXEC_BIN/xendomains start 2>&1 | /usr/bin/tee -a
> /tmp/xd-start.log
>         ;;
>     stop)
>         $LIBEXEC_BIN/xendomains stop 2>&1 | /usr/bin/tee -a
/tmp/xd-stop.log
>         ;;
>
> - The /tmp/xd-stop.log file is not created on dom0 shutdown (make sense,
but
> why?!).
> - The /tmp/xd-stop.log is created on dom0 boot, as expected.
> - Both files are created on manual /etc/init.d/xendomains start/stop
> - Manually added an rc link: /etc/rc.d/rc0.d/K01xendomains ->
> ../init.d/xendomains
> - The behavior remains unchanged.
>
> This is looking much more of an upstart/sysvinit issue that a Xen 4.6
issue.
> However, I believe others may have been through this experience.
>
> Can anyone provide any pointers in further diagnosing this?
> Thanks in advance.
Hey exvito,

I finally got a chance to look at this.  It turns out that one of the
paths was incorrectly changed between 4.4 and 4.6.

To fix your problem, after upgrading to 4.6, please edit the following file:

/etc/xen/scripts/hotplugpath.sh

and replace

XEN_LOCK_DIR="/var/lock"

with

XEN_LOCK_DIR="/var/lock/subsys"

I'll be working on a proper fix with upstream -- I'll post here when
I've got packages with the backported change to test.

What's happening is that the CentOS init scripts only shut down
services that indicate they have an active component running -- which
it takes to mean, "Has a lockfile in /var/lock/subsys".  As you can
probably guess, in 4.6 the lockfile is being created in /var/lock
instead; so when shutting down, "xendomains stop" was never called.

The /var/lock/subsys thing is apparently a RHEL thing; other systems
seem to call "${service} stop" unconditionally.

Thanks again for reporting this,
 -George

George,

Thanks a lot for your investigation and feedback.
I confirm it works as expected after the change you proposed.

Cheers,
--
exvito