Pkg xen devel - Oct 2017 - Bug#879751: xen-utils-common: /etc/init.d/xen restart kills running domU qemu processes

Package: xen-utils-common
Version: 4.8.1-1+deb9u3
Severity: important

On debian stretch with xen and qemu-system-x86 installed, when
upgrading the system if for some reason
# /etc/init.d/xen restart
is issued, all qemu processes belonging to domU's are killed.

Suggestion is to remove the second start-stop-daemon from
qemu_stop_real() in /etc/init.d/xen.


-- System Information:
Debian Release: 9.2
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable'), (1,
'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.9.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=hu_HU.UTF-8, LC_CTYPE=hu_HU.UTF-8 (charmap=UTF-8),
LANGUAGE=hu_HU.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages xen-utils-common depends on:
ii  lsb-base        9.20161125
ii  python          2.7.13-2
ii  ucf             3.0036
ii  udev            232-25+deb9u1
pn  xenstore-utils  <none>

xen-utils-common recommends no packages.

xen-utils-common suggests no packages.

Hi Richard,

it's recommended to only actually upgrade xen packages just before you
want to reboot the system, and when you already stopped all domUs.

This is e.g. a similar situation to upgrading the openvswitch package,
which will also restart processes and mess up all network interfaces of
running domUs.

For unattended-updates, you can just stuff "xen" inside your
Unattended-Upgrade::Package-Blacklist, or if you're not using that you
can put all xen related packages on hold manually and only forcibly
include them to be upgraded just before the reboot.

Upgrading the xen packages while domUs are running can result in a wide
range of scenarios which will cause problems. Solving all of those would
require more work and adding far more complexity to the packages than
there's programmers and testers (and hardware to test all scenarios etc)
available in the Debian project to make that happen currently.

Hans

Dear Hans,

Thanks for your reply. Actually, the problem is that the script kills ALL 
qemu processes, not only those attached to xen domU domains. I think that 
is a problem.

Would that be nice that a bash upgrade would kill all running bash 
processes? And also, in earlier debian releases, this was not the case, I 
could upgrade the xen-common package without any issues. Of course, a 
whole xen (hypervisor) upgrade might need a reboot, but just a xen-common 
upgrade might not. Right now it seems to me it breaks thing more than it 
helps.

Regards,
Kojedzinszky Rich?rd
Euronet Magyarorszag Informatika Zrt.

On Sun, 17 Dec 2017, Hans van Kranenburg wrote:
> Date: Sun, 17 Dec 2017 20:08:37 +0100
> From: Hans van Kranenburg <hans at knorrie.org>
> To: 879751 at bugs.debian.org,
>     Richard Kojedzinszky <kojedzinszky.richard at euronetrt.hu>
> Subject: Re: xen-utils-common: /etc/init.d/xen restart kills running domU
qemu
>      processes
> 
> Hi Richard,
>
> it's recommended to only actually upgrade xen packages just before you
> want to reboot the system, and when you already stopped all domUs.
>
> This is e.g. a similar situation to upgrading the openvswitch package,
> which will also restart processes and mess up all network interfaces of
> running domUs.
>
> For unattended-updates, you can just stuff "xen" inside your
> Unattended-Upgrade::Package-Blacklist, or if you're not using that you
> can put all xen related packages on hold manually and only forcibly
> include them to be upgraded just before the reboot.
>
> Upgrading the xen packages while domUs are running can result in a wide
> range of scenarios which will cause problems. Solving all of those would
> require more work and adding far more complexity to the packages than
> there's programmers and testers (and hardware to test all scenarios
etc)
> available in the Debian project to make that happen currently.
>
> Hans
>

Dear devs,

I still consider this issue a bug. Actually, I think the 'kill all qemu 
processes' way does not solve any issues that might arise during an 
upgrade, even it may cause more trouble. If that is a real concert, a 
preinst script might check for running qemu processes, and break if it 
founds one. I myself am not real familiar with xen internals, why is a 
dedicated qemu process needed for dom0, but for the init script it seems 
to be enough to just kill that one only, not all.

I do not understand fully the involved code, in qemu_stop_real() the 
script kills the qemu process for dom0 with a known pid in pidfile, and if 
it succeeds will kill any other qemu processes also. Is this the intent?

Regards,

Kojedzinszky Rich?rd
Euronet Magyarorszag Informatika Zrt.

On Mon, 18 Dec 2017, Kojedzinszky Rich?rd wrote:
> Date: Mon, 18 Dec 2017 08:41:06 +0100 (CET)
> From: Kojedzinszky Rich?rd <kojedzinszky.richard at euronetrt.hu>
> To: Hans van Kranenburg <hans at knorrie.org>
> Cc: 879751 at bugs.debian.org
> Subject: Re: xen-utils-common: /etc/init.d/xen restart kills running domU
qemu
>      processes
> 
> Dear Hans,
>
> Thanks for your reply. Actually, the problem is that the script kills ALL 
> qemu processes, not only those attached to xen domU domains. I think that
is
> a problem.
>
> Would that be nice that a bash upgrade would kill all running bash
processes?
> And also, in earlier debian releases, this was not the case, I could
upgrade
> the xen-common package without any issues. Of course, a whole xen 
> (hypervisor) upgrade might need a reboot, but just a xen-common upgrade
might
> not. Right now it seems to me it breaks thing more than it helps.
>
> Regards,
> Kojedzinszky Rich?rd
> Euronet Magyarorszag Informatika Zrt.
>
> On Sun, 17 Dec 2017, Hans van Kranenburg wrote:
>
>> Date: Sun, 17 Dec 2017 20:08:37 +0100
>> From: Hans van Kranenburg <hans at knorrie.org>
>> To: 879751 at bugs.debian.org,
>>     Richard Kojedzinszky <kojedzinszky.richard at euronetrt.hu>
>> Subject: Re: xen-utils-common: /etc/init.d/xen restart kills running
domU
>> qemu
>>      processes
>> 
>> Hi Richard,
>> 
>> it's recommended to only actually upgrade xen packages just before
you
>> want to reboot the system, and when you already stopped all domUs.
>> 
>> This is e.g. a similar situation to upgrading the openvswitch package,
>> which will also restart processes and mess up all network interfaces of
>> running domUs.
>> 
>> For unattended-updates, you can just stuff "xen" inside your
>> Unattended-Upgrade::Package-Blacklist, or if you're not using that
you
>> can put all xen related packages on hold manually and only forcibly
>> include them to be upgraded just before the reboot.
>> 
>> Upgrading the xen packages while domUs are running can result in a wide
>> range of scenarios which will cause problems. Solving all of those
would
>> require more work and adding far more complexity to the packages than
>> there's programmers and testers (and hardware to test all scenarios
etc)
>> available in the Debian project to make that happen currently.
>> 
>> Hans
>

Control: tags -1 + patch
>From 1bb0ee913b35da4d4e6a86b92beb01aa5f77e5c9 Mon Sep 17 00:00:00 2001
From: Ian Jackson <ian.jackson at eu.citrix.com>
Date: Fri, 23 Feb 2018 17:04:16 +0000
Subject: [PATCH] init scripts: Do not kill per-domain qemu processes.

Drop 2nd start-stop-daemon from qemu_stop_real xen init script.
Closes:#879751.

This code was introduced in
  8ad8a8221d9884b76f6942acabd9465e11ca23bd
  Start a qemu process in dom0 to service the toolstacks loopback disk attaches.

Ie the commit which introduced the global qemu.  There is no
indication that there was any intent to do anything to the per-domain
qemus.

I think the 2nd s-s-d invocation is there to stop a qemu which is for
some reason not mentioned in the pidfile and somehow escaped.  That
seems like a thing we should try to prevent.  I don't think if it
comes to pass, that it can be cleaned up without trashing the domUs.

So remove it.

Signed-off-by: Ian Jackson <Ian.Jackson at eu.citrix.com>
---
 debian/xen-utils-common.xen.init | 2 --
 1 file changed, 2 deletions(-)

diff --git a/debian/xen-utils-common.xen.init b/debian/xen-utils-common.xen.init
index a153885..3411c15 100644
--- a/debian/xen-utils-common.xen.init
+++ b/debian/xen-utils-common.xen.init
@@ -235,8 +235,6 @@ qemu_stop_real()
 	start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile
"$QEMU_PIDFILE" --exec "$QEMU"
 	RETVAL="$?"
 	[ "$RETVAL" = 2 ] && return 2
-	start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec
"$QEMU"
-	[ "$?" = 2 ] && return 2
 	rm -f $QEMU_PIDFILE
 	return "$RETVAL"
 }
-- 
2.1.4


-- 
Ian Jackson <ijackson at chiark.greenend.org.uk>   These opinions are my
own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.

Processing control commands:
> tags -1 + patch
Bug #879751 [xen-utils-common] xen-utils-common: /etc/init.d/xen restart kills
running domU qemu processes
Added tag(s) patch.

-- 
879751: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879751
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems

Richard, thanks for your perseverence.
Can you please test the patch I have just sent to the bug ?

You should be able to
  patch <0001-init-scripts-Do-not-kill-per-domain-qemu-processes.patch
/etc/init.d/xen

It WFM.  That is, I checked that it kills the global qemu but leaves
a domU one alone.

I am considering shipping this change in a stable update for Xen 4.8.

Ian.

Your message dated Sun, 04 Mar 2018 15:02:17 +0000
with message-id <E1esV9V-0002ks-1R at fasolo.debian.org>
and subject line Bug#879751: fixed in xen
4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
has caused the Debian Bug report #879751,
regarding xen-utils-common: /etc/init.d/xen restart kills running domU qemu
processes
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
879751: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=879751
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: Richard Kojedzinszky <kojedzinszky.richard at euronetrt.hu>
Subject: xen-utils-common: /etc/init.d/xen restart kills running domU qemu
processes
Date: Wed, 25 Oct 2017 11:35:35 +0200
Size: 3470
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20180304/a8288726/attachment.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Ian Jackson <ijackson at chiark.greenend.org.uk>
Subject: Bug#879751: fixed in xen 4.8.3+comet2+shim4.10.0+comet3-1+deb9u5
Date: Sun, 04 Mar 2018 15:02:17 +0000
Size: 10962
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20180304/a8288726/attachment-0001.mht>

Dear Ian,

For some reason, I've missed your reply. We are using the scripts with the 
same two lines removed, and I can confirm it works as expected, a simple 
upgrade now will not stop existing, running qemu processes.

Thanks for your efforts.

Regards,
Kojedzinszky Rich?rd
Euronet Magyarorszag Informatika Zrt.

On Fri, 23 Feb 2018, Ian Jackson wrote:
> Date: Fri, 23 Feb 2018 17:11:12 +0000
> From: Ian Jackson <ijackson at chiark.greenend.org.uk>
> To: Kojedzinszky Rich?rd <kojedzinszky.richard at euronetrt.hu>
> Cc: 879751 at bugs.debian.org
> Subject: Re: [PATCH] init scripts: Do not kill per-domain qemu processes.
> 
> Richard, thanks for your perseverence.
> Can you please test the patch I have just sent to the bug ?
>
> You should be able to
>  patch <0001-init-scripts-Do-not-kill-per-domain-qemu-processes.patch
/etc/init.d/xen
>
> It WFM.  That is, I checked that it kills the global qemu but leaves
> a domU one alone.
>
> I am considering shipping this change in a stable update for Xen 4.8.
>
> Ian.
>