Pkg xen devel - Jul 2019 - Bug#932759: After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled

Package:  xen-hypervisor-4.11-amd64
Version: 4.11.1+92-g6c33308a8d-2
System: Linux [host] 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5+deb10u1
(2019-07-19) x86_64 GNU/Linux
Debian Release:        10
Codename:       buster

In Reference To: Debian bug #851654 (archived)
> On Mon, 21 Jan 2019 00:48:09 +0100 Hans van Kranenburg
> <hans at knorrie.org> wrote:
>> reassign 851654 src:xen
>> thanks
>>
>> Hm!
>>
>> I think this is the same one that I've been observing during
upgrade
>> tests from 4.8 -> 4.11 and 4.10 -> 4.11.
>>
>> In my IRC logs I can find myself complaining about xenconsoled
that's
>> suddenly gone multiple times during 2018. I didn't manage to track
this
>> issue down yet. The IRC logs contain some links to expired pastebin
>> entries. :|
>>
>> Also, in some cases xenconsoled just got shut down during the night,
>> during some systemd reload whatever was happening.
>>
>> I suspect more users are going to run into this issue when upgrading to
>> Buster.
>>
>> Hans
What happened:
- upgraded Debian Xen Dom0 from stretch to buster and rebooted, as
described in
https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html

- started some Linux pv domu without problems

- removed obsolete packages with 'apt autoremove'. This removed (among
others)
xen-hypervisor-4.8-amd64:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11),
libxen-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11),
xen-utils-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11)

- starting another pv domu the next day using 'xl create -c [domu.cfg]'
raised an error message:
"xenconsole: Could not read tty from store: Success"

- opening a console for an already running domu raised another error
message that I believe was:
"xenconsole: Could not open tty `/dev/pts/2': No such file or
directory"

- some internet searching hinted that this could be caused by
xenconsoled not running.

- xenconsoled was not running

- searching system logs revealed that xenconsoled seemed to have stopped
when 'apt autoremove' removed the obsolete xen 4.8
packages after upgrading to xen 4.11.

- this coincided with a systemd reload as seen in journalctl:
Jul 21 07:38:40 host systemd[1]: Reloading.
Jul 21 07:38:40 host systemd[1]: serial-getty at hvc0.service: Current
command vanished from the unit file, execution of the command list won't
be resumed.
Jul 21 07:38:40 host systemd[1]: serial-getty at ttyS1.service: Current
command vanished from the unit file, execution of the command list won't
be resumed.
Jul 21 07:38:40 host systemd[1]: getty at tty1.service: Current command
vanished from the unit file, execution of the command list won't be resumed.
Jul 21 07:38:41 host systemd[1]: Stopping LSB: Xen daemons...
Jul 21 07:38:41 host xen[9092]: Stopping Xen daemons: xenconsoled.
Jul 21 07:38:41 host systemd[1]: xen.service: Succeeded.
Jul 21 07:38:41 host systemd[1]: Stopped LSB: Xen daemons.

Note that at this time domu's where already running under xen4.11.

'systemctl restart xen.service' solved the issue. I report the issue
because it could affect others that are performing upgrades.

Hi niek,

Thanks for the report!

On 7/22/19 8:32 PM, niek wrote:
> Package:  xen-hypervisor-4.11-amd64
> Version: 4.11.1+92-g6c33308a8d-2
> 
> What happened:
> - upgraded Debian Xen Dom0 from stretch to buster and rebooted, as
> described in
>
https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html
> 
> - started some Linux pv domu without problems
> 
> - removed obsolete packages with 'apt autoremove'. This removed
(among
> others)
> xen-hypervisor-4.8-amd64:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11),
> libxen-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11),
> xen-utils-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11)
> 
> [...]
> - xenconsoled was not running
> 
> - searching system logs revealed that xenconsoled seemed to have stopped
> when 'apt autoremove' removed the obsolete xen 4.8
> packages after upgrading to xen 4.11.
Well, there it is again. We tried to make a fix, exactly for this...

https://salsa.debian.org/xen-team/debian-xen/commit/ef242a700765a971a6afc12d25ee19944dd3a27a

...and apparently there's another scenario in which even this doesn't
work?

Can you show the lines from /var/log/dpkg.log from that moment, the
seconds around 07:38:40? It tells exactly what got removed, in what
second, just to confirm?

I'm pretty sure I tried to reproduce this after we added the fix I just
referenced, and I was unable to. So, I'm very interested in finding out
what's still going on here.

Usually being able to reproduce a problem is one of the biggest steps
towards finding a solution. (since it can be done over and over again,
finding out what exactly causes it). So, finding the right sequence of
steps to make it happen again is crucial here.

Do you think the systemd reload has anything to do with it? Maybe the
whole systemd init-script-wrapper-trickery is misbehaving in some way?

Can you reproduce this by manually grabbing the
xen-hypervisor-4.8-amd64, libxen-4.8 and xen-utils-4.8 from stretch
again, installing them and removing them again? Do you have any other idea?

Thanks,
Hans

On 22-7-2019 18:57, Hans van Kranenburg wrote:
> Hi niek,
> 
> Thanks for the report!
> 
> On 7/22/19 8:32 PM, niek wrote:
>> Package:  xen-hypervisor-4.11-amd64
>> Version: 4.11.1+92-g6c33308a8d-2
>>
>> What happened:
>> - upgraded Debian Xen Dom0 from stretch to buster and rebooted, as
>> described in
>>
https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html
>>
>> - started some Linux pv domu without problems
>>
>> - removed obsolete packages with 'apt autoremove'. This removed
(among
>> others)
>> xen-hypervisor-4.8-amd64:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11),
>> libxen-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11),
>> xen-utils-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11)
>>
>> [...]
>> - xenconsoled was not running
>>
>> - searching system logs revealed that xenconsoled seemed to have
stopped
>> when 'apt autoremove' removed the obsolete xen 4.8
>> packages after upgrading to xen 4.11.
> 
> Well, there it is again. We tried to make a fix, exactly for this...
> 
>
https://salsa.debian.org/xen-team/debian-xen/commit/ef242a700765a971a6afc12d25ee19944dd3a27a
> 
> ...and apparently there's another scenario in which even this
doesn't work?
> 
> Can you show the lines from /var/log/dpkg.log from that moment, the
> seconds around 07:38:40? It tells exactly what got removed, in what
> second, just to confirm?
> 
> I'm pretty sure I tried to reproduce this after we added the fix I just
> referenced, and I was unable to. So, I'm very interested in finding out
> what's still going on here.
> 
> Usually being able to reproduce a problem is one of the biggest steps
> towards finding a solution. (since it can be done over and over again,
> finding out what exactly causes it). So, finding the right sequence of
> steps to make it happen again is crucial here.
> 
> Do you think the systemd reload has anything to do with it? Maybe the
> whole systemd init-script-wrapper-trickery is misbehaving in some way?
> 
> Can you reproduce this by manually grabbing the
> xen-hypervisor-4.8-amd64, libxen-4.8 and xen-utils-4.8 from stretch
> again, installing them and removing them again? Do you have any other idea?
> 
> Thanks,
> Hans
> 
Hi Hans,

Thanks for your work with Ian getting xen 4.11 in buster! Very happy
with that.

I seem to remember seeing 'processing triggers for systemd' somewhere,
sometime, but I can't confirm that from the logs so it probably was at
another stage of the upgrade.

This is a production system, so I will not try to reproduce this by
reinstalling the 4.8 packages as you suggested. There is, however an
identical server that I still need to upgrade, so I will be able to look
closely what happens with that.

Here are the relevant lines from dpkg.conf:
2019-07-21 07:38:39 status installed python3.5-minimal:amd64 3.5.3-1+deb9u1
2019-07-21 07:38:39 remove python3.5-minimal:amd64 3.5.3-1+deb9u1 <none>
2019-07-21 07:38:39 status half-configured python3.5-minimal:amd64
3.5.3-1+deb9u1
2019-07-21 07:38:39 status half-installed python3.5-minimal:amd64
3.5.3-1+deb9u1
2019-07-21 07:38:40 status config-files python3.5-minimal:amd64
3.5.3-1+deb9u1
2019-07-21 07:38:40 status installed libpython3.5-minimal:amd64
3.5.3-1+deb9u1
2019-07-21 07:38:40 remove libpython3.5-minimal:amd64 3.5.3-1+deb9u1
<none>
2019-07-21 07:38:40 status half-configured libpython3.5-minimal:amd64
3.5.3-1+deb9u1
2019-07-21 07:38:40 status half-installed libpython3.5-minimal:amd64
3.5.3-1+deb9u1
2019-07-21 07:38:40 status config-files libpython3.5-minimal:amd64
3.5.3-1+deb9u1
2019-07-21 07:38:40 status installed xen-utils-4.8:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:38:40 remove xen-utils-4.8:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11 <none>
2019-07-21 07:38:40 status half-configured xen-utils-4.8:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:38:41 status half-installed xen-utils-4.8:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:38:41 status config-files xen-utils-4.8:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:38:41 status not-installed xen-utils-4.8:amd64 <none>
2019-07-21 07:38:41 status installed libxen-4.8:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:38:41 remove libxen-4.8:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11 <none>
2019-07-21 07:38:41 status half-configured libxen-4.8:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:38:41 status half-installed libxen-4.8:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:38:41 status config-files libxen-4.8:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:38:41 status not-installed libxen-4.8:amd64 <none>
2019-07-21 07:38:41 status installed python3-distutils:all 3.7.3-1
2019-07-21 07:38:41 remove python3-distutils:all 3.7.3-1 <none>
2019-07-21 07:38:41 status half-configured python3-distutils:all 3.7.3-1
2019-07-21 07:38:41 status half-installed python3-distutils:all 3.7.3-1
2019-07-21 07:38:41 status config-files python3-distutils:all 3.7.3-1
2019-07-21 07:38:41 status not-installed python3-distutils:all <none>
2019-07-21 07:38:41 status installed python3-lib2to3:all 3.7.3-1
2019-07-21 07:38:41 remove python3-lib2to3:all 3.7.3-1 <none>
2019-07-21 07:38:41 status half-configured python3-lib2to3:all 3.7.3-1
2019-07-21 07:38:41 status half-installed python3-lib2to3:all 3.7.3-1
2019-07-21 07:38:41 status config-files python3-lib2to3:all 3.7.3-1
2019-07-21 07:38:41 status not-installed python3-lib2to3:all <none>
2019-07-21 07:38:41 status installed rename:all 1.10-1
2019-07-21 07:38:41 remove rename:all 1.10-1 <none>
2019-07-21 07:38:41 status half-configured rename:all 1.10-1
2019-07-21 07:38:42 status half-installed rename:all 1.10-1
2019-07-21 07:38:42 status config-files rename:all 1.10-1
2019-07-21 07:38:42 status not-installed rename:all <none>
2019-07-21 07:38:42 status installed xml-core:all 0.18+nmu1
2019-07-21 07:38:42 remove xml-core:all 0.18+nmu1 <none>
2019-07-21 07:38:42 status triggers-pending sgml-base:all 1.29
2019-07-21 07:38:42 status triggers-awaited xml-core:all 0.18+nmu1
2019-07-21 07:38:42 status half-configured xml-core:all 0.18+nmu1
2019-07-21 07:38:42 status half-installed xml-core:all 0.18+nmu1
2019-07-21 07:38:42 status config-files xml-core:all 0.18+nmu1
2019-07-21 07:38:42 remove sgml-base:all 1.29 <none>
2019-07-21 07:38:42 status half-configured sgml-base:all 1.29
2019-07-21 07:38:42 status half-installed sgml-base:all 1.29
2019-07-21 07:38:42 status config-files sgml-base:all 1.29
2019-07-21 07:38:42 status installed tcpd:amd64 7.6.q-28
2019-07-21 07:38:42 remove tcpd:amd64 7.6.q-28 <none>
2019-07-21 07:38:42 status half-configured tcpd:amd64 7.6.q-28
2019-07-21 07:38:42 status half-installed tcpd:amd64 7.6.q-28
2019-07-21 07:38:42 status config-files tcpd:amd64 7.6.q-28
2019-07-21 07:38:42 status not-installed tcpd:amd64 <none>
2019-07-21 07:38:42 status installed xen-hypervisor-4.8-amd64:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:38:42 remove xen-hypervisor-4.8-amd64:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11 <none>
2019-07-21 07:38:42 status half-configured
xen-hypervisor-4.8-amd64:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:38:42 status half-installed xen-hypervisor-4.8-amd64:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:39:41 status config-files xen-hypervisor-4.8-amd64:amd64
4.8.5+shim4.10.2+xsa282-1+deb9u11
2019-07-21 07:39:41 startup packages configure
2019-07-21 07:39:41 trigproc libc-bin:amd64 2.28-10 <none>
2019-07-21 07:39:41 status half-configured libc-bin:amd64 2.28-10
2019-07-21 07:39:41 status installed libc-bin:amd64 2.28-10
2019-07-21 07:39:41 trigproc man-db:amd64 2.8.5-2 <none>
2019-07-21 07:39:41 status half-configured man-db:amd64 2.8.5-2

And here is the relevant portion of /var/log/history.log:
Start-Date: 2019-07-21  07:38:37
Commandline: apt autoremove
Requested-By: [user]
Remove: liblvm2cmd2.02:amd64 (2.02.168-2), libisccfg140:amd64
(1:9.10.3.dfsg.P4-12.3+deb9u5), python3-distutils:amd64 (3.7.3-1),
sgml-base:amd64 (1.29), rename:amd64 (1.10-1), libicu57:amd64
(57.1-6+deb9u2), python3.5:amd64 (3.5.3-1+deb9u1),
python3.5-minimal:amd64 (3.5.3-1+deb9u1), libisc160:amd64
(1:9.10.3.dfsg.P4-12.3+deb9u5), libperl5.24:amd64 (5.24.1-3+deb9u5),
xml-core:amd64 (0.18+nmu1), libapparmor-perl:amd64 (2.13.2-10),
guile-2.0-libs:amd64 (2.0.13+1-5.1), liblvm2app2.2:amd64 (2.02.168-2),
liblwres141:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u5), tcpd:amd64 (7.6.q-28),
python3-lib2to3:amd64 (3.7.3-1), xen-hypervisor-4.8-amd64:amd64
(4.8.5+shim4.10.2+xsa282-1+deb9u11), dh-python:amd64 (3.20190308),
libxen-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), libdns162:amd64
(1:9.10.3.dfsg.P4-12.3+deb9u5), libisccc140:amd64
(1:9.10.3.dfsg.P4-12.3+deb9u5), libpython3.5-stdlib:amd64
(3.5.3-1+deb9u1), libbind9-140:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u5),
libpython3.5-minimal:amd64 (3.5.3-1+deb9u1), xen-utils-4.8:amd64
(4.8.5+shim4.10.2+xsa282-1+deb9u11)
End-Date: 2019-07-21  07:39:44

niek

Some more lines from /var/log/syslog of that same time.
Looks like some process is running various scripts in
/usr/lib/linux-boot-probes/ and /usr/lib/os-probes.

Jul 21 07:38:41 host systemd[1]: xen.service: Succeeded.
Jul 21 07:38:41 host systemd[1]: Stopped LSB: Xen daemons.
Jul 21 07:38:53 host kernel: SGI XFS with ACLs, security attributes,
realtime, no debug enabled
Jul 21 07:38:53 host kernel: JFS: nTxBlock = 7137, nTxLock = 57100
Jul 21 07:38:53 host kernel: QNX4 filesystem 0.2.3 registered.
Jul 21 07:38:54 host kernel: raid6: sse2x1   gen()  4028 MB/s
Jul 21 07:38:54 host kernel: raid6: sse2x1   xor()  3036 MB/s
Jul 21 07:38:54 host kernel: raid6: sse2x2   gen()  4666 MB/s
Jul 21 07:38:54 host kernel: raid6: sse2x2   xor()  3596 MB/s
Jul 21 07:38:54 host kernel: raid6: sse2x4   gen()  5263 MB/s
Jul 21 07:38:54 host kernel: raid6: sse2x4   xor()  3941 MB/s
Jul 21 07:38:54 host kernel: raid6: using algorithm sse2x4 gen() 5263 MB/s
Jul 21 07:38:54 host kernel: raid6: .... xor() 3941 MB/s, rmw enabled
Jul 21 07:38:54 host kernel: raid6: using ssse3x2 recovery algorithm
Jul 21 07:38:54 host kernel: xor: measuring software checksum speed
Jul 21 07:38:54 host kernel:    prefetch64-sse:  6630.000 MB/sec
Jul 21 07:38:54 host kernel:    generic_sse:  5853.000 MB/sec
Jul 21 07:38:54 host kernel: xor: using function: prefetch64-sse
(6630.000 MB/sec)
Jul 21 07:38:54 host kernel: Btrfs loaded, crc32c=crc32c-intel
Jul 21 07:38:54 host kernel: fuse init (API version 7.27)
Jul 21 07:38:54 host systemd[1]: Mounting FUSE Control File System...
Jul 21 07:38:54 host systemd[1]: Mounted FUSE Control File System.
Jul 21 07:38:55 host os-prober[11566]: debug: running
/usr/lib/os-probes/mounted/05efi on mounted /dev/sda1
(...)
Jul 21 07:39:38 host 50mounted-tests[16816]: debug:
/usr/lib/linux-boot-probes/mounted/40grub2 succeeded
Jul 21 07:39:38 host systemd[1828]: var-lib-os\x2dprober-mount.mount:
Succeeded.
Jul 21 07:39:38 host systemd[1]: var-lib-os\x2dprober-mount.mount:
Succeeded.
Jul 21 07:39:38 host linux-boot-prober[16820]: debug: linux detected by
/usr/lib/linux-boot-probes/50mounted-tests


niek

On 7/23/19 4:07 PM, niek wrote:
> [...]
> 2019-07-21 07:38:40 status installed xen-utils-4.8:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11
> 2019-07-21 07:38:40 remove xen-utils-4.8:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11 <none>
> 2019-07-21 07:38:40 status half-configured xen-utils-4.8:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11
> 2019-07-21 07:38:41 status half-installed xen-utils-4.8:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11
> 2019-07-21 07:38:41 status config-files xen-utils-4.8:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11
> 2019-07-21 07:38:41 status not-installed xen-utils-4.8:amd64 <none>
> 2019-07-21 07:38:41 status installed libxen-4.8:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11
> 2019-07-21 07:38:41 remove libxen-4.8:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11 <none>
> 2019-07-21 07:38:41 status half-configured libxen-4.8:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11
> 2019-07-21 07:38:41 status half-installed libxen-4.8:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11
> 2019-07-21 07:38:41 status config-files libxen-4.8:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11
> 2019-07-21 07:38:41 status not-installed libxen-4.8:amd64 <none>
> [...]
> 2019-07-21 07:38:42 status installed xen-hypervisor-4.8-amd64:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11
> 2019-07-21 07:38:42 remove xen-hypervisor-4.8-amd64:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11 <none>
> 2019-07-21 07:38:42 status half-configured
> xen-hypervisor-4.8-amd64:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11
> 2019-07-21 07:38:42 status half-installed xen-hypervisor-4.8-amd64:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11
> 2019-07-21 07:39:41 status config-files xen-hypervisor-4.8-amd64:amd64
> 4.8.5+shim4.10.2+xsa282-1+deb9u11
Ok, so, the most interesting question for me is...

On line 50 in the init script:

https://salsa.debian.org/xen-team/debian-xen/blob/master/debian/xen-utils-common.xen.init

case $DPKG_MAINTSCRIPT_PACKAGE in
xen-utils-$VERSION) ;;  # xen-utils-V maintscript, under Xen X=V
xen-utils-*)        exit 0;; # xen-utils-V maintscript, but under Xen X!=V
*)          ;;  # maybe not under dpkg, etc.
esac

What is the value of this $DPKG_MAINTSCRIPT_PACKAGE when it happens?

Could it be something else than something beginning with xen-utils-?
I have a suspicion that the systemd[1]: Reloading. has something to do
with it. Or the triggers?

Anyway, if DPKG_MAINTSCRIPT_PACKAGE gets lost *anywhere* in whatever
happens, it might end up as empty, and then matching just *.

But, we really need find out how to reproduce it in a test environment. :|

Hans

This should be enough to finally fix the problem of the mysteriously
disappearing xenconsoled process. We have tried to fix this before, but
it turned out the fix was incomplete.

The two attached patches...
* revert the previous fix
* prevent xen-utils-V prerm and postinst to call the xen init script
  when V != running version X.
* remove even more additional extra bonus redundant superfluous
  supererogatory inordinate loquacious start/stop calls from the
  xen-utils-common maintainer scripts, which were put there by
  dh_installinit and went unnoticed so far.

Ian, can you give your A-B on this. It will have to go into buster as
well, to help users upgrade to Bullseye without these problems.

The test scenario I used (all on current Debian unstable):
[x] Reproduce the problem (disappearing xenconsoled) with current
    packages
[x] Install fixed 4.11 packages, and check that when upgrading 4.11 to
    4.11, the init script stop/start is called
[x] Install fixed 4.13 packages, and check that the init script is not
    called when installing xen-utils-4.13 and when upgrading
    xen-utils-common
[x] Reboot into Xen 4.13
[x] Remove xen-utils-4.11 and check that the stop action on the init
    script is not called.
[x] Install xen-utils-4.11 again and check that the start action is
    not called.
[x] Reboot into just Linux without Xen
[x] Remove xen-utils-4.11 and check that this works good enough. It's
    allowed to print some complaints on the screen and behave a little
    weird, but it should not totally explode.

Now, there's a last edge case I can think of, which is installing
xen-utils-V in a domU. In there, the /usr/lib/xen-common/bin/xen-version
script will return the Xen version of the host that is carrying this
domU and then do a thing. I do not think we actively support doing
interesting things inside a domU with these packages however.

Hans van Kranenburg (2):
  xen init/maint scripts: Do nothing if running for wrong Xen package
  debian/rules: --no-start for xen dh_installinit

 debian/rules                       |  2 +-
 debian/xen-utils-V.postinst.vsn-in | 10 +++++++++-
 debian/xen-utils-V.prerm.vsn-in    | 10 +++++++++-
 debian/xen-utils-common.xen.init   | 27 ---------------------------
 4 files changed, 19 insertions(+), 30 deletions(-)

-- 
2.20.1

After trying to fix this issue in the init script, we found out that the
problem still happened for systems running with systemd.

The xen-utils-V postinst and prerm have DPKG_MAINTSCRIPT_PACKAGE in
their environment. When calling invoke-rc.d xen <action> under systemd,
the whole circus of translation and compatibility layers is used to
finally end up running the /etc/init.d/xen script again. However, when
ending up there, the DPKG_MAINTSCRIPT_PACKAGE variable is lost.

So, instead of trying to fix this in the init script, avoid calling
invoke-rc.d altogether, when installing or removing for a different
version of Xen than the currently running one.

Since we only call this from two places, and the check is a one liner,
directly put it into the prerm and postinst.

Carefully quote the values on both sides of the comparison. For example,
when removing a xen-utils-V package after rebooting into just Linux
without Xen, the version retrieval helper will print an error like
"ERROR:  Can't find hypervisor information in sysfs!", there will
be no
useful output on stdout and it will compare an empty string with the
version of the xen-utils package, resulting in the right action, not
trying to stop or start anything.

To avoid hitting the disappearing xenconsoled scenario, the fix has to
be present in the maintainer scripts of the to be removed *old*
xen-utils-V package. This means users will have to first upgrade to a
package with this fix before upgrading to a different Xen version.

Signed-off-by: Hans van Kranenburg <hans at knorrie.org>
Closes: #932759 (1/2)
Fixes: cc85504103 "xen init script: Do nothing if running for wrong Xen
package"
---
 debian/xen-utils-V.postinst.vsn-in | 10 +++++++++-
 debian/xen-utils-V.prerm.vsn-in    | 10 +++++++++-
 debian/xen-utils-common.xen.init   | 27 ---------------------------
 3 files changed, 18 insertions(+), 29 deletions(-)

diff --git a/debian/xen-utils-V.postinst.vsn-in
b/debian/xen-utils-V.postinst.vsn-in
index 581327f09ffd..0acebf836bb2 100644
--- a/debian/xen-utils-V.postinst.vsn-in
+++ b/debian/xen-utils-V.postinst.vsn-in
@@ -6,7 +6,15 @@ case "$1" in
     configure)
         update-alternatives --remove xen-default /usr/lib/xen- at version@
         if [ -x "/etc/init.d/xen" ]; then
-	    invoke-rc.d xen start || exit $?
+            # Only call the init script when this xen-utils- at version@
package
+            # matches the currently running version of Xen. This means, doing
+            # in-place updates (e.g. a security update for same version).
+            #
+            # When installing a xen-utils package for any other Xen version,
+            # leave the running system alone.
+            if [ "$(/usr/lib/xen-common/bin/xen-version)" =
"@version@" ]; then
+                invoke-rc.d xen start || exit $?
+            fi
         fi
     ;;
 
diff --git a/debian/xen-utils-V.prerm.vsn-in b/debian/xen-utils-V.prerm.vsn-in
index 1aa2cae65fda..f1cb4299c30c 100644
--- a/debian/xen-utils-V.prerm.vsn-in
+++ b/debian/xen-utils-V.prerm.vsn-in
@@ -6,7 +6,15 @@ case "$1" in
     remove|upgrade)
         update-alternatives --remove xen-default /usr/lib/xen- at version@
         if [ -x "/etc/init.d/xen" ]; then
-            invoke-rc.d xen stop || exit $?
+            # Only call the init script when removing or while upgrading for
+            # the currently running version of Xen.
+            #
+            # Otherwise, for example after a Xen version upgrade, autoremoval
+            # of an obsolete xen-utils-V package would inadvertently stop
+            # running daemons like xenconsoled.
+            if [ "$(/usr/lib/xen-common/bin/xen-version)" =
"@version@" ]; then
+                invoke-rc.d xen stop || exit $?
+            fi
         fi
     ;;
 
diff --git a/debian/xen-utils-common.xen.init b/debian/xen-utils-common.xen.init
index f66ce6b8db18..05521733494e 100644
--- a/debian/xen-utils-common.xen.init
+++ b/debian/xen-utils-common.xen.init
@@ -26,33 +26,6 @@ xen)		;;
 esac
 
 VERSION=$(/usr/lib/xen-common/bin/xen-version)
-
-# The arrangements for the `xen' init script are a bit odd.
-# This script is part of xen-utils-common, of which there is one
-# version installed regardless of the Xen version.
-#
-# But it is called by the prerm and postinsts of xen-utils-VERSION.
-# The idea is that (for example) if xen-utils-VERSION is upgraded, the
-# daemons are restarted.
-#
-# However, this means that this script may be called by the
-# maintscript of a xen-utils-V package for a different V to the
-# running version of Xen (X, say).  Such a xen-utils-V package does
-# not actually want to start or stop its daemons.  Indeed, the version
-# selection machinery would redirect its efforts to the xen-utils-X
-# utilities.  But this is not right: we don't actually want to (for
-# example) stop xenconsoled from xen-utils-X just because some
-# not-currently-relevant xen-utils-V is installed/removed/whatever.
-#
-# So we use DPKG_MAINTSCRIPT_PACKAGE to detect this situation, and
-# turn these extraneous calls into no-ops.
-
-case $DPKG_MAINTSCRIPT_PACKAGE in
-xen-utils-$VERSION)	;;	# xen-utils-V maintscript, under Xen X=V
-xen-utils-*)		exit 0;; # xen-utils-V maintscript, but under Xen X!=V
-*)			;;	# maybe not under dpkg, etc.
-esac
-
 ROOT=$(/usr/lib/xen-common/bin/xen-dir)
 if [ $? -ne 0 ]; then
 	log_warning_msg "No compatible Xen utils for Xen $VERSION"
-- 
2.20.1

When debugging the xen-utils postinst/prerm to find the cause of the
mysteriously disappearing xenconsoled processes, I discovered that the
xen-utils-common postinst and prerm stop and start the xen init script
as well!

These commands are not visible in the packaging code, but they are added
by dh_installdeb into the postinst and prerm during package build time.

We only want to call the script from xen-utils-V, so disable this
behavior by using --no-start

Closes: #932759 (2/2)
Signed-off-by: Hans van Kranenburg <hans at knorrie.org>
---
 debian/rules | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/debian/rules b/debian/rules
index 23c982eb414b..73232ca20efe 100755
--- a/debian/rules
+++ b/debian/rules
@@ -282,7 +282,7 @@ override_dh_python2:
 
 # We have two init scripts.  (There used to be xend too.)
 override_dh_installinit:
-	dh_installinit --name xen -- defaults 20 21
+	dh_installinit --name xen --no-start -- defaults 20 21
 	dh_installinit --name xendomains --no-start -- defaults 21 20
 
 # dh_strip in dh compat 10 and earlier (which we are at so this
-- 
2.20.1

Hans van Kranenburg writes ("[PATCH 1/2] xen init/maint scripts: Do nothing
if running for wrong Xen package"):
> After trying to fix this issue in the init script, we found out that the
> problem still happened for systems running with systemd.
> 
> The xen-utils-V postinst and prerm have DPKG_MAINTSCRIPT_PACKAGE in
> their environment. When calling invoke-rc.d xen <action> under
systemd,
> the whole circus of translation and compatibility layers is used to
> finally end up running the /etc/init.d/xen script again. However, when
> ending up there, the DPKG_MAINTSCRIPT_PACKAGE variable is lost.
> 
> So, instead of trying to fix this in the init script, avoid calling
> invoke-rc.d altogether, when installing or removing for a different
> version of Xen than the currently running one.
> 
> Since we only call this from two places, and the check is a one liner,
> directly put it into the prerm and postinst.
> 
> Carefully quote the values on both sides of the comparison. For example,
> when removing a xen-utils-V package after rebooting into just Linux
> without Xen, the version retrieval helper will print an error like
> "ERROR:  Can't find hypervisor information in sysfs!", there
will be no
> useful output on stdout and it will compare an empty string with the
> version of the xen-utils package, resulting in the right action, not
> trying to stop or start anything.
> 
> To avoid hitting the disappearing xenconsoled scenario, the fix has to
> be present in the maintainer scripts of the to be removed *old*
> xen-utils-V package. This means users will have to first upgrade to a
> package with this fix before upgrading to a different Xen version.
> 
> Signed-off-by: Hans van Kranenburg <hans at knorrie.org>
> Closes: #932759 (1/2)
> Fixes: cc85504103 "xen init script: Do nothing if running for wrong
Xen package"
Reviewed-by: Ian Jackson <ijackson at chiark.greenend.org.uk>

Hans van Kranenburg writes ("[PATCH 2/2] debian/rules: --no-start for xen
dh_installinit"):
> When debugging the xen-utils postinst/prerm to find the cause of the
> mysteriously disappearing xenconsoled processes, I discovered that the
> xen-utils-common postinst and prerm stop and start the xen init script
> as well!
> 
> These commands are not visible in the packaging code, but they are added
> by dh_installdeb into the postinst and prerm during package build time.
> 
> We only want to call the script from xen-utils-V, so disable this
> behavior by using --no-start
> 
> Closes: #932759 (2/2)
> Signed-off-by: Hans van Kranenburg <hans at knorrie.org>
Reviewed-by: Ian Jackson <ijackson at chiark.greenend.org.uk>

I think it would be wise to look at the generated .debs and see that
they contain (only) the expected pieces in their maintscripts.

Ian.

Hi,

On 5/26/20 12:44 PM, Ian Jackson wrote:
> Hans van Kranenburg writes ("[PATCH 2/2] debian/rules: --no-start for
xen dh_installinit"):
>> When debugging the xen-utils postinst/prerm to find the cause of the
>> mysteriously disappearing xenconsoled processes, I discovered that the
>> xen-utils-common postinst and prerm stop and start the xen init script
>> as well!
>>
>> These commands are not visible in the packaging code, but they are
added
>> by dh_installdeb into the postinst and prerm during package build time.
>>
>> We only want to call the script from xen-utils-V, so disable this
>> behavior by using --no-start
>>
>> Closes: #932759 (2/2)
>> Signed-off-by: Hans van Kranenburg <hans at knorrie.org>
> 
> Reviewed-by: Ian Jackson <ijackson at chiark.greenend.org.uk>
Thanks.
> I think it would be wise to look at the generated .debs and see that
> they contain (only) the expected pieces in their maintscripts.
Yes, I did this while testing by diffing the files installed in
/var/lib/dpkg/info with the old ones and verifying that exactly that
part went away.

(for 4.11:)

-$ diff -u ~/xen-utils-common.postinst xen-utils-common.postinst
--- /home/beheer/xen-utils-common.postinst	2020-05-26 13:08:45.738926207
+0200
+++ xen-utils-common.postinst	2020-05-25 14:14:28.000000000 +0200
@@ -31,13 +31,7 @@
 # Automatically added by dh_installinit/13.1
 if [ "$1" = "configure" ] || [ "$1" =
"abort-upgrade" ] || [ "$1" "abort-deconfigure" ]
|| [ "$1" = "abort-remove" ] ; then
 	if [ -x "/etc/init.d/xen" ]; then
-		update-rc.d xen defaults 20 21 >/dev/null
-		if [ -n "$2" ]; then
-			_dh_action=restart
-		else
-			_dh_action=start
-		fi
-		invoke-rc.d xen $_dh_action || exit 1
+		update-rc.d xen defaults 20 21 >/dev/null || exit 1
 	fi
 fi
 # End automatically added section


-$ diff -u ~/xen-utils-common.prerm xen-utils-common.prerm
--- /home/beheer/xen-utils-common.prerm	2020-05-26 13:09:01.570617331 +0200
+++ xen-utils-common.prerm	2020-05-25 14:14:28.000000000 +0200
@@ -1,10 +1,5 @@
 #!/bin/sh
 set -e
-# Automatically added by dh_installinit/13.1
-if [ -x "/etc/init.d/xen" ] && [ "$1" = remove ];
then
-	invoke-rc.d xen stop || exit 1
-fi
-# End automatically added section
 # Automatically added by dh_installdeb/13.1
 dpkg-maintscript-helper rm_conffile /etc/default/xend 4.11.1-2\~ --
"$@"
 dpkg-maintscript-helper rm_conffile /etc/xen/xend-config.sxp 4.11.1-2\~
-- "$@"

Hans

Hans van Kranenburg writes ("Re: [PATCH 2/2] debian/rules: --no-start for
xen dh_installinit"):
> On 5/26/20 12:44 PM, Ian Jackson wrote:
> > I think it would be wise to look at the generated .debs and see that
> > they contain (only) the expected pieces in their maintscripts.
> 
> Yes, I did this while testing by diffing the files installed in
> /var/lib/dpkg/info with the old ones and verifying that exactly that
> part went away.
Cool.  LGTM.

Thanks,
Ian.

-- 
Ian Jackson <ijackson at chiark.greenend.org.uk>   These opinions are my
own.

Pronouns: they/he.  If I emailed you from @fyvzl.net or @evade.org.uk,
that is a private address which bypasses my fierce spamfilter.

Your message dated Wed, 27 May 2020 17:36:26 +0000
with message-id <E1jdzyc-0007Zm-1B at fasolo.debian.org>
and subject line Bug#932759: fixed in xen 4.11.4-1
has caused the Debian Bug report #932759,
regarding After upgrade from stretch to buster, removal of obsolete xen 4.8
packages seems to trigger shutdown of xenconsoled
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
932759: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932759
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: niek <niek at niek.org>
Subject: After upgrade from stretch to buster, removal of obsolete xen 4.8
packages seems to trigger shutdown of xenconsoled
Date: Mon, 22 Jul 2019 14:32:16 -0400
Size: 5705
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20200527/66efe7f5/attachment.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Debian FTP Masters <ftpmaster at ftp-master.debian.org>
Subject: Bug#932759: fixed in xen 4.11.4-1
Date: Wed, 27 May 2020 17:36:26 +0000
Size: 6138
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20200527/66efe7f5/attachment-0001.mht>

Hi,

On 5/27/20 7:39 PM, Debian Bug Tracking System wrote:
> Your message dated Wed, 27 May 2020 17:36:26 +0000
> with message-id <E1jdzyc-0007Zm-1B at fasolo.debian.org>
> and subject line Bug#932759: fixed in xen 4.11.4-1
> has caused the Debian Bug report #932759,
> regarding After upgrade from stretch to buster, removal of obsolete xen 4.8
packages seems to trigger shutdown of xenconsoled
> to be marked as done.
> 
> This means that you claim that the problem has been dealt with.
To avoid confusion, yes, this one closes with the upload of 4.11.4 to
unstable which has the fix.

However, it's still present in 4.11.3+24-g14b62ab3e5-1~deb10u1 in
buster. So, the same fix will also go into buster later, to in the end
help users upgrade from buster to bullseye.

Hans