niek
2019-Jul-22 18:32 UTC
[Pkg-xen-devel] Bug#932759: After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled
Package: xen-hypervisor-4.11-amd64 Version: 4.11.1+92-g6c33308a8d-2 System: Linux [host] 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5+deb10u1 (2019-07-19) x86_64 GNU/Linux Debian Release: 10 Codename: buster In Reference To: Debian bug #851654 (archived)> On Mon, 21 Jan 2019 00:48:09 +0100 Hans van Kranenburg > <hans at knorrie.org> wrote: >> reassign 851654 src:xen >> thanks >> >> Hm! >> >> I think this is the same one that I've been observing during upgrade >> tests from 4.8 -> 4.11 and 4.10 -> 4.11. >> >> In my IRC logs I can find myself complaining about xenconsoled that's >> suddenly gone multiple times during 2018. I didn't manage to track this >> issue down yet. The IRC logs contain some links to expired pastebin >> entries. :| >> >> Also, in some cases xenconsoled just got shut down during the night, >> during some systemd reload whatever was happening. >> >> I suspect more users are going to run into this issue when upgrading to >> Buster. >> >> HansWhat happened: - upgraded Debian Xen Dom0 from stretch to buster and rebooted, as described in https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html - started some Linux pv domu without problems - removed obsolete packages with 'apt autoremove'. This removed (among others) xen-hypervisor-4.8-amd64:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), libxen-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), xen-utils-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11) - starting another pv domu the next day using 'xl create -c [domu.cfg]' raised an error message: "xenconsole: Could not read tty from store: Success" - opening a console for an already running domu raised another error message that I believe was: "xenconsole: Could not open tty `/dev/pts/2': No such file or directory" - some internet searching hinted that this could be caused by xenconsoled not running. - xenconsoled was not running - searching system logs revealed that xenconsoled seemed to have stopped when 'apt autoremove' removed the obsolete xen 4.8 packages after upgrading to xen 4.11. - this coincided with a systemd reload as seen in journalctl: Jul 21 07:38:40 host systemd[1]: Reloading. Jul 21 07:38:40 host systemd[1]: serial-getty at hvc0.service: Current command vanished from the unit file, execution of the command list won't be resumed. Jul 21 07:38:40 host systemd[1]: serial-getty at ttyS1.service: Current command vanished from the unit file, execution of the command list won't be resumed. Jul 21 07:38:40 host systemd[1]: getty at tty1.service: Current command vanished from the unit file, execution of the command list won't be resumed. Jul 21 07:38:41 host systemd[1]: Stopping LSB: Xen daemons... Jul 21 07:38:41 host xen[9092]: Stopping Xen daemons: xenconsoled. Jul 21 07:38:41 host systemd[1]: xen.service: Succeeded. Jul 21 07:38:41 host systemd[1]: Stopped LSB: Xen daemons. Note that at this time domu's where already running under xen4.11. 'systemctl restart xen.service' solved the issue. I report the issue because it could affect others that are performing upgrades.
Hans van Kranenburg
2019-Jul-22 22:57 UTC
[Pkg-xen-devel] Bug#932759: Bug#932759: After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled
Hi niek, Thanks for the report! On 7/22/19 8:32 PM, niek wrote:> Package: xen-hypervisor-4.11-amd64 > Version: 4.11.1+92-g6c33308a8d-2 > > What happened: > - upgraded Debian Xen Dom0 from stretch to buster and rebooted, as > described in > https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html > > - started some Linux pv domu without problems > > - removed obsolete packages with 'apt autoremove'. This removed (among > others) > xen-hypervisor-4.8-amd64:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), > libxen-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), > xen-utils-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11) > > [...] > - xenconsoled was not running > > - searching system logs revealed that xenconsoled seemed to have stopped > when 'apt autoremove' removed the obsolete xen 4.8 > packages after upgrading to xen 4.11.Well, there it is again. We tried to make a fix, exactly for this... https://salsa.debian.org/xen-team/debian-xen/commit/ef242a700765a971a6afc12d25ee19944dd3a27a ...and apparently there's another scenario in which even this doesn't work? Can you show the lines from /var/log/dpkg.log from that moment, the seconds around 07:38:40? It tells exactly what got removed, in what second, just to confirm? I'm pretty sure I tried to reproduce this after we added the fix I just referenced, and I was unable to. So, I'm very interested in finding out what's still going on here. Usually being able to reproduce a problem is one of the biggest steps towards finding a solution. (since it can be done over and over again, finding out what exactly causes it). So, finding the right sequence of steps to make it happen again is crucial here. Do you think the systemd reload has anything to do with it? Maybe the whole systemd init-script-wrapper-trickery is misbehaving in some way? Can you reproduce this by manually grabbing the xen-hypervisor-4.8-amd64, libxen-4.8 and xen-utils-4.8 from stretch again, installing them and removing them again? Do you have any other idea? Thanks, Hans
niek
2019-Jul-23 14:07 UTC
[Pkg-xen-devel] Bug#932759: Bug#932759: After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled
On 22-7-2019 18:57, Hans van Kranenburg wrote:> Hi niek, > > Thanks for the report! > > On 7/22/19 8:32 PM, niek wrote: >> Package: xen-hypervisor-4.11-amd64 >> Version: 4.11.1+92-g6c33308a8d-2 >> >> What happened: >> - upgraded Debian Xen Dom0 from stretch to buster and rebooted, as >> described in >> https://www.debian.org/releases/buster/amd64/release-notes/ch-upgrading.en.html >> >> - started some Linux pv domu without problems >> >> - removed obsolete packages with 'apt autoremove'. This removed (among >> others) >> xen-hypervisor-4.8-amd64:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), >> libxen-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), >> xen-utils-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11) >> >> [...] >> - xenconsoled was not running >> >> - searching system logs revealed that xenconsoled seemed to have stopped >> when 'apt autoremove' removed the obsolete xen 4.8 >> packages after upgrading to xen 4.11. > > Well, there it is again. We tried to make a fix, exactly for this... > > https://salsa.debian.org/xen-team/debian-xen/commit/ef242a700765a971a6afc12d25ee19944dd3a27a > > ...and apparently there's another scenario in which even this doesn't work? > > Can you show the lines from /var/log/dpkg.log from that moment, the > seconds around 07:38:40? It tells exactly what got removed, in what > second, just to confirm? > > I'm pretty sure I tried to reproduce this after we added the fix I just > referenced, and I was unable to. So, I'm very interested in finding out > what's still going on here. > > Usually being able to reproduce a problem is one of the biggest steps > towards finding a solution. (since it can be done over and over again, > finding out what exactly causes it). So, finding the right sequence of > steps to make it happen again is crucial here. > > Do you think the systemd reload has anything to do with it? Maybe the > whole systemd init-script-wrapper-trickery is misbehaving in some way? > > Can you reproduce this by manually grabbing the > xen-hypervisor-4.8-amd64, libxen-4.8 and xen-utils-4.8 from stretch > again, installing them and removing them again? Do you have any other idea? > > Thanks, > Hans >Hi Hans, Thanks for your work with Ian getting xen 4.11 in buster! Very happy with that. I seem to remember seeing 'processing triggers for systemd' somewhere, sometime, but I can't confirm that from the logs so it probably was at another stage of the upgrade. This is a production system, so I will not try to reproduce this by reinstalling the 4.8 packages as you suggested. There is, however an identical server that I still need to upgrade, so I will be able to look closely what happens with that. Here are the relevant lines from dpkg.conf: 2019-07-21 07:38:39 status installed python3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:39 remove python3.5-minimal:amd64 3.5.3-1+deb9u1 <none> 2019-07-21 07:38:39 status half-configured python3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:39 status half-installed python3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 status config-files python3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 status installed libpython3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 remove libpython3.5-minimal:amd64 3.5.3-1+deb9u1 <none> 2019-07-21 07:38:40 status half-configured libpython3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 status half-installed libpython3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 status config-files libpython3.5-minimal:amd64 3.5.3-1+deb9u1 2019-07-21 07:38:40 status installed xen-utils-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:40 remove xen-utils-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 <none> 2019-07-21 07:38:40 status half-configured xen-utils-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status half-installed xen-utils-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status config-files xen-utils-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status not-installed xen-utils-4.8:amd64 <none> 2019-07-21 07:38:41 status installed libxen-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 remove libxen-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 <none> 2019-07-21 07:38:41 status half-configured libxen-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status half-installed libxen-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status config-files libxen-4.8:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:41 status not-installed libxen-4.8:amd64 <none> 2019-07-21 07:38:41 status installed python3-distutils:all 3.7.3-1 2019-07-21 07:38:41 remove python3-distutils:all 3.7.3-1 <none> 2019-07-21 07:38:41 status half-configured python3-distutils:all 3.7.3-1 2019-07-21 07:38:41 status half-installed python3-distutils:all 3.7.3-1 2019-07-21 07:38:41 status config-files python3-distutils:all 3.7.3-1 2019-07-21 07:38:41 status not-installed python3-distutils:all <none> 2019-07-21 07:38:41 status installed python3-lib2to3:all 3.7.3-1 2019-07-21 07:38:41 remove python3-lib2to3:all 3.7.3-1 <none> 2019-07-21 07:38:41 status half-configured python3-lib2to3:all 3.7.3-1 2019-07-21 07:38:41 status half-installed python3-lib2to3:all 3.7.3-1 2019-07-21 07:38:41 status config-files python3-lib2to3:all 3.7.3-1 2019-07-21 07:38:41 status not-installed python3-lib2to3:all <none> 2019-07-21 07:38:41 status installed rename:all 1.10-1 2019-07-21 07:38:41 remove rename:all 1.10-1 <none> 2019-07-21 07:38:41 status half-configured rename:all 1.10-1 2019-07-21 07:38:42 status half-installed rename:all 1.10-1 2019-07-21 07:38:42 status config-files rename:all 1.10-1 2019-07-21 07:38:42 status not-installed rename:all <none> 2019-07-21 07:38:42 status installed xml-core:all 0.18+nmu1 2019-07-21 07:38:42 remove xml-core:all 0.18+nmu1 <none> 2019-07-21 07:38:42 status triggers-pending sgml-base:all 1.29 2019-07-21 07:38:42 status triggers-awaited xml-core:all 0.18+nmu1 2019-07-21 07:38:42 status half-configured xml-core:all 0.18+nmu1 2019-07-21 07:38:42 status half-installed xml-core:all 0.18+nmu1 2019-07-21 07:38:42 status config-files xml-core:all 0.18+nmu1 2019-07-21 07:38:42 remove sgml-base:all 1.29 <none> 2019-07-21 07:38:42 status half-configured sgml-base:all 1.29 2019-07-21 07:38:42 status half-installed sgml-base:all 1.29 2019-07-21 07:38:42 status config-files sgml-base:all 1.29 2019-07-21 07:38:42 status installed tcpd:amd64 7.6.q-28 2019-07-21 07:38:42 remove tcpd:amd64 7.6.q-28 <none> 2019-07-21 07:38:42 status half-configured tcpd:amd64 7.6.q-28 2019-07-21 07:38:42 status half-installed tcpd:amd64 7.6.q-28 2019-07-21 07:38:42 status config-files tcpd:amd64 7.6.q-28 2019-07-21 07:38:42 status not-installed tcpd:amd64 <none> 2019-07-21 07:38:42 status installed xen-hypervisor-4.8-amd64:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:42 remove xen-hypervisor-4.8-amd64:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 <none> 2019-07-21 07:38:42 status half-configured xen-hypervisor-4.8-amd64:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:38:42 status half-installed xen-hypervisor-4.8-amd64:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:39:41 status config-files xen-hypervisor-4.8-amd64:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 2019-07-21 07:39:41 startup packages configure 2019-07-21 07:39:41 trigproc libc-bin:amd64 2.28-10 <none> 2019-07-21 07:39:41 status half-configured libc-bin:amd64 2.28-10 2019-07-21 07:39:41 status installed libc-bin:amd64 2.28-10 2019-07-21 07:39:41 trigproc man-db:amd64 2.8.5-2 <none> 2019-07-21 07:39:41 status half-configured man-db:amd64 2.8.5-2 And here is the relevant portion of /var/log/history.log: Start-Date: 2019-07-21 07:38:37 Commandline: apt autoremove Requested-By: [user] Remove: liblvm2cmd2.02:amd64 (2.02.168-2), libisccfg140:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u5), python3-distutils:amd64 (3.7.3-1), sgml-base:amd64 (1.29), rename:amd64 (1.10-1), libicu57:amd64 (57.1-6+deb9u2), python3.5:amd64 (3.5.3-1+deb9u1), python3.5-minimal:amd64 (3.5.3-1+deb9u1), libisc160:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u5), libperl5.24:amd64 (5.24.1-3+deb9u5), xml-core:amd64 (0.18+nmu1), libapparmor-perl:amd64 (2.13.2-10), guile-2.0-libs:amd64 (2.0.13+1-5.1), liblvm2app2.2:amd64 (2.02.168-2), liblwres141:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u5), tcpd:amd64 (7.6.q-28), python3-lib2to3:amd64 (3.7.3-1), xen-hypervisor-4.8-amd64:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), dh-python:amd64 (3.20190308), libxen-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11), libdns162:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u5), libisccc140:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u5), libpython3.5-stdlib:amd64 (3.5.3-1+deb9u1), libbind9-140:amd64 (1:9.10.3.dfsg.P4-12.3+deb9u5), libpython3.5-minimal:amd64 (3.5.3-1+deb9u1), xen-utils-4.8:amd64 (4.8.5+shim4.10.2+xsa282-1+deb9u11) End-Date: 2019-07-21 07:39:44 niek
niek
2019-Jul-23 20:04 UTC
[Pkg-xen-devel] Bug#932759: Bug#932759: After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled
Some more lines from /var/log/syslog of that same time. Looks like some process is running various scripts in /usr/lib/linux-boot-probes/ and /usr/lib/os-probes. Jul 21 07:38:41 host systemd[1]: xen.service: Succeeded. Jul 21 07:38:41 host systemd[1]: Stopped LSB: Xen daemons. Jul 21 07:38:53 host kernel: SGI XFS with ACLs, security attributes, realtime, no debug enabled Jul 21 07:38:53 host kernel: JFS: nTxBlock = 7137, nTxLock = 57100 Jul 21 07:38:53 host kernel: QNX4 filesystem 0.2.3 registered. Jul 21 07:38:54 host kernel: raid6: sse2x1 gen() 4028 MB/s Jul 21 07:38:54 host kernel: raid6: sse2x1 xor() 3036 MB/s Jul 21 07:38:54 host kernel: raid6: sse2x2 gen() 4666 MB/s Jul 21 07:38:54 host kernel: raid6: sse2x2 xor() 3596 MB/s Jul 21 07:38:54 host kernel: raid6: sse2x4 gen() 5263 MB/s Jul 21 07:38:54 host kernel: raid6: sse2x4 xor() 3941 MB/s Jul 21 07:38:54 host kernel: raid6: using algorithm sse2x4 gen() 5263 MB/s Jul 21 07:38:54 host kernel: raid6: .... xor() 3941 MB/s, rmw enabled Jul 21 07:38:54 host kernel: raid6: using ssse3x2 recovery algorithm Jul 21 07:38:54 host kernel: xor: measuring software checksum speed Jul 21 07:38:54 host kernel: prefetch64-sse: 6630.000 MB/sec Jul 21 07:38:54 host kernel: generic_sse: 5853.000 MB/sec Jul 21 07:38:54 host kernel: xor: using function: prefetch64-sse (6630.000 MB/sec) Jul 21 07:38:54 host kernel: Btrfs loaded, crc32c=crc32c-intel Jul 21 07:38:54 host kernel: fuse init (API version 7.27) Jul 21 07:38:54 host systemd[1]: Mounting FUSE Control File System... Jul 21 07:38:54 host systemd[1]: Mounted FUSE Control File System. Jul 21 07:38:55 host os-prober[11566]: debug: running /usr/lib/os-probes/mounted/05efi on mounted /dev/sda1 (...) Jul 21 07:39:38 host 50mounted-tests[16816]: debug: /usr/lib/linux-boot-probes/mounted/40grub2 succeeded Jul 21 07:39:38 host systemd[1828]: var-lib-os\x2dprober-mount.mount: Succeeded. Jul 21 07:39:38 host systemd[1]: var-lib-os\x2dprober-mount.mount: Succeeded. Jul 21 07:39:38 host linux-boot-prober[16820]: debug: linux detected by /usr/lib/linux-boot-probes/50mounted-tests niek
Hans van Kranenburg
2019-Aug-01 21:51 UTC
[Pkg-xen-devel] Bug#932759: Bug#932759: After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled
On 7/23/19 4:07 PM, niek wrote:> [...] > 2019-07-21 07:38:40 status installed xen-utils-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:40 remove xen-utils-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 <none> > 2019-07-21 07:38:40 status half-configured xen-utils-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status half-installed xen-utils-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status config-files xen-utils-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status not-installed xen-utils-4.8:amd64 <none> > 2019-07-21 07:38:41 status installed libxen-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 remove libxen-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 <none> > 2019-07-21 07:38:41 status half-configured libxen-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status half-installed libxen-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status config-files libxen-4.8:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:41 status not-installed libxen-4.8:amd64 <none> > [...] > 2019-07-21 07:38:42 status installed xen-hypervisor-4.8-amd64:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:42 remove xen-hypervisor-4.8-amd64:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 <none> > 2019-07-21 07:38:42 status half-configured > xen-hypervisor-4.8-amd64:amd64 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:38:42 status half-installed xen-hypervisor-4.8-amd64:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11 > 2019-07-21 07:39:41 status config-files xen-hypervisor-4.8-amd64:amd64 > 4.8.5+shim4.10.2+xsa282-1+deb9u11Ok, so, the most interesting question for me is... On line 50 in the init script: https://salsa.debian.org/xen-team/debian-xen/blob/master/debian/xen-utils-common.xen.init case $DPKG_MAINTSCRIPT_PACKAGE in xen-utils-$VERSION) ;; # xen-utils-V maintscript, under Xen X=V xen-utils-*) exit 0;; # xen-utils-V maintscript, but under Xen X!=V *) ;; # maybe not under dpkg, etc. esac What is the value of this $DPKG_MAINTSCRIPT_PACKAGE when it happens? Could it be something else than something beginning with xen-utils-? I have a suspicion that the systemd[1]: Reloading. has something to do with it. Or the triggers? Anyway, if DPKG_MAINTSCRIPT_PACKAGE gets lost *anywhere* in whatever happens, it might end up as empty, and then matching just *. But, we really need find out how to reproduce it in a test environment. :| Hans
Hans van Kranenburg
2020-May-25 21:38 UTC
[Pkg-xen-devel] Bug#932759: [PATCH 0/2] Bug#932759 Fix misfiring init scripts
This should be enough to finally fix the problem of the mysteriously disappearing xenconsoled process. We have tried to fix this before, but it turned out the fix was incomplete. The two attached patches... * revert the previous fix * prevent xen-utils-V prerm and postinst to call the xen init script when V != running version X. * remove even more additional extra bonus redundant superfluous supererogatory inordinate loquacious start/stop calls from the xen-utils-common maintainer scripts, which were put there by dh_installinit and went unnoticed so far. Ian, can you give your A-B on this. It will have to go into buster as well, to help users upgrade to Bullseye without these problems. The test scenario I used (all on current Debian unstable): [x] Reproduce the problem (disappearing xenconsoled) with current packages [x] Install fixed 4.11 packages, and check that when upgrading 4.11 to 4.11, the init script stop/start is called [x] Install fixed 4.13 packages, and check that the init script is not called when installing xen-utils-4.13 and when upgrading xen-utils-common [x] Reboot into Xen 4.13 [x] Remove xen-utils-4.11 and check that the stop action on the init script is not called. [x] Install xen-utils-4.11 again and check that the start action is not called. [x] Reboot into just Linux without Xen [x] Remove xen-utils-4.11 and check that this works good enough. It's allowed to print some complaints on the screen and behave a little weird, but it should not totally explode. Now, there's a last edge case I can think of, which is installing xen-utils-V in a domU. In there, the /usr/lib/xen-common/bin/xen-version script will return the Xen version of the host that is carrying this domU and then do a thing. I do not think we actively support doing interesting things inside a domU with these packages however. Hans van Kranenburg (2): xen init/maint scripts: Do nothing if running for wrong Xen package debian/rules: --no-start for xen dh_installinit debian/rules | 2 +- debian/xen-utils-V.postinst.vsn-in | 10 +++++++++- debian/xen-utils-V.prerm.vsn-in | 10 +++++++++- debian/xen-utils-common.xen.init | 27 --------------------------- 4 files changed, 19 insertions(+), 30 deletions(-) -- 2.20.1
Hans van Kranenburg
2020-May-25 21:38 UTC
[Pkg-xen-devel] Bug#932759: [PATCH 1/2] xen init/maint scripts: Do nothing if running for wrong Xen package
After trying to fix this issue in the init script, we found out that the problem still happened for systems running with systemd. The xen-utils-V postinst and prerm have DPKG_MAINTSCRIPT_PACKAGE in their environment. When calling invoke-rc.d xen <action> under systemd, the whole circus of translation and compatibility layers is used to finally end up running the /etc/init.d/xen script again. However, when ending up there, the DPKG_MAINTSCRIPT_PACKAGE variable is lost. So, instead of trying to fix this in the init script, avoid calling invoke-rc.d altogether, when installing or removing for a different version of Xen than the currently running one. Since we only call this from two places, and the check is a one liner, directly put it into the prerm and postinst. Carefully quote the values on both sides of the comparison. For example, when removing a xen-utils-V package after rebooting into just Linux without Xen, the version retrieval helper will print an error like "ERROR: Can't find hypervisor information in sysfs!", there will be no useful output on stdout and it will compare an empty string with the version of the xen-utils package, resulting in the right action, not trying to stop or start anything. To avoid hitting the disappearing xenconsoled scenario, the fix has to be present in the maintainer scripts of the to be removed *old* xen-utils-V package. This means users will have to first upgrade to a package with this fix before upgrading to a different Xen version. Signed-off-by: Hans van Kranenburg <hans at knorrie.org> Closes: #932759 (1/2) Fixes: cc85504103 "xen init script: Do nothing if running for wrong Xen package" --- debian/xen-utils-V.postinst.vsn-in | 10 +++++++++- debian/xen-utils-V.prerm.vsn-in | 10 +++++++++- debian/xen-utils-common.xen.init | 27 --------------------------- 3 files changed, 18 insertions(+), 29 deletions(-) diff --git a/debian/xen-utils-V.postinst.vsn-in b/debian/xen-utils-V.postinst.vsn-in index 581327f09ffd..0acebf836bb2 100644 --- a/debian/xen-utils-V.postinst.vsn-in +++ b/debian/xen-utils-V.postinst.vsn-in @@ -6,7 +6,15 @@ case "$1" in configure) update-alternatives --remove xen-default /usr/lib/xen- at version@ if [ -x "/etc/init.d/xen" ]; then - invoke-rc.d xen start || exit $? + # Only call the init script when this xen-utils- at version@ package + # matches the currently running version of Xen. This means, doing + # in-place updates (e.g. a security update for same version). + # + # When installing a xen-utils package for any other Xen version, + # leave the running system alone. + if [ "$(/usr/lib/xen-common/bin/xen-version)" = "@version@" ]; then + invoke-rc.d xen start || exit $? + fi fi ;; diff --git a/debian/xen-utils-V.prerm.vsn-in b/debian/xen-utils-V.prerm.vsn-in index 1aa2cae65fda..f1cb4299c30c 100644 --- a/debian/xen-utils-V.prerm.vsn-in +++ b/debian/xen-utils-V.prerm.vsn-in @@ -6,7 +6,15 @@ case "$1" in remove|upgrade) update-alternatives --remove xen-default /usr/lib/xen- at version@ if [ -x "/etc/init.d/xen" ]; then - invoke-rc.d xen stop || exit $? + # Only call the init script when removing or while upgrading for + # the currently running version of Xen. + # + # Otherwise, for example after a Xen version upgrade, autoremoval + # of an obsolete xen-utils-V package would inadvertently stop + # running daemons like xenconsoled. + if [ "$(/usr/lib/xen-common/bin/xen-version)" = "@version@" ]; then + invoke-rc.d xen stop || exit $? + fi fi ;; diff --git a/debian/xen-utils-common.xen.init b/debian/xen-utils-common.xen.init index f66ce6b8db18..05521733494e 100644 --- a/debian/xen-utils-common.xen.init +++ b/debian/xen-utils-common.xen.init @@ -26,33 +26,6 @@ xen) ;; esac VERSION=$(/usr/lib/xen-common/bin/xen-version) - -# The arrangements for the `xen' init script are a bit odd. -# This script is part of xen-utils-common, of which there is one -# version installed regardless of the Xen version. -# -# But it is called by the prerm and postinsts of xen-utils-VERSION. -# The idea is that (for example) if xen-utils-VERSION is upgraded, the -# daemons are restarted. -# -# However, this means that this script may be called by the -# maintscript of a xen-utils-V package for a different V to the -# running version of Xen (X, say). Such a xen-utils-V package does -# not actually want to start or stop its daemons. Indeed, the version -# selection machinery would redirect its efforts to the xen-utils-X -# utilities. But this is not right: we don't actually want to (for -# example) stop xenconsoled from xen-utils-X just because some -# not-currently-relevant xen-utils-V is installed/removed/whatever. -# -# So we use DPKG_MAINTSCRIPT_PACKAGE to detect this situation, and -# turn these extraneous calls into no-ops. - -case $DPKG_MAINTSCRIPT_PACKAGE in -xen-utils-$VERSION) ;; # xen-utils-V maintscript, under Xen X=V -xen-utils-*) exit 0;; # xen-utils-V maintscript, but under Xen X!=V -*) ;; # maybe not under dpkg, etc. -esac - ROOT=$(/usr/lib/xen-common/bin/xen-dir) if [ $? -ne 0 ]; then log_warning_msg "No compatible Xen utils for Xen $VERSION" -- 2.20.1
Hans van Kranenburg
2020-May-25 21:38 UTC
[Pkg-xen-devel] Bug#932759: [PATCH 2/2] debian/rules: --no-start for xen dh_installinit
When debugging the xen-utils postinst/prerm to find the cause of the mysteriously disappearing xenconsoled processes, I discovered that the xen-utils-common postinst and prerm stop and start the xen init script as well! These commands are not visible in the packaging code, but they are added by dh_installdeb into the postinst and prerm during package build time. We only want to call the script from xen-utils-V, so disable this behavior by using --no-start Closes: #932759 (2/2) Signed-off-by: Hans van Kranenburg <hans at knorrie.org> --- debian/rules | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/debian/rules b/debian/rules index 23c982eb414b..73232ca20efe 100755 --- a/debian/rules +++ b/debian/rules @@ -282,7 +282,7 @@ override_dh_python2: # We have two init scripts. (There used to be xend too.) override_dh_installinit: - dh_installinit --name xen -- defaults 20 21 + dh_installinit --name xen --no-start -- defaults 20 21 dh_installinit --name xendomains --no-start -- defaults 21 20 # dh_strip in dh compat 10 and earlier (which we are at so this -- 2.20.1
Ian Jackson
2020-May-26 10:43 UTC
[Pkg-xen-devel] Bug#932759: [PATCH 1/2] xen init/maint scripts: Do nothing if running for wrong Xen package
Hans van Kranenburg writes ("[PATCH 1/2] xen init/maint scripts: Do nothing if running for wrong Xen package"):> After trying to fix this issue in the init script, we found out that the > problem still happened for systems running with systemd. > > The xen-utils-V postinst and prerm have DPKG_MAINTSCRIPT_PACKAGE in > their environment. When calling invoke-rc.d xen <action> under systemd, > the whole circus of translation and compatibility layers is used to > finally end up running the /etc/init.d/xen script again. However, when > ending up there, the DPKG_MAINTSCRIPT_PACKAGE variable is lost. > > So, instead of trying to fix this in the init script, avoid calling > invoke-rc.d altogether, when installing or removing for a different > version of Xen than the currently running one. > > Since we only call this from two places, and the check is a one liner, > directly put it into the prerm and postinst. > > Carefully quote the values on both sides of the comparison. For example, > when removing a xen-utils-V package after rebooting into just Linux > without Xen, the version retrieval helper will print an error like > "ERROR: Can't find hypervisor information in sysfs!", there will be no > useful output on stdout and it will compare an empty string with the > version of the xen-utils package, resulting in the right action, not > trying to stop or start anything. > > To avoid hitting the disappearing xenconsoled scenario, the fix has to > be present in the maintainer scripts of the to be removed *old* > xen-utils-V package. This means users will have to first upgrade to a > package with this fix before upgrading to a different Xen version. > > Signed-off-by: Hans van Kranenburg <hans at knorrie.org> > Closes: #932759 (1/2) > Fixes: cc85504103 "xen init script: Do nothing if running for wrong Xen package"Reviewed-by: Ian Jackson <ijackson at chiark.greenend.org.uk>
Ian Jackson
2020-May-26 10:44 UTC
[Pkg-xen-devel] Bug#932759: [PATCH 2/2] debian/rules: --no-start for xen dh_installinit
Hans van Kranenburg writes ("[PATCH 2/2] debian/rules: --no-start for xen dh_installinit"):> When debugging the xen-utils postinst/prerm to find the cause of the > mysteriously disappearing xenconsoled processes, I discovered that the > xen-utils-common postinst and prerm stop and start the xen init script > as well! > > These commands are not visible in the packaging code, but they are added > by dh_installdeb into the postinst and prerm during package build time. > > We only want to call the script from xen-utils-V, so disable this > behavior by using --no-start > > Closes: #932759 (2/2) > Signed-off-by: Hans van Kranenburg <hans at knorrie.org>Reviewed-by: Ian Jackson <ijackson at chiark.greenend.org.uk> I think it would be wise to look at the generated .debs and see that they contain (only) the expected pieces in their maintscripts. Ian.
Hans van Kranenburg
2020-May-26 11:19 UTC
[Pkg-xen-devel] Bug#932759: [PATCH 2/2] debian/rules: --no-start for xen dh_installinit
Hi, On 5/26/20 12:44 PM, Ian Jackson wrote:> Hans van Kranenburg writes ("[PATCH 2/2] debian/rules: --no-start for xen dh_installinit"): >> When debugging the xen-utils postinst/prerm to find the cause of the >> mysteriously disappearing xenconsoled processes, I discovered that the >> xen-utils-common postinst and prerm stop and start the xen init script >> as well! >> >> These commands are not visible in the packaging code, but they are added >> by dh_installdeb into the postinst and prerm during package build time. >> >> We only want to call the script from xen-utils-V, so disable this >> behavior by using --no-start >> >> Closes: #932759 (2/2) >> Signed-off-by: Hans van Kranenburg <hans at knorrie.org> > > Reviewed-by: Ian Jackson <ijackson at chiark.greenend.org.uk>Thanks.> I think it would be wise to look at the generated .debs and see that > they contain (only) the expected pieces in their maintscripts.Yes, I did this while testing by diffing the files installed in /var/lib/dpkg/info with the old ones and verifying that exactly that part went away. (for 4.11:) -$ diff -u ~/xen-utils-common.postinst xen-utils-common.postinst --- /home/beheer/xen-utils-common.postinst 2020-05-26 13:08:45.738926207 +0200 +++ xen-utils-common.postinst 2020-05-25 14:14:28.000000000 +0200 @@ -31,13 +31,7 @@ # Automatically added by dh_installinit/13.1 if [ "$1" = "configure" ] || [ "$1" = "abort-upgrade" ] || [ "$1" "abort-deconfigure" ] || [ "$1" = "abort-remove" ] ; then if [ -x "/etc/init.d/xen" ]; then - update-rc.d xen defaults 20 21 >/dev/null - if [ -n "$2" ]; then - _dh_action=restart - else - _dh_action=start - fi - invoke-rc.d xen $_dh_action || exit 1 + update-rc.d xen defaults 20 21 >/dev/null || exit 1 fi fi # End automatically added section -$ diff -u ~/xen-utils-common.prerm xen-utils-common.prerm --- /home/beheer/xen-utils-common.prerm 2020-05-26 13:09:01.570617331 +0200 +++ xen-utils-common.prerm 2020-05-25 14:14:28.000000000 +0200 @@ -1,10 +1,5 @@ #!/bin/sh set -e -# Automatically added by dh_installinit/13.1 -if [ -x "/etc/init.d/xen" ] && [ "$1" = remove ]; then - invoke-rc.d xen stop || exit 1 -fi -# End automatically added section # Automatically added by dh_installdeb/13.1 dpkg-maintscript-helper rm_conffile /etc/default/xend 4.11.1-2\~ -- "$@" dpkg-maintscript-helper rm_conffile /etc/xen/xend-config.sxp 4.11.1-2\~ -- "$@" Hans
Ian Jackson
2020-May-26 11:31 UTC
[Pkg-xen-devel] Bug#932759: [PATCH 2/2] debian/rules: --no-start for xen dh_installinit
Hans van Kranenburg writes ("Re: [PATCH 2/2] debian/rules: --no-start for xen dh_installinit"):> On 5/26/20 12:44 PM, Ian Jackson wrote: > > I think it would be wise to look at the generated .debs and see that > > they contain (only) the expected pieces in their maintscripts. > > Yes, I did this while testing by diffing the files installed in > /var/lib/dpkg/info with the old ones and verifying that exactly that > part went away.Cool. LGTM. Thanks, Ian. -- Ian Jackson <ijackson at chiark.greenend.org.uk> These opinions are my own. Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter.
Debian Bug Tracking System
2020-May-27 17:39 UTC
[Pkg-xen-devel] Bug#932759: marked as done (After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled)
Your message dated Wed, 27 May 2020 17:36:26 +0000 with message-id <E1jdzyc-0007Zm-1B at fasolo.debian.org> and subject line Bug#932759: fixed in xen 4.11.4-1 has caused the Debian Bug report #932759, regarding After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled to be marked as done. This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact owner at bugs.debian.org immediately.) -- 932759: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932759 Debian Bug Tracking System Contact owner at bugs.debian.org with problems -------------- next part -------------- An embedded message was scrubbed... From: niek <niek at niek.org> Subject: After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled Date: Mon, 22 Jul 2019 14:32:16 -0400 Size: 5705 URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20200527/66efe7f5/attachment.mht> -------------- next part -------------- An embedded message was scrubbed... From: Debian FTP Masters <ftpmaster at ftp-master.debian.org> Subject: Bug#932759: fixed in xen 4.11.4-1 Date: Wed, 27 May 2020 17:36:26 +0000 Size: 6138 URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20200527/66efe7f5/attachment-0001.mht>
Hans van Kranenburg
2020-May-27 17:47 UTC
[Pkg-xen-devel] Bug#932759: marked as done (After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled)
Hi, On 5/27/20 7:39 PM, Debian Bug Tracking System wrote:> Your message dated Wed, 27 May 2020 17:36:26 +0000 > with message-id <E1jdzyc-0007Zm-1B at fasolo.debian.org> > and subject line Bug#932759: fixed in xen 4.11.4-1 > has caused the Debian Bug report #932759, > regarding After upgrade from stretch to buster, removal of obsolete xen 4.8 packages seems to trigger shutdown of xenconsoled > to be marked as done. > > This means that you claim that the problem has been dealt with.To avoid confusion, yes, this one closes with the upload of 4.11.4 to unstable which has the fix. However, it's still present in 4.11.3+24-g14b62ab3e5-1~deb10u1 in buster. So, the same fix will also go into buster later, to in the end help users upgrade from buster to bullseye. Hans