Pkg xen devel - Feb 2019 - Bug#921590: xen-utils-4.11: pygrub bails out with "ImportError: No module named fsimage"

Package: xen-utils-4.11,xen-utils-common
Version: 4.11.1-1
Severity: serious
Control: affects -1 xen-tools

Hi,

both, /usr/lib/xen-4.11/bin/pygrub as well as /usr/bin/pygrub bail out
for me as follows on Buster:

# /usr/bin/pygrub
Traceback (most recent call last):
  File "/usr/bin/pygrub", line 23, in <module>
    import fsimage
ImportError: No module named fsimage

This is a regression from Stretch and breaks many DomUs after upgrade as
well as most DomUs generated with xen-tools' defaults (which uses
pygrub). It especially severly harms the release testing for the
xen-tools pending 4.8 release. :-(

This seems related (and partially mentioned), but not identical to
#912441 as pygrub is present for me, just not working at all. Which
means: This bug report is about pygrub being present, but completely
broken.

There aren't many differences between pygrub in Stretch and Buster:

1c1
< #! /usr/bin/python2.7
---
> #! /usr/bin/python
22,23d21
< 
< sys.path.insert(1, sys.path[0] + '/../lib/python')

That's all differences.

The latter removal of a search path is very likely the reason for this
issue. (Already hinted towards that in #912441, too.)

As a (very ugly) workaround I tried to copy over pygrub from a Stretch
Dom0, i.e. from xen-utils-4.8, but this helped only partially:

# pygrub
Traceback (most recent call last):
  File "/usr/bin/pygrub", line 25, in <module>
    import fsimage
ImportError: libfsimage.so: cannot open shared object file: No such file or
directory

Note the different error message on the last line. Now it doesn't find
libfsimage.so.

"strace /usr/lib/xen-4.11/bin/pygrub" reveals that pygrub from
xen-utils-4.8, when copied onto a Buster system, indeed tries to access
that /usr/lib/xen-4.11/lib/python/fsimage.so, just doesn't use it:

# strace /usr/lib/xen-4.11/bin/pygrub |& fgrep -A20 lib/python/fsimage.so
--color
openat(AT_FDCWD, "/usr/lib/xen-4.11/bin/../lib/python/fsimage.so",
O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=16128, ...}) = 0
openat(AT_FDCWD, "/usr/lib/xen-4.11/bin/../lib/python/fsimage.so",
O_RDONLY|O_CLOEXEC) = 4
read(4,
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\21\0\0\0\0\0\0"...,
832) = 832
fstat(4, {st_mode=S_IFREG|0644, st_size=16128, ...}) = 0
mmap(NULL, 18352, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 0x7f1af1082000
mmap(0x7f1af1083000, 4096, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x1000) = 0x7f1af1083000
mmap(0x7f1af1084000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4,
0x2000) = 0x7f1af1084000
mmap(0x7f1af1085000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x2000) = 0x7f1af1085000
close(4)                                = 0
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=49493, ...}) = 0
mmap(NULL, 49493, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7f1af0f92000
close(4)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or
directory)
openat(AT_FDCWD,
"/lib/x86_64-linux-gnu/tls/x86_64/x86_64/libfsimage.so",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu/tls/x86_64/x86_64", 0x7fff92e70430) = -1
ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls/x86_64/libfsimage.so",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu/tls/x86_64", 0x7fff92e70430) = -1 ENOENT
(No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls/x86_64/libfsimage.so",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu/tls/x86_64", 0x7fff92e70430) = -1 ENOENT
(No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/tls/libfsimage.so",
O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
stat("/lib/x86_64-linux-gnu/tls", 0x7fff92e70430) = -1 ENOENT (No such
file or directory)

Reason for this seems to be that the linker doesn't find libfsimage.so:

# ldd /usr/lib/xen-4.11/lib/python/fsimage.so
        linux-vdso.so.1 (0x00007fff1bbb5000)
        libfsimage.so => not found
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f657bd3a000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f657bb72000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f657bd7a000)

What finally works is using pygrub from Stretch _and_ setting
LD_LIBRARY_PATH=/usr/lib/xen-4.11/lib/x86_64-linux-gnu:

# env LD_LIBRARY_PATH=/usr/lib/xen-4.11/lib/x86_64-linux-gnu pygrub
Usage: /usr/bin/pygrub [-q|--quiet] [-i|--interactive] [-l|--list-entries]
[-n|--not-really] [--output=] [--kernel=] [--ramdisk=] [--args=] [--entry=]
[--output-directory=] [--output-format=sxp|simple|simple0] [--offset=]
<image>

So using that LD_LIBRARY_PATH when calling xl and the pygrub from
stretch, I can boot pygrub/grub-legacy based DomUs again.

Even if this doesn't affect the general usage of Xen, this is a severe
regression from Stretch and needs to be fixed for Buster. Hence the RC
severity.

Another reason for RC severity is that shipped binaries and libraries
have non-working loading of dynamic libraries, i.e. some kind of missing
dependencies.

-- System Information:
Debian Release: buster/sid
  APT prefers testing
  APT policy: (500, 'testing'), (300, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: sysvinit (via /sbin/init)
LSM: AppArmor: enabled

Versions of packages xen-utils-4.11 depends on:
ii  libc6             2.28-5
ii  libncurses6       6.1+20181013-1
ii  libtinfo6         6.1+20181013-1
ii  libxenevtchn1     4.11.1-1
ii  libxengnttab1     4.11.1-1
ii  libxenmisc4.11    4.11.1-1
ii  libxenstore3.0    4.11.1-1
ii  libxentoolcore1   4.11.1-1
ii  libxentoollog1    4.11.1-1
ii  libyajl2          2.1.0-3
ii  python            2.7.15-4
ii  xen-utils-common  4.11.1-1

Versions of packages xen-utils-4.11 recommends:
ii  bridge-utils                                     1.6-2
ii  grub-xen-host                                    2.02+dfsg1-10
ii  qemu-system-x86                                  1:3.1+dfsg-2+b1
ii  xen-hypervisor-4.11-amd64 [xen-hypervisor-4.11]  4.11.1-1

Versions of packages xen-utils-4.11 suggests:
pn  ovmf        <none>
ii  qemu-utils  1:3.1+dfsg-2+b1
ii  seabios     1.12.0-1

-- no debconf information

Processing control commands:
> affects -1 xen-tools
Bug #921590 [xen-utils-4.11,xen-utils-common] xen-utils-4.11: pygrub bails out
with "ImportError: No module named fsimage"
Added indication that 921590 affects xen-tools

-- 
921590: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921590
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems

Your message dated Fri, 22 Feb 2019 21:22:03 +0100
with message-id <cf03885b-6878-b6d5-9f0c-fe849b1346f5 at knorrie.org>
and subject line Re: [Pkg-xen-devel] Bug#921590: xen-utils-4.11: pygrub bails
out with "ImportError: No module named fsimage"
has caused the Debian Bug report #921590,
regarding xen-utils-4.11: pygrub bails out with "ImportError: No module
named fsimage"
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
921590: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=921590
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: Axel Beckert <abe at debian.org>
Subject: xen-utils-4.11: pygrub bails out with "ImportError: No module
named fsimage"
Date: Thu, 07 Feb 2019 02:13:56 +0100
Size: 8114
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20190222/479d3f4d/attachment-0002.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Hans van Kranenburg <hans at knorrie.org>
Subject: Re: [Pkg-xen-devel] Bug#921590: xen-utils-4.11: pygrub bails out with
"ImportError: No module named fsimage"
Date: Fri, 22 Feb 2019 21:22:03 +0100
Size: 8086
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20190222/479d3f4d/attachment-0003.mht>