Pkg xen devel - Apr 2014 - Bug#745419: xen-utils-4.1: Pygrub fails to boot from LVM LV when something installed in the volume boot record

Package: xen-utils-4.1
Version: 4.1.4-3+deb7u1
Severity: important

When an LVM LV that serves as the root disk for a Xen DomU contains a boot
loader (or possibly other data) in its volume boot record, pygrub fails to boot
it, printing "Error: boot loader didn't return any data" before
exiting.

I think this is because of the function "is_disk_image" on line 45 of
/usr/lib/xen-4.1/bin/pygrub: it should return false on LVM LV's because they
are not disk images but virtual devices, but it returns true because of certain
bytes in the first block of 512 bytes being set. This causes
get_partition_offsets to seek fruitlessly for a partition instead of simply
returning [0], causing the bootloader not to find the right partition.

Detailed steps to reproduce:
We have a domU called blup.kassala.de, it is running Jessie, its root disk is
a LVM LV at /dev/vpsdisks/blup.kassala.de-disk. The first 512 bytes are null:

# dd if=/dev/vpsdisks/blup.kassala.de-disk bs=1 count=512 | hexdump -C
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000200

"xm create -c blup.kassala.de.cfg" shows the machine boots fine. After
boot, we
log in and "apt-get install grub2". During the installation, we say to
install
the bootloader to /dev/xvda2, the root disk. We then halt the system. The first
512 bytes are now non-null:

# dd if=/dev/vpsdisks/blup.kassala.de-disk bs=1 count=512 | hexdump -C
00000000  eb 63 90 00 00 00 00 00  00 00 00 00 00 00 00 00  |.c..............|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000050  00 00 00 00 00 00 00 00  00 00 00 80 80 12 46 00  |..............F.|
00000060  00 00 00 00 ff fa 90 90  f6 c2 80 74 05 f6 c2 70  |...........t...p|
[snip]
00000180  7d e8 2e 00 cd 18 eb fe  47 52 55 42 20 00 47 65  |}.......GRUB .Ge|
00000190  6f 6d 00 48 61 72 64 20  44 69 73 6b 00 52 65 61  |om.Hard Disk.Rea|
000001a0  64 00 20 45 72 72 6f 72  0d 0a 00 bb 01 00 b4 0e  |d. Error........|
000001b0  cd 10 ac 3c 00 75 f4 c3  00 00 00 00 00 00 00 00 
|...<.u..........|
000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa  |..............U.|
00000200

We try to boot the DomU again:

# xm create -c blup.kassala.de.cfg 
Using config file "./blup.kassala.de.cfg".
Error: Boot loader didn't return any data!

We write 512 nullbytes to the disk and retry booting the domU:

# dd if=/dev/zero of=/dev/vpsdisks/blup.kassala.de-disk bs=1 count=512
# xm create -c blup.kassala.de.cfg

The pyGRUB menu appears normally and the machine boots.

-- System Information:
Debian Release: 7.4
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 3.2.0-4-amd64 (SMP w/8 CPU cores)
Locale: LANG=nl_NL.UTF-8, LC_CTYPE=nl_NL.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages xen-utils-4.1 depends on:
ii  e2fslibs          1.42.5-1.1
ii  libc6             2.13-38+deb7u1
ii  libgnutls26       2.12.20-8+deb7u1
ii  libncurses5       5.9-10
ii  libpci3           1:3.1.9-6
ii  libtinfo5         5.9-10
ii  libuuid1          2.20.1-5.3
ii  libxen-4.1        4.1.4-3+deb7u1
ii  libxenstore3.0    4.1.4-3+deb7u1
ii  python            2.7.3-4+deb7u1
ii  python2.7         2.7.3-6+deb7u2
ii  xen-utils-common  4.1.4-3+deb7u1
ii  zlib1g            1:1.2.7.dfsg-13

Versions of packages xen-utils-4.1 recommends:
ii  bridge-utils                                   1.5-6
ii  qemu-keymaps                                   1.1.2+dfsg-6a+deb7u1
ii  qemu-utils                                     1.1.2+dfsg-6a+deb7u1
ii  xen-hypervisor-4.1-amd64 [xen-hypervisor-4.1]  4.1.4-3+deb7u1

Versions of packages xen-utils-4.1 suggests:
pn  xen-docs-4.1  <none>

-- no debconf information

Sjors Gielen schreef op 21-04-14 15:43:
> When an LVM LV that serves as the root disk for a Xen DomU contains a boot
> loader (or possibly other data) in its volume boot record, pygrub fails to
boot
> it, printing "Error: boot loader didn't return any data"
before exiting.
Forgot to include three details:

* filed as "important" because the bug has a major effect on the
usability of the package. Especially if the domU is maintained by
someone who has no acces to dom0, he might inexplicably break the
booting of his system without being able to fix it. (The dom0
administrator also gets a very vague error message, so he might be
unable to fix it too.)

* We couldn't reproduce this bug on a Wheezy domU because there, the
grub-pc configuration wouldn't write to /dev/xvda2. So if you try to
reproduce but can't, double-check this.

* Also, if you try to reproduce, make sure to halt the DomU before
checking if the first 512 bytes were updated; we noticed this update was
not always immediately visible on the dom0, even after running `sync` in
the domU. Probably waiting a while also works, but if you want to be
sure, halt it.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 884 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20140421/158d7978/attachment.sig>

On Mon, 2014-04-21 at 15:43 +0200, Sjors Gielen wrote:
> Package: xen-utils-4.1
> Version: 4.1.4-3+deb7u1
> Severity: important
> 
> When an LVM LV that serves as the root disk for a Xen DomU contains a boot
> loader (or possibly other data) in its volume boot record, pygrub fails to
boot
> it, printing "Error: boot loader didn't return any data"
before exiting.
> 
> I think this is because of the function "is_disk_image" on line
45 of
> /usr/lib/xen-4.1/bin/pygrub: it should return false on LVM LV's because
they
> are not disk images but virtual devices, but it returns true because of
certain
> bytes in the first block of 512 bytes being set. This causes
> get_partition_offsets to seek fruitlessly for a partition instead of simply
> returning [0], causing the bootloader not to find the right partition.
I'm afraid I don't follow your logic here. is_disk_image is trying to
determine whether or not the device (be it a virtual device or
otherwise) contains a partition table or not. A device with a partition
table is a "disk image" as in a "full disk image", as
opposed to a
device which is unpartitioned and could be used via the "per partition
block device" thing in the block driver.

It seems that your block device does include the DOS MBR/partition table
indicator (the 0x55aa signature at the end of the first sector) and
therefore is considered to be a disk image. But perhaps it doesn't
actually contain a valid partition table? What does "fdisk -l" report
for this device?

Does this still happen with pygrub from a newer Xen? I can see that much
has changed around this area since 4.1.x. If you can reproduce with
something newer then I think your best bet is to report this upstream:
http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen

Ian.
> Detailed steps to reproduce:
> We have a domU called blup.kassala.de, it is running Jessie, its root disk
is
> a LVM LV at /dev/vpsdisks/blup.kassala.de-disk. The first 512 bytes are
null:
> 
> # dd if=/dev/vpsdisks/blup.kassala.de-disk bs=1 count=512 | hexdump -C
> 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
> *
> 00000200
> 
> "xm create -c blup.kassala.de.cfg" shows the machine boots fine.
After boot, we
> log in and "apt-get install grub2". During the installation, we
say to install
> the bootloader to /dev/xvda2, the root disk. We then halt the system. The
first
> 512 bytes are now non-null:
> 
> # dd if=/dev/vpsdisks/blup.kassala.de-disk bs=1 count=512 | hexdump -C
> 00000000  eb 63 90 00 00 00 00 00  00 00 00 00 00 00 00 00 
|.c..............|
> 00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
> *
> 00000050  00 00 00 00 00 00 00 00  00 00 00 80 80 12 46 00 
|..............F.|
> 00000060  00 00 00 00 ff fa 90 90  f6 c2 80 74 05 f6 c2 70 
|...........t...p|
> [snip]
> 00000180  7d e8 2e 00 cd 18 eb fe  47 52 55 42 20 00 47 65  |}.......GRUB
.Ge|
> 00000190  6f 6d 00 48 61 72 64 20  44 69 73 6b 00 52 65 61  |om.Hard
Disk.Rea|
> 000001a0  64 00 20 45 72 72 6f 72  0d 0a 00 bb 01 00 b4 0e  |d.
Error........|
> 000001b0  cd 10 ac 3c 00 75 f4 c3  00 00 00 00 00 00 00 00 
|...<.u..........|
> 000001c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 
|................|
> *
> 000001f0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 aa 
|..............U.|
> 00000200
> 
> We try to boot the DomU again:
> 
> # xm create -c blup.kassala.de.cfg 
> Using config file "./blup.kassala.de.cfg".
> Error: Boot loader didn't return any data!
> 
> We write 512 nullbytes to the disk and retry booting the domU:
> 
> # dd if=/dev/zero of=/dev/vpsdisks/blup.kassala.de-disk bs=1 count=512
> # xm create -c blup.kassala.de.cfg
> 
> The pyGRUB menu appears normally and the machine boots.
> 
> -- System Information:
> Debian Release: 7.4
>   APT prefers stable
>   APT policy: (500, 'stable')
> Architecture: amd64 (x86_64)
> 
> Kernel: Linux 3.2.0-4-amd64 (SMP w/8 CPU cores)
> Locale: LANG=nl_NL.UTF-8, LC_CTYPE=nl_NL.UTF-8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
> 
> Versions of packages xen-utils-4.1 depends on:
> ii  e2fslibs          1.42.5-1.1
> ii  libc6             2.13-38+deb7u1
> ii  libgnutls26       2.12.20-8+deb7u1
> ii  libncurses5       5.9-10
> ii  libpci3           1:3.1.9-6
> ii  libtinfo5         5.9-10
> ii  libuuid1          2.20.1-5.3
> ii  libxen-4.1        4.1.4-3+deb7u1
> ii  libxenstore3.0    4.1.4-3+deb7u1
> ii  python            2.7.3-4+deb7u1
> ii  python2.7         2.7.3-6+deb7u2
> ii  xen-utils-common  4.1.4-3+deb7u1
> ii  zlib1g            1:1.2.7.dfsg-13
> 
> Versions of packages xen-utils-4.1 recommends:
> ii  bridge-utils                                   1.5-6
> ii  qemu-keymaps                                   1.1.2+dfsg-6a+deb7u1
> ii  qemu-utils                                     1.1.2+dfsg-6a+deb7u1
> ii  xen-hypervisor-4.1-amd64 [xen-hypervisor-4.1]  4.1.4-3+deb7u1
> 
> Versions of packages xen-utils-4.1 suggests:
> pn  xen-docs-4.1  <none>
> 
> -- no debconf information
> 
> _______________________________________________
> Pkg-xen-devel mailing list
> Pkg-xen-devel at lists.alioth.debian.org
> http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-xen-devel
>

Hi Ian,

Thanks for your reply. My thoughts are inline.

Ian Campbell schreef op 22-04-14 16:14:
> I'm afraid I don't follow your logic here. is_disk_image is trying
to
> determine whether or not the device (be it a virtual device or
> otherwise) contains a partition table or not. A device with a partition
> table is a "disk image" as in a "full disk image", as
opposed to a
> device which is unpartitioned and could be used via the "per partition
> block device" thing in the block driver.
Ah, that makes sense. So is_disk_image is used to decide whether we're
giving it /dev/sda or /dev/sda1, i.e. there is a partition
table/possibility of multiple partitions, or a single partition that
fills the device. Still, the problem remains there was a single
partition, but it thought there was a partition table.
> It seems that your block device does include the DOS MBR/partition table
> indicator (the 0x55aa signature at the end of the first sector) and
> therefore is considered to be a disk image. But perhaps it doesn't
> actually contain a valid partition table? What does "fdisk -l"
report
> for this device?
It would seem so. This was caused by the grub-pc install script; it may
have thought /dev/xvda2 was a disk instead of a partition. `fdisk -l`
seems to indicate a valid but empty list of partitions:

<<<
Disk /dev/vpsdisks/blup.kassala.de-disk: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Device Boot      Start         End      Blocks   Id  System
<<<

Do you think it would be possible for Pygrub to detect this situation
and continue booting from a single-partition image?
> Does this still happen with pygrub from a newer Xen? I can see that much
> has changed around this area since 4.1.x. If you can reproduce with
> something newer then I think your best bet is to report this upstream:
> http://wiki.xen.org/wiki/Reporting_Bugs_against_Xen
I've noticed and investigated this bug on a production Xen machine. At
this moment I have no spare machine to test with later versions of Xen.
Maybe it would be possible for someone else to test this, or I can try
looking through pygrub's history on Xen trunk soon.

Sjors

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 884 bytes
Desc: OpenPGP digital signature
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20140423/1cc1123d/attachment.sig>

Your message dated Thu, 5 Nov 2015 14:46:12 +0000
with message-id <1446734772-27405-1-git-send-email-ian.campbell at
citrix.com>
has caused the   report #745419,
regarding xen-utils-4.1: Pygrub fails to boot from LVM LV when something
installed in the volume boot record
to be marked as having been forwarded to the upstream software
author(s) <ian.jackson at eu.citrix.com>, <wei.liu2 at citrix.com>,
<xen-devel at lists.xen.org>

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
745419: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=745419
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: Ian Campbell <ian.campbell at citrix.com>
Subject: [PATCH] tools: pygrub: if partition table is empty, try treating as a
whole disk
Date: Thu, 5 Nov 2015 14:46:12 +0000
Size: 3423
URL:
<http://lists.alioth.debian.org/pipermail/pkg-xen-devel/attachments/20151105/e2306146/attachment.mht>

Processing control commands:
> tag -1 +fixed-upstream
Bug #745419 [src:xen] xen-utils-4.1: Pygrub fails to boot from LVM LV when
something installed in the volume boot record
Added tag(s) fixed-upstream.

-- 
745419: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=745419
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems

Orders up to yesterday are attached.

https://www.fourgonsrivesud.com/Invoice-receipt/




Sincerely,

Your message dated Sat, 7 Sep 2019 19:50:45 +0200
with message-id <1b0b0549-8781-8664-63d9-d1710f88cd4b at knorrie.org>
and subject line Pygrub bug fixed in Xen 4.7 upstream
has caused the Debian Bug report #745419,
regarding xen-utils-4.1: Pygrub fails to boot from LVM LV when something
installed in the volume boot record
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
745419: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=745419
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: Sjors Gielen <sjors at sjorsgielen.nl>
Subject: xen-utils-4.1: Pygrub fails to boot from LVM LV when something
installed in the volume boot record
Date: Mon, 21 Apr 2014 15:43:14 +0200
Size: 5261
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20190907/2223b2a1/attachment.mht>
-------------- next part --------------
An embedded message was scrubbed...
From: Hans van Kranenburg <hans at knorrie.org>
Subject: Pygrub bug fixed in Xen 4.7 upstream
Date: Sat, 7 Sep 2019 19:50:45 +0200
Size: 7423
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20190907/2223b2a1/attachment-0001.mht>