Pkg xen devel - Oct 2021 - Bug#994870: Bug#994870: Bug#994870: Bug#994870: Memory allocation problem for VM after xen security update

Package: xen-system-amd64
Version: 4.14.3-1~deb11u1

After applying the buster security update to xen, my VM won't start
any longer, complaining about a memory allocation error.

Switching back to the previous version 4.14.2+25-gb6a8c4f72d-2 lets
the VM start (again,) normally.

/var/log/libvirt/libxl/libxl-driver.log:
2021-09-21 14:01:44.645+0000: xc: panic: xc_dom_boot.c:120: 
xc_dom_boot_mem_init: can't allocate low memory for domain: Out of 
memory
2021-09-21 14:01:44.653+0000: libxl: libxl_dom.c:593:libxl__build_dom: 
xc_dom_boot_mem_init failed: Die Operation wird nicht unterst?tzt 
[means: the operation is not supported]
2021-09-21 14:01:44.662+0000: libxl: 
libxl_create.c:1576:domcreate_rebuild_done: Domain 1:cannot (re-)build 
domain: -3

The error is triggered, regardless if there was a boot-parameter
"dom0_mem=1024M:max=2048M" set or not.
/etc/xen/xl.conf was unaltered, i.e. 'autoballoon' was implicitely set
to "auto".

I am "on" Buster, kernel 5.10.0-8-amd64 (5.10.46-4), all relevant
fixes
included.

Best regards,
  Ruediger Oberhage
-- 
Dr. H.-R. Oberhage
Mail: Univ. Duisburg-Essen	E-Mail: oberhage at Uni-Duisburg-Essen.DE
       Fakultaet fuer Physik	ruediger at Theo.Physik.Uni-Duisburg-Essen.DE

Hi Ruediger,

On 9/22/21 11:37 AM, H.-R. Oberhage wrote:
> Package: xen-system-amd64
> Version: 4.14.3-1~deb11u1
> 
> After applying the buster security update to xen, my VM won't start
> any longer, complaining about a memory allocation error.
Can you confirm that this is a virtual machine that tries to boot a
32-bit kernel as PV type?

The error message you are seeing is not particularly helpful, but it is
most likely related to this.

The fact that with this package update 32-bit PV guests fail to start is
indeed a regression problem, which is quite inconvenient for you, right now.

At this point I would really recommend to not wait for a fix to arrive
which makes it start again, but change your VM to use a 64-bit kernel.

Let me know if you need help or run into problems while making this change.

Running 32-bit PV at all is already 'on life support' upstream for quite
a while now, and it also not under security support any more.

In the long run, I'd suggest working towards having 64-bit guests in PVH
mode, since that's one of the best options we have these days.

If there's a reason you really cannot switch to a 64-bit kernel or move
the functionality of this virtual machine to a new fully 64 bit system,
switching the virtualization type from PV to HVM would also be an option.
> Switching back to the previous version 4.14.2+25-gb6a8c4f72d-2 lets
> the VM start (again,) normally.
> 
> /var/log/libvirt/libxl/libxl-driver.log:
> 2021-09-21 14:01:44.645+0000: xc: panic: xc_dom_boot.c:120: 
> xc_dom_boot_mem_init: can't allocate low memory for domain: Out of 
> memory
> 2021-09-21 14:01:44.653+0000: libxl: libxl_dom.c:593:libxl__build_dom: 
> xc_dom_boot_mem_init failed: Die Operation wird nicht unterst?tzt 
> [means: the operation is not supported]
> 2021-09-21 14:01:44.662+0000: libxl: 
> libxl_create.c:1576:domcreate_rebuild_done: Domain 1:cannot (re-)build 
> domain: -3
> 
> The error is triggered, regardless if there was a boot-parameter
> "dom0_mem=1024M:max=2048M" set or not.
> /etc/xen/xl.conf was unaltered, i.e. 'autoballoon' was implicitely
set
> to "auto".
> 
> I am "on" Buster, kernel 5.10.0-8-amd64 (5.10.46-4), all relevant
fixes
> included.
Apologies for the inconvenience,

Hans

Hi!,

Please don't (accidentally) drop the debian bug email from the recipient
list. This information might also be useful for others later.

On 9/23/21 1:47 AM, H.-R. Oberhage wrote:
> Good evening Hans,
> 
> On 22.09.2021 20:54, Hans van Kranenburg wrote:
>> Hi Ruediger,
>>
>> On 9/22/21 11:37 AM, H.-R. Oberhage wrote:
>>> Package: xen-system-amd64
>>> Version: 4.14.3-1~deb11u1
>>>
>>> After applying the buster security update to xen, my VM won't
start
>>> any longer, complaining about a memory allocation error.
>>
>> Can you confirm that this is a virtual machine that tries to boot a
>> 32-bit kernel as PV type?
> 
> yes, your assumption ...
> 
>> The error message you are seeing is not particularly helpful, but it is
>> most likely related to this.
> 
> ... is correct.
> 
>> The fact that with this package update 32-bit PV guests fail to start 
>> is
>> indeed a regression problem, which is quite inconvenient for you, right
>> now.
> 
> Ok, then I will put the Xen-package on "hold" for now.
> 
>> At this point I would really recommend to not wait for a fix to arrive
>> which makes it start again, but change your VM to use a 64-bit kernel.
> 
> It really is a shame, that 32-bit isn't supported properly any longer.
> The address- and data-overhead in 64-bit machines only using a 32-bit
> address- and data-space is considerable.
> 
> I already experienced, "bullseye" not supporting a dom0-Kernel
for the
> i686-pae architecture any longer :-(. A shame that it doesn't come with
> a kernel before 5.9, which would still allow this.
> 
>> Let me know if you need help or run into problems while making this 
>> change.
> 
> Would you know of a "simple" way to convert/clone a 32-bit VM to
a
> work-alike 64-bit one? One has to replace all the .debs for this, after
> all.
The smallest amount of work to initially get your VM going again is to
only install the 64 bit kernel and keep running a 32 bit user land.

The process to fully change from a 32 to 64 system (in place) is called
'cross grading'. I found instructions at
https://wiki.debian.org/CrossGrading

I never did this myself, though.
>> Running 32-bit PV at all is already 'on life support' upstream
for
>> quite
>> a while now, and it also not under security support any more.
> 
> Well it's a Debian "stretch" one, so it's just working
for now :-).
One of the main reasons why it's so problematic to keep around is that
in the 32 bit PV case, there are no possibilities to implement fixes for
all the speculative vulnerabilities that have been very much in the news
in the last years.

More about this: https://xenbits.xen.org/xsa/advisory-370.html
>> In the long run, I'd suggest working towards having 64-bit guests
in
>> PVH
>> mode, since that's one of the best options we have these days.
> 
> Thanks, I'll consider this for any newer VMs.
> Are 64-bit PV VMs automatically "moved" to or executed as PVH?
> I would even be willing to edit the .xml/.cfg-file manually.
> I see "bullseye's" virt-manager/libvirt offering only choices
for
> "xen (fullvirt)", "xen (paravirt)", or xen", when
creating a new
> VM.
It should be as simple as changing type="pv" to type="pvh"
in the config
file. In Debian, using PVH this is possible since Buster. Also, using
the xen variant of grub2 (grub-xen and grub-xen-host) is possible.

More info:
https://wiki.xenproject.org/wiki/Understanding_the_Virtualization_Spectrum

Have fun,
Hans

Hello,

On Fri, Sep 24, 2021 at 03:33:21PM +0200, Hans van Kranenburg
wrote:
> The smallest amount of work to initially get your VM going again is to
> only install the 64 bit kernel and keep running a 32 bit user land.
Yep, I have a few 32-bit PV domUs that I run as 64-bit just the
kernel, works great.
> The process to fully change from a 32 to 64 system (in place) is called
> 'cross grading'. I found instructions at
> https://wiki.debian.org/CrossGrading
> 
> I never did this myself, though.
I've done it lots of times but for any non-trivial host you have to
pretty much be a Debian expert to know how to handle the many
problems that dpkg will encounter. It takes longer than
reinstalling and a mistake breaks everything. Not recommended.
> It should be as simple as changing type="pv" to
type="pvh" in the config
> file. In Debian, using PVH this is possible since Buster. Also, using
> the xen variant of grub2 (grub-xen and grub-xen-host) is possible.
I can confirm that I have a lot of Debian stretch domUs running PVH
but you do need to install backports kernel in the domU. The 4.9.x
stretch kernelis too old. You need the 4.19.x one from
stretch-backports. They boot fine for me with pvhgrub.

Cheers,
Andy

Hello,

thank you for your kind and helpful answers.

No including the "bugs" e-mail address in my first reply
was my mistake, I just didn't notice, that there was more than
one address. Shouldn't happen again :-).

On 24.09.2021 15:50, Andy Smith wrote:
> Hello,
> 
> On Fri, Sep 24, 2021 at 03:33:21PM +0200, Hans van Kranenburg wrote:
>> The smallest amount of work to initially get your VM going again is to
>> only install the 64 bit kernel and keep running a 32 bit user land.
> 
> Yep, I have a few 32-bit PV domUs that I run as 64-bit just the
> kernel, works great.
That information helps me a lot!
>> The process to fully change from a 32 to 64 system (in place) is 
>> called
>> 'cross grading'. I found instructions at
>> https://wiki.debian.org/CrossGrading
>> 
>> I never did this myself, though.
> 
> I've done it lots of times but for any non-trivial host you have to
> pretty much be a Debian expert to know how to handle the many
> problems that dpkg will encounter. It takes longer than
> reinstalling and a mistake breaks everything. Not recommended.
I also did it with a few machines some time ago, but I remembered
how cumbersome it was. That's why I asked about a "simple" method
:-).
I won't do it (cross grading), if not forced to do so.
>> It should be as simple as changing type="pv" to
type="pvh" in the
>> config
>> file. In Debian, using PVH this is possible since Buster. Also, using
>> the xen variant of grub2 (grub-xen and grub-xen-host) is possible.
> 
> I can confirm that I have a lot of Debian stretch domUs running PVH
> but you do need to install backports kernel in the domU. The 4.9.x
> stretch kernelis too old. You need the 4.19.x one from
> stretch-backports. They boot fine for me with pvhgrub.
This, too, helps me a lot. Thank you. All of the machines in question
here, are on buster or newer - although by upgrading. So no stress
here.
> Cheers,
> Andy
Thanks to both of you, Hans and Andy,
  Ruediger
-- 
Dr. H.-R. Oberhage
Mail: Univ. Duisburg-Essen	E-Mail: oberhage at Uni-Duisburg-Essen.DE
       Fakultaet fuer Physik	ruediger at Theo.Physik.Uni-Duisburg-Essen.DE

Hello,

Am 22.09.21 um 20:54 schrieb Hans van Kranenburg:
> Hi Ruediger,
> 
> On 9/22/21 11:37 AM, H.-R. Oberhage wrote:
>> Package: xen-system-amd64
>> Version: 4.14.3-1~deb11u1
>>
>> After applying the buster security update to xen, my VM won't start
>> any longer, complaining about a memory allocation error.
> 
> Can you confirm that this is a virtual machine that tries to boot a
> 32-bit kernel as PV type?
I can confirm this for three of my virtual machines.
> The error message you are seeing is not particularly helpful, but it is
> most likely related to this.
> 
> The fact that with this package update 32-bit PV guests fail to start is
> indeed a regression problem, which is quite inconvenient for you, right
now.
And for me. :-/
> At this point I would really recommend to not wait for a fix to arrive
> which makes it start again, but change your VM to use a 64-bit kernel.
How?
> Let me know if you need help or run into problems while making this change.
> 
> Running 32-bit PV at all is already 'on life support' upstream for
quite
> a while now, and it also not under security support any more.
Ack.

FWIW, Debian 10 VMs with 32 bit running with PVH work fine. My important 
VM is still Debian 9 however due to a software I can not simply upgrade.
> In the long run, I'd suggest working towards having 64-bit guests in
PVH
> mode, since that's one of the best options we have these days.
Ack.
> If there's a reason you really cannot switch to a 64-bit kernel or move
> the functionality of this virtual machine to a new fully 64 bit system,
> switching the virtualization type from PV to HVM would also be an option.
Thanks for your support.

Greets
Alex
>> Switching back to the previous version 4.14.2+25-gb6a8c4f72d-2 lets
>> the VM start (again,) normally.
>>
>> /var/log/libvirt/libxl/libxl-driver.log:
>> 2021-09-21 14:01:44.645+0000: xc: panic: xc_dom_boot.c:120:
>> xc_dom_boot_mem_init: can't allocate low memory for domain: Out of
>> memory
>> 2021-09-21 14:01:44.653+0000: libxl: libxl_dom.c:593:libxl__build_dom:
>> xc_dom_boot_mem_init failed: Die Operation wird nicht unterst?tzt
>> [means: the operation is not supported]
>> 2021-09-21 14:01:44.662+0000: libxl:
>> libxl_create.c:1576:domcreate_rebuild_done: Domain 1:cannot (re-)build
>> domain: -3
>>
>> The error is triggered, regardless if there was a boot-parameter
>> "dom0_mem=1024M:max=2048M" set or not.
>> /etc/xen/xl.conf was unaltered, i.e. 'autoballoon' was
implicitely set
>> to "auto".
>>
>> I am "on" Buster, kernel 5.10.0-8-amd64 (5.10.46-4), all
relevant fixes
>> included.
> 
> Apologies for the inconvenience,
> 
> Hans
> 
> _______________________________________________
> Pkg-xen-devel mailing list
> Pkg-xen-devel at alioth-lists.debian.net
> https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-xen-devel
>

Hi Alex,

On Thu, Sep 30, 2021 at 12:10:32AM +0200, Alexander Dahl
wrote:
> Am 22.09.21 um 20:54 schrieb Hans van Kranenburg:
> > At this point I would really recommend to not wait for a fix to arrive
> > which makes it start again, but change your VM to use a 64-bit kernel.
> 
> How?
This was answered in earlier comments on this bug; please see:

    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#15
    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#20

The brief summary is, "start out like a crossgrade, but only do the
kernel". Very simple and quite safe.

You haven't said how you boot your guest though (show us your
/etc/xen/guest.cfg file). If it's pvgrub, that has a 32-bit and a
64-bit version so you'll need to change those as well. If it's
pygrub you probably don't need to do anything, though pygrub has its
own issues outside the scope of this bug.
> FWIW, Debian 10 VMs with 32 bit running with PVH work fine. My important VM
> is still Debian 9 however due to a software I can not simply upgrade.
I've found PVH needs at least 4.19 guest kernel to work, which can
be achieved in Debian 9 (stretch) today by using kernel from
stretch-backports, so perhaps that is an option for you.

Cheers,
Andy

Hi!

On 9/30/21 12:45 AM, Andy Smith wrote:
> Hi Alex,
> 
> On Thu, Sep 30, 2021 at 12:10:32AM +0200, Alexander Dahl wrote:
>> Am 22.09.21 um 20:54 schrieb Hans van Kranenburg:
>>> At this point I would really recommend to not wait for a fix to
arrive
>>> which makes it start again, but change your VM to use a 64-bit
kernel.
>>
>> How?
> 
> This was answered in earlier comments on this bug; please see:
> 
>     https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#15
>     https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#20
> 
> The brief summary is, "start out like a crossgrade, but only do the
> kernel". Very simple and quite safe.
> 
> You haven't said how you boot your guest though (show us your
> /etc/xen/guest.cfg file). If it's pvgrub, that has a 32-bit and a
> 64-bit version so you'll need to change those as well. If it's
> pygrub you probably don't need to do anything, though pygrub has its
> own issues outside the scope of this bug.
> 
>> FWIW, Debian 10 VMs with 32 bit running with PVH work fine. My
important VM
>> is still Debian 9 however due to a software I can not simply upgrade.
> 
> I've found PVH needs at least 4.19 guest kernel to work, which can
> be achieved in Debian 9 (stretch) today by using kernel from
> stretch-backports, so perhaps that is an option for you.
You can certainly do that and then run PVH.

Since stretch-backports is not used any more since stretch became
oldoldstable, new 4.19 backports kernels for Stretch are released
through the security updates channel. Be aware of this.

https://lists.debian.org/debian-lts-announce/2020/08/msg00019.html

Latest in stretch-backports (frozen) is 4.19.118, and stretch security
is now at 4.19.194. So double check you end up following the right one.

Hans

Hei hei,

On Thu, Sep 30, 2021 at 01:13:54PM +0200, Hans van Kranenburg
wrote:
> Hi!
> 
> On 9/30/21 12:45 AM, Andy Smith wrote:
> > Hi Alex,
> > 
> > On Thu, Sep 30, 2021 at 12:10:32AM +0200, Alexander Dahl wrote:
> >> Am 22.09.21 um 20:54 schrieb Hans van Kranenburg:
> >>> At this point I would really recommend to not wait for a fix
to arrive
> >>> which makes it start again, but change your VM to use a 64-bit
kernel.
> >>
> >> How?
> > 
> > This was answered in earlier comments on this bug; please see:
> > 
> >     https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#15
> >     https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#20
> > 
> > The brief summary is, "start out like a crossgrade, but only do
the
> > kernel". Very simple and quite safe.
> > 
> > You haven't said how you boot your guest though (show us your
> > /etc/xen/guest.cfg file). If it's pvgrub, that has a 32-bit and a
> > 64-bit version so you'll need to change those as well. If it's
> > pygrub you probably don't need to do anything, though pygrub has
its
> > own issues outside the scope of this bug.
> > 
> >> FWIW, Debian 10 VMs with 32 bit running with PVH work fine. My
important VM
> >> is still Debian 9 however due to a software I can not simply
upgrade.
> > 
> > I've found PVH needs at least 4.19 guest kernel to work, which can
> > be achieved in Debian 9 (stretch) today by using kernel from
> > stretch-backports, so perhaps that is an option for you.
> 
> You can certainly do that and then run PVH.
This actually works.  I'm running 4.19 i686 kernel in the stretch VM
now with PVH, at least for the Debian stretch VM (I had to permanently
disable some old OpenWRT VMs, where I get no updates anymore).  Was a
little tricky to install it, because I had to install that kernel
without the vm being able to start, but it worked like this:

- mount the root filesystem of the vm in the host, e.g. to /mnt
- bind mount /dev, /sys to /mnt/dev and /mnt/sys
- mount procfs to /mnt/proc
- mount a tmpfs to /mnt/run and create /run/lock
- chroot into /mnt
- install the needed kernel with apt
- leave chroot, umount the things from above
- change domU config to PVH
- when in grub, edit the cmdline and change root= if it was changed by
  update-grub (might have been changed to the mount point from the
  chroot)
- in the now booted system, run update-grub again
> Since stretch-backports is not used any more since stretch became
> oldoldstable, new 4.19 backports kernels for Stretch are released
> through the security updates channel. Be aware of this.
> 
> https://lists.debian.org/debian-lts-announce/2020/08/msg00019.html
> 
> Latest in stretch-backports (frozen) is 4.19.118, and stretch security
> is now at 4.19.194. So double check you end up following the right one.
Thanks for all your hints. I really have to migrate my virtual
machines. :-/

Greets
Alex

-- 
/"\ ASCII RIBBON | ?With the first link, the chain is forged. The first
\ / CAMPAIGN     | speech censured, the first thought forbidden, the
 X  AGAINST      | first freedom denied, chains us all irrevocably.?
/ \ HTML MAIL    | (Jean-Luc Picard, quoting Judge Aaron Satie)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20211002/8162e00b/attachment.sig>

Your message dated Fri, 25 Feb 2022 17:29:20 +0100
with message-id <5226fd2e-5044-7662-d0d9-1dc0a8a50b4e at knorrie.org>
and subject line Re: Bug#994870: Memory allocation problem for VM after xen
security update
has caused the Debian Bug report #994870,
regarding Memory allocation problem for VM after xen security update
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact owner at bugs.debian.org
immediately.)


-- 
994870: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870
Debian Bug Tracking System
Contact owner at bugs.debian.org with problems
-------------- next part --------------
An embedded message was scrubbed...
From: "H.-R. Oberhage" <oberhage at Uni-Duisburg-Essen.DE>
Subject: Memory allocation problem for VM after xen security update
Date: Wed, 22 Sep 2021 11:37:28 +0200
Size: 3514
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20220225/b8a60b92/attachment.eml>
-------------- next part --------------
An embedded message was scrubbed...
From: Hans van Kranenburg <hans at knorrie.org>
Subject: Re: Bug#994870: Memory allocation problem for VM after xen security
update
Date: Fri, 25 Feb 2022 17:29:20 +0100
Size: 2946
URL:
<http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20220225/b8a60b92/attachment-0001.eml>