H.-R. Oberhage
2021-Sep-22 09:37 UTC
[Pkg-xen-devel] Bug#994870: Memory allocation problem for VM after xen security update
Package: xen-system-amd64 Version: 4.14.3-1~deb11u1 After applying the buster security update to xen, my VM won't start any longer, complaining about a memory allocation error. Switching back to the previous version 4.14.2+25-gb6a8c4f72d-2 lets the VM start (again,) normally. /var/log/libvirt/libxl/libxl-driver.log: 2021-09-21 14:01:44.645+0000: xc: panic: xc_dom_boot.c:120: xc_dom_boot_mem_init: can't allocate low memory for domain: Out of memory 2021-09-21 14:01:44.653+0000: libxl: libxl_dom.c:593:libxl__build_dom: xc_dom_boot_mem_init failed: Die Operation wird nicht unterst?tzt [means: the operation is not supported] 2021-09-21 14:01:44.662+0000: libxl: libxl_create.c:1576:domcreate_rebuild_done: Domain 1:cannot (re-)build domain: -3 The error is triggered, regardless if there was a boot-parameter "dom0_mem=1024M:max=2048M" set or not. /etc/xen/xl.conf was unaltered, i.e. 'autoballoon' was implicitely set to "auto". I am "on" Buster, kernel 5.10.0-8-amd64 (5.10.46-4), all relevant fixes included. Best regards, Ruediger Oberhage -- Dr. H.-R. Oberhage Mail: Univ. Duisburg-Essen E-Mail: oberhage at Uni-Duisburg-Essen.DE Fakultaet fuer Physik ruediger at Theo.Physik.Uni-Duisburg-Essen.DE
Hans van Kranenburg
2021-Sep-22 18:54 UTC
[Pkg-xen-devel] Bug#994870: Bug#994870: Memory allocation problem for VM after xen security update
Hi Ruediger, On 9/22/21 11:37 AM, H.-R. Oberhage wrote:> Package: xen-system-amd64 > Version: 4.14.3-1~deb11u1 > > After applying the buster security update to xen, my VM won't start > any longer, complaining about a memory allocation error.Can you confirm that this is a virtual machine that tries to boot a 32-bit kernel as PV type? The error message you are seeing is not particularly helpful, but it is most likely related to this. The fact that with this package update 32-bit PV guests fail to start is indeed a regression problem, which is quite inconvenient for you, right now. At this point I would really recommend to not wait for a fix to arrive which makes it start again, but change your VM to use a 64-bit kernel. Let me know if you need help or run into problems while making this change. Running 32-bit PV at all is already 'on life support' upstream for quite a while now, and it also not under security support any more. In the long run, I'd suggest working towards having 64-bit guests in PVH mode, since that's one of the best options we have these days. If there's a reason you really cannot switch to a 64-bit kernel or move the functionality of this virtual machine to a new fully 64 bit system, switching the virtualization type from PV to HVM would also be an option.> Switching back to the previous version 4.14.2+25-gb6a8c4f72d-2 lets > the VM start (again,) normally. > > /var/log/libvirt/libxl/libxl-driver.log: > 2021-09-21 14:01:44.645+0000: xc: panic: xc_dom_boot.c:120: > xc_dom_boot_mem_init: can't allocate low memory for domain: Out of > memory > 2021-09-21 14:01:44.653+0000: libxl: libxl_dom.c:593:libxl__build_dom: > xc_dom_boot_mem_init failed: Die Operation wird nicht unterst?tzt > [means: the operation is not supported] > 2021-09-21 14:01:44.662+0000: libxl: > libxl_create.c:1576:domcreate_rebuild_done: Domain 1:cannot (re-)build > domain: -3 > > The error is triggered, regardless if there was a boot-parameter > "dom0_mem=1024M:max=2048M" set or not. > /etc/xen/xl.conf was unaltered, i.e. 'autoballoon' was implicitely set > to "auto". > > I am "on" Buster, kernel 5.10.0-8-amd64 (5.10.46-4), all relevant fixes > included.Apologies for the inconvenience, Hans
Hans van Kranenburg
2021-Sep-24 13:33 UTC
[Pkg-xen-devel] Bug#994870: Memory allocation problem for VM after xen security update
Hi!, Please don't (accidentally) drop the debian bug email from the recipient list. This information might also be useful for others later. On 9/23/21 1:47 AM, H.-R. Oberhage wrote:> Good evening Hans, > > On 22.09.2021 20:54, Hans van Kranenburg wrote: >> Hi Ruediger, >> >> On 9/22/21 11:37 AM, H.-R. Oberhage wrote: >>> Package: xen-system-amd64 >>> Version: 4.14.3-1~deb11u1 >>> >>> After applying the buster security update to xen, my VM won't start >>> any longer, complaining about a memory allocation error. >> >> Can you confirm that this is a virtual machine that tries to boot a >> 32-bit kernel as PV type? > > yes, your assumption ... > >> The error message you are seeing is not particularly helpful, but it is >> most likely related to this. > > ... is correct. > >> The fact that with this package update 32-bit PV guests fail to start >> is >> indeed a regression problem, which is quite inconvenient for you, right >> now. > > Ok, then I will put the Xen-package on "hold" for now. > >> At this point I would really recommend to not wait for a fix to arrive >> which makes it start again, but change your VM to use a 64-bit kernel. > > It really is a shame, that 32-bit isn't supported properly any longer. > The address- and data-overhead in 64-bit machines only using a 32-bit > address- and data-space is considerable. > > I already experienced, "bullseye" not supporting a dom0-Kernel for the > i686-pae architecture any longer :-(. A shame that it doesn't come with > a kernel before 5.9, which would still allow this. > >> Let me know if you need help or run into problems while making this >> change. > > Would you know of a "simple" way to convert/clone a 32-bit VM to a > work-alike 64-bit one? One has to replace all the .debs for this, after > all.The smallest amount of work to initially get your VM going again is to only install the 64 bit kernel and keep running a 32 bit user land. The process to fully change from a 32 to 64 system (in place) is called 'cross grading'. I found instructions at https://wiki.debian.org/CrossGrading I never did this myself, though.>> Running 32-bit PV at all is already 'on life support' upstream for >> quite >> a while now, and it also not under security support any more. > > Well it's a Debian "stretch" one, so it's just working for now :-).One of the main reasons why it's so problematic to keep around is that in the 32 bit PV case, there are no possibilities to implement fixes for all the speculative vulnerabilities that have been very much in the news in the last years. More about this: https://xenbits.xen.org/xsa/advisory-370.html>> In the long run, I'd suggest working towards having 64-bit guests in >> PVH >> mode, since that's one of the best options we have these days. > > Thanks, I'll consider this for any newer VMs. > Are 64-bit PV VMs automatically "moved" to or executed as PVH? > I would even be willing to edit the .xml/.cfg-file manually. > I see "bullseye's" virt-manager/libvirt offering only choices for > "xen (fullvirt)", "xen (paravirt)", or xen", when creating a new > VM.It should be as simple as changing type="pv" to type="pvh" in the config file. In Debian, using PVH this is possible since Buster. Also, using the xen variant of grub2 (grub-xen and grub-xen-host) is possible. More info: https://wiki.xenproject.org/wiki/Understanding_the_Virtualization_Spectrum Have fun, Hans
Andy Smith
2021-Sep-24 13:50 UTC
[Pkg-xen-devel] Bug#994870: Bug#994870: Memory allocation problem for VM after xen security update
Hello, On Fri, Sep 24, 2021 at 03:33:21PM +0200, Hans van Kranenburg wrote:> The smallest amount of work to initially get your VM going again is to > only install the 64 bit kernel and keep running a 32 bit user land.Yep, I have a few 32-bit PV domUs that I run as 64-bit just the kernel, works great.> The process to fully change from a 32 to 64 system (in place) is called > 'cross grading'. I found instructions at > https://wiki.debian.org/CrossGrading > > I never did this myself, though.I've done it lots of times but for any non-trivial host you have to pretty much be a Debian expert to know how to handle the many problems that dpkg will encounter. It takes longer than reinstalling and a mistake breaks everything. Not recommended.> It should be as simple as changing type="pv" to type="pvh" in the config > file. In Debian, using PVH this is possible since Buster. Also, using > the xen variant of grub2 (grub-xen and grub-xen-host) is possible.I can confirm that I have a lot of Debian stretch domUs running PVH but you do need to install backports kernel in the domU. The 4.9.x stretch kernelis too old. You need the 4.19.x one from stretch-backports. They boot fine for me with pvhgrub. Cheers, Andy
H.-R. Oberhage
2021-Sep-24 14:15 UTC
[Pkg-xen-devel] Bug#994870: Bug#994870: Memory allocation problem for VM after xen security update
Hello, thank you for your kind and helpful answers. No including the "bugs" e-mail address in my first reply was my mistake, I just didn't notice, that there was more than one address. Shouldn't happen again :-). On 24.09.2021 15:50, Andy Smith wrote:> Hello, > > On Fri, Sep 24, 2021 at 03:33:21PM +0200, Hans van Kranenburg wrote: >> The smallest amount of work to initially get your VM going again is to >> only install the 64 bit kernel and keep running a 32 bit user land. > > Yep, I have a few 32-bit PV domUs that I run as 64-bit just the > kernel, works great.That information helps me a lot!>> The process to fully change from a 32 to 64 system (in place) is >> called >> 'cross grading'. I found instructions at >> https://wiki.debian.org/CrossGrading >> >> I never did this myself, though. > > I've done it lots of times but for any non-trivial host you have to > pretty much be a Debian expert to know how to handle the many > problems that dpkg will encounter. It takes longer than > reinstalling and a mistake breaks everything. Not recommended.I also did it with a few machines some time ago, but I remembered how cumbersome it was. That's why I asked about a "simple" method :-). I won't do it (cross grading), if not forced to do so.>> It should be as simple as changing type="pv" to type="pvh" in the >> config >> file. In Debian, using PVH this is possible since Buster. Also, using >> the xen variant of grub2 (grub-xen and grub-xen-host) is possible. > > I can confirm that I have a lot of Debian stretch domUs running PVH > but you do need to install backports kernel in the domU. The 4.9.x > stretch kernelis too old. You need the 4.19.x one from > stretch-backports. They boot fine for me with pvhgrub.This, too, helps me a lot. Thank you. All of the machines in question here, are on buster or newer - although by upgrading. So no stress here.> Cheers, > AndyThanks to both of you, Hans and Andy, Ruediger -- Dr. H.-R. Oberhage Mail: Univ. Duisburg-Essen E-Mail: oberhage at Uni-Duisburg-Essen.DE Fakultaet fuer Physik ruediger at Theo.Physik.Uni-Duisburg-Essen.DE
Alexander Dahl
2021-Sep-29 22:10 UTC
[Pkg-xen-devel] Bug#994870: Bug#994870: Bug#994870: Memory allocation problem for VM after xen security update
Hello, Am 22.09.21 um 20:54 schrieb Hans van Kranenburg:> Hi Ruediger, > > On 9/22/21 11:37 AM, H.-R. Oberhage wrote: >> Package: xen-system-amd64 >> Version: 4.14.3-1~deb11u1 >> >> After applying the buster security update to xen, my VM won't start >> any longer, complaining about a memory allocation error. > > Can you confirm that this is a virtual machine that tries to boot a > 32-bit kernel as PV type?I can confirm this for three of my virtual machines.> The error message you are seeing is not particularly helpful, but it is > most likely related to this. > > The fact that with this package update 32-bit PV guests fail to start is > indeed a regression problem, which is quite inconvenient for you, right now.And for me. :-/> At this point I would really recommend to not wait for a fix to arrive > which makes it start again, but change your VM to use a 64-bit kernel.How?> Let me know if you need help or run into problems while making this change. > > Running 32-bit PV at all is already 'on life support' upstream for quite > a while now, and it also not under security support any more.Ack. FWIW, Debian 10 VMs with 32 bit running with PVH work fine. My important VM is still Debian 9 however due to a software I can not simply upgrade.> In the long run, I'd suggest working towards having 64-bit guests in PVH > mode, since that's one of the best options we have these days.Ack.> If there's a reason you really cannot switch to a 64-bit kernel or move > the functionality of this virtual machine to a new fully 64 bit system, > switching the virtualization type from PV to HVM would also be an option.Thanks for your support. Greets Alex>> Switching back to the previous version 4.14.2+25-gb6a8c4f72d-2 lets >> the VM start (again,) normally. >> >> /var/log/libvirt/libxl/libxl-driver.log: >> 2021-09-21 14:01:44.645+0000: xc: panic: xc_dom_boot.c:120: >> xc_dom_boot_mem_init: can't allocate low memory for domain: Out of >> memory >> 2021-09-21 14:01:44.653+0000: libxl: libxl_dom.c:593:libxl__build_dom: >> xc_dom_boot_mem_init failed: Die Operation wird nicht unterst?tzt >> [means: the operation is not supported] >> 2021-09-21 14:01:44.662+0000: libxl: >> libxl_create.c:1576:domcreate_rebuild_done: Domain 1:cannot (re-)build >> domain: -3 >> >> The error is triggered, regardless if there was a boot-parameter >> "dom0_mem=1024M:max=2048M" set or not. >> /etc/xen/xl.conf was unaltered, i.e. 'autoballoon' was implicitely set >> to "auto". >> >> I am "on" Buster, kernel 5.10.0-8-amd64 (5.10.46-4), all relevant fixes >> included. > > Apologies for the inconvenience, > > Hans > > _______________________________________________ > Pkg-xen-devel mailing list > Pkg-xen-devel at alioth-lists.debian.net > https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-xen-devel >
Andy Smith
2021-Sep-29 22:45 UTC
[Pkg-xen-devel] Bug#994870: Bug#994870: Bug#994870: Bug#994870: Memory allocation problem for VM after xen security update
Hi Alex, On Thu, Sep 30, 2021 at 12:10:32AM +0200, Alexander Dahl wrote:> Am 22.09.21 um 20:54 schrieb Hans van Kranenburg: > > At this point I would really recommend to not wait for a fix to arrive > > which makes it start again, but change your VM to use a 64-bit kernel. > > How?This was answered in earlier comments on this bug; please see: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#15 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#20 The brief summary is, "start out like a crossgrade, but only do the kernel". Very simple and quite safe. You haven't said how you boot your guest though (show us your /etc/xen/guest.cfg file). If it's pvgrub, that has a 32-bit and a 64-bit version so you'll need to change those as well. If it's pygrub you probably don't need to do anything, though pygrub has its own issues outside the scope of this bug.> FWIW, Debian 10 VMs with 32 bit running with PVH work fine. My important VM > is still Debian 9 however due to a software I can not simply upgrade.I've found PVH needs at least 4.19 guest kernel to work, which can be achieved in Debian 9 (stretch) today by using kernel from stretch-backports, so perhaps that is an option for you. Cheers, Andy
Hans van Kranenburg
2021-Sep-30 11:13 UTC
[Pkg-xen-devel] Bug#994870: Bug#994870: Bug#994870: Bug#994870: Memory allocation problem for VM after xen security update
Hi! On 9/30/21 12:45 AM, Andy Smith wrote:> Hi Alex, > > On Thu, Sep 30, 2021 at 12:10:32AM +0200, Alexander Dahl wrote: >> Am 22.09.21 um 20:54 schrieb Hans van Kranenburg: >>> At this point I would really recommend to not wait for a fix to arrive >>> which makes it start again, but change your VM to use a 64-bit kernel. >> >> How? > > This was answered in earlier comments on this bug; please see: > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#15 > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#20 > > The brief summary is, "start out like a crossgrade, but only do the > kernel". Very simple and quite safe. > > You haven't said how you boot your guest though (show us your > /etc/xen/guest.cfg file). If it's pvgrub, that has a 32-bit and a > 64-bit version so you'll need to change those as well. If it's > pygrub you probably don't need to do anything, though pygrub has its > own issues outside the scope of this bug. > >> FWIW, Debian 10 VMs with 32 bit running with PVH work fine. My important VM >> is still Debian 9 however due to a software I can not simply upgrade. > > I've found PVH needs at least 4.19 guest kernel to work, which can > be achieved in Debian 9 (stretch) today by using kernel from > stretch-backports, so perhaps that is an option for you.You can certainly do that and then run PVH. Since stretch-backports is not used any more since stretch became oldoldstable, new 4.19 backports kernels for Stretch are released through the security updates channel. Be aware of this. https://lists.debian.org/debian-lts-announce/2020/08/msg00019.html Latest in stretch-backports (frozen) is 4.19.118, and stretch security is now at 4.19.194. So double check you end up following the right one. Hans
Alexander Dahl
2021-Oct-02 18:06 UTC
[Pkg-xen-devel] Bug#994870: Bug#994870: Bug#994870: Bug#994870: Memory allocation problem for VM after xen security update
Hei hei, On Thu, Sep 30, 2021 at 01:13:54PM +0200, Hans van Kranenburg wrote:> Hi! > > On 9/30/21 12:45 AM, Andy Smith wrote: > > Hi Alex, > > > > On Thu, Sep 30, 2021 at 12:10:32AM +0200, Alexander Dahl wrote: > >> Am 22.09.21 um 20:54 schrieb Hans van Kranenburg: > >>> At this point I would really recommend to not wait for a fix to arrive > >>> which makes it start again, but change your VM to use a 64-bit kernel. > >> > >> How? > > > > This was answered in earlier comments on this bug; please see: > > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#15 > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870#20 > > > > The brief summary is, "start out like a crossgrade, but only do the > > kernel". Very simple and quite safe. > > > > You haven't said how you boot your guest though (show us your > > /etc/xen/guest.cfg file). If it's pvgrub, that has a 32-bit and a > > 64-bit version so you'll need to change those as well. If it's > > pygrub you probably don't need to do anything, though pygrub has its > > own issues outside the scope of this bug. > > > >> FWIW, Debian 10 VMs with 32 bit running with PVH work fine. My important VM > >> is still Debian 9 however due to a software I can not simply upgrade. > > > > I've found PVH needs at least 4.19 guest kernel to work, which can > > be achieved in Debian 9 (stretch) today by using kernel from > > stretch-backports, so perhaps that is an option for you. > > You can certainly do that and then run PVH.This actually works. I'm running 4.19 i686 kernel in the stretch VM now with PVH, at least for the Debian stretch VM (I had to permanently disable some old OpenWRT VMs, where I get no updates anymore). Was a little tricky to install it, because I had to install that kernel without the vm being able to start, but it worked like this: - mount the root filesystem of the vm in the host, e.g. to /mnt - bind mount /dev, /sys to /mnt/dev and /mnt/sys - mount procfs to /mnt/proc - mount a tmpfs to /mnt/run and create /run/lock - chroot into /mnt - install the needed kernel with apt - leave chroot, umount the things from above - change domU config to PVH - when in grub, edit the cmdline and change root= if it was changed by update-grub (might have been changed to the mount point from the chroot) - in the now booted system, run update-grub again> Since stretch-backports is not used any more since stretch became > oldoldstable, new 4.19 backports kernels for Stretch are released > through the security updates channel. Be aware of this. > > https://lists.debian.org/debian-lts-announce/2020/08/msg00019.html > > Latest in stretch-backports (frozen) is 4.19.118, and stretch security > is now at 4.19.194. So double check you end up following the right one.Thanks for all your hints. I really have to migrate my virtual machines. :-/ Greets Alex -- /"\ ASCII RIBBON | ?With the first link, the chain is forged. The first \ / CAMPAIGN | speech censured, the first thought forbidden, the X AGAINST | first freedom denied, chains us all irrevocably.? / \ HTML MAIL | (Jean-Luc Picard, quoting Judge Aaron Satie) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: not available URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20211002/8162e00b/attachment.sig>
Debian Bug Tracking System
2022-Feb-25 16:42 UTC
[Pkg-xen-devel] Bug#994870: marked as done (Memory allocation problem for VM after xen security update)
Your message dated Fri, 25 Feb 2022 17:29:20 +0100 with message-id <5226fd2e-5044-7662-d0d9-1dc0a8a50b4e at knorrie.org> and subject line Re: Bug#994870: Memory allocation problem for VM after xen security update has caused the Debian Bug report #994870, regarding Memory allocation problem for VM after xen security update to be marked as done. This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact owner at bugs.debian.org immediately.) -- 994870: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=994870 Debian Bug Tracking System Contact owner at bugs.debian.org with problems -------------- next part -------------- An embedded message was scrubbed... From: "H.-R. Oberhage" <oberhage at Uni-Duisburg-Essen.DE> Subject: Memory allocation problem for VM after xen security update Date: Wed, 22 Sep 2021 11:37:28 +0200 Size: 3514 URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20220225/b8a60b92/attachment.eml> -------------- next part -------------- An embedded message was scrubbed... From: Hans van Kranenburg <hans at knorrie.org> Subject: Re: Bug#994870: Memory allocation problem for VM after xen security update Date: Fri, 25 Feb 2022 17:29:20 +0100 Size: 2946 URL: <http://alioth-lists.debian.net/pipermail/pkg-xen-devel/attachments/20220225/b8a60b92/attachment-0001.eml>