James Dingwall
2012-Dec-19 08:47 UTC
kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17
Hi, I have encountered an apparently benign error on two systems where the dom0 kernel log is flooded with messages like: [52482.163855] System RAM resource [mem 0x1b8000000-0x1bfffffff] cannot be added [52482.163860] xen_balloon: reserve_additional_memory: add_memory() failed: -17 The first line is from drivers/xen/xen-balloon.c, the second from mm/memory_hotplug.c The trigger for the messages seems to be the first occasion that a Xen guest is shutdown. I have noted this in a vanilla 3.6.7 and kernel 3.5.0-18 built from Ubuntu sources. Xen version is 4.2.0. It is not clear why the dom0 is kernel is trying to balloon up as the Xen command line is specifies a fixed dom0 memory allocation and noselfballooning is specified for the kernel and ballooning is also disabled in the xend-config.sxp / xl.conf (one system using xm, another xl) xen command line: placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1 dom0_mem=max:6144m kernel command line: root=/dev/loop0 ro console=tty1 console=hvc0 earlyprintk=xen nomodeset noselfballooning Examining /proc/iomem does show that the dom0 memory allocation is actually 64kb short of 6144Mb: cat /proc/iomem | grep System\ RAM 00010000-0009bfff : System RAM [573440 bytes] 00100000-cb2dffff : System RAM [3407740928 bytes] 100000000-1b4d83fff : System RAM [3034071040 bytes] Total system ram: 6442385408 - 6x2^30 = 65536 The memory range indicated in the log message is "Unusable memory" in /proc/iomem: 1b4d84000-82fffffff : Unusable memory Another point of interest is that we have multiple "identical" hardware platforms (Dell T320) for the system running the 3.5.0-18 kernel but only see this error on a slightly more recent system. Older systems show in /proc/iomem that all memory is System RAM. 100000000-82fffffff : System RAM [older system BIOS 1.0] 100000000-1b4d83fff : System RAM [newer system BIOS 1.3] 1b4d84000-82fffffff : Unusable memory The BIOS revision between the old and new has changed so I was wondering if it is possible that there is a white list which affects the impact of the kernel option: CONFIG_X86_RESERVE_LOW=64 This is only a guess since the amount of memory reserved is equivalent to the short fall calculated above. If this is the right area perhaps the dom0 calculation for its memory entitlement needs to be taught to not to try and hotplug the missing 64k when it has been reserved. If any other information would be useful then please let me know. Thanks, James
James Dingwall
2012-Dec-19 10:55 UTC
Re: kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17
On 2012-12-19 08:47, James Dingwall wrote:> Hi, > > I have encountered an apparently benign error on two systems where > the dom0 kernel log is flooded with messages like: > > [52482.163855] System RAM resource [mem 0x1b8000000-0x1bfffffff] > cannot be added > [52482.163860] xen_balloon: reserve_additional_memory: add_memory() > failed: -17 > > The first line is from drivers/xen/xen-balloon.c, the second from > mm/memory_hotplug.c > > The trigger for the messages seems to be the first occasion that a > Xen guest is shutdown. I have noted this in a vanilla 3.6.7 and > kernel 3.5.0-18 built from Ubuntu sources. Xen version is 4.2.0. It > is not clear why the dom0 is kernel is trying to balloon up as the > Xen > command line is specifies a fixed dom0 memory allocation and > noselfballooning is specified for the kernel and ballooning is also > disabled in the xend-config.sxp / xl.conf (one system using xm, > another xl) > > xen command line: > placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1 > dom0_mem=max:6144m > > kernel command line: > root=/dev/loop0 ro console=tty1 console=hvc0 earlyprintk=xen > nomodeset noselfballooning > > Examining /proc/iomem does show that the dom0 memory allocation is > actually 64kb short of 6144Mb: > > cat /proc/iomem | grep System\ RAM > 00010000-0009bfff : System RAM [573440 bytes] > 00100000-cb2dffff : System RAM [3407740928 bytes] > 100000000-1b4d83fff : System RAM [3034071040 bytes] > > Total system ram: 6442385408 - 6x2^30 = 65536 > > The memory range indicated in the log message is "Unusable memory" in > /proc/iomem: > 1b4d84000-82fffffff : Unusable memory > > Another point of interest is that we have multiple "identical" > hardware platforms (Dell T320) for the system running the 3.5.0-18 > kernel but only see this error on a slightly more recent system. > Older systems show in /proc/iomem that all memory is System RAM. > > 100000000-82fffffff : System RAM [older system BIOS 1.0] > > 100000000-1b4d83fff : System RAM [newer system BIOS 1.3] > 1b4d84000-82fffffff : Unusable memory > > The BIOS revision between the old and new has changed so I was > wondering if it is possible that there is a white list which affects > the impact of the kernel option: > CONFIG_X86_RESERVE_LOW=64 > This is only a guess since the amount of memory reserved is > equivalent to the short fall calculated above. If this is the right > area perhaps the dom0 calculation for its memory entitlement needs to > be taught to not to try and hotplug the missing 64k when it has been > reserved. > > If any other information would be useful then please let me know.With some further investigation we have determined that the different BIOS version does not seem to be a factor and the key point is actually the Xen command line. The reason that we had max: specified is that without it we could not boot the kernel/xen combination on an AMD platform. I will do some further testing to see what the result of dom0_mem=6144m,max:6144m as suggested http://wiki.xen.org/wiki/Xen_Best_Practices gets us. placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1 dom0_mem=max:6144m results in /proc/iomem having an unusable range and top reports 6083900k of memory in dom0 placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1 dom0_mem=6144m no unusable range, top reports 5605976k of memory in dom0 and no log messages
Jacek Konieczny
2012-Dec-20 14:50 UTC
Re: kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17
On Wed, Dec 19, 2012 at 08:47:22AM +0000, James Dingwall wrote:> Hi, > > I have encountered an apparently benign error on two systems where the > dom0 kernel log is flooded with messages like: > > [52482.163855] System RAM resource [mem 0x1b8000000-0x1bfffffff] cannot > be added > [52482.163860] xen_balloon: reserve_additional_memory: add_memory() > failed: -17I have seen this bug too (under Xen 4.2.0). I am using the following workaround to stop those messages: cat /sys/devices/system/xen_memory/xen_memory0/info/current_kb > \ /sys/devices/system/xen_memory/xen_memory0/target_kb I have not verified yet if Xen 4.2.1 is also affected. Greets, Jacek
James Dingwall
2012-Dec-20 15:55 UTC
Re: kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17
On 2012-12-20 14:50, Jacek Konieczny wrote:> On Wed, Dec 19, 2012 at 08:47:22AM +0000, James Dingwall wrote: >> Hi, >> >> I have encountered an apparently benign error on two systems where >> the >> dom0 kernel log is flooded with messages like: >> >> [52482.163855] System RAM resource [mem 0x1b8000000-0x1bfffffff] >> cannot >> be added >> [52482.163860] xen_balloon: reserve_additional_memory: add_memory() >> failed: -17 > > I have seen this bug too (under Xen 4.2.0). > > I am using the following workaround to stop those messages: > > cat /sys/devices/system/xen_memory/xen_memory0/info/current_kb > \ > /sys/devices/system/xen_memory/xen_memory0/target_kb > > I have not verified yet if Xen 4.2.1 is also affected.Thanks for the cat tip, that works for me too. The issue is still present in Xen 4.2.1 with kernel. 3.7.1. James
Konrad Rzeszutek Wilk
2012-Dec-21 20:23 UTC
Re: kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17
On Wed, Dec 19, 2012 at 10:55:49AM +0000, James Dingwall wrote:> On 2012-12-19 08:47, James Dingwall wrote: > >Hi, > > > >I have encountered an apparently benign error on two systems where > >the dom0 kernel log is flooded with messages like: > > > >[52482.163855] System RAM resource [mem 0x1b8000000-0x1bfffffff] > >cannot be added > >[52482.163860] xen_balloon: reserve_additional_memory: > >add_memory() failed: -17 > > > >The first line is from drivers/xen/xen-balloon.c, the second from > >mm/memory_hotplug.c > > > >The trigger for the messages seems to be the first occasion that a > >Xen guest is shutdown. I have noted this in a vanilla 3.6.7 and > >kernel 3.5.0-18 built from Ubuntu sources. Xen version is 4.2.0. It > >is not clear why the dom0 is kernel is trying to balloon up as the > >Xen > >command line is specifies a fixed dom0 memory allocation and > >noselfballooning is specified for the kernel and ballooning is also > >disabled in the xend-config.sxp / xl.conf (one system using xm, > >another xl) > > > >xen command line: > >placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1 > >dom0_mem=max:6144m > > > >kernel command line: > >root=/dev/loop0 ro console=tty1 console=hvc0 earlyprintk=xen > >nomodeset noselfballooning > > > >Examining /proc/iomem does show that the dom0 memory allocation is > >actually 64kb short of 6144Mb: > > > >cat /proc/iomem | grep System\ RAM > >00010000-0009bfff : System RAM [573440 bytes] > >00100000-cb2dffff : System RAM [3407740928 bytes] > >100000000-1b4d83fff : System RAM [3034071040 bytes] > > > >Total system ram: 6442385408 - 6x2^30 = 65536 > > > >The memory range indicated in the log message is "Unusable memory" in > >/proc/iomem: > >1b4d84000-82fffffff : Unusable memory > > > >Another point of interest is that we have multiple "identical" > >hardware platforms (Dell T320) for the system running the 3.5.0-18 > >kernel but only see this error on a slightly more recent system. > >Older systems show in /proc/iomem that all memory is System RAM. > > > >100000000-82fffffff : System RAM [older system BIOS 1.0]Wow. That is a lot of memory :-)> > > >100000000-1b4d83fff : System RAM [newer system BIOS 1.3] > >1b4d84000-82fffffff : Unusable memory > > > >The BIOS revision between the old and new has changed so I was > >wondering if it is possible that there is a white list which affects > >the impact of the kernel option: > >CONFIG_X86_RESERVE_LOW=64 > >This is only a guess since the amount of memory reserved is > >equivalent to the short fall calculated above. If this is the right > >area perhaps the dom0 calculation for its memory entitlement needs to > >be taught to not to try and hotplug the missing 64k when it has been > >reserved. > > > >If any other information would be useful then please let me know. > > With some further investigation we have determined that the > different BIOS version does not seem to be a factor and the key > point is actually the Xen command line. The reason that we had max: > specified is that without it we could not boot the kernel/xen > combination on an AMD platform. I will do some further testing toWhat happend? Did you report it in another email on this mailing list?> see what the result of dom0_mem=6144m,max:6144m as suggested > http://wiki.xen.org/wiki/Xen_Best_Practices gets us. > > placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1 > dom0_mem=max:6144mI think you don''t need xsave=0 anymore? It was only needed with a Fedora stock kernel (and the underlaying issue I believe is now fixed).> results in /proc/iomem having an unusable range and top reports > 6083900k of memory in dom0 > > placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1 > dom0_mem=6144m > no unusable range, top reports 5605976k of memory in dom0 and no log > messagesHmm.. If you were to run this with ''loglevel=8 debug'' with the dom0_mem=6144m and dom0_mem=max:6144m could you send both ''xl dmesg'' and ''dmesg'' outputs?> > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
Konrad Rzeszutek Wilk
2012-Dec-21 20:25 UTC
Re: kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17
On Wed, Dec 19, 2012 at 08:47:22AM +0000, James Dingwall wrote:> Hi, > > I have encountered an apparently benign error on two systems where > the dom0 kernel log is flooded with messages like: > > [52482.163855] System RAM resource [mem 0x1b8000000-0x1bfffffff] > cannot be added > [52482.163860] xen_balloon: reserve_additional_memory: add_memory() > failed: -17Daniel tells me it is due to the recent changes in the balloon code. CC-ing him here.> > The first line is from drivers/xen/xen-balloon.c, the second from > mm/memory_hotplug.c > > The trigger for the messages seems to be the first occasion that a > Xen guest is shutdown. I have noted this in a vanilla 3.6.7 and > kernel 3.5.0-18 built from Ubuntu sources. Xen version is 4.2.0. > It is not clear why the dom0 is kernel is trying to balloon up as > the Xen command line is specifies a fixed dom0 memory allocation and > noselfballooning is specified for the kernel and ballooning is also > disabled in the xend-config.sxp / xl.conf (one system using xm, > another xl) > > xen command line: > placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1 > dom0_mem=max:6144m > > kernel command line: > root=/dev/loop0 ro console=tty1 console=hvc0 earlyprintk=xen > nomodeset noselfballooning > > Examining /proc/iomem does show that the dom0 memory allocation is > actually 64kb short of 6144Mb: > > cat /proc/iomem | grep System\ RAM > 00010000-0009bfff : System RAM [573440 bytes] > 00100000-cb2dffff : System RAM [3407740928 bytes] > 100000000-1b4d83fff : System RAM [3034071040 bytes] > > Total system ram: 6442385408 - 6x2^30 = 65536 > > The memory range indicated in the log message is "Unusable memory" > in /proc/iomem: > 1b4d84000-82fffffff : Unusable memory > > Another point of interest is that we have multiple "identical" > hardware platforms (Dell T320) for the system running the 3.5.0-18 > kernel but only see this error on a slightly more recent system. > Older systems show in /proc/iomem that all memory is System RAM. > > 100000000-82fffffff : System RAM [older system BIOS 1.0] > > 100000000-1b4d83fff : System RAM [newer system BIOS 1.3] > 1b4d84000-82fffffff : Unusable memory > > The BIOS revision between the old and new has changed so I was > wondering if it is possible that there is a white list which affects > the impact of the kernel option: > CONFIG_X86_RESERVE_LOW=64 > This is only a guess since the amount of memory reserved is > equivalent to the short fall calculated above. If this is the right > area perhaps the dom0 calculation for its memory entitlement needs > to be taught to not to try and hotplug the missing 64k when it has > been reserved. > > If any other information would be useful then please let me know. > > Thanks, > James > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >
Carsten Schiers
2012-Dec-23 10:41 UTC
Re: kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17
Hi folks, I had the same messages flooding logs, but it was in DomUs. It came together with e.g.: System RAM resource [mem 0x20000000-0x3fffffff] cannot be added I am not 100% sure, but it was only for DomUs with PCI and PCIe devices passed-through. Xen 4.2.1, Dom0&DomU Kernel 3.7.1. Xen commandline with dom0_mem=2G and dom0_mem=2G,max:2G. Machine is an Intel 64 Bit 16 GB installation. Difficult to check deeper, as it made me so nervous that I reverted to Xen 4.1, but I could certainly re-install it after Xmas, if it is helping... BR, Carsten. -----Ursprüngliche Nachricht----- Von: xen-devel-bounces@lists.xen.org [mailto:xen-devel-bounces@lists.xen.org] Im Auftrag von Konrad Rzeszutek Wilk Gesendet: Freitag, 21. Dezember 2012 21:25 An: James Dingwall Cc: daniel.kiper@oracle.com; xen-devel@lists.xen.org Betreff: Re: [Xen-devel] kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17 On Wed, Dec 19, 2012 at 08:47:22AM +0000, James Dingwall wrote:> Hi, > > I have encountered an apparently benign error on two systems where the > dom0 kernel log is flooded with messages like: > > [52482.163855] System RAM resource [mem 0x1b8000000-0x1bfffffff] > cannot be added [52482.163860] xen_balloon: reserve_additional_memory: > add_memory() > failed: -17Daniel tells me it is due to the recent changes in the balloon code. CC-ing him here.> > The first line is from drivers/xen/xen-balloon.c, the second from > mm/memory_hotplug.c > > The trigger for the messages seems to be the first occasion that a Xen > guest is shutdown. I have noted this in a vanilla 3.6.7 and kernel > 3.5.0-18 built from Ubuntu sources. Xen version is 4.2.0. > It is not clear why the dom0 is kernel is trying to balloon up as the > Xen command line is specifies a fixed dom0 memory allocation and > noselfballooning is specified for the kernel and ballooning is also > disabled in the xend-config.sxp / xl.conf (one system using xm, > another xl) > > xen command line: > placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1 > dom0_mem=max:6144m > > kernel command line: > root=/dev/loop0 ro console=tty1 console=hvc0 earlyprintk=xen nomodeset > noselfballooning > > Examining /proc/iomem does show that the dom0 memory allocation is > actually 64kb short of 6144Mb: > > cat /proc/iomem | grep System\ RAM > 00010000-0009bfff : System RAM [573440 bytes] > 00100000-cb2dffff : System RAM [3407740928 bytes] > 100000000-1b4d83fff : System RAM [3034071040 bytes] > > Total system ram: 6442385408 - 6x2^30 = 65536 > > The memory range indicated in the log message is "Unusable memory" > in /proc/iomem: > 1b4d84000-82fffffff : Unusable memory > > Another point of interest is that we have multiple "identical" > hardware platforms (Dell T320) for the system running the 3.5.0-18 > kernel but only see this error on a slightly more recent system. > Older systems show in /proc/iomem that all memory is System RAM. > > 100000000-82fffffff : System RAM [older system BIOS 1.0] > > 100000000-1b4d83fff : System RAM [newer system BIOS 1.3] > 1b4d84000-82fffffff : Unusable memory > > The BIOS revision between the old and new has changed so I was > wondering if it is possible that there is a white list which affects > the impact of the kernel option: > CONFIG_X86_RESERVE_LOW=64 > This is only a guess since the amount of memory reserved is equivalent > to the short fall calculated above. If this is the right area perhaps > the dom0 calculation for its memory entitlement needs to be taught to > not to try and hotplug the missing 64k when it has been reserved. > > If any other information would be useful then please let me know. > > Thanks, > James > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >_______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Daniel Kiper
2012-Dec-24 14:39 UTC
Re: kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17
Hi,> I had the same messages flooding logs, but it was in DomUs. It came together with e.g.: > > System RAM resource [mem 0x20000000-0x3fffffff] cannot be added > > I am not 100% sure, but it was only for DomUs with PCI and PCIe devices passed-through. > > Xen 4.2.1, Dom0&DomU Kernel 3.7.1. Xen commandline with dom0_mem=2G and dom0_mem=2G,max:2G. > Machine is an Intel 64 Bit 16 GB installation. > > Difficult to check deeper, as it made me so nervous that I reverted to Xen 4.1, but I could > certainly re-install it after Xmas, if it is helping...Thanks for update. I fill that it could be linked with recent balloon driver updates. I will take a look at that bug in new year. Daniel
James Dingwall
2013-Jan-02 10:28 UTC
Re: kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17
On 2012-12-21 20:23, Konrad Rzeszutek Wilk wrote:> On Wed, Dec 19, 2012 at 10:55:49AM +0000, James Dingwall wrote: >> On 2012-12-19 08:47, James Dingwall wrote:Hi Konrad, Thanks for your interest in this problem, sorry for the delay in replying but today is the first day I''ve had access to the machine to do more testing.>> With some further investigation we have determined that the >> different BIOS version does not seem to be a factor and the key >> point is actually the Xen command line. The reason that we had max: >> specified is that without it we could not boot the kernel/xen >> combination on an AMD platform. I will do some further testing to > > What happend? Did you report it in another email on this mailing > list?I reported this on xen-users http://lists.xen.org/archives/html/xen-users/2012-07/msg00062.html>> see what the result of dom0_mem=6144m,max:6144m as suggested >> http://wiki.xen.org/wiki/Xen_Best_Practices gets us. >> >> placeholder xsave=0 iommu=0 console=vga,com2 com2=115200,8n1 >> dom0_mem=max:6144m > > I think you don''t need xsave=0 anymore? It was only needed with > a Fedora stock kernel (and the underlaying issue I believe is > now fixed).IIRC the stock Ubuntu Precise kernel needed this but as we are running our own build now it is probably unnecessary.> Hmm.. > > If you were to run this with ''loglevel=8 debug'' with the > dom0_mem=6144m and dom0_mem=max:6144m could you send both > ''xl dmesg'' and ''dmesg'' outputs?Output attached. Thicky question, loglevel=8 debug was for the Xen command line? James _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Carsten Schiers
2013-Jan-24 21:38 UTC
Re: kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17
Hello Daniel, have you already had a chance to look at the issue? BR, Carsten. -----Ursprüngliche Nachricht----- Von: xen-devel-bounces@lists.xen.org [mailto:xen-devel-bounces@lists.xen.org] Im Auftrag von Daniel Kiper Gesendet: Montag, 24. Dezember 2012 15:39 An: Carsten Schiers Cc: xen-devel@lists.xen.org; james-xen@dingwall.me.uk; konrad.wilk@oracle.com Betreff: Re: [Xen-devel] kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17 Hi,> I had the same messages flooding logs, but it was in DomUs. It came together with e.g.: > > System RAM resource [mem 0x20000000-0x3fffffff] cannot be added > > I am not 100% sure, but it was only for DomUs with PCI and PCIe devices passed-through. > > Xen 4.2.1, Dom0&DomU Kernel 3.7.1. Xen commandline with dom0_mem=2G and dom0_mem=2G,max:2G. > Machine is an Intel 64 Bit 16 GB installation. > > Difficult to check deeper, as it made me so nervous that I reverted to > Xen 4.1, but I could certainly re-install it after Xmas, if it is helping...Thanks for update. I fill that it could be linked with recent balloon driver updates. I will take a look at that bug in new year. Daniel _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ----- E-Mail ist virenfrei. Von AVG überprüft - www.avg.de Version: 2013.0.2805 / Virendatenbank: 2637/5993 - Ausgabedatum: 28.12.2012
Possibly Parallel Threads
- xen_balloon: reserve_additional_memory: add_memory() Errors
- messages logs flooded with xen_balloon: reserve_additional_memory: add_memory() failed
- Re: kernel log flooded with: xen_balloon: reserve_additional_memory: add_memory() failed: -17
- Where to get a newer xen dom0 kernel?
- Where to get a newer xen dom0 kernel?