Hi all, my setup: dom0 and domU - Debian Squeeze (updated) dom0 kernel: linux-image-2.6.32-5-xen-amd64 (Debian revision 2.6.32-45) Xen hypervisor: 4.0.1 (Debian revision 4.0.1-5.4) with domU running kernels 2.6.32-5-xen-amd64 and 2.6.32-5-amd64 the memory extension via mem-set is not working with domU running older kernel 2.6.26-2-xen-amd64 (Debian revision 2.6.26-27) the memory extension via mem-set is working fine These 2.6.32 kernels should be pvops kernels. Is there any special procedure for the memory extension on pvops kernels? Can somebody tell me what am I doing wrong and how I could get this online memory increasing working? Thank you. Best regards, -- Peter Viskup
Ian Campbell
2012-Nov-14 10:12 UTC
Re: dynamic memory extension not working on Debian Squeeze
On Wed, 2012-11-14 at 00:19 +0000, Peter Viskup wrote:> Hi all, > my setup: > dom0 and domU - Debian Squeeze (updated) > dom0 kernel: linux-image-2.6.32-5-xen-amd64 (Debian revision 2.6.32-45) > Xen hypervisor: 4.0.1 (Debian revision 4.0.1-5.4) > with domU running kernels 2.6.32-5-xen-amd64 and 2.6.32-5-amd64 the > memory extension via mem-set is not working > with domU running older kernel 2.6.26-2-xen-amd64 (Debian revision > 2.6.26-27) the memory extension via mem-set is working fine > > These 2.6.32 kernels should be pvops kernels. Is there any special > procedure for the memory extension on pvops kernels? > > Can somebody tell me what am I doing wrong and how I could get this > online memory increasing working?can you supply your guest configuration please, as well as the specific values you are trying to mem-set (and the output of those commands). Also the output of # xm list # xenstore-ls -fp | grep target after creation and after attempting to mem-set would be handy. Also the content of /sys/devices/system/xen_memory/xen_memory0/target_kb , MemTotal from /proc/meminfo. Ian.
Peter Viskup
2012-Nov-14 10:35 UTC
Re: dynamic memory extension not working on Debian Squeeze
On 11/14/2012 11:12 AM, Ian Campbell wrote:> On Wed, 2012-11-14 at 00:19 +0000, Peter Viskup wrote: >> Hi all, >> my setup: >> dom0 and domU - Debian Squeeze (updated) >> dom0 kernel: linux-image-2.6.32-5-xen-amd64 (Debian revision 2.6.32-45) >> Xen hypervisor: 4.0.1 (Debian revision 4.0.1-5.4) >> with domU running kernels 2.6.32-5-xen-amd64 and 2.6.32-5-amd64 the >> memory extension via mem-set is not working >> with domU running older kernel 2.6.26-2-xen-amd64 (Debian revision >> 2.6.26-27) the memory extension via mem-set is working fine >> >> These 2.6.32 kernels should be pvops kernels. Is there any special >> procedure for the memory extension on pvops kernels? >> >> Can somebody tell me what am I doing wrong and how I could get this >> online memory increasing working? > can you supply your guest configuration please, as well as the specific > values you are trying to mem-set (and the output of those commands). > > Also the output of > # xm list > # xenstore-ls -fp | grep target > after creation and after attempting to mem-set would be handy. Also the > content of /sys/devices/system/xen_memory/xen_memory0/target_kb , > MemTotal from /proc/meminfo. > > Ian. > >Hi Ian, here are the outputs and additiona info: server1:~# xm list ldap Name ID Mem VCPUs State Time(s) ldap 2 256 1 -b---- 449.3 server1:~# xm mem-set ldap 512 server1:~# xm list ldap Name ID Mem VCPUs State Time(s) ldap 2 512 1 -b---- 449.4 ldap:~# grep MemTotal /proc/meminfo MemTotal: 250320 kB ldap:~# cat /sys/devices/system/xen_memory/xen_memory0/target_kb 524288 ldap:~# uname -a Linux ldap 2.6.32-5-amd64 #1 SMP Sun Sep 23 10:07:46 UTC 2012 x86_64 GNU/Linux server1:~# xenstore-ls -fp | grep target <this command is hanging> xend.log entry from today: [2012-11-14 11:27:24 2879] DEBUG (XendDomainInfo:1477) Setting memory target of domain ldap (2) to 512 MiB. messages errors: Nov 14 11:29:06 localhost kernel: : [37380.393789] XENBUS error -22 while reading message xenfs is not mounted on dom0 - could it be the reason of this issues? Any thoughts? -- Peter
Ian Campbell
2012-Nov-14 10:39 UTC
Re: dynamic memory extension not working on Debian Squeeze
On Wed, 2012-11-14 at 10:35 +0000, Peter Viskup wrote: [...]> server1:~# xenstore-ls -fp | grep target > <this command is hanging>[...]> xenfs is not mounted on dom0 - could it be the reason of this issues?Could be, I''m surprised you were able to start a domain at all without it though. Also if xenstore is hanging then there is something else majorly wrong too. Probably best to remedy both of those first. Mounting xenfs might be enough to kick things back to life but otherwise you may need to reboot the host. Ian.
Peter Viskup
2012-Nov-14 10:46 UTC
Re: dynamic memory extension not working on Debian Squeeze
On 11/14/2012 11:39 AM, Ian Campbell wrote:> On Wed, 2012-11-14 at 10:35 +0000, Peter Viskup wrote: > [...] >> server1:~# xenstore-ls -fp | grep target >> <this command is hanging> > [...] >> xenfs is not mounted on dom0 - could it be the reason of this issues? > Could be, I''m surprised you were able to start a domain at all without > it though. > > Also if xenstore is hanging then there is something else majorly wrong > too. Probably best to remedy both of those first. > > Mounting xenfs might be enough to kick things back to life but otherwise > you may need to reboot the host. > > Ian.Sorry I was wrong - xenfs mounted on dom0 and also on domU. Server rebooted last night (6-7 hours before) - no additional installation tasks were performed. These are xen-related packages installed: ii xen-tools 4.2-1 Tools to manage Xen virtual servers ii xen-utils-4.0 4.0.1-5.4 XEN administrative tools ii xen-utils-common 4.0.0-1 XEN administrative tools - common files ii xen-hypervisor-4.0-amd64 4.0.1-5.4 The Xen Hypervisor on AMD64 ii linux-image-2.6.26-2-xen-amd64 2.6.26-27 Linux 2.6.26 image on AMD64, oldstyle Xen support ii linux-image-2.6.32-5-xen-amd64 2.6.32-46 Linux 2.6.32 for 64-bit PCs, Xen dom0 support These is configurations of domU: kernel = ''/boot/vmlinuz-2.6.32-5-amd64'' ramdisk = ''/boot/initrd.img-2.6.32-5-amd64'' memory = ''256'' memmax = ''4092'' cpus = ''1-2'' vcpus = ''1'' cpu_weight = memory cpu_cap = 100 extra = ''selinux=0'' root = ''/dev/xvda1 ro'' disk = [ ''phy:system_vhosts/ldap-disk,xvda1,w'', ''phy:system_vhosts/ldap-swap,xvda2,w'', ''phy:system_vhosts/ldap-var,xvda3,w'', ''phy:system_vhosts/ldap-varlog,xvda4,w'', ''phy:system_vhosts/ldap-tmp,xvda5,w'', ''phy:system_vhosts/ldap-usr,xvda6,w'' ] name = ''ldap'' vif = [ ''ip=10.0.0.3, vifname=ldap-pub'', ''mac=00:16:3e:1a:7b:46, ip=192.168.0.3, vifname=ldap-pri, script=vif-bridge bridge=xenin''] Command xenstore-ls still hanging. -- Peter
Ian Campbell
2012-Nov-14 10:51 UTC
Re: dynamic memory extension not working on Debian Squeeze
On Wed, 2012-11-14 at 10:46 +0000, Peter Viskup wrote:> On 11/14/2012 11:39 AM, Ian Campbell wrote: > > On Wed, 2012-11-14 at 10:35 +0000, Peter Viskup wrote: > > [...] > >> server1:~# xenstore-ls -fp | grep target > >> <this command is hanging> > > [...] > >> xenfs is not mounted on dom0 - could it be the reason of this issues? > > Could be, I''m surprised you were able to start a domain at all without > > it though. > > > > Also if xenstore is hanging then there is something else majorly wrong > > too. Probably best to remedy both of those first. > > > > Mounting xenfs might be enough to kick things back to life but otherwise > > you may need to reboot the host. > > > > Ian. > Sorry I was wrong - xenfs mounted on dom0 and also on domU. > Server rebooted last night (6-7 hours before) - no additional > installation tasks were performed. > These are xen-related packages installed: > > ii xen-tools > 4.2-1 Tools to manage Xen virtual servers > ii xen-utils-4.0 > 4.0.1-5.4 XEN administrative tools > ii xen-utils-common > 4.0.0-1 XEN administrative tools - common files > ii xen-hypervisor-4.0-amd64 > 4.0.1-5.4 The Xen Hypervisor on AMD64 > ii linux-image-2.6.26-2-xen-amd64 > 2.6.26-27 Linux 2.6.26 image on AMD64, oldstyle Xen > support > ii linux-image-2.6.32-5-xen-amd64 > 2.6.32-46 Linux 2.6.32 for 64-bit PCs, Xen dom0 supportWhat version of xenstore-utils do you have installed?> memory = ''256'' > memmax = ''4092''[...] This is good, it *should* mean you can balloon up to 4G.> Command xenstore-ls still hanging.Are you running this in dom0 or domU? You should run it in dom0. If the xenstore tools don''t work in dom0 then there is a pretty big problem. You could try strace''ing the client and/or daemon I suppose? One not uncommon problem with the C xenstored is a corrupted DB, so removing /var/run/xenstored/tdb and rebooting might help. In general removing this on boot (before starting the daemon) is a good idea, not sure if the Debian scripts do this for you. Ian.
Peter Viskup
2012-Nov-14 11:08 UTC
Re: dynamic memory extension not working on Debian Squeeze
On 11/14/2012 11:51 AM, Ian Campbell wrote:> On Wed, 2012-11-14 at 10:46 +0000, Peter Viskup wrote: >> On 11/14/2012 11:39 AM, Ian Campbell wrote: >>> On Wed, 2012-11-14 at 10:35 +0000, Peter Viskup wrote: >>> [...] >>>> server1:~# xenstore-ls -fp | grep target >>>> <this command is hanging> >>> [...] >>>> xenfs is not mounted on dom0 - could it be the reason of this issues? >>> Could be, I''m surprised you were able to start a domain at all without >>> it though. >>> >>> Also if xenstore is hanging then there is something else majorly wrong >>> too. Probably best to remedy both of those first. >>> >>> Mounting xenfs might be enough to kick things back to life but otherwise >>> you may need to reboot the host. >>> >>> Ian. >> Sorry I was wrong - xenfs mounted on dom0 and also on domU. >> Server rebooted last night (6-7 hours before) - no additional >> installation tasks were performed. >> These are xen-related packages installed: >> >> ii xen-tools >> 4.2-1 Tools to manage Xen virtual servers >> ii xen-utils-4.0 >> 4.0.1-5.4 XEN administrative tools >> ii xen-utils-common >> 4.0.0-1 XEN administrative tools - common files >> ii xen-hypervisor-4.0-amd64 >> 4.0.1-5.4 The Xen Hypervisor on AMD64 >> ii linux-image-2.6.26-2-xen-amd64 >> 2.6.26-27 Linux 2.6.26 image on AMD64, oldstyle Xen >> support >> ii linux-image-2.6.32-5-xen-amd64 >> 2.6.32-46 Linux 2.6.32 for 64-bit PCs, Xen dom0 support > What version of xenstore-utils do you have installed? > >> memory = ''256'' >> memmax = ''4092'' > [...] > > This is good, it *should* mean you can balloon up to 4G. > >> Command xenstore-ls still hanging. > Are you running this in dom0 or domU? You should run it in dom0. > > If the xenstore tools don''t work in dom0 then there is a pretty big > problem. You could try strace''ing the client and/or daemon I suppose? > > One not uncommon problem with the C xenstored is a corrupted DB, so > removing /var/run/xenstored/tdb and rebooting might help. In general > removing this on boot (before starting the daemon) is a good idea, not > sure if the Debian scripts do this for you. > > Ian. > >I ran that xenstore command in dom0. These are versions of xenstore packages: ii libxenstore3.0 4.0.1-5.4 Xenstore communications library for Xen ii xenstore-utils 4.0.1-5.4 Xenstore utilities for Xen Strange is that xenstore tdb file is missing: server1:~# ls -la /var/run/xenstored/ total 8 drwxr-xr-x 2 root root 4096 Nov 14 01:06 . drwxr-xr-x 20 root root 4096 Nov 14 01:08 .. srw------- 1 root root 0 Nov 14 01:06 socket srw-rw---- 1 root root 0 Nov 14 01:06 socket_ro Debian is not removing that file during the shutdown (checked /etc/init.d/xend). I also checked checksums of all files of all xen-related packages and all is OK. As this is production box it''s hard to strace it during the office hours. Any way to get the xenstore tdb re-created without rebooting? -- Peter
Ian Campbell
2012-Nov-14 11:13 UTC
Re: dynamic memory extension not working on Debian Squeeze
On Wed, 2012-11-14 at 11:08 +0000, Peter Viskup wrote:> On 11/14/2012 11:51 AM, Ian Campbell wrote: > > On Wed, 2012-11-14 at 10:46 +0000, Peter Viskup wrote: > >> On 11/14/2012 11:39 AM, Ian Campbell wrote: > >>> On Wed, 2012-11-14 at 10:35 +0000, Peter Viskup wrote: > >>> [...] > >>>> server1:~# xenstore-ls -fp | grep target > >>>> <this command is hanging> > >>> [...] > >>>> xenfs is not mounted on dom0 - could it be the reason of this issues? > >>> Could be, I''m surprised you were able to start a domain at all without > >>> it though. > >>> > >>> Also if xenstore is hanging then there is something else majorly wrong > >>> too. Probably best to remedy both of those first. > >>> > >>> Mounting xenfs might be enough to kick things back to life but otherwise > >>> you may need to reboot the host. > >>> > >>> Ian. > >> Sorry I was wrong - xenfs mounted on dom0 and also on domU. > >> Server rebooted last night (6-7 hours before) - no additional > >> installation tasks were performed. > >> These are xen-related packages installed: > >> > >> ii xen-tools > >> 4.2-1 Tools to manage Xen virtual servers > >> ii xen-utils-4.0 > >> 4.0.1-5.4 XEN administrative tools > >> ii xen-utils-common > >> 4.0.0-1 XEN administrative tools - common files > >> ii xen-hypervisor-4.0-amd64 > >> 4.0.1-5.4 The Xen Hypervisor on AMD64 > >> ii linux-image-2.6.26-2-xen-amd64 > >> 2.6.26-27 Linux 2.6.26 image on AMD64, oldstyle Xen > >> support > >> ii linux-image-2.6.32-5-xen-amd64 > >> 2.6.32-46 Linux 2.6.32 for 64-bit PCs, Xen dom0 support > > What version of xenstore-utils do you have installed? > > > >> memory = ''256'' > >> memmax = ''4092'' > > [...] > > > > This is good, it *should* mean you can balloon up to 4G. > > > >> Command xenstore-ls still hanging. > > Are you running this in dom0 or domU? You should run it in dom0. > > > > If the xenstore tools don''t work in dom0 then there is a pretty big > > problem. You could try strace''ing the client and/or daemon I suppose? > > > > One not uncommon problem with the C xenstored is a corrupted DB, so > > removing /var/run/xenstored/tdb and rebooting might help. In general > > removing this on boot (before starting the daemon) is a good idea, not > > sure if the Debian scripts do this for you. > > > > Ian. > > > > > > I ran that xenstore command in dom0. > These are versions of xenstore packages: > ii libxenstore3.0 > 4.0.1-5.4 Xenstore communications library for Xen > ii xenstore-utils > 4.0.1-5.4 Xenstore utilities for XenLooks ok to me.> Strange is that xenstore tdb file is missing:Very odd. /var/run/xenstored was the path on my system, which I''ve just remembered is running Wheezy, perhaps it was elsewhere for Squeeze? "find /var -name tdb" might locate it. Otherweise /proc/<PID>/fds for the daemon''s PID might give a clue.> Debian is not removing that file during the shutdown (checked > /etc/init.d/xend). > I also checked checksums of all files of all xen-related packages and > all is OK. > > As this is production box it''s hard to strace it during the office hours. > Any way to get the xenstore tdb re-created without rebooting?Unfortunately not. I don''t suppose you can reproduce the issue on a non-production system? (I suspect not, it seems like the sort of oddity which has befallen just this one system) Ian.
Peter Viskup
2012-Nov-14 11:13 UTC
Re: dynamic memory extension not working on Debian Squeeze
On 11/14/2012 12:08 PM, Peter Viskup wrote:> On 11/14/2012 11:51 AM, Ian Campbell wrote: >> On Wed, 2012-11-14 at 10:46 +0000, Peter Viskup wrote: >>> On 11/14/2012 11:39 AM, Ian Campbell wrote: >>>> On Wed, 2012-11-14 at 10:35 +0000, Peter Viskup wrote: >>>> [...] >>>>> server1:~# xenstore-ls -fp | grep target >>>>> <this command is hanging> >>>> [...] >>>>> xenfs is not mounted on dom0 - could it be the reason of this issues? >>>> Could be, I''m surprised you were able to start a domain at all without >>>> it though. >>>> >>>> Also if xenstore is hanging then there is something else majorly wrong >>>> too. Probably best to remedy both of those first. >>>> >>>> Mounting xenfs might be enough to kick things back to life but >>>> otherwise >>>> you may need to reboot the host. >>>> >>>> Ian. >>> Sorry I was wrong - xenfs mounted on dom0 and also on domU. >>> Server rebooted last night (6-7 hours before) - no additional >>> installation tasks were performed. >>> These are xen-related packages installed: >>> >>> ii xen-tools >>> 4.2-1 Tools to manage Xen virtual servers >>> ii xen-utils-4.0 >>> 4.0.1-5.4 XEN administrative tools >>> ii xen-utils-common >>> 4.0.0-1 XEN administrative tools - common files >>> ii xen-hypervisor-4.0-amd64 >>> 4.0.1-5.4 The Xen Hypervisor on AMD64 >>> ii linux-image-2.6.26-2-xen-amd64 >>> 2.6.26-27 Linux 2.6.26 image on AMD64, oldstyle Xen >>> support >>> ii linux-image-2.6.32-5-xen-amd64 >>> 2.6.32-46 Linux 2.6.32 for 64-bit PCs, Xen dom0 >>> support >> What version of xenstore-utils do you have installed? >> >>> memory = ''256'' >>> memmax = ''4092'' >> [...] >> >> This is good, it *should* mean you can balloon up to 4G. >> >>> Command xenstore-ls still hanging. >> Are you running this in dom0 or domU? You should run it in dom0. >> >> If the xenstore tools don''t work in dom0 then there is a pretty big >> problem. You could try strace''ing the client and/or daemon I suppose? >> >> One not uncommon problem with the C xenstored is a corrupted DB, so >> removing /var/run/xenstored/tdb and rebooting might help. In general >> removing this on boot (before starting the daemon) is a good idea, not >> sure if the Debian scripts do this for you. >> >> Ian. >> >> > > I ran that xenstore command in dom0. > These are versions of xenstore packages: > ii libxenstore3.0 > 4.0.1-5.4 Xenstore communications library for Xen > ii xenstore-utils > 4.0.1-5.4 Xenstore utilities for Xen > > Strange is that xenstore tdb file is missing: > > server1:~# ls -la /var/run/xenstored/ > total 8 > drwxr-xr-x 2 root root 4096 Nov 14 01:06 . > drwxr-xr-x 20 root root 4096 Nov 14 01:08 .. > srw------- 1 root root 0 Nov 14 01:06 socket > srw-rw---- 1 root root 0 Nov 14 01:06 socket_ro > > Debian is not removing that file during the shutdown (checked > /etc/init.d/xend). > I also checked checksums of all files of all xen-related packages and > all is OK. > > As this is production box it''s hard to strace it during the office hours. > Any way to get the xenstore tdb re-created without rebooting? > > -- > PeterI found the tdb file - in Debian it is in /var/lib/xenstored: server1:~# ls -la /var/lib/xenstored/tdb -rw-r----- 1 root root 1368064 Nov 14 11:27 /var/lib/xenstored/tdb I will try to remove it and reboot this server at night. Is there any way to check the xenstore tdb file consistency? -- Peter
Peter Viskup
2012-Nov-14 13:07 UTC
Re: dynamic memory extension not working on Debian Squeeze
On 11/14/2012 12:13 PM, Peter Viskup wrote:> On 11/14/2012 12:08 PM, Peter Viskup wrote: >> On 11/14/2012 11:51 AM, Ian Campbell wrote: >>> On Wed, 2012-11-14 at 10:46 +0000, Peter Viskup wrote: >>>> On 11/14/2012 11:39 AM, Ian Campbell wrote: >>>>> On Wed, 2012-11-14 at 10:35 +0000, Peter Viskup wrote: >>>>> [...] >>>>>> server1:~# xenstore-ls -fp | grep target >>>>>> <this command is hanging> >>>>> [...] >>>>>> xenfs is not mounted on dom0 - could it be the reason of this >>>>>> issues? >>>>> Could be, I''m surprised you were able to start a domain at all >>>>> without >>>>> it though. >>>>> >>>>> Also if xenstore is hanging then there is something else majorly >>>>> wrong >>>>> too. Probably best to remedy both of those first. >>>>> >>>>> Mounting xenfs might be enough to kick things back to life but >>>>> otherwise >>>>> you may need to reboot the host. >>>>> >>>>> Ian. >>>> Sorry I was wrong - xenfs mounted on dom0 and also on domU. >>>> Server rebooted last night (6-7 hours before) - no additional >>>> installation tasks were performed. >>>> These are xen-related packages installed: >>>> >>>> ii xen-tools >>>> 4.2-1 Tools to manage Xen virtual servers >>>> ii xen-utils-4.0 >>>> 4.0.1-5.4 XEN administrative tools >>>> ii xen-utils-common >>>> 4.0.0-1 XEN administrative tools - common files >>>> ii xen-hypervisor-4.0-amd64 >>>> 4.0.1-5.4 The Xen Hypervisor on AMD64 >>>> ii linux-image-2.6.26-2-xen-amd64 >>>> 2.6.26-27 Linux 2.6.26 image on AMD64, oldstyle Xen >>>> support >>>> ii linux-image-2.6.32-5-xen-amd64 >>>> 2.6.32-46 Linux 2.6.32 for 64-bit PCs, Xen dom0 >>>> support >>> What version of xenstore-utils do you have installed? >>> >>>> memory = ''256'' >>>> memmax = ''4092'' >>> [...] >>> >>> This is good, it *should* mean you can balloon up to 4G. >>> >>>> Command xenstore-ls still hanging. >>> Are you running this in dom0 or domU? You should run it in dom0. >>> >>> If the xenstore tools don''t work in dom0 then there is a pretty big >>> problem. You could try strace''ing the client and/or daemon I suppose? >>> >>> One not uncommon problem with the C xenstored is a corrupted DB, so >>> removing /var/run/xenstored/tdb and rebooting might help. In general >>> removing this on boot (before starting the daemon) is a good idea, not >>> sure if the Debian scripts do this for you. >>> >>> Ian. >>> >>> >> >> I ran that xenstore command in dom0. >> These are versions of xenstore packages: >> ii libxenstore3.0 >> 4.0.1-5.4 Xenstore communications library for Xen >> ii xenstore-utils >> 4.0.1-5.4 Xenstore utilities for Xen >> >> Strange is that xenstore tdb file is missing: >> >> server1:~# ls -la /var/run/xenstored/ >> total 8 >> drwxr-xr-x 2 root root 4096 Nov 14 01:06 . >> drwxr-xr-x 20 root root 4096 Nov 14 01:08 .. >> srw------- 1 root root 0 Nov 14 01:06 socket >> srw-rw---- 1 root root 0 Nov 14 01:06 socket_ro >> >> Debian is not removing that file during the shutdown (checked >> /etc/init.d/xend). >> I also checked checksums of all files of all xen-related packages and >> all is OK. >> >> As this is production box it''s hard to strace it during the office >> hours. >> Any way to get the xenstore tdb re-created without rebooting? >> >> -- >> Peter > > I found the tdb file - in Debian it is in /var/lib/xenstored: > > server1:~# ls -la /var/lib/xenstored/tdb > -rw-r----- 1 root root 1368064 Nov 14 11:27 /var/lib/xenstored/tdb > > I will try to remove it and reboot this server at night. Is there any > way to check the xenstore tdb file consistency? > > -- > PeterJust for reference - I opened bugreport at Debian [1], and I have been told it is already implemented in xen-utils-common/4.1.0~rc6-1 (means it''s not in current stable branch). Will try to remove this tdb file and inform you if it helped me. [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=693217 -- Peter
Alexandre Kouznetsov
2012-Nov-14 18:29 UTC
Re: dynamic memory extension not working on Debian Squeeze
Hello, Ian.
I have been living with the same problem for years. I guess it worked on
Etch with kernel 2.6.18, not sure about Lenny and 2.6.26. In my case,
xenstore works fine.
The symptom is: after using mem-set, it apparently works, but the DomU
still thinks it has the old memory size. The issue shows up only when
increasing memory above the original size, the reduction work fine.
I have read the rest of the thread, and it seems Peter is stuck in some
xenstore issue. I suspect xenstore might be unrelated: in my case it
works fine, but mem-set still have the issue.
The OS on Dom0 and DomU is a regular Debian Squeeze, no backports or
custom build packages. It has been seen on different models of Dell
PowerEdge servers. If relevant, I can confirm for a commodity hardware
case, or for a Debian Wheezy case (upgraded from Squeeze, not a clean
install). I have no other OS at hand than Debian to test.
The output of "xenstore-ls -fp" that you asked remains unchanged, so I
will post it only once:
root@on05:~# xenstore-ls -fp | grep scoutapi-dev
/vm/8b7714b8-eeb0-2120-cee7-59273be76f79/name = "scoutapi-dev" (n0)
/local/domain/0/backend/vbd/12/51715/domain = "scoutapi-dev"
(n0,r12)
/local/domain/0/backend/vbd/12/51715/params =
"/dev/r5VG/scoutapi-dev-srv" (n0,r12)
/local/domain/0/backend/vbd/12/51714/domain = "scoutapi-dev"
(n0,r12)
/local/domain/0/backend/vbd/12/51714/params =
"/dev/r5VG/scoutapi-dev-root" (n0,r12)
/local/domain/0/backend/vbd/12/51713/domain = "scoutapi-dev"
(n0,r12)
/local/domain/0/backend/vbd/12/51713/params =
"/dev/r5VG/scoutapi-dev-swap" (n0,r12)
/local/domain/0/backend/vif/12/0/domain = "scoutapi-dev" (n0,r12)
/local/domain/0/backend/console/12/0/domain = "scoutapi-dev"
(n0,r12)
/local/domain/12/name = "scoutapi-dev" (n0,r12)
# DomU scoutapi-dev initially has assigned 256MB
root@on05:~# xm list|grep scoutapi-dev
scoutapi-dev 12 256 1 -b---- 8.8
# ...and it can see that amount
root@scoutapi-dev:~# free -m|grep -B 1 "Mem"
total used free shared buffers cached
Mem: 252 70 182 0 4 38
root@scoutapi-dev:~# cat /proc/meminfo |grep MemTotal
MemTotal: 258908 kB
root@scoutapi-dev:~# cat
/sys/devices/system/xen_memory/xen_memory0/target_kb
262144
# I increase scoutapi-dev''s memory on Dom0
root@on05:~# xm mem-set scoutapi-dev 512
root@on05:~# xm list|grep scoutapi-dev
scoutapi-dev 12 512 1 -b---- 9.0
# DomU seems to be aware of the change (target_kb) but does not use it
root@scoutapi-dev:~# free -m|grep -B 1 "Mem"
total used free shared buffers cached
Mem: 252 70 182 0 4 38
root@scoutapi-dev:~# cat /proc/meminfo |grep MemTotal
MemTotal: 258908 kB
root@scoutapi-dev:~# cat
/sys/devices/system/xen_memory/xen_memory0/target_kb
262144
# Now I reduce scoutapi-dev''s memory
root@on05:~# xm mem-set scoutapi-dev 128
root@on05:~# xm list|grep scoutapi-dev
scoutapi-dev 12 128 1 -b---- 9.1
# DomU can see the change as expected
root@scoutapi-dev:~# free -m|grep -B 1 "Mem"
total used free shared buffers cached
Mem: 128 70 58 0 4 38
root@scoutapi-dev:~# cat /proc/meminfo |grep MemTotal
MemTotal: 131676 kB
root@scoutapi-dev:~# cat
/sys/devices/system/xen_memory/xen_memory0/target_kb
131072
# Finally, I increase DomU memory to it''s original size:
root@on05:~# xm mem-set scoutapi-dev 256
root@on05:~# xm list|grep scoutapi-dev
scoutapi-dev 12 256 1 -b---- 9.2
# DomU can see the increase fine, it seems to have problem
# only if it the new size is bigger than the original.
root@scoutapi-dev:~# free -m|grep -B 1 "Mem"
total used free shared buffers cached
Mem: 252 70 182 0 4 38
root@scoutapi-dev:~# cat /proc/meminfo |grep MemTotal
MemTotal: 258908 kB
root@scoutapi-dev:~# cat
/sys/devices/system/xen_memory/xen_memory0/target_kb
262144
# This is my config.
# It has been created by xen-tools, but i use a custom
# xen-tools.con and xm.tmpl
root@on05:~# cat /etc/xen/scout-dev.cfg
#
# Configuration file for the Xen instance scout-dev, created
# by xen-tools 4.2 on Tue Nov 13 19:02:28 2012.
name = ''scout-dev''
memory = ''256''
maxmem = ''8192''
vcpus = ''1''
kernel = ''/boot/vmlinuz-2.6.32-5-xen-amd64''
ramdisk = ''/boot/initrd.img-2.6.32-5-xen-amd64''
root = ''/dev/xvda2 ro''
disk = [
''phy:/dev/r5VG/scout-dev-srv,xvda3,w'',
''phy:/dev/r5VG/scout-dev-root,xvda2,w'',
''phy:/dev/r5VG/scout-dev-swap,xvda1,w'',
]
vif = [
''ip=11.22.33.44,mac=00:16:3E:72:7B:5A,bridge=xenbr55'',
]
on_poweroff = ''destroy''
on_reboot = ''restart''
on_crash = ''restart''
root@on05:~# dpkg -la|grep xen|awk ''{print $1"\t
"$2"\t"$3}''
ii libxenstore3.0 4.0.1-5.4
ii linux-image-2.6.32-5-xen-amd64 2.6.32-46
ii xen-hypervisor-4.0-amd64 4.0.1-5.4
ii xen-linux-system-2.6.32-5-xen-amd64 2.6.32-46
ii xen-qemu-dm-4.0 4.0.1-2+squeeze2
ii xen-tools 4.2-1
ii xen-utils-4.0 4.0.1-5.4
ii xen-utils-common 4.0.0-1
ii xenstore-utils 4.0.1-5.4
ii xenwatch 0.5.4-2
root@on05:~# uname -a
Linux on05 2.6.32-5-xen-amd64 #1 SMP Sun Sep 23 13:49:30 UTC 2012 x86_64
GNU/Linux
If you can bring some light on it, that would be just wonderful.
Greetings.
--
Alexandre Kouznetsov
Peter Viskup
2012-Nov-15 00:17 UTC
Re: dynamic memory extension not working on Debian Squeeze
On 11/14/2012 07:29 PM, Alexandre Kouznetsov wrote:> Hello, Ian. > > I have been living with the same problem for years. I guess it worked > on Etch with kernel 2.6.18, not sure about Lenny and 2.6.26. In my > case, xenstore works fine. > > The symptom is: after using mem-set, it apparently works, but the DomU > still thinks it has the old memory size. The issue shows up only when > increasing memory above the original size, the reduction work fine. > I have read the rest of the thread, and it seems Peter is stuck in > some xenstore issue. I suspect xenstore might be unrelated: in my case > it works fine, but mem-set still have the issue. > > The OS on Dom0 and DomU is a regular Debian Squeeze, no backports or > custom build packages. It has been seen on different models of Dell > PowerEdge servers. If relevant, I can confirm for a commodity hardware > case, or for a Debian Wheezy case (upgraded from Squeeze, not a clean > install). I have no other OS at hand than Debian to test. > > The output of "xenstore-ls -fp" that you asked remains unchanged, so I > will post it only once: > root@on05:~# xenstore-ls -fp | grep scoutapi-dev > /vm/8b7714b8-eeb0-2120-cee7-59273be76f79/name = "scoutapi-dev" (n0) > /local/domain/0/backend/vbd/12/51715/domain = "scoutapi-dev" (n0,r12) > /local/domain/0/backend/vbd/12/51715/params = > "/dev/r5VG/scoutapi-dev-srv" (n0,r12) > /local/domain/0/backend/vbd/12/51714/domain = "scoutapi-dev" (n0,r12) > /local/domain/0/backend/vbd/12/51714/params = > "/dev/r5VG/scoutapi-dev-root" (n0,r12) > /local/domain/0/backend/vbd/12/51713/domain = "scoutapi-dev" (n0,r12) > /local/domain/0/backend/vbd/12/51713/params = > "/dev/r5VG/scoutapi-dev-swap" (n0,r12) > /local/domain/0/backend/vif/12/0/domain = "scoutapi-dev" (n0,r12) > /local/domain/0/backend/console/12/0/domain = "scoutapi-dev" (n0,r12) > /local/domain/12/name = "scoutapi-dev" (n0,r12) > > > # DomU scoutapi-dev initially has assigned 256MB > root@on05:~# xm list|grep scoutapi-dev > scoutapi-dev 12 256 1 -b---- 8.8 > > # ...and it can see that amount > root@scoutapi-dev:~# free -m|grep -B 1 "Mem" > total used free shared buffers cached > Mem: 252 70 182 0 4 38 > root@scoutapi-dev:~# cat /proc/meminfo |grep MemTotal > MemTotal: 258908 kB > root@scoutapi-dev:~# cat > /sys/devices/system/xen_memory/xen_memory0/target_kb > 262144 > > > # I increase scoutapi-dev''s memory on Dom0 > root@on05:~# xm mem-set scoutapi-dev 512 > root@on05:~# xm list|grep scoutapi-dev > scoutapi-dev 12 512 1 -b---- 9.0 > > # DomU seems to be aware of the change (target_kb) but does not use it > root@scoutapi-dev:~# free -m|grep -B 1 "Mem" > total used free shared buffers cached > Mem: 252 70 182 0 4 38 > root@scoutapi-dev:~# cat /proc/meminfo |grep MemTotal > MemTotal: 258908 kB > root@scoutapi-dev:~# cat > /sys/devices/system/xen_memory/xen_memory0/target_kb > 262144 > > > # Now I reduce scoutapi-dev''s memory > root@on05:~# xm mem-set scoutapi-dev 128 > root@on05:~# xm list|grep scoutapi-dev > scoutapi-dev 12 128 1 -b---- 9.1 > > # DomU can see the change as expected > root@scoutapi-dev:~# free -m|grep -B 1 "Mem" > total used free shared buffers cached > Mem: 128 70 58 0 4 38 > root@scoutapi-dev:~# cat /proc/meminfo |grep MemTotal > MemTotal: 131676 kB > root@scoutapi-dev:~# cat > /sys/devices/system/xen_memory/xen_memory0/target_kb > 131072 > > # Finally, I increase DomU memory to it''s original size: > root@on05:~# xm mem-set scoutapi-dev 256 > root@on05:~# xm list|grep scoutapi-dev > scoutapi-dev 12 256 1 -b---- 9.2 > > # DomU can see the increase fine, it seems to have problem > # only if it the new size is bigger than the original. > root@scoutapi-dev:~# free -m|grep -B 1 "Mem" > total used free shared buffers cached > Mem: 252 70 182 0 4 38 > root@scoutapi-dev:~# cat /proc/meminfo |grep MemTotal > MemTotal: 258908 kB > root@scoutapi-dev:~# cat > /sys/devices/system/xen_memory/xen_memory0/target_kb > 262144 > > > # This is my config. > # It has been created by xen-tools, but i use a custom > # xen-tools.con and xm.tmpl > root@on05:~# cat /etc/xen/scout-dev.cfg > # > # Configuration file for the Xen instance scout-dev, created > # by xen-tools 4.2 on Tue Nov 13 19:02:28 2012. > > name = ''scout-dev'' > memory = ''256'' > maxmem = ''8192'' > vcpus = ''1'' > > kernel = ''/boot/vmlinuz-2.6.32-5-xen-amd64'' > ramdisk = ''/boot/initrd.img-2.6.32-5-xen-amd64'' > > root = ''/dev/xvda2 ro'' > disk = [ > ''phy:/dev/r5VG/scout-dev-srv,xvda3,w'', > ''phy:/dev/r5VG/scout-dev-root,xvda2,w'', > ''phy:/dev/r5VG/scout-dev-swap,xvda1,w'', > ] > vif = [ > ''ip=11.22.33.44,mac=00:16:3E:72:7B:5A,bridge=xenbr55'', > ] > on_poweroff = ''destroy'' > on_reboot = ''restart'' > on_crash = ''restart'' > > root@on05:~# dpkg -la|grep xen|awk ''{print $1"\t "$2"\t"$3}'' > ii libxenstore3.0 4.0.1-5.4 > ii linux-image-2.6.32-5-xen-amd64 2.6.32-46 > ii xen-hypervisor-4.0-amd64 4.0.1-5.4 > ii xen-linux-system-2.6.32-5-xen-amd64 2.6.32-46 > ii xen-qemu-dm-4.0 4.0.1-2+squeeze2 > ii xen-tools 4.2-1 > ii xen-utils-4.0 4.0.1-5.4 > ii xen-utils-common 4.0.0-1 > ii xenstore-utils 4.0.1-5.4 > ii xenwatch 0.5.4-2 > root@on05:~# uname -a > Linux on05 2.6.32-5-xen-amd64 #1 SMP Sun Sep 23 13:49:30 UTC 2012 > x86_64 GNU/Linux > > > If you can bring some light on it, that would be just wonderful. > > Greetings. >Hi Ian and Alexander, it''s good to hear I am not the onlyone experienced this issue. The removal of tdb file solved the xenstore issue. Now I am able to provide you the outputs you requested. They are very similar to those provided by Alexander. server1:~# xenstore-ls -fp | grep ldap /vm/2fad68ca-0fc8-b3ea-dac2-b54f713af36d/name = "ldap" (n0) /local/domain/0/backend/vbd/2/51713/domain = "ldap" (n0,r2) /local/domain/0/backend/vbd/2/51713/params = "system_vhosts/ldap-disk" (n0,r2) /local/domain/0/backend/vbd/2/51714/domain = "ldap" (n0,r2) /local/domain/0/backend/vbd/2/51714/params = "system_vhosts/ldap-swap" (n0,r2) /local/domain/0/backend/vbd/2/51715/domain = "ldap" (n0,r2) /local/domain/0/backend/vbd/2/51715/params = "system_vhosts/ldap-var" (n0,r2) /local/domain/0/backend/vbd/2/51716/domain = "ldap" (n0,r2) /local/domain/0/backend/vbd/2/51716/params = "system_vhosts/ldap-varlog" (n0,r2) /local/domain/0/backend/vbd/2/51717/domain = "ldap" (n0,r2) /local/domain/0/backend/vbd/2/51717/params = "system_vhosts/ldap-tmp" (n0,r2) /local/domain/0/backend/vbd/2/51718/domain = "ldap" (n0,r2) /local/domain/0/backend/vbd/2/51718/params = "system_vhosts/ldap-usr" (n0,r2) /local/domain/0/backend/vif/2/0/domain = "ldap" (n0,r2) /local/domain/0/backend/vif/2/0/vifname = "ldap-pub" (n0,r2) /local/domain/0/backend/vif/2/1/domain = "ldap" (n0,r2) /local/domain/0/backend/vif/2/1/vifname = "ldap-pri" (n0,r2) /local/domain/0/backend/console/2/0/domain = "ldap" (n0,r2) /local/domain/2/name = "ldap" (n0,r2) I just increased the memory for ldap domU to 512MB: ldap:~# grep MemTotal /proc/meminfo MemTotal: 250320 kB ldap:~# cat /sys/devices/system/xen_memory/xen_memory0/target_kb 262144 ldap:~# grep MemTotal /proc/meminfo MemTotal: 250320 kB ldap:~# cat /sys/devices/system/xen_memory/xen_memory0/target_kb 524288 but the real memory size didn''t changed. Shrinking of the memory is working well - but I am not able to increase the size of the memory above the initial size. Let me state once again, that this memory increase is working well with kernel 2.6.26-2-xen. Hope we will be able to solve this issue. Best regards, -- Peter Viskup
Ian Campbell
2012-Nov-15 09:20 UTC
Re: dynamic memory extension not working on Debian Squeeze
On Thu, 2012-11-15 at 00:17 +0000, Peter Viskup wrote:> Hi Ian and Alexander, > it''s good to hear I am not the onlyone experienced this issue. > The removal of tdb file solved the xenstore issue.Great!> Now I am able to provide you the outputs you requested. They are very > similar to those provided by Alexander. > > server1:~# xenstore-ls -fp | grep ldapWhen I asked for "xenstore-ls -fp | grep target" I meant the literal string "target" not the name of the target VM, sorry if this wasn''t clear!> I just increased the memory for ldap domU to 512MB: > ldap:~# grep MemTotal /proc/meminfo > MemTotal: 250320 kB > ldap:~# cat /sys/devices/system/xen_memory/xen_memory0/target_kb > 262144OOI what happens if you echo -n 524288 > /sys/devices/system/xen_memory/xen_memory0/target_kb ? Do you get anything in your dom0 or domU dmesg when you try and balloon up in this way? What about Xen''s dmesg? Actually, it would be useful to see the full domU dmesg at least. Ian.
Alexandre Kouznetsov
2012-Nov-15 19:18 UTC
Re: dynamic memory extension not working on Debian Squeeze
Peter Viskup
2012-Nov-15 19:59 UTC
Re: dynamic memory extension not working on Debian Squeeze
On 11/15/2012 10:20 AM, Ian Campbell wrote:> On Thu, 2012-11-15 at 00:17 +0000, Peter Viskup wrote: >> Hi Ian and Alexander, >> it''s good to hear I am not the onlyone experienced this issue. >> The removal of tdb file solved the xenstore issue. > Great! > >> Now I am able to provide you the outputs you requested. They are very >> similar to those provided by Alexander. >> >> server1:~# xenstore-ls -fp | grep ldap > When I asked for "xenstore-ls -fp | grep target" I meant the literal > string "target" not the name of the target VM, sorry if this wasn''t > clear! > >> I just increased the memory for ldap domU to 512MB: >> ldap:~# grep MemTotal /proc/meminfo >> MemTotal: 250320 kB >> ldap:~# cat /sys/devices/system/xen_memory/xen_memory0/target_kb >> 262144 > OOI what happens if you > echo -n 524288> /sys/devices/system/xen_memory/xen_memory0/target_kb > ? > > Do you get anything in your dom0 or domU dmesg when you try and balloon > up in this way? What about Xen''s dmesg? > > Actually, it would be useful to see the full domU dmesg at least. > > Ian. > >Here is additional info: - attached whole domU''s dmesg since the reboot, xen balloon driver initialized - attached is ''xm dmesg'' output - could help probably - tried echo <number> into target_kb with no effect and no messages - during the xm memory commands no kernel messages were logged by dom0 nor domU - the only messages from xend.log are: [2012-11-15 20:24:14 2948] DEBUG (XendDomainInfo:1477) Setting memory target of domain ldap (2) to 512 MiB. [2012-11-15 20:24:19 2948] DEBUG (XendDomainInfo:1504) Setting memory maximum of domain ldap (2) to 512 MiB. [2012-11-15 20:24:21 2948] DEBUG (XendDomainInfo:1477) Setting memory target of domain ldap (2) to 512 MiB. [2012-11-15 20:25:50 2948] DEBUG (XendDomainInfo:1477) Setting memory target of domain ldap (2) to 256 MiB. - I probably found something in Xen''s domain-builder-ng.log log file - messages from last domU build process are saved in attached file domU-build.log (there are some xen-3.0 statements) - not sure if these are ok or not elf_xen_parse_note: GUEST_OS = "linux" elf_xen_parse_note: GUEST_VERSION = "2.6" elf_xen_parse_note: XEN_VERSION = "xen-3.0" ... x86_compat: guest xen-3.0-x86_64, address size 64 ... xc_dom_compat_check: supported guest type: xen-3.0-x86_64 <= matches This is dom0 boot config: multiboot /xen-4.0-amd64.gz placeholder dom0_mem=756M numa=off loglvl=all guest_loglvl=all module /vmlinuz-2.6.32-5-xen-amd64 placeholder root=/dev/mapper/system_xen-root ro numa=off quiet Just for your information - the disk and cpu hotplug is working like a charm - the only issue is with memory ballooning. Not sure what else I could provide to all of you. Best regards, -- Peter Viskup _______________________________________________ Xen-users mailing list Xen-users@lists.xen.org http://lists.xen.org/xen-users
Ian Campbell
2012-Nov-20 09:33 UTC
Re: dynamic memory extension not working on Debian Squeeze
On Thu, 2012-11-15 at 19:59 +0000, Peter Viskup wrote: In dmesg.2012-11-15.log I see: [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable) [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved) [ 0.000000] Xen: 0000000000100000 - 0000000010000000 (usable) which indicates that the maximum RAM size the kernel can cope with is 0x10000000 == 256M, which matches your experience. I suspect that if you mem-set to e.g. 128M then you could increase back to 256M successfully. What is missing is booting "pre-ballooned" e.g. with a maximum memory size of 1G but only using 256M on boot. You have memmax=''4092'' in your guest config, which should work with a modern toolstack/kernel (BTW why 4092 not 4096?). I''m a little bit rusty on the details but older kernels and/or toolstack needed a little more manual intervention. Can you try adding "mem=4G" (or 4092M to match your config) to your domU command-line? Ian.
Alexandre Kouznetsov
2012-Nov-20 16:57 UTC
Re: dynamic memory extension not working on Debian Squeeze
El 20/11/12 03:33, Ian Campbell escribió:> On Thu, 2012-11-15 at 19:59 +0000, Peter Viskup wrote: > > In dmesg.2012-11-15.log I see: > [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable) > [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved) > [ 0.000000] Xen: 0000000000100000 - 0000000010000000 (usable) > > which indicates that the maximum RAM size the kernel can cope with is > 0x10000000 == 256M, which matches your experience. > > I suspect that if you mem-set to e.g. 128M then you could increase back > to 256M successfully.Yes. I can confirm that behavior. The RAM size can be shrinked and grown correctly, but only under the limit of the initial size.> What is missing is booting "pre-ballooned" e.g. with a maximum memory > size of 1G but only using 256M on boot.Some Gentoo wiki http://en.gentoo-wiki.com/wiki/Xen/Creating_DomUs says they get a feedback from balloon in their dmesg, like this: xen_mem: Initialising balloon driver. My DomU''s dmesg does not mentions anything about xen_mem, but does shows a xen_balloon message: xen_balloon: Initialising balloon driver with page order 0> I''m a little bit rusty on the details but older kernels and/or toolstack > needed a little more manual intervention. Can you try adding > "mem=4G" (or 4092M to match your config) to your domU command-line?In my case, same result. Tried "mem=8G" and "mem=8192M" (matching the config, maxmem=8192). root@scoutapi-dev:~# dmesg| grep "Command line" [ 0.000000] Command line: root=/dev/xvda2 ro mem=8G root@scoutapi-dev:~# dmesg | grep Xen [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable) [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved) [ 0.000000] Xen: 0000000000100000 - 0000000010000000 (usable) [ 0.000000] Booting paravirtualized kernel on Xen [ 0.000000] Xen version: 4.0.1 (preserve-AD) [ 0.000000] Xen: using vcpu_info placement [ 0.000000] Xen: using vcpuop timer interface [ 0.000000] installing Xen timer for CPU 0 [ 0.009451] PCI: setting up Xen PCI frontend stub [ 0.282324] Initialising Xen virtual ethernet driver. root@scoutapi-dev:~# cat /proc/meminfo |grep MemTotal MemTotal: 258908 kB root@scoutapi-dev:~# cat /sys/devices/system/xen_memory/xen_memory0/target_kb 262144 root@on05:~# xm mem-set scoutapi-dev 512 root@scoutapi-dev:~# cat /proc/meminfo |grep MemTotal MemTotal: 258908 kB root@scoutapi-dev:~# cat /sys/devices/system/xen_memory/xen_memory0/target_kb 524288 -- Alexandre Kouznetsov
Ian Campbell
2012-Nov-20 17:07 UTC
Re: dynamic memory extension not working on Debian Squeeze
On Tue, 2012-11-20 at 16:57 +0000, Alexandre Kouznetsov wrote:> root@scoutapi-dev:~# dmesg| grep "Command line" > [ 0.000000] Command line: root=/dev/xvda2 ro mem=8G > root@scoutapi-dev:~# dmesg | grep Xen > [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable) > [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved) > [ 0.000000] Xen: 0000000000100000 - 0000000010000000 (usable)0x10000000 != 8G so maybe that''s not the right parameter name, but I thought so. If that doesn''t help then I''m afraid I''m all out of ideas. Perhaps experiment with upgrading the dom0 to e.g. the Wheezy hypervisor + kernel and/or the Wheezy dom0 kernel and see if that helps. Ian.
Alexandre Kouznetsov
2012-Nov-20 17:30 UTC
Re: dynamic memory extension not working on Debian Squeeze
Ian Campbell
2012-Nov-20 17:43 UTC
Re: dynamic memory extension not working on Debian Squeeze
On Tue, 2012-11-20 at 17:30 +0000, Alexandre Kouznetsov wrote:> > I have attached the complete dmesg for a better reference. This one is > after I changed from "mem=8G" to "mem=8192M".It''s interesting that it has both: [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable) [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved) [ 0.000000] Xen: 0000000000100000 - 0000000010000000 (usable) and [ 0.000000] user-defined physical RAM map: [ 0.000000] user: 0000000000000000 - 00000000000a0000 (usable) [ 0.000000] user: 00000000000a0000 - 0000000000100000 (reserved) [ 0.000000] user: 0000000000100000 - 0000000010000000 (usable) Where I think the second one is the result of passing mem=, yet it is exactly the same! I suspect it has taken the command line provided limit and clamped it... Another thing you could try is using the memmap command line option to generate a memmap like those above but with the limit on the 3rd one =8G. You''ll have to read Documentation/kernel-parameters.txt to figure out exactly how though since I''m not sure how that stuff works. Another workaround would be to boot with 8G and then balloon down ASAP. This means you need the spare RAM available to boot the guest though, which isn''t ideal... Ian.
Peter Viskup
2012-Nov-20 23:45 UTC
Re: dynamic memory extension not working on Debian Squeeze
On 11/20/2012 06:43 PM, Ian Campbell wrote:> On Tue, 2012-11-20 at 17:30 +0000, Alexandre Kouznetsov wrote: >> I have attached the complete dmesg for a better reference. This one is >> after I changed from "mem=8G" to "mem=8192M". > It''s interesting that it has both: > [ 0.000000] BIOS-provided physical RAM map: > [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable) > [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved) > [ 0.000000] Xen: 0000000000100000 - 0000000010000000 (usable) > > and > [ 0.000000] user-defined physical RAM map: > [ 0.000000] user: 0000000000000000 - 00000000000a0000 (usable) > [ 0.000000] user: 00000000000a0000 - 0000000000100000 (reserved) > [ 0.000000] user: 0000000000100000 - 0000000010000000 (usable) > > Where I think the second one is the result of passing mem=, yet it is > exactly the same! I suspect it has taken the command line provided limit > and clamped it... > > Another thing you could try is using the memmap command line option to > generate a memmap like those above but with the limit on the 3rd one => 8G. You''ll have to read Documentation/kernel-parameters.txt to figure > out exactly how though since I''m not sure how that stuff works. > > Another workaround would be to boot with 8G and then balloon down ASAP. > This means you need the spare RAM available to boot the guest though, > which isn''t ideal... > > Ian.I was just followed this wiki [1] and found that these options/features are not available in Debian''s domU paravirt_ops kernel (2.6.32-5-amd64): CONFIG_PARAVIRT_SPINLOCKS (really needed?) CONFIG_XEN_XENBUS_FRONTEND (available since 2.6.38) Looks like that wiki needs rewrite and having more information about every kernel option would be great. Anyway I am quite tired of checking those kernel options and decided to open an bug report for it [2]. Will see what will happen. [1] http://wiki.xensource.com/xenwiki/XenParavirtOps [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=693851 -- Peter
Alexandre Kouznetsov
2012-Nov-21 05:51 UTC
Re: dynamic memory extension works on Debian Wheezy
Hello.
I have made the tests on a Wheezy box, a much smaller machine. It''s not
a fresh install, but upgraded from Squeeze, it''s visible in the
installed packages list. The dmesg shows some residues of my IOMMU
tests, I believe they do not affect mem-set behavior.
Using Xen 4.1, kernel 3.2.0 on Dom0 and kernel 2.6.32 on DomU system
gave the same result as before, the memory can''t be grown beyond
initial
size.
Using the same setup, but kernel 3.2.0 on DomU, mem-set was successful
beyond the initial size. I just tested to grow from .5G to 1G, which
proves the concept. Somebody ate 62MB of RAM from my DomU (difference
between meminfo and xen_memory0/target_kb).
I conclude that Squeeze''s Xen kernel has the ballooning functionality
broken somehow. Moving from 2.6.32 to 3.2.0 made the difference. There
are little hopes of a fix: Squeeze is very close to become
"oldstable",
so we shall just move on forward. I believe (although did not tested)
the upgrade of the hypervisor itself from 4.0 to 4.1 had little to do.
root@lor01:~# uname -a
Linux lor01 3.2.0-2-amd64 #1 SMP Mon Jun 11 17:24:18 UTC 2012 x86_64
GNU/Linux
root@lor01:~# dpkg -la|grep xen|awk ''{print $1"\t
"$2"\t"$3}''
ii libxen-4.1 4.1.2-6
ii libxenstore3.0 4.1.2-6
ii linux-image-2.6.32-5-xen-amd64 2.6.32-41squeeze2
ii xen-hypervisor-4.0-amd64 4.0.1-4
ii xen-hypervisor-4.1-amd64 4.1.2-6
ii xen-linux-system-2.6.32-5-xen-amd64 2.6.32-41squeeze2
ii xen-qemu-dm-4.0 4.0.1-2+squeeze1
ii xen-tools 4.2.1-1
ii xen-utils-4.0 4.0.1-4
ii xen-utils-4.1 4.1.2-6
ii xen-utils-common 4.1.2-6
ii xenstore-utils 4.1.2-6
ii xenwatch 0.5.4-3
root@lor01:~# cat /etc/xen/sso.cfg
#
# Configuration file for the Xen instance sso, created
# by xen-tools 4.2 on Wed Feb 29 22:40:17 2012.
name = ''sso''
memory = ''512''
maxmem = ''3328''
vcpus = ''1''
kernel = ''/boot/vmlinuz-2.6.32-5-xen-amd64''
ramdisk = ''/boot/initrd.img-2.6.32-5-xen-amd64''
root = ''/dev/xvda2 ro''
disk = [
''phy:/dev/mirrorVG/sso-data,xvda3,w'',
''phy:/dev/mirrorVG/sso-root,xvda2,w'',
''phy:/dev/mirrorVG/sso-swap,xvda1,w'',
]
vif = [
''ip=1.2.3.4,mac=00:16:3E:86:59:3E,bridge=xenbr01'',
]
on_poweroff = ''destroy''
on_reboot = ''restart''
on_crash = ''restart''
root@lor01:~# xm dmesg
(XEN) Xen version 4.1.2 (Debian 4.1.2-6) [...]
[ ... output attached as Wheezy_Dom0_xm-dmesg.txt ... ]
root@lor01:~# xm create sso.cfg
Using config file "/etc/xen/sso.cfg".
Started domain sso (id=10)
root@lor01:~# xm list|grep sso
sso 10 512 1 -b---- 2.2
root@lor01:~# xenstore-ls -fp | grep target | grep "(n10)"
/local/domain/10/memory/target = "524288" (n10)
root@sso:~# uname -a
Linux sso 2.6.32-5-xen-amd64 #1 SMP Thu Mar 22 21:14:26 UTC 2012 x86_64
GNU/Linux
root@sso:~# dmesg
[ ... output attached as DomU_dmesg_2.6.32.txt ... ]
root@sso:~# cat /proc/meminfo |grep MemTotal
MemTotal: 521052 kB
root@sso:~# cat /sys/devices/system/xen_memory/xen_memory0/target_kb
524288
root@lor01:~# xm mem-set sso 1024
root@lor01:~# xm list|grep sso
sso 10 1024 1 -b---- 2.3
root@lor01:~# xenstore-ls -fp | grep target | grep "(n10)"
/local/domain/10/memory/target = "1048576" (n10)
root@sso:~# cat /proc/meminfo |grep MemTotal
MemTotal: 521052 kB
root@sso:~# cat /sys/devices/system/xen_memory/xen_memory0/target_kb
1048576
root@sso:~# dmesg
[ ... no change ... ]
# Now I copy Dom0''s /lib/modules/3.2.0-2-amd64 to "sso" host
and change
sso.cfg to use 3.2.0 kernel.
root@lor01:~# xm create sso.cfg
Using config file "/etc/xen/sso.cfg".
Started domain sso (id=11)
root@lor01:~# xm list|grep sso
sso 11 512 1 -b---- 2.2
root@lor01:~# xenstore-ls -fp | grep target | grep "(n11)"
/local/domain/11/memory/target = "524288" (n11)
root@sso:~# uname -a
Linux sso 3.2.0-2-amd64 #1 SMP Mon Jun 11 17:24:18 UTC 2012 x86_64 GNU/Linux
root@sso:~# dmesg
[ ... output attached as DomU_dmesg_3.2.0.txt ... ]
root@sso:~# cat /proc/meminfo |grep MemTotal
MemTotal: 460880 kB
root@sso:~# cat /sys/devices/system/xen_memory/xen_memory0/target_kb
524288
root@lor01:~# xm mem-set sso 1024
root@lor01:~# xm list|grep sso
sso 11 1024 1 -b---- 2.4
root@lor01:~# xenstore-ls -fp | grep target | grep "(n11)"
/local/domain/11/memory/target = "1048576" (n11)
root@lor01:~# xm dmesg
[ ... same as before, plus the following ... ]
(XEN) traps.c:2432:d10 Domain attempted WRMSR 00000000c0010048 from
0x0000000000780000 to 0x0000000000780400.
(XEN) traps.c:2432:d11 Domain attempted WRMSR 00000000c0010048 from
0x0000000000780000 to 0x0000000000780400.
(XEN) traps.c:2432:d11 Domain attempted WRMSR 00000000c0010004 from
0x0000d7dea1ae17df to 0x000000000000abcd.
root@sso:~# cat /proc/meminfo |grep MemTotal
MemTotal: 985168 kB
root@sso:~# cat /sys/devices/system/xen_memory/xen_memory0/target_kb
1048576
root@sso:~# dmesg
[ ... no change ... ]
root@sso:~# free -m
total used free shared buffers cached
Mem: 962 39 922 0 3 12
-/+ buffers/cache: 22 939
Swap: 2047 0 2047
--
Alexandre Kouznetsov
_______________________________________________
Xen-users mailing list
Xen-users@lists.xen.org
http://lists.xen.org/xen-users
Alexandre Kouznetsov
2012-Nov-21 06:02 UTC
Re: dynamic memory extension not working on Debian Squeeze
El 20/11/2012 11:43 a.m., Ian Campbell escribió:> Another thing you could try is using the memmap command line option toChecked the docs. Even if it work, it will be unmaintainable: I create/destroy VM''s much more frequently than changing their''s configurations. I guess I''ll stay with a kernel upgrade. Next time I need this ballooning, I''ll try leaving Dom0 with it''s 2.6.32 and giving DomU 3.2 kernel from backports. Probably that would be enough.> Another workaround would be to boot with 8G and then balloon down ASAP. > This means you need the spare RAM available to boot the guest though, > which isn''t ideal...Tried that. It works, but very unpractical. Thank you, all that was helpful. Greetings. -- Alexandre Kouznetsov
Ian Campbell
2012-Nov-21 07:13 UTC
Re: dynamic memory extension not working on Debian Squeeze
On Tue, 2012-11-20 at 23:45 +0000, Peter Viskup wrote:> On 11/20/2012 06:43 PM, Ian Campbell wrote: > > On Tue, 2012-11-20 at 17:30 +0000, Alexandre Kouznetsov wrote: > >> I have attached the complete dmesg for a better reference. This one is > >> after I changed from "mem=8G" to "mem=8192M". > > It''s interesting that it has both: > > [ 0.000000] BIOS-provided physical RAM map: > > [ 0.000000] Xen: 0000000000000000 - 00000000000a0000 (usable) > > [ 0.000000] Xen: 00000000000a0000 - 0000000000100000 (reserved) > > [ 0.000000] Xen: 0000000000100000 - 0000000010000000 (usable) > > > > and > > [ 0.000000] user-defined physical RAM map: > > [ 0.000000] user: 0000000000000000 - 00000000000a0000 (usable) > > [ 0.000000] user: 00000000000a0000 - 0000000000100000 (reserved) > > [ 0.000000] user: 0000000000100000 - 0000000010000000 (usable) > > > > Where I think the second one is the result of passing mem=, yet it is > > exactly the same! I suspect it has taken the command line provided limit > > and clamped it... > > > > Another thing you could try is using the memmap command line option to > > generate a memmap like those above but with the limit on the 3rd one => > 8G. You''ll have to read Documentation/kernel-parameters.txt to figure > > out exactly how though since I''m not sure how that stuff works. > > > > Another workaround would be to boot with 8G and then balloon down ASAP. > > This means you need the spare RAM available to boot the guest though, > > which isn''t ideal... > > > > Ian. > > I was just followed this wiki [1] and found that these options/features > are not available in Debian''s domU paravirt_ops kernel (2.6.32-5-amd64): > CONFIG_PARAVIRT_SPINLOCKS (really needed?) > CONFIG_XEN_XENBUS_FRONTEND (available since 2.6.38)Neither of these have any relevance to the problem at hand.> Looks like that wiki needs rewrite and having more information about > every kernel option would be great. Anyway I am quite tired of checking > those kernel options and decided to open an bug report for it [2]. Will > see what will happen. > > [1] http://wiki.xensource.com/xenwiki/XenParavirtOps > [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=693851I''m afraid that I''m the Debian maintainer too, and I''m still fresh out of theories, so the above suggestions and the ones in <1353431265.13542.83.camel@zakaz.uk.xensource.com> are as good as it is going to get. Sorry. (note that wiki.xensource.com/xenwiki is the old wiki, which is no longer maintainer. wiki.xen.org/wiki is the maintained one). Ian.
Ian Campbell
2012-Nov-21 07:15 UTC
Re: dynamic memory extension not working on Debian Squeeze
On Tue, 2012-11-20 at 23:45 +0000, Peter Viskup wrote:> I was just followed this wiki [1] and found that these options/features > are not available in Debian''s domU paravirt_ops kernel (2.6.32-5-amd64):This version number just triggered an idea. Please try using the Xen flavour (2.6.32-5-xen-amd64) in the domU. I''m not sure how I didn''t notice this before. Ian.
Peter Viskup
2012-Nov-21 08:01 UTC
Re: dynamic memory extension not working on Debian Squeeze
On 11/21/2012 08:15 AM, Ian Campbell wrote:> On Tue, 2012-11-20 at 23:45 +0000, Peter Viskup wrote: >> I was just followed this wiki [1] and found that these options/features >> are not available in Debian''s domU paravirt_ops kernel (2.6.32-5-amd64): > This version number just triggered an idea. Please try using the Xen > flavour (2.6.32-5-xen-amd64) in the domU. I''m not sure how I didn''t > notice this before. > > Ian.We already tested both ''xen'' and ''no-xen'' flavour kernels. Both of them doesn''t work. -- Peter